[routing-wg] RPKI Validator 3 Disk Space Requirement
- Previous message (by thread): [routing-wg] RPKI Validator 3 Disk Space Requirement
- Next message (by thread): [routing-wg] RPKI Validator 3 Disk Space Requirement
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
nusenu
nusenu-lists at riseup.net
Tue Oct 9 20:37:00 CEST 2018
>> the rpki-validator.h2.mv.db file is already >22GB in size, I assume >> there is no way to reduce the disk usage requirements via a >> configuration change? >> >> How big will the db file become? > > There is a config setting regulating how often the validator will > remove the "garbage", i.e. unused objects that it still stores but > they are not really used but kept just in case (no connectivity to > download updated records, broken manifests, CRLs, etc.). The setting > is called "rpki.validator.rpki.object.cleanup.grace.duration" and by > default it is set to "P7D", i.e. a period of 7 days, which is pretty > conservative. You can set it to much smaller periods, 1-2 day or 12 > hours and still have reasonable behaviour. There is another parameter > “rpki.validator.validation.run.cleanup.grace.duration" with the > similar meaning, but it's less crucial when it comes to storage > space. Also Validator version 3 is lighter than 2 in a sense that it > consumes less memory, but it can only do it by storing more data on > the disk. > > I hope this helps. thanks for your reply. I changed it to: rpki.validator.rpki.object.cleanup.grace.duration=P1D in conf/application.properties, restarted the validator and expected some reduction in disk usage after a day or so, but the db folder uses almost 50GB now.. What disk usage should I expect / prepare for? There are also some errors shown on stdout (details see bellow). Since there doesn't appear much activity on the RPKI validator 3 issue tracker a general question: What is the general maintenance status of RPKI Validator 3? thanks, nusenu ERROR pool-1-thread-1 o.h.i.ExceptionMapperStandardImpl HHH000346: Error during managed flush [org.hibernate.exception.ConstraintViolationException: could not execute statement] ERROR pool-1-thread-1 o.s.s.s.TaskUtils$LoggingErrorHandler Unexpected error occurred in scheduled task. org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint ["RPKI_OBJECT__SHA256_IDX ON PUBLIC.RPKI_OBJECT(SHA256) VALUES (X'e17d74b2082b15701d300e24d97a3d7b0eb081d59034c6d026b9555b0509bc0e', 3443950)"; SQL statement: /* insert net.ripe.rpki.validator3.domain.RpkiObject */ insert into rpki_object (created_at, updated_at, version, authority_key_identifier, last_marked_reachable_at, serial_number, sha256, signing_time, type, id) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [23505-197]]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:259) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:225) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:540) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:746) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:714) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:532) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:304) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) at net.ripe.rpki.validator3.domain.validation.RpkiRepositoryValidationService$$EnhancerBySpringCGLIB$$3380f99b.validateRsyncRepositories(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:178) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3032) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3547) at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:600) at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:474) at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337) at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39) at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1437) at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:494) at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3245) at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2451) at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:473) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:156) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38) at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:231) at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68) at org.springframework.orm.jpa.JpaTransactionManager.doCommit(JpaTransactionManager.java:536) ... 21 common frames omitted -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: <https://lists.ripe.net/ripe/mail/archives/routing-wg/attachments/20181009/cc5a3918/attachment.sig>
- Previous message (by thread): [routing-wg] RPKI Validator 3 Disk Space Requirement
- Next message (by thread): [routing-wg] RPKI Validator 3 Disk Space Requirement
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]