[routing-wg] Subject: RPKI ROA Deletion: Post-mortem
- Previous message (by thread): [routing-wg] Subject: RPKI ROA Deletion: Post-mortem
- Next message (by thread): [routing-wg] rsync RPKI Repository downtime
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Job Snijders
job at ntt.net
Sun Apr 5 18:08:57 CEST 2020
On Sun, Apr 05, 2020 at 12:56:48PM +0200, Marcus Stoegbauer wrote: > thanks a lot for collecting the information and the post-mortem, and > apologies for nitpicking, but: > > > Following an update to our internal registry software on 1 April at > > 18:16 (UTC+2), 2,669 ROAs were deleted from Provider Independent > > (PI) address assignments. > [..] > > Affected users with alerts set up in the LIR Portal received a > > notification email on 31 March at 22:23, stating that their ROAs > > were missing. > [..] > > Our engineers were able to reinstate all of the missing ROAs by > > 13:15 on 2 April. > > The timeline does not completely match up here. I assume the users received > alarms on April 1 after the update of the internal registry software? I've constructed the following timeline based on my own validator data archives and the notification I received for my PI block. The minute timestamps are a little bit different than the above because NTT's validators run roughly every 15 minutes and i'm basing this off those snapshots. 2020-04-01T16:32Z - 2,666 VRPs disappeared from the RPKI. 2020-04-01T21:23Z - affected users with alerts set up in the LIR Portal received a notification, stating that their ROAs were missing. 2020-04-02T11:32Z - The missing VRPs returned. It is interesting the NCC's outreach about the incident (or their ROA state alert notification emails) seems to have triggered not just a rebound to the original level (which was expected to happen because of the undelete action), but also a tiny increase in new RPKI ROA creation. Date VRP Count 2020-03-29 80,915 2020-03-30 80,933 2020-03-31 81,016 2020-04-01 81,089 2020-04-02 78,626 2020-04-03 81,530 2020-04-04 81,655 2020-04-05 81,732 Kind regards, Job
- Previous message (by thread): [routing-wg] Subject: RPKI ROA Deletion: Post-mortem
- Next message (by thread): [routing-wg] rsync RPKI Repository downtime
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]