[routing-wg] Subject: RPKI ROA Deletion: Post-mortem

On Sun, Apr 05, 2020 at 12:56:48PM +0200, Marcus Stoegbauer wrote:
> thanks a lot for collecting the information and the post-mortem, and
> apologies for nitpicking, but:
> 
> > Following an update to our internal registry software on 1 April at
> > 18:16 (UTC+2), 2,669 ROAs were deleted from Provider Independent
> > (PI) address assignments.
> [..]
> > Affected users with alerts set up in the LIR Portal received a
> > notification email on 31 March at 22:23, stating that their ROAs
> > were missing.
> [..]
> > Our engineers were able to reinstate all of the missing ROAs by
> > 13:15 on 2 April.
> 
> The timeline does not completely match up here. I assume the users received
> alarms on April 1 after the update of the internal registry software?

I've constructed the following timeline based on my own validator data
archives and the notification I received for my PI block. The minute
timestamps are a little bit different than the above because NTT's
validators run roughly every 15 minutes and i'm basing this off those
snapshots.

2020-04-01T16:32Z - 2,666 VRPs disappeared from the RPKI.

2020-04-01T21:23Z - affected users with alerts set up in the LIR Portal
                    received a notification, stating that their ROAs
                    were missing.

2020-04-02T11:32Z - The missing VRPs returned.

It is interesting the NCC's outreach about the incident (or their ROA
state alert notification emails) seems to have triggered not just a
rebound to the original level (which was expected to happen because of
the undelete action), but also a tiny increase in new RPKI ROA creation. 

    Date        VRP Count
    2020-03-29  80,915
    2020-03-30  80,933
    2020-03-31  81,016
    2020-04-01  81,089
    2020-04-02  78,626
    2020-04-03  81,530
    2020-04-04  81,655
    2020-04-05  81,732

Kind regards,

Job