[routing-wg] Subject: RPKI ROA Deletion: Post-mortem

Dear colleagues,

After our accidental deletion of RPKI ROAs on Wednesday evening, we have 
a post-mortem report to share with the working group.

Following an update to our internal registry software on 1 April at 
18:16 (UTC+2), 2,669 ROAs were deleted from Provider Independent (PI) 
address assignments.

This was caused by our registry software classifying these assignments 
as not-certifiable. From our logs, we can confirm that these blocks 
never left the RIPE Registry, and within 15 minutes the registry was 
back to normal. However, by that time the ROAs had already been deleted 
and could not be restored without intervention from our engineers.

Affected users with alerts set up in the LIR Portal received a 
notification email on 31 March at 22:23, stating that their ROAs were 
missing. Some of these users emailed our Customer Service Department to 
ask why their ROAs had been deleted. As this was outside of office 
hours, our staff did not discover the issue until the next morning. 

Our engineers were able to reinstate all of the missing ROAs by 13:15 on 
2 April. We then informed our membership via ncc-announce and notified 
the affected users directly.

We have since implemented stricter checks on both our registry and RPKI 
software.

We are also investigating whether any of these PI assignments suffered 
from route-leaks or hijacks after their ROAs were deleted.

We apologise for any inconvenience this may have caused and we are 
taking all necessary steps to ensure this does not happen again in the 
future.

Kind regards,

Nathalie Trenaman
Routing Security Programme Manager
RIPE NCC