[routing-wg] Post-mortem: Error in Contractual Status of Legacy Resources Impacted RPKI

Dear colleagues,

On Wednesday, 16 December 2020, we carried out work to implement 
Numbered Work Item 10 (NWI-10), which applies a consistent 
interpretation of the country code attribute in the RIPE Database and 
the Extended Delegated Statistics. At some point between 18:00-19:00 
(UTC+1), 105 legacy resources mistakenly lost their contractual status 
in our internal systems. As a result, these resources were no longer 
able to be certified using RPKI.

Details:
Legacy resources can only be certified with RPKI when there is a 
contractual relationship in place with the RIPE NCC. These resources are 
registered differently from RIPE NCC-issued resources in our internal 
systems. A programming error in our NWI-10 implementation overlooked 
this aspect, which caused the contractual status for these legacy 
resources to be set to “none”. As a consequence, the resources were 
unable to be certified.

Once RPKI detected this change:

* The resource certificates for 36 Certificate Authorities (CAs) were 
updated to contain fewer certifiable resources.
* The ROAs for legacy resources held by affected hosted CAs were 
revoked. 41 ROAs (with 202 Validated ROA Payloads) from 24 CAs were deleted.
* The RPKI certificates issued for affected delegated CAs shrank, which 
caused their ROAs to disappear or be rejected due to overclaiming, 
depending on their CA software.

We recovered the contractual status of the 105 affected resources at 
17:10 (UTC+1) on Thursday, 17 December, re-established the correct 
resource list for the affected certificates, and recreated the 41 
affected/deleted ROAs (with 202 VRPs) that are hosted by us.

Recommendations:
We have already recovered the ROAs belonging to the affected hosted CAs. 
We recommend that these CAs double-check this.

Affected delegated CAs should check whether they need to recreate any 
ROAs. Our Customer Services team will be in contact to follow up.

To prevent this from happening again in future, we will improve our 
Quality Assurance and Acceptance Testing. We will also improve our 
testing and prepare more detailed impact analyses when making changes to 
our registry software, applying a more risk-based testing approach to 
mitigate any concerns that are identified.

Kind regards,

Marco Schmidt
Registry Services Assistant Manager