[routing-wg] Invalid RPKI Certificate Published / Update RPKI Validator Software

Dear colleagues,

Summary:
An issue with our RPKI software caused an invalid certificate to be published
from 9.40-10:43 (UTC+1) today. This has resulted in outages. We strongly
recommend network operators update their Relying Party software to the latest
version.


At 9:32 (UTC+1) this morning, we processed an outgoing transfer of IP resources
to another RIR service region. This caused our system to update the corresponding
RPKI certificates in our Certificate Authority (CA). Unfortunately, a bug in our
software caused the publication of the updated child certificate ahead of its
parent. As a result, our CA published incorrect information from 9:40-10:43 (UTC+1).

Some Relying Parties had applied a strict interpretation in their validator
software. This meant that they were configured to reject all certificates in the
manifest if a single entry was invalid. As a consequence, all RPKI certificates
covering RIPE resources were rejected by these validators during this period.

While RPKI is designed to "fail-open", an unrelated issue with some routers seems
to have prevented this from happening, which resulted in outages.

Some Relying Parties have since updated their validator software to apply a
less-strict approach in light of this issue[1]. We have published a release
candidate of our own RIPE NCC RPKI Validator and we plan to release this to
production tomorrow.

While our CA is now publishing correct information, the underlying cause remains
unsolved. We are continuing to investigate and will share more details when we
have them, followed by a post mortem report once everything is resolved.

In the meantime, we recommend that network operators update their RPKI validator
software to the latest version.

Regards,

Ties de Kock
Software Engineer
RIPE NCC

[1] Note: Routinator 0.8.2, Fort, rpki-client and octorpki 1.2.2 are either
unaffected or contain the updated interpretation.