[routing-wg] RFO for RIPE NCC RRDP outage 24 March 2022

Dear colleagues,

At 08:03 UTC, we had a short (partial) outage of about 10 minutes, where rrdp.ripe.net could not be resolved in DNS. Rsync and the RPKI dashboard were not affected. 

The outage happened after our weekly automated switching between our CDN providers. During the switch, we verify if our CDN providers are accessible and resolve correctly. Right after the switch, it appears that there were issues in resolving the CDN host, which resulted in rrdp.ripe.net being inaccessible for a short period of time.

This was almost immediately noticed by our monitoring and we reverted the change in DNS, moving rrdp.ripe.net to point back to our secondary CDN provider. We have set the TTL to 60 seconds, so the DNS update was quickly propagated.

Timeline:
08:03 UTC: DNS switch
08:10 UTC: The RPKI team gets alerted that rrdp.ripe.net is down
08:20 UTC: Change is reverted

We have contacted the CDN provider to investigate what the exact issue is.

Kind regards,
Stella
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ripe.net/ripe/mail/archives/routing-wg/attachments/20220324/abf655a1/attachment.html>