[routing-wg] RFO for RIPE NCC RRDP outage 24 March 2022

Dear colleagues,

Recently, the RPKI team has upgraded the RRDP setup to use two CDNs in an active-passive state. This switch is done automatically and on a fixed schedule. On Thursday, the 24th of March, we switched the CDN over to our second provider for the first time. After the switch, they noticed a peak in DNS requests compared to their regular traffic, triggering query limits that they had not encountered before, causing DNS requests to fail.

They have now solved the issue by improving the underlying performance limitations in their DNS-based traffic steering implementation and adding more DNS servers. They have also optimised their setup to speed up the query response time.

After multiple rounds of tests, including the provider’s synthetic and real-world tests using Atlas and NLNog-ring, we felt sure that the provider had resolved the issue. This morning at 9 AM UTC, we switched the active CDN for rrdp.ripe.net again, with engineers on both sides collectively monitoring the systems. The switch went smoothly.

We will switch between the CDN providers regularly, with a fallback option available for operational needs and when we notice issues.

Regards,

Stella

> On 24 Mar 2022, at 18:31, Lukas Tribus <lukas at ltri.eu> wrote:
> 
> Hello Stella,
> 
> 
> On Thu, 24 Mar 2022 at 10:39, Stella Vouteva <svouteva at ripe.net> wrote:
>> it appears that there were issues in resolving the CDN host
> 
> Could you elaborate?
> 
> 
> Lukas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ripe.net/ripe/mail/archives/routing-wg/attachments/20220407/3d66068d/attachment.html>