[routing-wg] HTTP 404 Errors for rrdp.ripe.net

Dear colleagues,

This morning, 12 September 2022, during scheduled maintenance, the state of
rrdp.ripe.net was inconsistent between 09:35 and 09:50 UTC. After becoming aware
of a potential issue, we switched to secondary RRDP infrastructure before
starting our investigation, mitigating the problem.

During the incident, the RRDP repository https://rrdp.ripe.net/notification.xml
referred to files that were not available, and 3480 requests resulted in a 404
response out of more than 10.97k requests (lower bound, excludes traffic after
failover). According to our logs and external monitoring, the rsync service at
rsync://rpki.ripe.net was fully available during this period.

The alert we received indicated that notification.xml was missing while the
delta/snapshot files were unavailable, which delayed our response to the
incident. As an immediate action, we have already started making necessary
changes to improve our alerting. 

We will continue investigating the root cause and focus on possible race
conditions during the shutdown of the publication server (which could have
happened during patching).

Kind regards,

Ties de Kock