You are here: Home > Get Support > Service and Security Announcements > Issue with RRDP Repository

Issue with RRDP Repository

On 12 September 2022, during scheduled maintenance, the state of was inconsistent between 09:35 and 09:50 UTC. After becoming aware of a potential issue, we switched to secondary RRDP infrastructure before starting our investigation, mitigating the problem.

During the incident, the RRDP repository referred to files that were not available, and 3480 requests resulted in a 404 response out of more than 10.97k requests (lower bound, excludes traffic after failover). According to our logs and external monitoring, the rsync service at rsync:// was fully available during this period.

The alert we received indicated that notification.xml was missing while the delta/snapshot files were unavailable, which delayed our response to the incident. As an immediate action, we have already started making necessary changes to improve our alerting.

We will continue investigating the root cause and focus on possible race conditions during the shutdown of the publication server (which could have happened during patching).