[ncc-services-wg] Interruption of RIPE Database and Related Services

Dear colleagues,

Starting at approximately 14:30 UTC on 4 October 2015, RIPE Database queries and updates, the LIR Portal and other related services experienced problems. Our investigation found that the service interruptions were due to a hardware failure. The queries were restored at 16:00 UTC and other affected services were restored at around 21:00 UTC.
The ability of the RIPE Database to process incoming mail-updates was restored at 12:30 UTC on 5 October 2015. Please be aware that all mail-updates sent to Whois during this outage period must be considered lost.

A summary of the hardware failure is as follows:
The RAID controller on the master MySQL server reinitialised all of a sudden, causing IO errors. Unfortunately, we also experienced another (unrelated) hardware problem at the same time with the external load balancers. This combination of issues significantly affected the time it took us to rectify the situation.

We apologise for any inconvenience this may have caused.

Kind regards,
Trudy Prins
Software Engineering Manager

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2599 bytes
Desc: not available
URL: <https://lists.ripe.net/ripe/mail/archives/ncc-services-wg/attachments/20151005/d34dff1f/attachment.p7s>