[db-wg] Further details of RIPE Database update service outage on June 30

Dear colleagues,

On Saturday 30 June 2012 there was an outage on the RIPE Database update
service from 05:58 UTC (07:58 CEST) until 07:55 UTC (9:55 CEST) due to
hardware issues. During this failure, no email updates were lost and no
corruption of the RIPE Database occurred. The query service, which is
running our new whois software, was not affected at all.

Currently the update service runs on a single, active server backed up
by a redundant failover machine. We use database replication to keep the
hot standby ready, but there is no automatic failover. A decision was
made to first try to recover the failed primary update server. When this
was unsuccessful, our 24/7 engineers manually switched to the secondary
server and the slave database was set as the master.

Unfortunately the old legacy update system does not allow for operation
of a cluster of update servers or automated failover from master to
slave database. We have looked at building this into the system in the
past, but the whole legacy software was simply not designed with this in
mind.

As announced at RIPE 64, the RIPE NCC is completely re-writing all of
the legacy RIPE Database software. The query service is already fully
running new, re-written code, load balanced across a cluster of servers
with a high degree of redundancy. We are currently working on re-writing
the update service. A load balanced cluster of servers is one of the
primary goals of a new RIPE Database update service. This would avoid a
repeat of the outage this weekend.

We'd like to apologise again for any inconvenience caused. If you have
any further questions, please do not hesitate to get in touch.

Regards,

Denis Walker
Business Analyst
RIPE NCC Database Group