Database crash

Mojn guys
I had a server crash last week, and when I got it up and running again, the database was not able to receive updates - the whois-server core dumped. I've been there before, so I kept the old pseudo defect database running and rebuild a new DB during the night (downloading latest ripe.db.gz and RIPE.CURRENTSERIAL)

Rebuilding ended up with an error. I can't remember seeing this before, but my first thought was that it something to do with the already running database.
*** ERROR: Loader failure [139]. Exiting\n
************** ERROR ***************
*** 04:56:54 Error loading database=WHOISDB for source=RIPE, /usr/local/whoisserver/bin/make_db exiting
************************************
I've started NRTM through the console, and everything looks fine, but apparently my current serial is 0 (zero) and not 28071882(+) as I would expect.

20140227 14:31:32 whois_rip-25050/2356136848 INFO connecting to NRTM server [whois.ripe.net:4444] (current serial=0)
20140227 14:31:32 whois_rip-25050/2356136848 INFO ******** report **********
20140227 14:31:32 whois_rip-25050/2356136848 INFO 0 objects OK ( 0.0000 obj/s)
20140227 14:31:32 whois_rip-25050/2356136848 INFO 0 objects failed
20140227 14:31:32 whois_rip-25050/2356136848 INFO average processing time 0.0000 obj/s ( 0.00 obj/min)
20140227 14:31:32 whois_rip-25050/2356136848 INFO forwarded to serial:0

The old database (table serials) contains some 5051203 rows ending with serial=28072454
The new database (table serials) is pretty empty:
mysql> desc serials;
+-------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+----------------+
| thread_id | int(10) unsigned | NO | MUL | 0 | |
| serial_id | int(11) | NO | PRI | NULL | auto_increment |
| object_id | int(10) unsigned | NO | MUL | 0 | |
| sequence_id | int(10) unsigned | NO | | 0 | |
| atlast | tinyint(4) unsigned | NO | | 0 | |
| operation | tinyint(4) unsigned | NO | | 0 | |
+-------------+---------------------+------+-----+---------+----------------+
6 rows in set (0.00 sec)

mysql> select serial_id from serials;
Empty set (0.00 sec)

Digging deeper into the the problem I found this - this is the actual error when it crashed during make_db the first night:
WARNING - 'Feb 27 04:56:54 snowblind kernel: [67641.487347] loader[22291]: segfault at 0 ip 00000000 sp bfb7fddc error 4 in loader[8048000+1a6000]' (1 matches for 'WARNING,[Ww]arning,ERROR,[Ee]rror') found in '/var/log/messages'

I wonder if the ripe.db.gz is broken ("ip 00000000") ?

So, I ran another make_db - but with the same result. Then I recovered an mysqldump from the start of the week, and it started up OK. Then I activated NRTM and updates started to roll in ...
Happy I was for some hours; the server crash again saturday night :-(

Today even startup of whois-server fails completely with core dumps, so I've picked another mysql backup and it's curently recovering.

It seems me like one of the updates are causing problems - it's just a feeling - not a solid "Hey, here's the culprit!" fact.

I've attached the error log (from today - core dumping (crash recovery mode + roll back))

Anyone ?

Best Regards,
Peter Landbo



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.ripe.net/ripe/mail/archives/db-help/attachments/20140303/2a1703ce/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: whois-server.err.log.20140303
Type: application/octet-stream
Size: 9679 bytes
Desc: not available
URL: <https://www.ripe.net/ripe/mail/archives/db-help/attachments/20140303/2a1703ce/attachment.obj>