[db-wg] 2017 and RIPEdb only handles 700 updates per *MINUTE*

HI Jeroen
The three DB-WG co-chairs (2 of whom are former RIPE NCC staff) are still actively involved in overseeing the maintenance of the RIPE Database. We don't have any 'hands on' involvement.
Things may have changed a little since I was working on the RIPE Database so I am sure Tim will correct me if I am wrong here. Updates sent by email are queued and processed sequentially. It is a batch process, not real time as with Syncupdates or Webupdates. If you submit a single email with a huge number of objects, each of those objects are processed one at a time sequentially. If you submit several emails in quick succession with that huge list of updates spread over them, the emails will be queued and processed sequentially. Not sure now if there is only one queue or several for email updates. AFAIK the database can only update a single object at a time, no matter how many processors are running to feed updates to the database.
The bottom line is, if you submit a huge number of objects to be updated it will take some time to complete. The number of daily queries massively outweighs the number of updates. So performance is naturally optimised to serve the query load. The average number of updates is easily handled, but when you get spikes it does slow down the update process for a while.
The number of objects in the database makes no difference to performance on either queries or updates. So a large amount of junk will not affect performance.

To address a couple of your points from your previous email. No one has ever defined what the "country:" attribute in an INET(6)NUM object means. It could be the location of the registries head office, their data centre, their service desk, their user base... It is not uncommon to see this value 'apparently' conflict with netname or address.

If you believe 40% of the INET6NUM objects in the RIPE Database are junk, that is something the RIPE NCC Registration Services needs to comment on.

cheersdenisco-chair DB-WG

      From: Jeroen Massar via db-wg <db-wg at ripe.net>
 To: db-wg at ripe.net 
 Sent: Tuesday, 20 June 2017, 14:23
 Subject: [db-wg] 2017 and RIPEdb only handles 700 updates per *MINUTE*
   
https://www.ripe.net/analyse/statistics/ripe-database/ripedb-updates

(see attached png for the current version as over time it will disappear)

shows a nice peak with a max of 723.47 updates.

Thus either the software is really bad, the updates are rate limited, or
the queries are so high that no further performance is to be expected.


As alluded to in the other thread, if there was less junk (those 400k
inet6nums for instance) then likely this would all scale better....

Are the RIPE DB-WG chairs still actively involved in the maintenance of
the RIPE Database!?

As two of the co-chairs are working for RIPE NCC, are they able to tell
why such a process is slow and why these problems that are being raised
over and over on the various lists are not being resolved?

Greets,
 Jeroen


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ripe.net/ripe/mail/archives/db-wg/attachments/20170620/8cd3d787/attachment.html>