Proposal for the stored/processed attribute (action 19.12)

Dear all,

When I published the release notes of the RIPE database v1.1 beta version
I also made some small notes regarding the format of the serial numbers
that are needed for the (nearly) real-time mirroring of the RIPE
database. Though, I know that you all like debating I didn't get any
reactions :-(. Therefore I would like to stir up the discussion somewhat
by highlighting the point (again) and formulate a (not the) proposal. If
I understand our action list correct, the proposal will also make the
historic action point on Marten Terpstra (19.12) about the
'stored:/processed:' attribute obsolete.


What is (nearly) real-time mirroring support:
  
The RIPE database software now labels all incoming (atomic) updates with
an ever increasing serial number and stores the updates. It is now
possible to retrieve all the changes to the database every ten minutes by
use of a program that has been added to the RIPE database distribution.
Before using the program it is required that you ask the people that
maintain the database server near you (at RIPE: <ripe-dbm at ripe.net>), to
be put in an accessslist. We haven't had any major problems with the beta
version software (so far) except for some small bugs and support files
that needed some more explanation. We are now using the software for a
mirror server at RIPE and plan to change the roles of mirror and backup
somewhere at the end of January. Also, some registries (GARR-NIS, SWITCH,
ANS) have already such a mirror server running.

                   
Problem description:


The current real-time mirroring system stores all updates together with
an ever increasing serial number in separate files. Although it works, it
doesn't allow for growth (computers tend not to be able to handle ever
increasing integer values). Also other features like building a history
of an object are not possible and system failures are difficult to
handle. From the start of the real-time mirror project we intended to
first get the thing working and to make a design that would make adding
these features possible. Since it appears to work now, I want to publish
a proposal.

There are two important things that needs to be addressed:

- We need a reliable and unique serial number.

- We want to be able to roll back *and* forth to a random date/serial
  by using the collection of all updates 
  

Proposal:  
  

We add a 'stored:' attribute to every object in the database that will
contain the following information:

stored: (NEW|UPD) DATE TIME serialnumber
        
        NEW|UPD  - are we dealing with a new object or 
                   is it already updated once ?
                   
        DATE     - YYYYMMDD date of storing including
                   the leading 19 or 20
                   
        TIME     - time in HH:MM.SS, UTC time zone
        
        SERIAL   - serial number to avoid updates with same time & date
                   This should be an (nearly) ever increasing number. It
                   could start from 0 for every date & time combination,
                   but then we still need another ever increasing number
                   for finding out if we missed updates or not.
      
Of course every detail is open to discussion, I just wrote down one of
the possibilities.

In addition to the stored attribute we need a 'replaced:' attribute that
will point to the object that it replaced and will thus nearly have the
same syntax:

replaced: SOURCE DATE TIME SERIAL

We add the SOURCE here, for the time when we start integrating the
different registry databases.

Furthermore, we need to decide if we want to have both attributes to be
visible when doing a whois query, one of them visible or all hidden
except when using a special option.

As always, please advise!

David Kessens
RIPE NCC