2nd draft DB-WG minutes, RIPE39 @ BLQ (v4)

  Attached please find the 2nd draft for the DB-WG minutes (v4).
  
  A big thank you to Shane for taking notes,
  and to Markos, Holger and Carsten for submitting corrections regarding
  missing names.

  We are still working on collecting the URLs for the presentations, and
  on aggregating the open actions - before putting the minutes up on the
  web as well.
  
  Comments and corrections still welcome - 
  in particular if you feel mis-quoted or represented as name??.
  
  Thank you, regards,
  Wilfried.
 _________________________________:_____________________________________
  Wilfried Woeber                 : e-mail: Woeber at CC.UniVie.ac.at
  UniVie Computer Center - ACOnet : Tel: +43 1 4277 - 140 33
  Universitaetsstrasse 7          : Fax: +43 1 4277 - 9 140
  A-1010 Vienna, Austria, Europe  : RIPE-DB: WW144, PGP keyID 0xF0ACB369
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Draft Minutes, Database Working Group                v4, 08-May-2001
RIPE39 @ Bologna 

A. Admin stuff
  - list of participants, scribe
    	Scribe: Shane Kerr
  - agenda bashing
  	Draft agenda accepted as final
  - minutes
    	approved - content-wise,
	action on Wilfried and webmaster at ripe.net to insert links to
	presentations


B. Actions
  From previous meetings
  - On Joao
    to present "unique handle proposal" @ ARIN meeting
    Joao: still needs more work, so it wasn't presented.
    	Randy Bush is going to submit an IETF draft.
    Wilfried: can the draft be submitted to DB-WG mailing list?
    Joao: still needs some brush up...
    Wilfried: can you keep track of this?
    Joao: yes,
    	Action on Joao to track development of draft and circulate the
	draft (or a pointer to the draft) on the DB-WG mailing list
    
  - On NCC DB Group (Joao)
    implement "lir-allocated" attribute fo inetnum objects
    Wilfried: still on the to-do-list,
    	for obvious reasons (DB software reimplementation and RPSL
	transition)

  - On Wilfried
    Start discussion about legacy object migration on the list
    [ on the agenda ]

  - On Carsten
    Specify the "identity override" 
    [ on the agenda ]


C. DB Operations Update (Andrei R., RIPE NCC)

  "RIPE Database Operations Update"
  Andrei Robachevsky <andrei at ripe.net>

  Outline
    - Whois Service before migration
    - Whois Service after the migration
    - Related service

  RIPE Whois Service
    - Database Statistics
    - Database Operations
    - <ripe-dbm at ripe.net>
  
  3.8 million objects
    [ graph ]
    sharp decrease caused by migration of German domains
    since then # objects grows 
    slight slowdown caused by Austrian and Italian domain migration

  Distribution by object type
    [ chart ]
    Most objects were domain objects
    Now most objects are person objects (about 85%)

  Data protection
    [ graph ]
    Small number of unmaintained aut-nums
    Still maintained, but legacy objects that noone can change directly
    [ graph ]
    Person object protection - almost 30% unmaintained
    Wilfried: what about percentage?
    A: don't know what happened at RIPE 37 to cause numbers so low
    Wilfried: wasn't there a time when DENIC was requested to withdraw their
      maintainers?
    Ulf: needed to protect person objects to prevent someone else from
      changing them
    Andrei: *increase* was caused by German migration
    Ulf: DENIC has CRYPT-PW that every German ISP is supposed to know
    [ graph ]

  PGP deployment
    2% of objects are PGP protected

  Domain migration
    - Domain objects migrated
     .at February 2001
     .it April 2001
    - Top 5
     529956 .arpa
      34334 .sk
      11791 .hu
      11132 .lt
       8682 .lv
    - CENTR
      migration information

  Unreferenced person objects grow to 88%
    28 June 2000 only 12% (before German domain migration)
    Today 88%
    Two factors: ccTLD's still use RIPE registry as nic-hdl generator
                 a lot of person objects were left behind

  Personal Data in the DB
    3,2 M person objects
    66% unreferenced and maintained
    22% unreferenced and unmaintained
    12% referenced
    Legal consequences (details in Joao's presentation)

  Operations
    - Performance
      - Queries
      - Updates
    - Uptime

  Queries
    About 8/sec average
    [ graph ]
    Spike in March for Italian domains
    Spike in January for Austrian domains
    Needed to find all references

  Distribution of Queries
    48% domains (before domain migration)
    32% domains (after domain migration - almost half now IP numbers though)

  Updates
    About 5/min average
    [ graph ]

  Min/Max values since RIPE-38
    Queries: 13 q/sec max, 6 q/sec min
    Updates: 50427/day max, 10/day min (caused by server slowdown)

  Update rate by object type
    [ chart ]
    Was 54% person
    [ chart ]
    Now 65% person after migration
    56% are new persons

  Downtime and Actions Taken
    - No downtime
    - Main server performance slowdown
      9 March hardware was upgraded to cope with load
    - Migration 2001-04-23
      Deliberate query/update outage for 6 hours

  Communication with Database Users
    - Database status
      http://www.ripe.net/ripencc/pub-services/db/DBstatus.html
    - Database operational graphs
      http://www.ripe.net/ripencc/pub-services/db/mrtg/whois.html
    - FAQ
      http://www.ripe.net/ripencc/faq/database/
    - RIPE-DBM

  RIPE DBM 
  The Human Face of the Database
    - Engin Gündüz
    - Magnus Karlsson
    - Shane Kerr
    - Ambrose Magee
    - Denis Walker
    Magnus and Denis are new.
    Unfortunately Ambrose decided to leave RIPE NCC - May is his last month.

  RIPE-DBM - Activities
    - User support <ripe-dbm at ripe.net>
    - DB usage monitoring
      - AUP
      - NRTM
    - DB Server Performance Monitoring
    - Mntner object requests
    - PGP license requests
      - proposal to discontinue this service
      two reasons: 
        1. licenses are for (old) PGPi version with known bugs
	2. new database uses GnuPG, which is open source and compatible
         
  <ripe-dbm at ripe.net>
    Typical week
    1700 messages
      - maintainer creation requests (25-36/week)
      - PGP license requests (1-2/week)
      - general questions (50/week)
      etc.
      lots of country-to-ip mapping questions

  Whois Service After the Migration
    - Migration timeline
    - New Server Operatoipn
    - Presentation given at Joint DB/Routing WG session:
      http://www.ripe.net/ripe/meetings/current/ripe-39/presentations.html

  Migration Timeline
    - 2 April, ripe-db 3.0 release
    - 12 April, Advised Mirror switchover
      4 servers mirror the new database
    - 19 April, Migration of the TEST database
      Dress rehersal of the migration
    - 23 April, Migration of the RIPE database
      Migratiopn night that lasted 12 hours
      8-page migration manual
      no real problems
    - 14 May, default updates switch to RPSL
    - 15 October, RIPE-181 updates will no longer be accepted

  New Version of the RIPE Database
    - Supports RPSL (RFC2622)
      - extended syntax (for all object types)
      - new objects and attributes
    - Supports RPSS (RFC2725)
      - new authorization rules
    - Supports RAToolSet (4.7.0)
      RtConfig -protocol ripe
    - Code is completely rewritten

  Server Architecture
    [ diagram ]

  Queries per second weeks 14-17
    No surprises, even though there were no surprises!
    Slight decrease, but no explanation
    [ graph ]

  Updates per minutes weeks 14-17
    Again, no surprises
    [ graph ]

  Migration TODO list
    - still need to resolve minor issues
    - publish consistent documentation
    - make ripe-dbase-3.0.1 release
    - start next generation of DB consistency project
    - host and support development of RAToolSet

  Related RIPE Database Services
    - Database consistency project
      - Will be postponed until autumn when new version is available
    - Routing Reality Check
      - Not much progress since last RIPE meeting
    - TEST database - now in RPSL format
      - <test-dbm at ripe.net>, RIPE-181 format
      - <test-rpsl at ripe.net>, RPSL format
      - test-whois.ripe.net, port 43

  Summary
    - The database was successfully migrated to v3.0
    - domain migration is almost complete
      - still 2.5M unreferenced person objects left
    - things to do
      - DB consistency project
      - RRCC project
      - ripe-dbase-3.1.0 release
      - DBToolSet (incl. RAToolSet)

  Questions?
    Holger Münx (UUnet DE): as a multinational, really concerned about the
    RFC2725 issue.  Cannot use it because some objects are in other
    registries, but they are still in the RIPE database, and security not
    enabled.  Because they have to go to other objects as well, it would
    be really helpful to have mirroring between the registries.

    Andrei: we have a workaround to create foreign object - but you're right
    that security is low
    another problem is that even if we have a perfect solution, other
    registries don't implement RFC2725 (yet), so we can't guarantee
    security, so this depends on collaboration with other registries - we
    can't do it alone

    Holger: is this high priority?

    Andrei: we think this is very important - and it breaks the advantage of
    RFC2725.  We're going to discuss this issue in more detail at NANOG, and
    probably we can find some kind of solution.
 
    Holger: it would be helpful if you could post some progress to the
    database working group list

    APNIC: going to migrate to the database (RIPE v3 code), but about 6 to
    9 months behind the RIPE NCC

    Wilfried: does it make a difference to have the same software deployed in
    two registries?

    Joao: the software is not relavent, but right now the new RIPE database
    software is the only code that implements RFC725.  We could solve
    certain authentication problems, but we would still have one problem
    with ARIN's authentication/authorization model, which is quite different.
    
**  Wilfried: I'm going to put this on the plate as the Next Big Project.  The
    NCC should find out what the problems are, so we can start working on it.

**  Gert Döring from spacenet: Would like to see some sort of synchronous
    update mechnanism for the database.  It might be useful for others as
    well.

    Andrei: we appreciate this kind of input.  This is on the list for 
    "DBToolSet", so maybe this kind of thing can be done there.

    Wilfried: thanks to RIPE NCC

    Andrei: if you have problems, contact us

  Wilfried's short list of questions:

    - How many have digital signature technology available everyday?
      less than half answered

    - How many use commercial PGP implementation?
      some

    - How many use GPG?
      less

    - Who would be interested in a workshop or training in PGP?
      some

    We'll try to organize some sort of BOF or training for RIPE40

    Ulf: who is still using MAIL-FROM?
    [ laughter ]

D. Report from the ReImp/Mig TF (Ulf K., and NCC DB Group)
   Ulf Kieber: Thanks to RIPE NCC for smooth transition!
   Basically a short revamp of presentation yesterday

   4 different types of problems
     PGP Update problems
     - took about 1 day to resolve
     - 181 translation path does not process PGP correctly
     Acknowledgement problems
     - no ack for some updates
     Conversion problems
     - minor indentation issues
     If you have had problems with some updates, please double-check
     and/or resend updates. NCC should probably discard old update
     messages, as re-processing them might rol back objects to an older
     state

   Wilfried: proposes to close down task force, agreed.

E. "Orphaned" person object (Joao)

   How many attendees are LIR?
     A fair number

   Problem with several million unreferenced person objects is that we have a
   potential legal problem.  Certain European Union directives about what you
   can store in databases.  Basically you should not store information about
   anyone that is not required and directly related to your operations.  Not
   only do we do this, but it is publicly accessible.

   It's not necessary.

   There for historical reasons, but not directly related to current
   operations.

   This is becoming a big problem.  Afraid that the RIPE NCC will be held
   liable for holding information that it shouldn't.

   Is this something we should try to fix?  If so, when?  How?

   Ulf: internal attribute that marks with timestamp "unused since", after a
   certain period of time purge the object

   Joao: but do people think that we want to proceed with deleting these
   objects?

   Ulf: usually maintainer objects, and this person would be held responsible

   ??1: DE registry is planning on deleting all of ours, and they should be
   deleted.  The ccTLD domain registries should pull their person objects.

   Carsten Schiefner: Kind of "echo" of DE domains, should be made clear
   that this status is a transitional state.  A quick fix would not be
   very helpful.

   Wilfried: how do we overlap between unreferenced but maintained and those
   necessary for DENIC?

   Joao: let's not just concentrate on DENIC

   Andrei: 600000 by DENIC, 400000 by (next highest)

   Marcos Sanz: DENIC does not accept registration if person object is unmaintained.

   Joao: problem is not that it is maintained or not, but rather that it is
   there and not related to other objects in the database.  If DENIC is a
   problem, should we put a deadline?  In the meantime, we need to solve the
   problem.

   ??4: If person object is unreferenced, we have to remove them ASAP.  Send a
   note to the maintainer?

   Carsten: RIPE NCC should be aware of the problem.  Has the RIPE NCC already
   been the target of data protection authorities?  Is this really a problem
   now or is it just to be watched carefully?

   Joao: not worried about authorities, but rather individuals complaining
   
   Ulf: need an automatic timeout

   ???: As soon as object remains unreferenced, delete it.

   Wilfried: concensus that should only contain data related to the purpose of
   the database.  We have watched the domain name thing for quite a while,
   (setting a deadline) is helpful.  Propose to delete unreferenced objects by
   June 1, and inform maintainers beforehand.  If DENIC (or whoever) can
   provide documentation of requirement, then RIPE NCC should be safe.

   Andrei: are we trying to solve only DENIC problem?

   Joao: no, no, no.  We need a clear statement for ALL sources of information
   like this.

   Andrei: although DENIC contains most numbers, when DENIC removes objects,
   we don't expect complaints.  For *other* objects, we DO expect complaints.

** Wilfried: put an action on the NCC about what we intend to do
    -> Joao will generate problem statement, proposals, and target date of
       June 1. <-

   ??5: proposes to send e-mail to people, like ARIN did it

   Ray Plzk (ARIN): we did this, then deleted

   ???: not useful for generic case where objects are maintained.  Do we
   want to handle the 0.5 million questions about this, and how?


F. Enhancements to IRRd to support SQL, Jake K. and Karen W.

  "A Relational DB Approach to the IRR"
  Jake Khuon, Global Crossing
  Gerald Winters, Global Crossing
  Karen Williams, TimesTen

  "Yet Another Relational DB Approach to the IRR"  :)
  Gerald did most of coding at Global Crossing

  Goals
  - Replacement of the radix tree data structures used by IRRd to store
    objects with relational tables and an SQL backend database.
  - Maintain comparable performance

  Motivations
  - Flexible query
    - "Power" of SQL
    - Can make development of tools simpler
  - Flexible development platform
    - hard coded data structures required some hard coded query routines
    - can make extensions to the database simpler
  - Some data best represented as relational tables

  Concerns/Requirements
  - Performance
    - Need to compete with speed of in-memory radix structures
  - Dual datastore philosophy
    - "Fast path" (IRRd native radix)
    - "Slow path" (SQL tables)
    - Some data would have to exist in both
    - Synchronization problems
    - Path decision maker routine would add latency
  - SQL tables only philosophy
    - Single datastore: data exists in only one place
    - Would need to be in-memory for speed
  - Scalability
    - Needs to handle large number of objects (3+ million)
    - Needs to model multiple registries
  - Robustness
    - Transaction logging
    - replication
  
  System Characteristics  
  - IRRd is now a query/update core server
    - Both RPSL and RIPE-181 compliant
      will not handle mixed-datatypes - all in a source must be alike
    - SQL pass-thru interface
    - Service front-ends implemented as modules (e.g. email)
  - TimesTen is the database manager.
    - No IRRd indexing
    - A 100% TimesTen relational DB backend

  What is TimesTen?
  - An in-memory relational database
  - 1000% faster than disk-based databases
  - Standard database features:
    - SQL2 language and full SQL transaction semantics
    - Fully durable (always two database copies on disk)
    - Replication for high availability
    - Thread-safe

  General Schema Design
  - Keep a copy of each object referenced by unique 'ID'
    Misc_Tbl
  - All the data to search on kept in auxiliary index tables

  Pre-Compiled vs. User-Defined Queries
  - Use pre-compiled queries where feasible for increased performance
    - IRRd ! queries
    - ripewhois queries
  - Allow the user to define queries (SQL select)
    - select * from mntner_tbl where ...

  Query Path Overview
  - Pre-compiled path already bound (msec faster)
  [ no number brought though ]

  Whois Query Processing
  - Use knowledge of object types to speed up lookups by reducing the number
    of object tabes that must be searched
    e.g. "rs-..." can only be a route set object
  - Need radix tree '1-level-less-specific' search capability to support whois
    queries

  Radix Lookups
  - four types
    - 1 level less specific
    - all levels less specific
    - 1 level more specific
    - all levels more specific
  - use numeric representation suitable for relational tables
    first, last, and middle address

  Radix Lookup Examples
  - select route from Route_Tbl where FA >= FAr and LA >= LAr and source = '..'

  Whois Query Processing (continued)
  - Need secondary indexing to support inverse lookups and person lookups
    - build an inverse table as index into main table

  Joao: Database is 1000% faster than disk-based?  What would comparison
  look like if using a RAMdisk or a big cache?
  
  Karen: we compare with full caching, we know where our data is.  We're
  still 10x faster!  I've seen between 20 and 40x faster at customer sites.

  Joao: How to implement more or less specific queries.  We use a special
  index external to database.  What are the advantages of using full
  relational model?

  Jake: When doing updates - with a radix tree, you need to synchronize tree
  and databases.  This was simpler.

  Cengiz: More specific required radix tree, less specific was no problem.
  How do you do more specific?  Most of the time I just want the keys.

  Cengiz: Request to implement more specific queries.

  Antony Antony: to x10, how much memory for 3 million objects

  Karen: less than 0.5 Gbyte in memory


G. 
  Something like a "force:" (or force-delete) attribute.

  Unable to change a person's *name* on a person object.  If not
  referenced, it is easy to delete, but otherwise have to go through
  RIPE-DBM mailbox.

  Want a "force" attribute to specify "I really know what I'm doing" when
  changing a person name.
  
  Is this on the "wish-list" or not?

  ???: Pretty important to have a way to change a name.  Why do we need a
  special attribute?  While I like the attribute, perhaps it is the wrong
  way.  Should just do it!

  Joao: internally currently not possible.  Still some objects with
  references by name exist, and this prevents it.  Something like 4000
  objects with references by name.  If we can clean up these objects, we
  can release this artificial constraints.

  David Kessens: thought was that a typo in the nic-hdl could overwrite the
  information in a person object

  Wilfried: if reference by name, change contact to to all nic-hdl list

  Joao: if by name, kind of undefined, but if by nic-hdl seems like an
  assignment

  ???: Add a remark if we do this!

  [Speaker]: Is this a priority?

  Santos: Allow to change name of person?  If change initial shouldn't handle
  change as well?

  Wilfried: that's what we have now!

  Andrei: two issues: 
    1. reference by name and cleaning up is important and a good idea
    2. releasing primary key constraint - concern is unintentional hijacking
       of person objects
    Also from RIPE-DBM point of view, don't see a lot of requests to change
    these kinds of typos.  If really an issue, then we can do it, other wise
    we can have a more clear procedure.

  Ulf: 3 persons were left wrong because it was easier than sending to
  RIPE-DBM.  If I could do it myself, then would do it.

  ???: really not something that should go to a human
            
  Joao: I actually support users doing it themselves.  Would prefer fixing
  references and removing restrictions rather than adding a new feature.

  Wilfried: after unreferenced objects, how many are left?

  Carsten: many objects that reference person by name and not unique

  Joao: *all* name-based references left are non-unique

  Wilfried: replace name with however many handles are necessary, plus put in
  a remarks line

  Gert: force thing is not that bad - because of typos, just a gut feeling
  because I'll be the one to fix it


H. Legacy Object Migration, Richard J.

  "Early Registration Record Transfers"
  Richard Jimmerson, ARIN

  Background
    1980's
      - All IP and AS registrations made by entities under a US Department of
      Defense contract
    1990's
      - APNIC and RIPE NCC began registering IP and AS numbers for their
      respective regions

  Current Situation
    - Many registrations not maintained by appropriate regional registry
    - Many organizations have to interact with more than one RIR to modify
      their registration records
    - Have difficulty making timely in-addr.arpa updates
    - The in-addr.arpa zone contains delegations longer than a /8 (/16's and
      /24's)

  Goals
    - Registration records will be maintained in the appropriate RIR database
    - Organizations will be able to:
      - interact with a single RIR
      - make more timely in-addr.arpa updates
    - in-addr.arpa zone will contain only /8 delegations

  Tasks
    - transfer all IPv4 registrations to appropriate RIR
    - transfer all AS registration to appropriate RIR
    - establish in-addr.arpa domains

  Maintaining in-addr.arpa
    - each RIR will maintain a suite of in-addr.arpa servers
      - APNIC/RIPE NCC have already deployed
      - ARIN is testing
    - non-shared /8 maintained by appropriate RIR
    - shared /8 maintained by majority record holder

  RIR coordination efforts
    - sample dump provided by ARIN in early 2000
    - candidate list provided by ARIN 2000-08-22
    - companion dump of all network data provided 2000-08-23
    - updated/additional data provided 2001-03

  "Pre-RIR address space"
  Mirjam Kuehne, RIPE NCC

  Related Items
    - Transfer of DB objects
      - technically easy
      - conflicts need to be resolved
    - Merger algorithm for reverse delegation
      - techincally easy
      - ARIN setting up reverse DNS server
      - start testing in Q2/2001

  Current Issues
    - What determines transfer?
      - location of network
    - Conflicts in inetnum objects
    - Conflicts in person/role objects
    - What if contact person is not approachable
    - If in doubt, keep as is?

  Possible inetnum conflicts
    1. Exact match in ARIN and RIPE DB (4325)
      - override one with the other
      Wilfried: what do you mean by exact match?
      Mirjam: address range exact match, not whole object
    2. More specific network in RIPE DB
       (e.g. allocation in ARIN, assignements in RIPE)
      - keep both?
    3. Minor mismatch
      - override one with the other (latest data?)
      - human intervention
      - contact maintainer
    4. More specific network in ARIN DB (3060)

  Next Steps
    - publish list of objects to be transferred
      - operators encouraged to actively transfer data
      - clear procedure needed
    - try to resolve easy conflicts
    - try to alert all contacts affected by transfer
    - find out what operators need to prepare

  Joao: exact match - solution is override one with the other, 
  	but which one? we want suggestions!

  David Kessens: how much data?

  Joao: numbers in parenthesis

  David Kessens: contact those people first, case by case

  Mirjam: what if can't find?

  David: then leave as is

  Wilfried: could label space as "up for reclaim" (joke)

  David: might be a small group of people

  Wilfried: what about person objects?  It could be messy.

  Ray Plzak: suggestion:  for exact matches, make a synthesis and
  timestamp the effective data of the merger, and timestamp when the
  record was created in the ARIN database

  Ulf: first ask which database do I need to hold my objects

  Joao: now maintained in ARIN, will be in RIPE NCC

  Wilfried: got community start thinking about it - please keep us
  informed!

Y. Input from other WG's
  (From Local-IR)
  Try to make database transactions more accessible and easy to understand
  Make it more transparent for the end users

  Nurani: we spend a lot of time answering questions on how to interact with
  the RIPE database
  suggestion of improving documentation
  perhaps also easier tools (web end or other tools)

  Wilfried: also not relying on NCC staff for easy-to-answer questions
  also, can we recruit a small team of individuals to subscribe to a db-help
  mailing list?

  Andrei: planning on publishing user manual

  Wilfried: database working group will take this task on before the next
  meeting

  See you again in Prague!
  ______________________________________________________________________