Draft Minutes, Database Working Group v4, 08-May-2001
RIPE39 @ Bologna
A. Admin stuff
- list of participants, scribe
Scribe: Shane Kerr
- agenda bashing
Draft agenda accepted as final
- minutes
approved - content-wise,
action on Wilfried and webmaster@ripe.net to insert links to
presentations
B. Actions
From previous meetings
- On Joao
to present "unique handle proposal" @ ARIN meeting
Joao: still needs more work, so it wasn't presented.
Randy Bush is going to submit an IETF draft.
Wilfried: can the draft be submitted to DB-WG mailing list?
Joao: still needs some brush up...
Wilfried: can you keep track of this?
Joao: yes,
Action on Joao to track development of draft and circulate the
draft (or a pointer to the draft) on the DB-WG mailing list
- On NCC DB Group (Joao)
implement "lir-allocated" attribute fo inetnum objects
Wilfried: still on the to-do-list,
for obvious reasons (DB software reimplementation and RPSL
transition)
- On Wilfried
Start discussion about legacy object migration on the list
[ on the agenda ]
- On Carsten
Specify the "identity override"
[ on the agenda ]
C. DB Operations Update (Andrei R., RIPE NCC)
"RIPE Database Operations Update"
Andrei Robachevsky
Outline
- Whois Service before migration
- Whois Service after the migration
- Related service
RIPE Whois Service
- Database Statistics
- Database Operations
-
3.8 million objects
[ graph ]
sharp decrease caused by migration of German domains
since then # objects grows
slight slowdown caused by Austrian and Italian domain migration
Distribution by object type
[ chart ]
Most objects were domain objects
Now most objects are person objects (about 85%)
Data protection
[ graph ]
Small number of unmaintained aut-nums
Still maintained, but legacy objects that noone can change directly
[ graph ]
Person object protection - almost 30% unmaintained
Wilfried: what about percentage?
A: don't know what happened at RIPE 37 to cause numbers so low
Wilfried: wasn't there a time when DENIC was requested to withdraw their
maintainers?
Ulf: needed to protect person objects to prevent someone else from
changing them
Andrei: *increase* was caused by German migration
Ulf: DENIC has CRYPT-PW that every German ISP is supposed to know
[ graph ]
PGP deployment
2% of objects are PGP protected
Domain migration
- Domain objects migrated
.at February 2001
.it April 2001
- Top 5
529956 .arpa
34334 .sk
11791 .hu
11132 .lt
8682 .lv
- CENTR
migration information
Unreferenced person objects grow to 88%
28 June 2000 only 12% (before German domain migration)
Today 88%
Two factors: ccTLD's still use RIPE registry as nic-hdl generator
a lot of person objects were left behind
Personal Data in the DB
3,2 M person objects
66% unreferenced and maintained
22% unreferenced and unmaintained
12% referenced
Legal consequences (details in Joao's presentation)
Operations
- Performance
- Queries
- Updates
- Uptime
Queries
About 8/sec average
[ graph ]
Spike in March for Italian domains
Spike in January for Austrian domains
Needed to find all references
Distribution of Queries
48% domains (before domain migration)
32% domains (after domain migration - almost half now IP numbers though)
Updates
About 5/min average
[ graph ]
Min/Max values since RIPE-38
Queries: 13 q/sec max, 6 q/sec min
Updates: 50427/day max, 10/day min (caused by server slowdown)
Update rate by object type
[ chart ]
Was 54% person
[ chart ]
Now 65% person after migration
56% are new persons
Downtime and Actions Taken
- No downtime
- Main server performance slowdown
9 March hardware was upgraded to cope with load
- Migration 2001-04-23
Deliberate query/update outage for 6 hours
Communication with Database Users
- Database status
http://www.ripe.net/ripencc/pub-services/db/DBstatus.html
- Database operational graphs
http://www.ripe.net/ripencc/pub-services/db/mrtg/whois.html
- FAQ
http://www.ripe.net/ripencc/faq/database/
- RIPE-DBM
RIPE DBM
The Human Face of the Database
- Engin G|nd|z
- Magnus Karlsson
- Shane Kerr
- Ambrose Magee
- Denis Walker
Magnus and Denis are new.
Unfortunately Ambrose decided to leave RIPE NCC - May is his last month.
RIPE-DBM - Activities
- User support
- DB usage monitoring
- AUP
- NRTM
- DB Server Performance Monitoring
- Mntner object requests
- PGP license requests
- proposal to discontinue this service
two reasons:
1. licenses are for (old) PGPi version with known bugs
2. new database uses GnuPG, which is open source and compatible
Typical week
1700 messages
- maintainer creation requests (25-36/week)
- PGP license requests (1-2/week)
- general questions (50/week)
etc.
lots of country-to-ip mapping questions
Whois Service After the Migration
- Migration timeline
- New Server Operatoipn
- Presentation given at Joint DB/Routing WG session:
http://www.ripe.net/ripe/meetings/current/ripe-39/presentations.html
Migration Timeline
- 2 April, ripe-db 3.0 release
- 12 April, Advised Mirror switchover
4 servers mirror the new database
- 19 April, Migration of the TEST database
Dress rehersal of the migration
- 23 April, Migration of the RIPE database
Migratiopn night that lasted 12 hours
8-page migration manual
no real problems
- 14 May, default updates switch to RPSL
- 15 October, RIPE-181 updates will no longer be accepted
New Version of the RIPE Database
- Supports RPSL (RFC2622)
- extended syntax (for all object types)
- new objects and attributes
- Supports RPSS (RFC2725)
- new authorization rules
- Supports RAToolSet (4.7.0)
RtConfig -protocol ripe
- Code is completely rewritten
Server Architecture
[ diagram ]
Queries per second weeks 14-17
No surprises, even though there were no surprises!
Slight decrease, but no explanation
[ graph ]
Updates per minutes weeks 14-17
Again, no surprises
[ graph ]
Migration TODO list
- still need to resolve minor issues
- publish consistent documentation
- make ripe-dbase-3.0.1 release
- start next generation of DB consistency project
- host and support development of RAToolSet
Related RIPE Database Services
- Database consistency project
- Will be postponed until autumn when new version is available
- Routing Reality Check
- Not much progress since last RIPE meeting
- TEST database - now in RPSL format
- , RIPE-181 format
- , RPSL format
- test-whois.ripe.net, port 43
Summary
- The database was successfully migrated to v3.0
- domain migration is almost complete
- still 2.5M unreferenced person objects left
- things to do
- DB consistency project
- RRCC project
- ripe-dbase-3.1.0 release
- DBToolSet (incl. RAToolSet)
Questions?
Holger M|nx (UUnet DE): as a multinational, really concerned about the
RFC2725 issue. Cannot use it because some objects are in other
registries, but they are still in the RIPE database, and security not
enabled. Because they have to go to other objects as well, it would
be really helpful to have mirroring between the registries.
Andrei: we have a workaround to create foreign object - but you're right
that security is low
another problem is that even if we have a perfect solution, other
registries don't implement RFC2725 (yet), so we can't guarantee
security, so this depends on collaboration with other registries - we
can't do it alone
Holger: is this high priority?
Andrei: we think this is very important - and it breaks the advantage of
RFC2725. We're going to discuss this issue in more detail at NANOG, and
probably we can find some kind of solution.
Holger: it would be helpful if you could post some progress to the
database working group list
APNIC: going to migrate to the database (RIPE v3 code), but about 6 to
9 months behind the RIPE NCC
Wilfried: does it make a difference to have the same software deployed in
two registries?
Joao: the software is not relavent, but right now the new RIPE database
software is the only code that implements RFC725. We could solve
certain authentication problems, but we would still have one problem
with ARIN's authentication/authorization model, which is quite different.
** Wilfried: I'm going to put this on the plate as the Next Big Project. The
NCC should find out what the problems are, so we can start working on it.
** Gert Dvring from spacenet: Would like to see some sort of synchronous
update mechnanism for the database. It might be useful for others as
well.
Andrei: we appreciate this kind of input. This is on the list for
"DBToolSet", so maybe this kind of thing can be done there.
Wilfried: thanks to RIPE NCC
Andrei: if you have problems, contact us
Wilfried's short list of questions:
- How many have digital signature technology available everyday?
less than half answered
- How many use commercial PGP implementation?
some
- How many use GPG?
less
- Who would be interested in a workshop or training in PGP?
some
We'll try to organize some sort of BOF or training for RIPE40
Ulf: who is still using MAIL-FROM?
[ laughter ]
D. Report from the ReImp/Mig TF (Ulf K., and NCC DB Group)
Ulf Kieber: Thanks to RIPE NCC for smooth transition!
Basically a short revamp of presentation yesterday
4 different types of problems
PGP Update problems
- took about 1 day to resolve
- 181 translation path does not process PGP correctly
Acknowledgement problems
- no ack for some updates
Conversion problems
- minor indentation issues
If you have had problems with some updates, please double-check
and/or resend updates. NCC should probably discard old update
messages, as re-processing them might rol back objects to an older
state
Wilfried: proposes to close down task force, agreed.
E. "Orphaned" person object (Joao)
How many attendees are LIR?
A fair number
Problem with several million unreferenced person objects is that we have a
potential legal problem. Certain European Union directives about what you
can store in databases. Basically you should not store information about
anyone that is not required and directly related to your operations. Not
only do we do this, but it is publicly accessible.
It's not necessary.
There for historical reasons, but not directly related to current
operations.
This is becoming a big problem. Afraid that the RIPE NCC will be held
liable for holding information that it shouldn't.
Is this something we should try to fix? If so, when? How?
Ulf: internal attribute that marks with timestamp "unused since", after a
certain period of time purge the object
Joao: but do people think that we want to proceed with deleting these
objects?
Ulf: usually maintainer objects, and this person would be held responsible
??1: DE registry is planning on deleting all of ours, and they should be
deleted. The ccTLD domain registries should pull their person objects.
Carsten Schiefner: Kind of "echo" of DE domains, should be made clear
that this status is a transitional state. A quick fix would not be
very helpful.
Wilfried: how do we overlap between unreferenced but maintained and those
necessary for DENIC?
Joao: let's not just concentrate on DENIC
Andrei: 600000 by DENIC, 400000 by (next highest)
Marcos Sanz: DENIC does not accept registration if person object is unmaintained.
Joao: problem is not that it is maintained or not, but rather that it is
there and not related to other objects in the database. If DENIC is a
problem, should we put a deadline? In the meantime, we need to solve the
problem.
??4: If person object is unreferenced, we have to remove them ASAP. Send a
note to the maintainer?
Carsten: RIPE NCC should be aware of the problem. Has the RIPE NCC already
been the target of data protection authorities? Is this really a problem
now or is it just to be watched carefully?
Joao: not worried about authorities, but rather individuals complaining
Ulf: need an automatic timeout
???: As soon as object remains unreferenced, delete it.
Wilfried: concensus that should only contain data related to the purpose of
the database. We have watched the domain name thing for quite a while,
(setting a deadline) is helpful. Propose to delete unreferenced objects by
June 1, and inform maintainers beforehand. If DENIC (or whoever) can
provide documentation of requirement, then RIPE NCC should be safe.
Andrei: are we trying to solve only DENIC problem?
Joao: no, no, no. We need a clear statement for ALL sources of information
like this.
Andrei: although DENIC contains most numbers, when DENIC removes objects,
we don't expect complaints. For *other* objects, we DO expect complaints.
** Wilfried: put an action on the NCC about what we intend to do
-> Joao will generate problem statement, proposals, and target date of
June 1. <-
??5: proposes to send e-mail to people, like ARIN did it
Ray Plzk (ARIN): we did this, then deleted
???: not useful for generic case where objects are maintained. Do we
want to handle the 0.5 million questions about this, and how?
F. Enhancements to IRRd to support SQL, Jake K. and Karen W.
"A Relational DB Approach to the IRR"
Jake Khuon, Global Crossing
Gerald Winters, Global Crossing
Karen Williams, TimesTen
"Yet Another Relational DB Approach to the IRR" :)
Gerald did most of coding at Global Crossing
Goals
- Replacement of the radix tree data structures used by IRRd to store
objects with relational tables and an SQL backend database.
- Maintain comparable performance
Motivations
- Flexible query
- "Power" of SQL
- Can make development of tools simpler
- Flexible development platform
- hard coded data structures required some hard coded query routines
- can make extensions to the database simpler
- Some data best represented as relational tables
Concerns/Requirements
- Performance
- Need to compete with speed of in-memory radix structures
- Dual datastore philosophy
- "Fast path" (IRRd native radix)
- "Slow path" (SQL tables)
- Some data would have to exist in both
- Synchronization problems
- Path decision maker routine would add latency
- SQL tables only philosophy
- Single datastore: data exists in only one place
- Would need to be in-memory for speed
- Scalability
- Needs to handle large number of objects (3+ million)
- Needs to model multiple registries
- Robustness
- Transaction logging
- replication
System Characteristics
- IRRd is now a query/update core server
- Both RPSL and RIPE-181 compliant
will not handle mixed-datatypes - all in a source must be alike
- SQL pass-thru interface
- Service front-ends implemented as modules (e.g. email)
- TimesTen is the database manager.
- No IRRd indexing
- A 100% TimesTen relational DB backend
What is TimesTen?
- An in-memory relational database
- 1000% faster than disk-based databases
- Standard database features:
- SQL2 language and full SQL transaction semantics
- Fully durable (always two database copies on disk)
- Replication for high availability
- Thread-safe
General Schema Design
- Keep a copy of each object referenced by unique 'ID'
Misc_Tbl
- All the data to search on kept in auxiliary index tables
Pre-Compiled vs. User-Defined Queries
- Use pre-compiled queries where feasible for increased performance
- IRRd ! queries
- ripewhois queries
- Allow the user to define queries (SQL select)
- select * from mntner_tbl where ...
Query Path Overview
- Pre-compiled path already bound (msec faster)
[ no number brought though ]
Whois Query Processing
- Use knowledge of object types to speed up lookups by reducing the number
of object tabes that must be searched
e.g. "rs-..." can only be a route set object
- Need radix tree '1-level-less-specific' search capability to support whois
queries
Radix Lookups
- four types
- 1 level less specific
- all levels less specific
- 1 level more specific
- all levels more specific
- use numeric representation suitable for relational tables
first, last, and middle address
Radix Lookup Examples
- select route from Route_Tbl where FA >= FAr and LA >= LAr and source = '..'
Whois Query Processing (continued)
- Need secondary indexing to support inverse lookups and person lookups
- build an inverse table as index into main table
Joao: Database is 1000% faster than disk-based? What would comparison
look like if using a RAMdisk or a big cache?
Karen: we compare with full caching, we know where our data is. We're
still 10x faster! I've seen between 20 and 40x faster at customer sites.
Joao: How to implement more or less specific queries. We use a special
index external to database. What are the advantages of using full
relational model?
Jake: When doing updates - with a radix tree, you need to synchronize tree
and databases. This was simpler.
Cengiz: More specific required radix tree, less specific was no problem.
How do you do more specific? Most of the time I just want the keys.
Cengiz: Request to implement more specific queries.
Antony Antony: to x10, how much memory for 3 million objects
Karen: less than 0.5 Gbyte in memory
G.
Something like a "force:" (or force-delete) attribute.
Unable to change a person's *name* on a person object. If not
referenced, it is easy to delete, but otherwise have to go through
RIPE-DBM mailbox.
Want a "force" attribute to specify "I really know what I'm doing" when
changing a person name.
Is this on the "wish-list" or not?
???: Pretty important to have a way to change a name. Why do we need a
special attribute? While I like the attribute, perhaps it is the wrong
way. Should just do it!
Joao: internally currently not possible. Still some objects with
references by name exist, and this prevents it. Something like 4000
objects with references by name. If we can clean up these objects, we
can release this artificial constraints.
David Kessens: thought was that a typo in the nic-hdl could overwrite the
information in a person object
Wilfried: if reference by name, change contact to to all nic-hdl list
Joao: if by name, kind of undefined, but if by nic-hdl seems like an
assignment
???: Add a remark if we do this!
[Speaker]: Is this a priority?
Santos: Allow to change name of person? If change initial shouldn't handle
change as well?
Wilfried: that's what we have now!
Andrei: two issues:
1. reference by name and cleaning up is important and a good idea
2. releasing primary key constraint - concern is unintentional hijacking
of person objects
Also from RIPE-DBM point of view, don't see a lot of requests to change
these kinds of typos. If really an issue, then we can do it, other wise
we can have a more clear procedure.
Ulf: 3 persons were left wrong because it was easier than sending to
RIPE-DBM. If I could do it myself, then would do it.
???: really not something that should go to a human
Joao: I actually support users doing it themselves. Would prefer fixing
references and removing restrictions rather than adding a new feature.
Wilfried: after unreferenced objects, how many are left?
Carsten: many objects that reference person by name and not unique
Joao: *all* name-based references left are non-unique
Wilfried: replace name with however many handles are necessary, plus put in
a remarks line
Gert: force thing is not that bad - because of typos, just a gut feeling
because I'll be the one to fix it
H. Legacy Object Migration, Richard J.
"Early Registration Record Transfers"
Richard Jimmerson, ARIN
Background
1980's
- All IP and AS registrations made by entities under a US Department of
Defense contract
1990's
- APNIC and RIPE NCC began registering IP and AS numbers for their
respective regions
Current Situation
- Many registrations not maintained by appropriate regional registry
- Many organizations have to interact with more than one RIR to modify
their registration records
- Have difficulty making timely in-addr.arpa updates
- The in-addr.arpa zone contains delegations longer than a /8 (/16's and
/24's)
Goals
- Registration records will be maintained in the appropriate RIR database
- Organizations will be able to:
- interact with a single RIR
- make more timely in-addr.arpa updates
- in-addr.arpa zone will contain only /8 delegations
Tasks
- transfer all IPv4 registrations to appropriate RIR
- transfer all AS registration to appropriate RIR
- establish in-addr.arpa domains
Maintaining in-addr.arpa
- each RIR will maintain a suite of in-addr.arpa servers
- APNIC/RIPE NCC have already deployed
- ARIN is testing
- non-shared /8 maintained by appropriate RIR
- shared /8 maintained by majority record holder
RIR coordination efforts
- sample dump provided by ARIN in early 2000
- candidate list provided by ARIN 2000-08-22
- companion dump of all network data provided 2000-08-23
- updated/additional data provided 2001-03
"Pre-RIR address space"
Mirjam Kuehne, RIPE NCC
Related Items
- Transfer of DB objects
- technically easy
- conflicts need to be resolved
- Merger algorithm for reverse delegation
- techincally easy
- ARIN setting up reverse DNS server
- start testing in Q2/2001
Current Issues
- What determines transfer?
- location of network
- Conflicts in inetnum objects
- Conflicts in person/role objects
- What if contact person is not approachable
- If in doubt, keep as is?
Possible inetnum conflicts
1. Exact match in ARIN and RIPE DB (4325)
- override one with the other
Wilfried: what do you mean by exact match?
Mirjam: address range exact match, not whole object
2. More specific network in RIPE DB
(e.g. allocation in ARIN, assignements in RIPE)
- keep both?
3. Minor mismatch
- override one with the other (latest data?)
- human intervention
- contact maintainer
4. More specific network in ARIN DB (3060)
Next Steps
- publish list of objects to be transferred
- operators encouraged to actively transfer data
- clear procedure needed
- try to resolve easy conflicts
- try to alert all contacts affected by transfer
- find out what operators need to prepare
Joao: exact match - solution is override one with the other,
but which one? we want suggestions!
David Kessens: how much data?
Joao: numbers in parenthesis
David Kessens: contact those people first, case by case
Mirjam: what if can't find?
David: then leave as is
Wilfried: could label space as "up for reclaim" (joke)
David: might be a small group of people
Wilfried: what about person objects? It could be messy.
Ray Plzak: suggestion: for exact matches, make a synthesis and
timestamp the effective data of the merger, and timestamp when the
record was created in the ARIN database
Ulf: first ask which database do I need to hold my objects
Joao: now maintained in ARIN, will be in RIPE NCC
Wilfried: got community start thinking about it - please keep us
informed!
Y. Input from other WG's
(From Local-IR)
Try to make database transactions more accessible and easy to understand
Make it more transparent for the end users
Nurani: we spend a lot of time answering questions on how to interact with
the RIPE database
suggestion of improving documentation
perhaps also easier tools (web end or other tools)
Wilfried: also not relying on NCC staff for easy-to-answer questions
also, can we recruit a small team of individuals to subscribe to a db-help
mailing list?
Andrei: planning on publishing user manual
Wilfried: database working group will take this task on before the next
meeting
See you again in Prague!
______________________________________________________________________
you are here: RIPE -> RIPE Working Groups -> Database
