[ncc-services-wg] Re: [db-wg] Clean up of unreferenced person/role objects
- Previous message (by thread): [ncc-services-wg] Clean up of unreferenced person/role objects
- Next message (by thread): [ncc-services-wg] Re: [db-wg] Clean up of unreferenced person/role objects
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Andreas Polyrakis
apolyr at noc.ntua.gr
Wed Apr 18 13:50:07 CEST 2007
Denis Walker wrote: > Introduction > ------------ > > According to our database consistency statistics program (dbconstat [1]) we > currently have 460,573 unreferenced person/role objects [2]. Some of these > may be maintained, but are still unreferenced. Any personal data not > referenced by Internet resources do not fit within the purpose of the RIPE > Database. They should not be stored in the RIPE Database beyond a reasonable > 'work in progress' period. > > The RIPE NCC has had a mandate to delete these unreferenced person/role > objects since RIPE 40: > > http://www.ripe.net/ripe/wg/db/minutes/ripe-40.html (2001) > > At RIPE 41, the Database Working Group agreed that "maintained objects will > now be removed" and "gave a mandate to the RIPE NCC to continue with the > cleanup process." > > http://www.ripe.net/ripe/wg/db/minutes/ripe-41.html (2002) > > The cleanup process of 2003 > (http://www.ripe.net/db/news/unref-cleanup-200304.html) involved using a > script to run periodic cleanups. This script was put in place. However, it > failed about 18 months ago. Because of other priorities, we have not had > the time to examine this issue again until now. > > The graph showing the increase in these unreferenced person objects [3] > indicates that the cleanup script appeared not to be performing correctly. > At the start of 2006, there were about 300,000 unreferenced person/role > objects. Since then there has been a steady increase with a large increase > of almost 50,000 in February 2007. The graph also shows a slightly higher > rate of increases in these objects since February 2007. > > Because of the high number now of unreferenced person objects, we want to > raise the issue with the community again and propose the below procedure > to clean them up. > > Because redundant personal data is a serious data protection issue, we want > to take a new approach to this in the future. Once the initial cleanup is in > progress, we will create a new proposal for a new, regular cleanup > procedure. This will be sent to the Data Protection Task Force [4] and then > to the rest of the community. > > > One time bulk cleanup procedure > ------------------------------- > > Month 1 > > * Select 80,000 unreferenced person/role objects. > > Month 2 > > * Check selected person/role objects. > * Those still unreferenced: > o Delete using normal update process. > o 2000 objects per update message. > o Run updates overnight (Saturday/Sunday). > o One update every 15 minutes. > o This should avoid any unnecessary load on the servers. Hello, I suppose you have already foreseen this, but just in case: What about RIPE DB mirroring? Is there a chance that this mass deletion will cause anomalies in the NRTM mirroring? Regards, Andreas > * Select next 80,000 unreferenced person/role objects. > > Month 3 > > * Repeat process until complete. > > > Notes > ----- > > This procedure will take about 6 months to clear the current backlog, not > including the extra time that may be necessary due to the increases over > that period. > > Because of the high numbers involved, we prefer NOT to send out individual > e-mail notifications, either before or after deletion. This is to prevent a > high load on our mail servers, especially in the event of a high number of > bounced e-mails. > > This means that there will be no individual announcements to listed e-mail > addresses before the deletion and none of the usual update notifications. > This also means that even if the objects are maintained, the maintainer will > not be notified directly about the deletion of their unreferenced > person/role object. > > We will, however, announce the cleanup to the Working Group mailing lists > and as a news item on our web site home page. > > The worst problem that can occur is that someone will enter a reference to > their person/role object just as we delete it. However, as we are only > deleting unreferenced person/role objects, the time needed to re-create them > is minimal. We suspect that a very large proportion of the unreferenced > person/role objects that we will be deleting are abandoned objects that are > no longer used. > > > References > ---------- > > [1] dbconstat > http://www.ripe.net/projects/dbconstat/index.html > > [2] current unreferenced person/role objects > http://www.ripe.net/projects/dbconstat/html/cons-current.html > > [3] graph of unreferenced person object increase > http://www.ripe.net/projects/dbconstat/cons-unrefpero.html > > [4] Data Protection Task Force > http://www.ripe.net/ripe/tf/dp/index.html > -- ========================================= Andreas Polyrakis GRNET/NTUA NOC apolyr at noc.ntua.gr Office: +302107722409 Cell: +306972832445 =========================================
- Previous message (by thread): [ncc-services-wg] Clean up of unreferenced person/role objects
- Next message (by thread): [ncc-services-wg] Re: [db-wg] Clean up of unreferenced person/role objects
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]