[ripe-list] [db-wg] Reminder: RIPE Database Requirements Task Force – Draft Report Published

Hi Bijal

Thanks for the detailed response. I see my role now, as this process
moves towards completion, to question some of the thinking and logic
behind the Task Force's (TF) decisions and statements. Hopefully
generating some thoughts in the minds of community (and TF) members
and maybe some discussion leading up to your final report. It would be
a pity if 2 years of work by the TF simply passed by with a whimper
and no critical review.

Let's start with another moment of reflection on purposes. You just
said "We chose to use existing RIPE Database purposes to build our
requirements list as they were still relevant." I would question the
use of the word 'existing'. You have used the 'historical' purposes
and yes they are still relevant. But let's now put this into context.
The current version of the RIPE Database was developed using the old
waterfall methodology. All the design work, including defining the
purposes and requirements, was done in the late 1990s. The software
was written mostly throughout 2000 and it went live in April 2001. By
choosing only to use the 5 historical purposes you have discussed in
your report, the case you are presenting is that absolutely nothing
has changed regarding the purpose of the RIPE Database since the late
1990s. (Except for some historical purposes that have already been
dropped like forward domains.) To say that as we move forward into
2022 and beyond, the RIPE Database is not being, and not expected to
be, used for any purpose other than what it was designed for in the
1990s, is quite a bold statement to make. And let's be clear, this IS
the statement the TF seems to be making, the purpose of the RIPE
Database has not evolved in any way in more than 20 years. If this is
not the statement the TF intended to put forward and the purposes have
evolved in any way, then you have not built your requirements list on
the full set of currently existing purposes.

I suggested previously that geolocation should be considered as a
purpose. You commented "The RIPE Database only offers partial
geolocation data and the “geoloc:” attribute data is user-maintained
and mainly unreliable." The RIPE Database has moved on from "geoloc:".
There was a long discussion early this year on "geofeed:" which is now
progressing. This is a totally different context. It is an external
service relying on a new feature of the RIPE Database. It will be user
maintained data feeding into the geolocation service providers. But as
a matter of principle, there is nothing wrong with a purpose depending
on user maintained data. The IRR and rDNS purposes both rely totally
on user maintained data. If data is unreliable that is a data accuracy
issue and should be addressed separately. Rather than define
geolocation specifically as a purpose of the RIPE Database, it could
be more generalised into a category of external services that use,
require or rely on data in the RIPE Database. CSIRT use of the IRT
object could also fit into this category, which was not covered by the
original purposes defined in the 1990s. Also some RIPE Atlas users
reference PERSON/ROLE objects for contact details. There may be other
examples that could fit this generalised purpose.

In my previous message I also asked this question "Should it be
generally accepted now that the RIPE Database is a public registry of
all users of blocks of IP addresses? This would be similar to the
public set of domain name registries. Anyone who wants to know who
operates a domain name can look it up in the appropriate domain
registry. It has become quite common that anyone who wants to know who
is using a block of IP addresses will look up those addresses in the
RIPE Database (or other RIR databases). Has it evolved from just being
a tool for network operators to a more general public service? Is it
time to recognise this as a purpose of the RIPE Database?"

I see this as a fundamental question on the purpose of the RIPE
Database that needs to be answered. It can be a qualified yes or no
but should not be ignored. In any future discussion on assignments in
working groups this will be a central issue. Moving into 2022 and
beyond, has the RIPE Database evolved (from it's original concept in
the 1990s) into a more general public service registry of IP blocks
providing more than contact details for operational issues only?

Now data minimisation. Let's be clear what the term means. My
understanding is to minimise the data needed to be gathered, processed
and stored in order to serve the purposes of the RIPE Database. If you
have a different meaning perhaps you can explain your interpretation.
What it does not mean is removing or rejecting purposes in order to
minimise overall data in the database. If that is your interpretation
then let's dump the IRR and rDNS and focus on the original resource
registration purpose only and we can have a massive minimisation of
overall data. (Just to illustrate the extremes of that logic.)

So if we agree on the definition of 'data minimisation' then this
comment makes no sense whatsoever "As stated in the document, IPAM
goes against the data minimisation principle and is not directly
matching the purposes of the RIPE Database. This is why we didn’t
consider it as a requirement." A purpose can not 'go against the data
minimisation principle'. The purposes stand above this principle. You
can then apply this principle to the data needed to serve a purpose.
Then I think what you really meant to say was "not directly matching
the 'historical' purposes of the RIPE Database". This now goes right
to the core of what the TF is doing. By only accepting the 5
historical purposes of the RIPE Database and rejecting anything else
as a possible new purpose, you are reinforcing the view that the RIPE
Database has not evolved in any way over the last 20+ years. I thought
one of the aims of the TF was to look forwards at how the RIPE
Database has and/or may evolve. You seem to only be looking backwards
and rejecting anything that does not fit in with the historical view.
According to your own survey results, almost 54% of the respondents
said they use the RIPE Database for their IPAM. But you won't consider
it as a new requirement/purpose now because it does not fit in with
this historical view. This reasoning makes no sense to me. The TF
should be looking at how the RIPE Database is being used today and
identifying possible new purposes from the way people use the
database. IPAM looks like a prime candidate to be looked into in more
detail to see if this is an acceptable and manageable use of the
database. Maybe the TF has already done this, but you can't reject it
for the reasons you have specified.

I won't restate my views here on removing assignments but will add
that this statement in support of your argument is also reversed logic
"removing this policy requirement will be in line with the data
minimisation principle." You can NOT use the data minimisation
principle to argue against a requirement or purpose. The requirements
and purposes are at a higher level than this principle.

RPSL, I admit to not being a routing expert by any means, but when the
working groups look into deprecating this 'old standard' what is the
'new standard' that will replace it?

I think publishing a non-exhaustive list of known stakeholders (both
maintainers and consumers of data) would be a good starting point for
identifying them. Anyone who feels they have been left out can then
comment and we can extend the list over time and have a better view of
who really uses the RIPE Database and why.

Finally I think this document needs an introduction that makes it
clear what this document is and what you expect to achieve with it.
There is such a confusing mix of terms. You are the RIPE Database
Requirements Task Force. You refer to the document as a 'report'. In
it you talk about reviewing the database functionality and "a
high-level approach was needed to establish a general consensus about
the functionality of the RIPE Database". You also state that it
provides "a list of high-level requirements and recommendations that
attempt to resolve ongoing and possible future issues regarding the
functionality of the RIPE Database and the data it contains." Some
clarification of this mix of terms would be very helpful. Also why you
selected the specific issues you have discussed rather than all
requirements or functionalities or ongoing issues.

cheers
denis
co-chair DB-WG


On Wed, 29 Sept 2021 at 14:31, Bijal Sanghani <bijal at euro-ix.net> wrote:
>
> Dear Denis,
>
> Thank you for your feedback. You will find the task force’s reply to your suggestions and comments below. If you need clarifications feel free to contact us.
>
> -- Data management principles
> Regarding data accuracy, we decided to stick with the current wording as we think it encompasses all database users and not only network operators. Regarding data minimisation, the question of “data inheritance” is interesting but probably out of scope for this document.
>
> -- Purposes
> We chose to use existing RIPE Database purposes to build our requirements list as they were still relevant. Even though there is a use case for geolocation, we didn’t think that it constituted a purpose. The RIPE Database only offers partial geolocation data and the “geoloc:” attribute data is user-maintained and mainly unreliable.
>
> The task force suggested to rephrase the research purpose to “enabling research and analysis into IP networking in the RIPE region”. This way, it will also include IP research from the general public which was one of your suggestions. We will also give this purpose its own section and add more background information.
>
> — Requirements
> 5.1.2 Requirement: IPv4 PA assignments
> Under the current policy which we are recommending to remove, for reasons previously explained (https://ripe82.ripe.net/archives/video/542/), there are a lot of discrepancies in how resource holders document assignments and it’s already possible for them to delete assignments. Also, removing this policy requirement will be in line with the data minimisation principle.
>
> 5.1.3 Other consideration: Using the RIPE Database as an IPAM solution
> As stated in the document, IPAM goes against the data minimisation principle and is not directly matching the purposes of the RIPE Database. This is why we didn’t consider it as a requirement.
>
> 5.1.4 Requirement: Historical data and personal data filtering
> The historical data was added since 2001, but we implemented a feature to return historical data in 2013. We will edit the draft to make the distinction. We also appreciate your suggestion regarding historical data to: “Make as much historical data publicly available as is legally permitted.” but decided to stick with the current recommendation. This recommendation came as the result of discussions we had at previous BoFs and seems to be responding to the needs of the community.
>
> 5.2 Purpose: Provisioning of Reverse Domain Name System (rDNS)
> We didn’t identify any specific requirements that will be worth mentioning in this section. Regarding ENUM data, we will mention it as a side note in the document as it holds a special status in the RIPE Database.
>
> 5.3.1 Requirement: Routing information and 5.3.2 Requirement: Maintaining accurate routing origin information
> These requirements were developed in line with today’s best practices and might indeed reflect some of the status quo.
>
> 5.3.3 Other Consideration: Routing Policy Specification Language (RPSL)
> We do think that it’s worth clarifying that RPSL is not a requirement of the RIPE Database and that the relevant working groups should look into deprecating this old standard.
>
> 5.4 Purpose: Facilitating Internet operations and coordination
> Adding a stakeholder list is complicated as you’ll always end up excluding some people even if it’s a non-exhaustive list. This is why the task force decided to stick with a broader definition of RIPE Database users.
>
> Kind regards,
> Bijal Sanghani
>
>
> On 11 Aug 2021, at 13:50, denis walker via db-wg <db-wg at ripe.net> wrote:
>
> Colleagues
>
> The DBTF asked for comments about their latest draft document by 13
> August. So far I have seen only one comment. So I am taking off my
> co-chairs hat and putting my devil's advocate hat on again and I will
> make some comments of my own and raise a number of questions. I make
> no apologies for the length of this email and I accept some people
> won't read it for that reason. But as the DBTF is approaching the end
> of it's work, there are many things that need to be said about this
> document. Whether the DBTF can or will or should take these into
> account is for others to decide. The document can be found here
> https://www.ripe.net/participate/ripe/tf/rdb-requirements-tf/the-ripe-database-requirements-task-force-draft-report-july-2021.pdf
>
> '4.1 Data accuracy'
> Here you say data should be accurate for "all parties involved in
> network operations.". But In section 3 you accept that there are
> different user groups who use the database now. So data should be
> accurate for "all parties.". It may sound irrelevant, but you also
> accept in section 3 that there is friction between different user
> groups. If this document appears to favour one user group, network
> operators, it may be used in the future to argue against changes that
> benefit user groups other than network operators.
>
> '4.3 Data minimisation'
> You only refer to personal data and your definition only relates to
> limiting data to match the purposes. While they are valid examples,
> they are not the only issues regarding data minimisation. You haven't
> covered the issue of inheritance for example. Having one "tech-c:"
> attribute which is inherited by 10,000 INETNUM objects is far better
> than having 10,000 identical copies of that one piece of data. The
> concept of inheritance is certainly within scope of the business
> requirements as it can lead to improved data accuracy.
>
> '5. Purposes, Requirements and Recommendations'
> You start by saying:
>
> "To produce its requirements, the task force looked at four purposes
> of the RIPE Database.
>
> ● Providing authoritative and accurate registration of Internet number resources
> ● Provisioning of the Reverse Domain Name System (rDNS)
> ● Publishing routing policies by network operators (RIPE IRR)
> ● Facilitating Internet operations and coordination
>
> Even though this document is based around the four purposes mentioned
> above, the task force is aware that there is a fifth purpose that
> should be taken into consideration:
> ● Enabling scientific research into network operations and topology"
>
> This suggests that the DBTF considers that these 5 items are the only
> purposes of the RIPE Database. Although you say the fifth purpose
> "should be taken into consideration", you don't consider it in this
> document. This confuses me about the nature of this document. If it is
> to be the business requirements of the RIPE Database how can you not
> discuss one of the defined purposes?
>
> When I wrote the first set of Terms & Conditions for the RIPE Database
> 10+ years ago (which were approved at the time by the community) the
> purposes were listed as:
>
> ● Ensuring the uniqueness of Internet number resource usage through
> registration of information related to the resources and Registrants
> ● Publishing routing policies by network operators (IRR)
> ● Facilitating coordination between network operators (network problem
> resolution, outage notification etc.)
> ● Provisioning of Reverse Domain Name System (DNS) and ENUM delegations
> ● Providing information about the Registrant and Maintainer of
> Internet number resources when the resources are suspected of being
> used for unlawful activities, to parties who are authorised under the
> law to receive such information.
> ● Scientific research into network operations and topology
> ● Providing information to parties involved in disputes over Internet
> number resource registrations to parties who are authorised under the
> law to receive such information.
>
> Do you not regard these items as valid purposes of the RIPE Database now:
>
> ● Providing information about the Registrant and Maintainer of
> Internet number resources when the resources are suspected of being
> used for unlawful activities, to parties who are authorised under the
> law to receive such information.
> ● Providing information to parties involved in disputes over Internet
> number resource registrations to parties who are authorised under the
> law to receive such information.
>
> All of the 5 purposes you have listed are the historic purposes of the
> RIPE Database. Has the DBTF not recognised any new, valid purposes of
> the RIPE Database? (Besides IPAM that you have rejected.) Accepting
> that some historic purposes have been dropped (like forward domain
> registrations) do you consider that the purpose of the RIPE Database
> in 2021 has not moved on or developed from what it was, say, in 2001?
> It seems to me that there is a fundamental piece of information
> missing from this document. Who are the main user groups who enter
> data into and consume data from the RIPE Database, for what purposes
> and what data do they need? Do they have an equal stake or are there
> priorities? Do conflicts exist between stakeholders and data
> requirements? I don't see how you can document the business
> requirements and functionality without first defining the main
> stakeholders, actors, end users, data consumers and their data
> requirements. (Technically the Business Enterprise Requirements would
> be followed by the Stakeholder Requirements then the Solution
> Requirements. But as this document is shifting between requirements
> and functional review it seems to have merged all these into one, but
> without including some key elements.)
>
> In section 3 you say:
>
> "Some database users, such as ISPs or IXPs, have been part of the RIPE
> community for years, while others are relatively new, such as Law
> Enforcement Agencies (LEAs) or regulators. These user groups have
> different needs and expectations regarding the database which is
> creating friction within the community."
>
> So you have recognised that there are new user groups with different
> needs but this is not reflected in the historic purposes of the
> database. The purposes clearly have changed but these changes have not
> been recognised or formally accepted. Hence the friction you refer to.
> Unless we address the purposes these new user groups want from the
> database this friction is not going to be resolved. If these issues
> are not addressed then I would consider this document does not fully
> represent the business/stakeholder requirements.
>
> Geolocation is another purpose the database has long been used for and
> this has taken on a higher profile recently by a significant user
> group. This does not fit under the purpose 'Facilitating Internet
> operations and coordination' as some other features do like abuse
> contact definitions. Being a completely separate and external service
> that requires some new data to be stored in the RIPE Database for this
> service to function it is a different type of purpose to the other
> defined purposes. This should be listed in this document as a new
> purpose.
>
> Should it be generally accepted now that the RIPE Database is a public
> registry of all users of blocks of IP addresses? This would be similar
> to the public set of domain name registries. Anyone who wants to know
> who operates a domain name can look it up in the appropriate domain
> registry.  It has become quite common that anyone who wants to know
> who is using a block of IP addresses will look up those addresses in
> the RIPE Database. Has it evolved from just being a tool for network
> operators to a more general public service? Is it time to recognise
> this as a purpose of the RIPE Database?
>
> '5.1.2 Requirement: IPv4 PA assignments'
> You refer to "A core reason for registration of IPv4 PA assignments"
> in the address policy. It should be pointed out that this is also a
> 'historic' reason. Perhaps in 2021 it is not the only reason. If it is
> accepted that the RIPE Database has evolved into a public service
> registry and this is defined as a purpose of the database then that
> provides another reason for documenting all assignments in the
> database.
>
> For the DBTF to recommend not deleting assignments after making them
> optional is like a government recommending wearing masks after
> dropping the legal requirement. Most people don't wear masks now. Most
> optional assignments will probably be deleted. Why would operators
> spend time and money maintaining optional assignment data? Unless
> there is a clear benefit to the operator or some greater need covered
> by a requirement to provide assignments, I suspect most will simply
> ditch this public data. This would destroy the RIPE Database as a
> public service registry. If some operators maintained the optional
> data and some deleted it then we end up with an incomplete and
> inconsistent data set, which is what happened with forward domain
> data.
>
> '5.1.3 Other consideration: Using the RIPE Database as an IPAM solution'
> According to your survey results almost 54% of the respondents said
> they use the RIPE Database for their IPAM. That is the second largest
> percentage of all the listed uses. Only slightly behind general number
> resource management at 57%. Given the level of usage, why do you not
> accept IPAM as a valid, new purpose of the RIPE Database? Many years
> ago forward domain data was stored in the database. Many of the
> smaller domain registries used the database as their primary registry
> database. Most of the bigger registries only published a limited data
> set and used a referral mechanism to direct queries to their whois.
> One of the main reasons for deleting this data was the inconsistent
> and incomplete, public facing, data set. What is your reasoning for
> recommending not accepting IPAM as a new purpose of the database and
> maybe a resource holder benefit?
>
> '5.1.4 Requirement: Historical data and personal data filtering'
> This is the third draft doc you have published containing this statement:
> "Since 2013, the RIPE Database has stored historical data, as
> requested by the RIPE community"
> and this is the third time I have commented that this is simply not
> correct. The database contains every version of every object that has
> ever existed since April 2001. Historical queries were implemented in
> 2013, but the data was already there going all the way back to 2001. I
> trust the DBTF will finally correct this in the next draft.
>
> Your recommendation makes no sense whatsoever. Availability of
> historical data should depend on the type of data, not on the use case
> for accessing it. To suggest that researchers could be given more data
> access on a case by case basis is madness. The principle of
> availability of historical data is basically that operational data is
> available, personal data and any data subject to privacy is not
> available. There is no reason to limit any of the available data by
> user group or use case. It should all be publicly available. No user
> group or use case is going to make the excluded personal and privacy
> data available to anyone for very strict legal reasons. (In particular
> the right to be forgotten granted by GDPR. Of course that can be
> overruled by a court order but that is an exceptional use case.)
>
> The recommendation should be to make as much historical data publicly
> available as is legally permitted. I believe some of the data
> currently redacted could still be made available. I won't go into
> details here as it gets too technical for this doc.
>
> '5.2 Purpose: Provisioning of Reverse Domain Name System (rDNS)'
> ENUM may be a small part of this but it is still significant and
> should not be overlooked in terms of defining the purpose of the RIPE
> Database.
>
> "The task force didn’t find the need for any requirements or
> recommendations attached to this purpose and therefore recommend
> maintaining the current status quo."
> Comments like this really confuse me. It again makes me question what
> is this document? If it is a business requirements document then you
> cannot say there is no need to document any requirement for one of the
> purposes of the database.
>
> '5.3.1 Requirement: Routing information'
> The DBTF makes a long list of recommendations, but the whole list is
> simply a definition of the status quo. If you want to document this
> functionality that is fine, but that is then inconsistent with the way
> you have handled other functionality. There is nothing new or
> different in these recommendations so you could have said the same as
> you did for rDNS "we recommend the status quo".
>
> '5.3.2 Requirement: Maintaining accurate routing origin information'
> Again the list of recommendations are simply the status quo. I find
> this very confusing. Initially you appear to be recommending
> something. But you are only documenting the current practise.
>
> '5.3.3 Other Consideration: Routing Policy Specification Language (RPSL)'
> I think this section is out of scope for this document. This is choice
> of technology and technical implementation. First of all, what the
> DBTF says here is not strictly accurate. The RIPE Database data model
> is 'based' on RPSL and RPSLng. It never has been a strict
> implementation. Although the RPSL standard may not have been
> maintained for decades, the implementation in the RIPE Database has
> been modified, added to and cut down many times according to the needs
> of the RIPE community. The way data is input and output does not need
> to exactly match the way data is stored internally in the database.
> All the data, not only the routing information, is currently still
> stored internally in an RPSL like format. This is because the
> community has never had the appetite to redesign the data model, not
> even with a small iterative approach (despite numerous attempts by
> myself to push for this over the last 10 years).
>
> If this document is the business requirements then it may be in scope
> to recommend a review of the format(s) of data input to and output
> from the database. The internal representation of that data is
> definitely out of scope. Even discussing what detailed routing
> information is useful is more for the level of technical specification
> than a high level business requirements document.
>
> '5.4 Purpose: Facilitating Internet operations and coordination'
> "The RIPE Database should facilitate communication and cooperation
> among stakeholders for the following reasons:"
> This needs to say, "for reasons including the following:". The list is
> not exhaustive now and there may well be other reasons in the future.
>
> 'Accuracy'
> (hair splitting comment) RFC7020 does not define 'registration
> accuracy'. It simply uses the words as in "provide accurate
> registration information". So what 'accurate' means in terms of
> registration information does not appear to have been defined
> anywhere. I think correct would be a more appropriate word than
> accurate.
>
> 'Resource Holder:'
> It should say "allocated or assigned....from the RIPE NCC directly" to
> be correct.
>
> General closing comment from me..
> Having read this document several times I am still confused about what
> it is. It is not (some form of) a Requirements document, nor is it a
> review of RIPE Database functionality. In either of those cases it
> only presents a select subset of what would be needed. What it appears
> to be, to me, is mostly a review of RIPE Database functionality that
> has been the subject of (recent) discussion and has outstanding open
> issues, with some requirements added in some cases. But even that is
> not complete.
>
> I really do appreciate the work the task force members have done. I am
> just not sure how to interpret the outcome.
>
> In the first bof in Iceland, Daniel's opening remark was "it is time
> to stop tinkering around the edges...". Looking around the room at the
> time I saw a lot of heads nodding in approval at that comment, even
> from some of the 'old grandees' of the community. Unfortunately most
> of this document, in my opinion, is still tinkering around the edges.
>
> I don't know if the DBTF was meant to document where we are now with
> the database or produce a guiding light to move onwards and upwards.
> Given that there are many recommendations it suggests it was to be the
> way forward. For me it falls well short of taking me to the place I
> imagined after listening to Daniel's inspiring speech. Perhaps an
> opportunity missed....or maybe I misunderstood the destination and/or
> route...or maybe that is the next step?
>
> cheers
> denis
>
> On Wed, 11 Aug 2021 at 12:26, Bijal Sanghani via db-wg <db-wg at ripe.net> wrote:
>
>
> Dear all,
>
> A gentle reminder for feedback on the report the RIPE Database Requirements Task Force published last month.
>
> Please let us know if you have any questions or concerns on the draft or if you’re happy with our recommendations. This will give us input into finalising the report which we plan to publish before RIPE82.
>
> We look forward to your feedback, thank you for your time - https://www.ripe.net/participate/ripe/tf/rdb-requirements-tf/the-ripe-database-requirements-task-force-draft-report-july-2021.pdf
>
> Kind regards,
> Bijal Sanghani
>
>
>
> On 1 Jul 2021, at 13:31, Bijal Sanghani <bijal at euro-ix.net> wrote:
>
> Dear colleagues,
>
> The RIPE Database Requirements Task Force has just published a draft report for your review: https://www.ripe.net/participate/ripe/tf/rdb-requirements-tf/the-ripe-database-requirements-task-force-draft-report-july-2021.pdf
>
> In this third draft, we’ve added new recommendations and edited our document based on feedback received via emails, survey and during our third BoF in May.
>
> Please review this draft and share any comments on the ripe-list before Friday, 13 August 2021.
>
> The task force will reconvene in August to discuss the feedback received and work on its final report that will be published ahead of RIPE 83.
>
> We look forward to your feedback.
>
> Best Regards,
>
> Bijal Sanghani
>
>
>
>
>
>
>
>