[ncc-services-wg] Updated Hostcount++ proposal

Dear Colleagues,

Please find below an updated proposal for Hostcount++.  Hostcount ++ is
a set of enhancements to the hostcount service that the RIPE NCC has
been running for 15 years.

The Hostcount++ concept was presented at RIPE 50:
http://www.ripe.net/ripe/meetings/ripe-50/presentations/ripe50-serv-hostcount.pdf

The general feedback was positive, and there were some follow-up
discussions. For examples of these discussions see:

http://www.ripe.net/ripe/maillists/archives/ncc-services-wg/2005/msg00046.html
http://www.ripe.net/ripe/maillists/archives/ncc-services-wg/2005/msg00051.html

The Hostcount ++ proposal has been refined based on this feedback. Your
comments and suggestions are appreciated.

The RIPE NCC will make a short update on this project at the RIPE NCC
Services Working Group at RIPE 51.

Best regards,

Andrei Robachevsky
RIPE NCC


Hostcount++ for the RIPE NCC Service Region
===========================================

Motivation
----------

Every month the RIPE NCC coordinates the collection of data from the DNS
zones of ccTLDs in the RIPE NCC Service Region and publishes summary
statistics derived from this data.

The first hostcount was in October 1990 and involved 19 ccTLDs. Since
then, the hostcount has expanded to include around 100 ccTLDs.

The hostcount is provided as an informational service to the Internet
community.

The current implementation of the Hostcount has several problems:

- Connectivity problems with certain hosts
- Lack of data accuracy due to access policies of some ccTLD domains
- The increasing number of domains

There are also issues related to privacy and acceptable use of the results.

Objectives and Deliverables
---------------------------

1) To increase the precision and reliability of data, especially for
trend analysis, more sources will be added. Our final aim is to collect
data from:

  - Forward DNS
  - Reverse DNS
  - BGP view on availability of prefixes

2) Re-designing the software to be more scalable is very important,
since the amount of input to be analysed is already of considerable size
and is increasing every day.

3) To protect the privacy of domain holders and data, the following
changes will be made:

  - Use cases will be defined in more detail and "Acceptable Use Policy"
    documents will be written accordingly.

  - Raw data will not be made available.

4) The following information will be made available, via textual dumps
and graphs and tables on the web site:

   - Average visibility of RIPE NCC allocations:
     Visibility will be calculated by the information provided from
     the Routing Information Service (RIS) project.

   - Number of hosts and Average number of hosts:
     Host information will be retrieved via zone transfers. If the zone
     transfers fail due to blocking, partial zone information will be
     generated from reverse scan. Blocked zone transfers will be
     constructed by enumeration of the zones.

   - Breakdown of DNS record types:
     This information will depend on zone transfers of each allocation.

5) Results will be sorted into:

   - visible/invisible: This is based on whether a given IP range
     has been announced in the preceding month for longer than a
     particular threshold.

   - ccTLDs: This is based on the ccTLDs in the RIPE NCC service
     region.

   - RIPE NCC allocations: These are the allocations made by the
     RIPE NCC for its members.

   - IANA allocations: These are the /8 ranges for IPv4, and /23 ranges
     IPv6, allocated to the RIPE NCC.

6) In addition, timestamps of the results will be more consistent.

7) Hostcounts for certain ccTLDs will be performed locally by
contributors. We look forward to increasing the number of contributors
to distribute the load of the overall process.

8) Resulting software will be open-source, enabling a do-it-yourself kit
for local statistics gathering.