[db-wg] Looking for good way to find blocks allocated directly by ripe

On Mon, 22 Sep 2003, Shane Kerr wrote:

> > I'd like to find what is the best way to determine if particular inetnum 
> > was allocated directly by RIPE, this would include all blocks allocated 
> > to LIRs, all blocks allocated to large organizations and blocks allocated 
> > as small PI blocks.
> 
> Please have a look at:
> ftp://ftp.ripe.net/ripe/stats/issued
> 
> The README explains the contents, and you can review the source code there 
> if you want slightly different versions of this data.
> 
> You can also look at:
> ftp://ftp.ripe.net/ripe/stats/
> 
> The two are slightly different views on the same data.  The first comes 
> from the Whois Database itself, and the second is produced from our 
> internal registry database.

I have examined both of the above. Interesting to find set of scripts in 
issued that are somewhat parallel to what I wrote (and I did not know I 
could issue command to get entire list of inetnums from ripe whois for /8
so easily...), however the comparison to the scripts that I have shows 
that neither stats nor issued are accurate representation of the allocations
as neither one accounts well for small PI blocks (where you added "super 
block" which is the one being counted as if that entire range is allocated).
This particular part is important enough because of each block within PI 
range is accounted separately you will find that 4% of the 193/8 block 
(2836 /24s) is actually not allocated or at least not listed in whois. 
The details about difference between calculations done based on RIPE
stats calucations and direct from whois dump by my scripts are as follows:
http://www.completewhois.com/bogons/data/data-fromrirstatfiles/difference-stats_notin_whois-netrange-ripe.txt

I also ran comparison to see if my scripts improperly filter many actively 
used ip blocks and while the difference as seen from above files is 
significant, I only found the following that would be filtered:
  193.43.12.0/23
  193.57.12.0/24
  193.57.180.0/24
  193.58.186.0/23
  193.148.96.0/20
  193.168.0.0/24
  193.218.202.0/24
  193.221.208.0/22
So I would like to find of the above which ones are REALLY allocated (as 
such that there is no whois should be considered an omission, that I'm hope
you will fix). But if they are not allocated or allocations/assignments were
withdrawn by RIPE, then I will consider it acceptable to filter those blocks

Also regarding statistic files produced by RIRs, I'm fairly certain now 
that data can not be relied any better then whois. Those files are generally
equivalent to whois data for newwer /8 blocks but for legacy blocks a lot 
of data is simply misssing and there whois is a LOT better.

> There are some records in the issued files that are not in the stats files. 
There aren't any such records for new /8, the difference is legacy, see my
comment above. I track these kind of difference at: 
http://www.completewhois.com/bogons/data/data-fromrirstatfiles/difference-whois_notin_stats-cidr-ripe.txt

(Do note though that for APNIC there are blocks in whois but not in stats)

> We are in the planning phase of a project to account for this and 
> other differences, including inter-RIR conflicts, based on the excellent 
> work that Geoff Huston has been doing on projecting the exhaustion of IPv4.
I've talked to him and he also thinks proper whois search is better. And
if there is a difference between RIR allocation statistics fiels and 
whois, it should RIR job as maintainers of public records to fix WHOIS.

> If you intend to filter based on this data, I recommend using the issued 
> files, because it contains a superset, and you are less likely to block 
> legitimate traffic.
My whois scripts provide similar results to issued, they are just a lot
more specific whenever possible to look for any whois blocks that have 
RIR itself as first superblock.

P.S. For those interested, I'm almost ready now to release my whois-based 
allocation statistics and bogons ip list data which will happen next week 
with announcement at NANOG, I can post a copy here for RIPE. But I would like 
to get clear answer regarding the particular 8 ip blocks I listed above.

Thanks for your help and suggestions

-- 
William Leibzon
Elan Networks
william at elan.net