DNS vs. RIPE DB checking tool

Hi,

as promised at the last RIPE meeting, please find below a copy of
the perl script I have which checks for the consistency between
what is registered in the RIPE database for the domains and what
is registered in the DNS.


Do note, this script does nothing more than report what it thinks
is inconsistencies, you still need to sift through the error
reports this script produces and fix any inconsistencies
yourselves.

This script relies on a number of conventions and is a bit picky
about what needs to be in the RIPE database if it is to not
complain too much.

First, the script has to know which domains it should check.  It
performs this decision by "traversing" the domain objects in the
RIPE database starting from the object for the top-level domain.
If the object in question has at least one sub-dom: attbibute, my
script assumes that all names registered in this domain should be
registered in the RIPE database with it's own domain object.
This way, you use the "sub-dom" attbiutes to span out a subset of
the DNS hierarchy under a given top-level domain where
administrative registration in the RIPE database is required for
DNS registration.

The script subsequently does zone transfers for the DNS domains
corresponding to the domains which should be checked as decided
above.

When querying the RIPE database, the set of name server records
for each domain is recorded, so that the script can compare these
with what is registered in the DNS.  This is one point where my
script deviates slightly from or is slightly more strict than the
RIPE-49 document, in that it

1) does not permit IP addresses in the nserver: attribute (since
   you do not delegate to an IP address in the DNS, this makes
   sense)

2) does not permit more than one word in each nserver: attribute

This makes registration such as:

nserver:  aun.uninett.no / 129.241.1.99

illegal, and

nserver:  aun.uninett.no
nserver:  129.241.1.99

would also be flagged as an error when compared to the DNS
output.  It is my beleif that the RIPE database does not
adequatly support the registration of glue records (I do not
think it needs to either), and putting the glue record
information in together in the nserver: attribute is bad when
seen from the database normalization and maintenance side.

If a domain is registerd in the DNS with only MX information, my
script demands that the corresponding registration in the RIPE
database should contain the string "MX-only" in one of the
remarks: attributes of the domain.

Subsequently, what has been harvested via the RIPE database and
what has been fetched from the DNS is compared, and any
inconsistencies are reported.  These are:

1) Check for domains registered in the RIPE database but not in
   the DNS, report these as errors

2) Check for domains registered in the DNS but not in the RIPE
   database, report these as errors

3) For delegated domains, report any inconsistencies between the
   set of nserver: attribute values and the NS records in the DNS

4) Check for domains with both MX and NS records in the DNS,
   report these as errors

5) Check that only domains marked in the RIPE database as
   "MX-only" really have an MX-only registration in the DNS.

Lastly, this script depends on dig version 2.0 (forget the one
which comes with BIND 4.9.3 -- the external interface changed for
the worse, IMHO) and on a whois client which uses getopt() to
parse it's arguments (so that I can use the '--' construct to
separate the options and the arguments).  If you have another
version of whois than this, the fix should be trivial (at the top
of "get_ripe_data".

Note: the script creates a "cache file" with name cc.ripe-cache
in the current directory containing all the domain objects
without sub-dom attributes, so that multiple runs in quick
succession can be a little faster (do however remember to remove
any objects you sent in updates for, or remove the cache after
substantial modifications).

Well, other than this, read the script -- it's some 450 lines of
simple straight-forward Perl code.


- Håvard
-------------- next part --------------
#!/local/bin/perl

# $Id: domain-check.pl,v 1.7 1995/03/12 18:37:22 he Exp $
#
# Perl script to check the consistency between what is registered in the
# RIPE database and what is registered in the DNS for the domain objects
# under a given domain.  Missing pieces (i.e. inconsistencies) for either
# registration is reported.


# Pull in data for a given domain from the DNS using dig

sub get_dns_data {
    local($domain) = @_;
    local($ns, $label, $zoneok);
    local($wildcard_label);

    $zoneok = 0;
    $ns = &get_primary_ns($domain);
    if (!defined($ns)) {
	&err("Could not get SOA for $domain\n");
	return undef;
    }
    $digcmd = "dig @$ns $domain axfr +nodefname +pfset=0xa024";
    $digcmd = $digcmd . " | tr '[A-Z]' '[a-z]' |";
    open(dig, $digcmd) || return undef;
    while(<dig>) {
	chop; split;

	if ( $_[1] eq "in" ) {
	    chop($_[0]);
	    $label = $_[0];
	    $wildcard_label = 0;
	    if ($label =~ /^\*\./) {
		$label = substr($label, 2);
		$wildcard_label = 1;
	    }
	} else {
	    if (( $_[1] eq "matching" && $_[0] eq ";" ) ||
		( $_[1] eq "received" && $_[0] eq ";;" && $_[3] eq
		 "records" )) {
		$zoneok = 1;
	    }
	    next;
	}

	if ( $_[2] eq "a" ) {
	    $dns_glued{$label} = 1;
	} elsif ( $_[2] eq "ns" ) {
	    chop($_[3]);
	    $dns_ns{$label} = $dns_ns{$label} . ":" . $_[3];
	    $dns_delegated{$label} = 1;
	    $in_dns{$label} = 1;
	} elsif ( $_[2] eq "mx" ) {
	    $dns_mx_domain{$label} = 1;
	    if ($wildcard_label) {
		$dns_mx_wildcard{$label} = 1;
	    }
	    $in_dns{$label} = 1;
	} elsif ( $_[2] eq "cname" ) {
	    $dns_cname{$label} = 1;

	} elsif ( $_[2] eq "soa" ) {
	    # nothing to do, ignore
	} elsif ( $_[2] eq "hinfo" ) {
	    # nothing to do, ignore
	} elsif ( $_[2] eq "wks" ) {
	    # nothing to do, ignore
	} else {
	    printf stderr "spurious data: $_\n";
	}
    }
    close(dig);
    if ($zoneok) {
	return 1;
    } else {
	&err("No matching SOA record found from $ns\n");
	return undef;
    }
}


sub get_primary_ns {
    local($domain) = @_;
    local($ns, $digcmd);

    $digcmd = "dig $domain soa +nodefname +aa +pfset=0xa020";
    $digcmd = $digcmd . " | tr '[A-Z]' '[a-z]' |";
    open(dig, $digcmd) || return undef;
    while(<dig>) {
	chop; split;
	if ( $_[1] eq "soa" ) {
	    close(dig);
	    return $_[2];
	}
    }
    close(dig);
    return undef;
}


# Pull in any cached data from the RIPE database previously saved to disk
# by get_ripe_data

sub get_ripe_cached_data {
    local($domain) = @_;
    local($tag);

    open(cache, $ripe_cache_name) || return undef;
    while (<cache>) {
        tr/A-Z/a-z/;
	chop; split;

	chop($_[0]);
	$tag = $_[0];

	if ( $tag eq "nserver" ) {
	    $ripe_ns{$domain} = $ripe_ns{$domain} . ":" . $_[1];
	    if ( $#_ > 1 ) {
		&err("RIPE: more than one nserver on a line for $domain\n");
	    }
	}
	if ( $tag eq "domain" ) {
	    $domain = $_[1];
	    $in_ripe_db{$domain} = 1;
	}
	if ( $tag eq "sub-dom" ) {
	    shift;
	    while ($subdom = shift) {
		$fqdn = $subdom . "." . $domain;
		push(domains_to_check, $fqdn);
		if ( ! $has_subdom{$domain} ) {
		    $has_subdom{$domain} = 1;
		    push(needs_dns_xfer, $domain);
		}
	    }
	}
	if ( $tag eq "remarks" ) {
	    if ( /mx-only/ ) {
		$ripe_mx_only{$domain} = 1;
	    }
	}
	
    }
    close(cache);
    return 1;
}
    

# Pull in data for a given domain from the RIPE database using whois
# Put received data in "ripe-cache" (append it).  The assumption is that
# get_ripe_data will not be called unless data isn't found in the cache.

sub get_ripe_data {
    local($domain) = @_;
    local($tag, $subdom, $fqdn);
    local($do_cache) = 1;
    local(@entry, $s);

    $whoiscmd = "whois -h whois.ripe.net -- '-r '$domain |";
    open(whois, $whoiscmd) || return undef;
    open(cache, ">>$ripe_cache_name") || undef $do_cache;
    while(<whois>) {
	if ( /No entries found for the selected source/ ) {
	    close(whois);
	    return undef;
	}
	if ($do_cache) {
	    push(@entry, $_);
	}

        tr/A-Z/a-z/;
	chop; split;

	chop($_[0]);
	$tag = $_[0];
	if ( $tag eq "nserver" ) {
	    $ripe_ns{$domain} = $ripe_ns{$domain} . ":" . $_[1];
	    if ( $#_ > 1 ) {
		&err("RIPE: more than one nserver on a line for $domain\n");
	    }
	}
	if ( $tag eq "domain" ) {
	    $in_ripe_db{$domain} = 1;
	}

	if ( $tag eq "sub-dom" ) {
	    if ($do_cache) {
		close(cache);
	    }
	    undef $do_cache;	# Do not cache entries with sub-dom entries
	    shift;
	    while ($subdom = shift) {
		$fqdn = $subdom . "." . $domain;
		push(domains_to_check, $fqdn);
		if ( ! $has_subdom{$domain} ) {
		    $has_subdom{$domain} = 1;
		    push(needs_dns_xfer, $domain);
		}
	    }
	}
	if ( $tag eq "remarks" ) {
	    if ( /mx-only/ ) {
		$ripe_mx_only{$domain} = 1;
	    }
	}
    }
    close(whois);
    if ($do_cache) {
	while($s = shift(@entry)) {
	    print cache $s;
	}
	close(cache);
    }
    return 1;
}


# Check existence of data in RIPE database for a given domain
# (independent of sub-dom tree).  Might as well use code above
# which pulls in data from RIPE database entry (if it exists).

sub in_ripe_db {
    local($domain) = @_;

    return &get_ripe_data($domain);
}


# Get all RIPE data (decending recursively through sub-dom) 
# for a given domain, noting registered sub-doms that do not have
# their own entry in the RIPE database

sub get_all_ripe_data {
    local($domain) = @_;
    local(@domains_to_check) = @_;

    &notify("Getting cached RIPE data for domain $domain:");

    $ripe_cache_name = $domain . "." . "ripe-cache";

    &get_ripe_cached_data($domain);

    &notify("\nGetting all RIPE DB data: ");
    while ($domain = shift(domains_to_check)) {
	if (! $in_ripe_db{$domain} ) { # Not already seen in cache
	    &notify_progress();
	    if ( ! &get_ripe_data($domain) ) {
		&err("RIPE database entry missing for $domain\n");
	    }
	}
    }
}


# Pull inn all relevant data for doing a check.  Decend through sub-dom
# entries in the RIPE database and pull in data for the relevant DNS zones
# as well.

sub pull_in_data {
    local($domain) = @_;

    &get_all_ripe_data($domain);
    &notify("Getting corresponding DNS data\n");
    while ( $domain = shift(needs_dns_xfer) ) {
	&notify("Getting zone data for $domain\n");
	if (! &get_dns_data($domain) ) {
	    &err("Zone transfer of $domain failed\n");
	}
    }
}


# Perform an extensive consistency check between the data fetched from DNS
# and those fetched from the RIPE database.  In particular, check:
# 
# 1) Check that all entries in the RIPE database are in the DNS and
#    vice versa; report inconsistencies.
#    
# 2) Where entries exist in both RIPE and the DNS, check:
#    
#    a) for delegated domains, the set of NS records in DNS and those
#       registered in the RIPE nserver entries match up
#    
#    b) for domains with both NS and MX records in the DNS, report
#       this as an inconsistency
#    
#    c) for MX-only domains (in the DNS), check that the corresponding
#       entry in the RIPE database contains the remark "MX-only".

sub do_consistency_check {
    &report_missing_ripe();
    &report_missing_dns();
    &report_ns_inconsistencies();
    &report_inconsistent_domains();
    &report_mx_comments();
}


# Report entries in the DNS which do not exist in the RIPE database

sub report_missing_ripe {
    local($domain);

    foreach $domain (keys %in_dns) {
	if ( ! $in_ripe_db{$domain} ) {
	    if ( &in_ripe_db($domain) ) {
		&err("Missing as sub-dom in parent object: $domain\n");
	    } else {
		&err("Missing in RIPE db: $domain\n");
	    }
	    $ignore{$domain} = 1;
	}
    }
}


# Report entries in the RIPE database missing in the DNS

sub report_missing_dns {
    local($domain);

    foreach $domain (keys %in_ripe_db) {
	if ( ! $in_dns{$domain} ) {
	    if ( ! $dns_cname{$domain} ) {
		&err("Missing in DNS: $domain\n");
		$ignore{$domain} = 1;
	    }
	}
    }
}


sub split_ns_list {
    local($ns_str) = @_;
    local(%array, @tmp, $elt);

    $ns_str =~ s/^://;
    @tmp = split(/[:]+/, $ns_str);
    while ($elt = shift(@tmp)) {
	$array{$elt} = 1;
    }
    return %array;
}


# Report inconsistencies between registered name servers DNS/RIPE

sub report_ns_inconsistencies {
    local(%dns_nses, %ripe_nses);
    local($heading_written) = 0;
    local($domain, $ns);

    foreach $domain (keys %in_dns) {
	if ( ! $ignore{$domain} ) {
	    %dns_nses = &split_ns_list($dns_ns{$domain});
	    %ripe_nses = &split_ns_list($ripe_ns{$domain});
	    foreach $ns (keys %dns_nses) {
		if ( ! $ripe_nses{$ns} ) {
		    &err("Missing NS in RIPE for $domain: $ns\n");
		}
	    }
	    foreach $ns (keys %ripe_nses) {
		if ( ! $dns_nses{$ns} ) {
		    &err("Missing NS in DNS for $domain: $ns\n");
		}
	    }
	}
    }
}


# Report domains with both NS and MX records in the DNS
# and domains in RIPE database with both nserver and remark: MX-only.

sub report_inconsistent_domains {
    local($domain);

    foreach $domain (keys %in_dns) {
	if (defined($dns_ns{$domain}) &&
	    defined($dns_mx_domain{$domain})) {
	    &err("Both MX and NS records for $domain in DNS\n");
	}
    }
    foreach $domain (keys %in_ripe_db) {
	if (defined($ripe_mx_only{$domain}) &&
	    defined($ripe_ns{$domain})) {
	    &err("Both nserver and remark: MX-only in RIPE db for $domain\n");
	}
    }
}


# Report MX-only domains where RIPE entry doesn't contain remark "MX-only"
# and MX-only domains in RIPE database which are fully delegated in DNS

sub report_mx_comments {
    local($domain);

    foreach $domain (keys %in_dns) {
	if ($in_ripe_db{$domain}) {
	    if ($dns_mx_domain{$domain}) {
		if (! $ripe_mx_only{$domain}) {
		    &err("MX-domain in DNS, RIPE db not marked for $domain\n");
		}
	    }
	}
    }
    foreach $domain (keys %in_dns) {
	if ($dns_delegated{$domain}) {
	    if ($ripe_mx_only{$domain}) {
		&err("MX-domain in RIPE, DNS fully delegated for $domain\n");
	    }
	}
    }
}


sub err {
    local(@args) = @_;

    if ($dot_written) {
	$dot_written = 0;
	printf stderr "\n";
    }
    printf stderr @args;
}

sub notify {
    local(@args) = @_;

    if ($dot_written) {
	$dot_written = 0;
	printf stderr "\n";
    }
    printf stderr @args;
}

sub notify_progress {

    printf stderr ".";
    $dot_written = 1;
}


$arg = $ARGV[0];

if (defined($arg) &&
    ($arg =~ /^..$/)) {
    &pull_in_data($arg);
    &do_consistency_check();
} else {
    &err("usage: domain-check.pl cc\n");
    &err("\twhere cc is a two-letter country code\n");
}