Practicalities

All sponsored hosts are responsible for the provision of:

• Space in a data centre for the TTM equipment
• Sufficient power and cooling
• All of the infrastructure that is required to connect the node to the Internet
• Remote hands support.

The RIPE NCC will provide TTM equipment for fully sponsored hosts. Partially sponsored hosts will need to provide server hardware. The RIPE NCC will provide the necessary Global Positioning System (GPS) antenna equipment to all sponsored hosts. No TTM service fees will be charged to any sponsored host.

Sponsorship Criteria

This document lays out the criteria used to select a site for sponsorship. Although these guidelines cover a range of site/host requirements, fulfilling all these requirements does not automatically entitle the host/site to receive a sponsored node. Similarly, those hosts/sites not meeting all the requirements are not automatically excluded.

Preference will be given to neutral and independent bodies, such as academic or not-for-profit organisations, and to those organisations that work to support and promote the development of the Internet. Commercial organisations are not normally considered as candidates for sponsorship.

Requirements

1. TTM nodes should be located:

• Within a major Autonomous System (AS) or;
• At a point where a large number of ASs peer, such as a major Internet Exchange or;
• At any other point of significant interest, such in proximity to a root DNS server

This is to ensure that the node serves as a good reference point and so any conditions, such as bottlenecks, which might distort the general connectivity overview, can be avoided.

Due to heavy graphic use, this RIPE Document is best read in .pdf format.

On request, we will supply a .html version.

This document discusses the Routing Information Service (RIS), a project proposed as a new activity of the RIPE NCC in the RIPE NCC activity plan for 1999 and 2000 [1, 2]. The document gives an overview of the project, discusses the implementation and mentions a couple of points that have to be addressed in future design documents. This document is intended to solicit input from interested parties.

Introduction

Outline of this document

This document discusses the Routing Information Service (RIS), a project proposed as a new activity of the RIPE NCC in the RIPE NCC activity plan for 1999 and 2000.

The outline of this document is as follows: section 1.2 gives the necessary background for this project. The next section 1.3 briefly discusses which tools are already available and show why the RIS is needed. The next section (1.4) gives an overview of the goals for the RIS. Section 1.5 shows the
basic setup up the RIS.

In order to prove the concept and to get a feeling for the data-volumes and other problems involved in setting up this service, a prototype was developed. This prototype is discussed in section 2.

The design of the final system is discussed in section 3. The prototype also showed that there are a number of open questions that have to be answered before the design is finalized. These are discussed in section 4.The project is closely related to the IRR/Reality Checking project [3] and
section 5 discusses the interface between the two.  Collaboration with other, related projects is discussed in section 6. Finally, the implementation schedule is discussed in section 7.

Background

In general, the Internet consists of large number of interconnected regional national and international backbones. The backbones often peer or exchange traffic and routing information with one another at Internet exchange points. These exchange points can be considered the "core'' of the Internet. Backbone providers participating in the core must maintain a complete reachability information or default free routing table of all globally visible network-layer address reachable throughout the Internet.

From a routing point of view, however, the Internet can be considered to be partitioned into a number of independent sub-networks, the so-called Autonomous Systems (AS's). The Autonomous System's are connected to one or more Autonomous Systems. From this viewpoint, it follows that routing in the Internet can be divided into two domains: internal, or inside an AS, and external, or between AS's. At the boundary of each AS, border routers exchange reachability information to destination IP address blocks or prefix, for both transit networks and networks originating in in that domain.

This project only deals with external routing, tools to study internal routing have been developed by a number of router vendors as well as others. This project also focuses on so-called default-free routers. A default-free router is a router that actively decides where to send packets with a destination outside the AS to which the router belongs, and not forward it, by default, to another router.

How IP packets are routed from one AS to another, is is based on propagating information about network entities (IP address prefixes) and routing policies between them through the network. The commonly used protocol for exchanging this information is the Border Gateway Protocol, version 4 (BGP4) [4]. BGP is an incremental protocol that sends update information only upon changes in network topology or routing policy. As a path vector routing protocol, BGP limits the distribution of a router's reachability information to its peers or neighbor routers.

A path is a sequence of intermediate autonomous systems between source and destination routers that form a directed route for packets to travel. Router configuration files allow the stipulation of routing policies that may specify the filtering of specific routers, or modification of path attributes sent to neighbor routers. Routers may be configured to make policy decisions based on both the announcement of routes from peers and accompanying attributes and local policies. The attributes in the announcements may serve as hints to routers to choose from alternate paths to a given destination.

Backbone border routers at core locations may have 50 or more external peers as well as large number of intra-domain peering sessions with internal backbone routers. After each router makes a new local decision on the best route to a destination, it will sent that route, or path information along with accompanying distance metrics and path attributes to each of its peers. As this reachability information travels through the network, each router along the path appends its unique AS number to a list in the BGP message.This list is the route's AS-PATH. An AS-PATH in connection with a prefix provides a specific handle for a one-way transit route through the network.

Routing information in BGP appears as a series of announcements that either update or withdraw a route within the network. A route update indicates a router either has learned of a new network attachment or has made a policy decision to prefer another route to a certain network destination. Route withdrawals are sent when a router makes a new decision that a network is no longer reachable. A BGP update may contain multiple route announcements and withdrawals. In stable routing environment routing updates should be very less except when policies changes and Addison of new physical networks. In reality the number of BGP updates exchanged per day in the Internet core is one or more orders of magnitude larger than expected [6].

From routing updates routers maintain a so-called routing table containing information about topology and policies (see [5] for more details). From the information in the routing table, a forwarding table is extracted every time a route is modified. The forwarding table tells the router to which interface a packet with a certain IP-prefix has to be sent.

Both the routing and forwarding tables are large, 64k prefixes or more and can change very rapidly. Routers therefore only maintain the current tables, as they do not need more information for their operations. This set-up has several operational consequences.

First of all, it is easy to determine for a network operator of certain AS, if and how another AS can be reached. All he has to do, is look at the current routing table for the path from his AS to the target AS. This would be sufficient for one-way communication, but in practice, most communication is two-way and there is no way for the network operator to tell if the target AS has a route back to his AS.

Related to this is that reachability metrics are limited to what an observer at a certain AS can see. For example, it will be easy to define a metric to describe which part of the Internet can be reached from this AS but the equally interesting opposite metric, which part of the Internet can reach our AS, is impossible to extract.

Finally, the fact that only the current table is maintained, means that any history about the route will be lost forever. However, historic information is essential when trying to detect instabilities or debug problems that only show up every once in a while. For example, if a user complains that a certain site cannot be reached at certain times, looking at the current table is not going to be of any use, as the site is either reachable or it is not. The routing table does not give any clue when the route was last announced or withdrawn.

Existing tools

Over the years, several tools to access routing information have been developed, such as: BGP-routing table dump tools and the well-known traceroute and ping programs. Traceroute combined with path information shows how site can be reached, ping shows if end-to-end two-way communication to a site is possible. All these programs have in common that they only deal with the present situation and one way from observers network. In order get the complete picture of end-end-end communication we often have to look for reachability from remote location to your own network.

One possibility is to use a so-called Looking Glass. Looking Glasses provide access to the routers of the remote AS, allowing an observer to see how traffic from the remote AS is routed back to his network (similar to a traceroute on the remote AS) and thus determine the reachability of the observer's network from the remote AS (prefix and  AS-PATH).

However, not all sites have installed this facility and the Looking Glass only provides information about the AS where it has been installed. In order to find which sites can reach a certain AS, one would have to loop over the Looking Glasses at all AS's. This is not practical. Finally, the Looking Glass, again, provides no information about the development of the routing over time.

Other tools include ASExplorer [9] and RouteTracker. Both are developed by Merit and are being used to collect routing announcements from 5 core locations in the United States. However, these are tools primarily intended for research purposes and not for network operators.

In short, it is time for a tool that combines information from several default-free core locations in RIPE region. This is the Routing Information Service.

Figure 1: Basic set-up of the Routing Information Service

Goals of the Routing Information Service

The goal of the Routing Information Service (RIS) is to collect routing information between Autonomous Systems (AS) and their development over time from default free core the Internet. The RIS will collect and store default free BGP announcements as a function of time from several locations and provide that information to the users of the service, allowing them to see the full picture with all routes that are currently anywhere and their development over time. In other words, it can be regarded as one integrated Looking-Glass for the entire Internet that includes history information.

One important application for this data will be debugging. For example, if a user complains that a certain site could not be reached earlier, the RIS will provide the necessary information to discover what caused this problem. This is particularly useful for problems that occur periodically.

There are numerous other applications for the data, for example;

1. The routing table and its development over time can be used to check for local and global convergence of the table and routing flaps.
2. The routing table reflects the policies announced by the sites operating the routers. At a local level, the data can be used to verify the setup of the routers and correct any errors.
3. On global level, the RIS data can be used to compare policies registered in the Routing Registry against the policies that are actually announced. The RIS will therefore provide essential input for the Reality Checking project for the IRR.
4. Related to this is that the RIS will provide information about fake routes inserted into the network, for example by spammers.

The routing table also reflects the addressable IP-networks that can be reached, which AS-numbers are actually being used and such. These are valuable statistics.

The project is aimed at NOC's and engineers at ISP's, though it is a priority not excluded that end users will have access to the data as well.

Setup of the RIS

The basic setup of the RIS is shown in figure 1. At the RIPE NCC a main collection machine will be installed. This machine runs 3 programs: a route collector, a database and user interface. At a limited number of core locations (IX1, IX2, ...in figure 1) a dedicated machine will be installed.
This machine will run a copy of the route collector program. The route collector programs collects routes, either directly from the border routers at that site or via Multi-hop BGP peering from other nearby routers. All data is transferred to the RIPE NCC and stored in a database. The database
can be accessed by the sites participating in the project and will provide them with both the raw data as well as derived statistics.

The NCC already has experience with running remote data-collection points for the test-traffic project [7]. It is planned to use this experience for the RIS.

Prototype Setup

Figure 2: Set-up of the prototype RIS

In order to prove the concept and to get a feeling for the data volumes involved, a prototype of the RIS has been developed and has been used for data collection during the last months.

The setup of the prototype is shown in figure 2 and consists of a route-collector, a database and a rudimentary  user-interface.

We tested two products as route-collector software: GNU-Zebra [8] (versions 0.74 and 0.76, under the Linux operating system) and the Multi Threaded Routing toolkit (MRT) [9] (versions 1.4.xa and 2.0.0a under the Linux and
FreeBSD operating systems).

GNU-Zebra is a free software package that manages TCP/IP based routing protocols. It supports the BGP-4 protocol [4] as well as RIPv1, RIPv2 and OSPFv2. GNU-Zebra can be used with both IPv4 and IPv6. Zebra is intended be used as a Route Server and Route Reflector.

The Multi-threaded Routing Toolkit (MRT) is a project developed by Merit at the University of Michigan, to be used for routing architecture and protocol research. Software developed until now includes multi-protocol IPv4/IPv6
routing daemons as well as routing analysis and simulation tools.

At peak times, the route collector may have to deal with 70,000 prefix updates per second. The prototype shows that the route collectors can keep up with these these loads, however, the database is a point of concern. These studies show that generic database systems have problems coping with these numbers. The best solution at the moment seems to be store the data in regular files and only use a generic database product to maintain a summary file indicating which information is stored in which file. However, further studies are needed here.

The Multi-hop BGP4 peering mode has been tested with the prototype. This mode can be used to reduce the number of collection points.

The RIPE NCC is happy to make the software of the prototype available to interested parties [10]. However, the software will be made available on an "as-is'' basis and the RIPE NCC, unfortunately, does not have the resources to provide more than basic installation support.

The conclusion that can be drawn from this prototype is that it is possible to collect routing information from routers using BGP. It also gave us a good idea of the data volumes involved. This experience will be used in the final design. Data collection with dedicated machines at remote points has not yet been tested but no special problems in this respect are foreseen.

Figure 3: Overview of the final RIS set-up.

Final Setup

Figure 4 shows the proposed setup for the RIS in more detail.

Data-collection machines (the ``Remote Route Collector'' (RRC) in figure 4) will be installed at a few major topologically interesting points on the Internet, such as major exchange points and the like. These Remote Route Collectors will collect routing information either directly over the exchange LAN of the site or using the BGP-multi-hop mode to nearby ISP's. All peerings will be set up by hand and only data from ISP's interested in this project will be collected.

The optimal number of Remote Route Collectors and their location still has to be investigated. In first order, it is expected that around 10 such RRC's will be installed in the RIPE service area.

In principle, all peerings via BGP-multi-hop could be done by the central machine at the RIPE NCC. However, that would involve the transfer of a large amount of data over the networks involved. One central collector will not scale to several hundred peers. It is therefore more efficient to do some of the BGP-multi-hop peerings with the RRC's, reduce and compress the data there and only transfer the results to the RIPE NCC.

The RRC's are discussed in more detail in section 3.2.1.

Data is then transferred to a central machine at the NCC and stored into a database. Design considerations for this database are discussed in section 3.1.2.

Users of the RIS can access the database through various interfaces discussed in section 3.1.3. It is a goal of the project to transfer the data as often as practically possible without overloading the networks or databases, in order to minimize the time between data-taking and availability of the data to the users.

Figure 4: Detailed overview of the final set-up

Software for the RIS

An overview of the various software modules used in the final setup is shown in figure 4.

Software for the Remote Route Collectors

Remote Route collectors will run BGP listener which logs all updates received from its peers and also maintains a routing table. In the first instance, it is planned to run the MRT [9] and GNU-Zebra [8] software under a supported UNIX platform on each RRC. These products were described in section 2.

Depending on the development paths of MRT and Zebra, the RIPE NCC might decide to re-write parts of those software packages in order to add missing features or to optimize performance, or to use parts of other software
packages. The parts of any external software essential for the RIS will be reviewed and documented according to appropriate RIPE NCC guidelines in order to remain maintainable in the future.

The BGP listener software will collect time stamped BGP4 announcements, including:

• The route prefix and length.
• The origin AS.
• The AS paths for which the announcements are visible.
• Any additional BGP attributes which are propagated  through the system.
• Errors in the BGP announcements.
• Periodic dump of routing table.

As both Zebra and MRT supports IPv4 and IPv6, it will be easy to extend the RIS to include IPv6 routing information when native IPv6 networks start to be deployed on a large scale.

Figure 5: Number of updates seen by the prototype RIS peering with a single router during the first three minutes of operation.

Receiving speed:

Figure 5 shows the number of updates received by an RRC as a function of time. When the RRC is started, it receives some 65,000 initial prefix announcements in 90 seconds from a single peer. After the initial setup, the rate, averaged over a 1 week period, will drop to 2 updates/second/peer.

Assuming 60 to 100 peerings at a single collection point, the typical number seen with the prototype will translates to a peek of 50,000 to 80,000 announcements per second to be record by a RRC. Collector software will be resource intensive in terms of memory, to keep several views of routing table and I/O intensive to record the updates and snapshots.

During major instabilities, the number of updates might even be higher. In order not to interfere with operation of the routers, it will be considered to limit the data-flow from the routers to the RRC's.

Database software

The main requirements for the database are:

Insertion speed:

Assuming all RRC's are restarted once on a same day total number of updates collected from 10 RRC's during one day will be about 110 M prefix updates. To summerize and insert this data into database in about 3 hours, required processing speed will be 10,000 prefixes/sec. The database will have to buffer incoming data and the situation where not all data has been inserted in the database will be flagged. The acceptable time elapsed between data-taking and availability in the data-base still has to be investigated.

Query speed:

This should be such that an individual user will receive an answer to a simple query (e.g. what was the route from A to B at a given time) in O(10). If this takes much longer, extracting data will be too cumbersome and the RIS will not be used.

Merging data:

The software should be able to merge data from remote points into database if they have been disconnected from the RIPE NCC while still collecting data without "falling behind'' with the regular updates. The amount of data to be merged at once here may be of the order of the data collected during a week. (For example, an RRC that got disconnected from the Internet at the start of a long weekend, followed by a few days necessary to solve the problem.)

RRC                 NCC Central
                                     (GByte)                (GByte)
                           Peer/Day   Daily   Monthly  Monthly   Yearly
   BGP announcements
   Uncompressed            0  . 007 0  . 490  14 . 3  143  .0     -  . -
   Compressed              0  . 005 0  . 350   - . -    -  .-  1002  . 0
   Routing table dumps
   Uncompressed            0  . 021 1  . 430  43 . 0  430  .0     -  . -
   Compressed              0  . 002 0  . 143   - . -    -  .-   430  . 0
   Storage needed
   Hard disk               0  . 028 0  . 633  57 . 3  573  .0     -  . -
   Tape                    -  . -   -  . -     - . -    -  .-  1700  . 0
   Data to be transferred

   Compressed              -  . -   0  . 500   - . -    -  .-     -  . -

Storage requirements for the RIS. The first 3 columns refer to the individual RRC collecting BGP-announcements and routing table dumps, the last 2 columns to the central collection point at the NCC. The top 3 rows show the size of the BGP-announcements, routing table dumps and total storage space needed at that stage. The last row shows the amount of data to be transferred. Data will be stored either compressed or uncompressed,``-.-''  indicates that data will not be stored in the (un-)compressed format at that stage. A total of 10 collection points has been assumed. The numbers in this table are based on the prototype and will be further refined as we gain more experience. The numbers also do not include index-files and other overhead introduced by the data-base program.

Data volumes:

The prototype shows that the data at a typical collection point consists of:

• 7Mbyte/day/peer of BGP announcements when all logging options are switched on.
• 7Mbyte for a full dump of the routing table for each peer.
  Assuming that the data is dumped every 8 hours (3 times a day), this translates into 21Mbyte per peer, per  day. However, the routing table can be compressed quite efficiently by a factor of +/-10 using standard file compression tools.

The number of dumps each day is a trade-off between data-volume and CPU power. One can reconstruct the routing table at any point in time from an initial dump plus all updates received since the time the table was dumped. However, the longer the time elapsed since the routing table was dumped, the more updates will have been received and thus the more CPU power is needed to reconstruct the table. Dumping the table moreoften, means that less CPU power is needed but requires more storage space. 3 dumps a day appears to be a reasonable compromise.

The maximum number of collection points is of the order of 40, approximately 30 core locations in the RIPE region and about 10 core locations elsewhere in the world. However, there is a considerable overlap between the data that would be collected at all 40 points, so we assume that data collection at 10 points will be sufficient. Data-collection might also be done by exchanging data with similar projects (see 6), though this does not change the total data-volume.

An estimate of the upper limit of the total data volume based on these results can be found in table 1. The numbers in these table are based on our studies with the prototype and will be further refined as we gain more experience. Assuming that data will be available on disk for 1 month and is then moved to tape, the RIS will require some 600Gbyte of disk-space as well as a tape storage device capable of handling some 2Tbyte.

Handling such an amount of data is by no means trivial. The database should be able to handle this amount of data and be able to cope with future expansion.

Experience with the prototype has shown that the popular MySQL database cannot cope with these requirements. Other products are being investigated.

User Interface

Regardless of the method chosen to store the data, it is planned to implement the following ways to access the data (in this order):

All (or a selected subset of) the raw data (both the full routing tables and the individual updates) will be made available to the sites participating in the project via FTP or HTTP.

Command line queries, a user enters a command and the resulting output will be shown on the screen, similar to the well known RIPE NCC whois server. Typical queries that one can think of are, for example:

• What was the route between two ASN's at a particular time, as seen at a particular site?
• When was a route between those two ASN's first announced, which updates were made and when was the last route withdrawn?
• Which prefixes were announced by a certain ASN
• Who announced a route to a certain ASN.

These queries will be refined and expanded based on the input from the users.

Queries through a cgi-script. An interface to command-line queries via a web-interface will be provided. This interface will also allow us to provide graphical output, for example, a map of ASN's and their connections.

Reports and statistics. The data will be collected and reduced to reports showing statistics of Internet routing. For example:

• Rates of updates and withdrawals for certain routes.
• Number of flaps

This will again be expanded based on user input.

In the first instance, it is foreseen that all queries will be done to the RIPE NCC RIS server. At a later stage, the possibility to do local queries to the individual RRC's might be added. The advantages of this are two-fold, it will reduce the load on the central server and it will allow the users to use the RIS data when a major network problem causes the central server to be unreachable.

Data collection at the Remote Points

The RRC's will need careful monitoring and security since they are interacting with critical network elements like border routers, are located at important network points, and problems may trigger alarms at peer NOC's. For example, if a box goes down NOC monitors may generate alarms equivalent to peer going down even though an RRC is not announcing any routes. This should be avoided.

For security reasons, the RIS will get a separate AS-number. The RRC's will also not announce any routes though peers may wish to set up filters to block any announcements from the RRC's.

Although it is planned to control the RRC's from a central point with no operators or service required at the local sites, it is expected that each site that hosts a RRC appoints a local contact. This contact should take care of things that cannot be done remotely, such as rebooting the machine or copy information from the console in case of network or hardware problems, keep the RIPE NCC informed about planned maintenance of network and such. The local contact, obviously, has to be reachable by phone or email.

Hardware

Hardware for the RRC's.

The prototype shows that route collection is a memory and I/O intensive process, CPU power is less important. Approximately 256Mbyte of memory is needed to run all software without significant swapping.

Although data will be pulled to the RIPE NCC at regular intervals, one should consider the situation where  connectivity to the RIPE NCC is lost for some time and that data will have to be buffered at the RRC's. In order to cope with this situation, the RRC's will have to be equipped with at least 20Gbyte of disk space.

Over the last year's the RIPE NCC has gained experience with the operation of remote machines for data-collection purposes [7]. It is planned to use that experience to run the RRC's, no particular problems are foreseen in this respect.

There is also considerable interest in the installation of Test-boxes at the same location as the RRC's and it has been suggested to use the same hardware for both projects. This idea will be investigated further.

Hardware for the central collection point

The central machine will collect data in the same way as the RRC's. Besides that, it will run the database where all collected data is stored as well as handle external queries. The machine will be equipped with sufficient disk-space to keep approximately one month of data online. Older data will be stored on tape.

Peering with the route collector

The RIS project will use AS Number 12654 for peering at all collection points. This ASN 12654 ill not announce any routes[gif] . Ideally, the RIS should peer with all members of the site for their entire default free routing table. These peerings will be set up using BGP4. The peering policy will be added to auto-num objects in the RIPE whois database.

The RIPE NCC central collector will also setup a few  multi-hop BGP4 peerings to ISP/NAP's border routers.

Bandwidth requirements for the RRC's.

Approximately 500&thicksp;Mbyte (see table 1) will be transferred from each RRC to the RIPE NCC daily. This translates into a constant flow of about 6&thicksp;kbyte/s. The central collection point at the NCC will see an incoming flow of about 60&thicksp;kbyte/s. However, the data volume that has to be transfered can reduced only by sending the differences between the periodic dumps, rather than the full dumps. We will ensure that these data-transfers will not affect normal operations.

Research issues, Testing

Open Questions

While writing this proposal, a number of questions came up that need further investigation. These are listed in this section.

1. Empirically investigate the added value in collecting information at different places. Determine how one can best select the sites were data is collected. Determine how to merge identical data collected at different sites.
2. As can be seen from table 1, the RIS will have to handle large amounts of data. What is the best database to use for this project and how can its performance be optimized/tuned for this project?
3. Find out, in collaboration with the first users, how long the data should remain in the database?
4. Is the full data still useful after a while, or can it be reduced to a smaller sample that is only sufficient to produce statistics?
5. Determine the optimal number of routing table dumps per day based on practical experience with the first RRC's.
6. Investigate if default routes should be collected as well. Default routes are routes that are used by routers if no information about a certain IP-prefix is known.
7. It will be necessary idea to maintain a mirror data-base of the aut-num objects (and their history) in the RIR  databases. When analyzing data at a later point in time this information will be essential. The maximum number of ASN's is 64&thicksp;k so this will not create a large volume of data. However, the data is stored into several whois-servers so a common format and tools to extract information from the servers will have to be defined.
8. Investigate what kind of support is necessary for RRC and Server machines at NCC on a ``24/7''-basis, in order to make the RIS available to its users full time.

Alpha- and Beta-testing

For a service as the RIS, it is essential that the potential users of the service are involved from the first stage of development. The RIPE NCC has therefore started to look for users of the alpha- and beta-versions. Inside the NCC, there is interest in using the first versions and work on the definition of the user interface, database queries and other services based on the data.

Outside the NCC, at least four sites have shown interest in providing us with data through peering sessions and it is hoped that these sites will also act as the first external users of the project.

In order to get optimal feedback from the first users, documentation and a user's manual will be written as the project is being developed. The documentation will describe the program, the user's manual will show how the data should be interpreted and how it can be used for operational purposes. It is also planned to give regular progress reports in the appropriate RIPE-WG's and solicit input from all interested parties.

When the project is in a more mature state, turning the RIS into a regular service will be discussed in the RIPE-WG's.

Interface to the IRR Reality Checking Project

One possible use of the RIS is in the IRR Reality Checking Project [3]. The RIS provides the actual routing and this can be checked against the routing announced in the DB for consistency. At the moment, the IRR Reality Checking Project seems to be most interested in the dumps of the routing tables and not in the individual BGP updates.

The interface between the two projects will be defined at a later stage.

Related projects

There are several ongoing projects similar to the RIS. The most important ones are:

• Merit/IPMA [11] has been doing BGP logging since 1996. BGP announcements are collected directly from 5 major exchange points in North America as well as via multi-hop BGP from 20 providers. More on this project can be found at: ftp://ftp.merit.edu/statistics/ipma/data
• The University of Oregon Route Views Project, see:
  http://www.routeviews.org/

The RIPE NCC plans to collaborate as well as try to exchange data with those projects. The NLANR-group at SDSC [12, 13] has already shown interest in collaborating with the RIPE NCC on RIS-related research.

Other projects focus on the visualization of the data and this might be an other area for collaboration in the future.

Schedule, milestones and resources.

Based on discussions both inside the RIPE NCC and with the chairmen of the appropriate working groups, the  following milestones have been set for the project:

July 1, 1999:

Work started when AA joined the project.

October 14, 1999:

Draft project plan presented to the community.

September 1999/RIPE-34:

• Presentation of the project plan in the Routing-WG.
• Demonstration of the prototype.

October 14, 1999:
Final project plan presented to the community.

October 1999-February 2000:

Development work, intermediate steps:

• Documentation of the prototype.
• Develop the first version of the RRC software.
• Set up peering sessions with interested parties and start collecting data with (prototype) RRC software.  This data will be used to develop the user-interface and to finalize the estimates of the data-volumes.
• Make the raw data available for analysis.
• Build a prototype user interface and get user feedback on that.

The prototype will be documented.

• Start to extract statistics from the raw data.
• Finalize and order hardware.

February 2000/RIPE-35:
Present answers to the research issues mentioned in section 4.

Have a more advanced system ready to be tested inside and outside the NCC. User documentation will be provided and show how the data can be used in day-to-day operations.

February - May 2000:
Install several RRC's.

May 2000/RIPE-36:

• Add more features based on user feed-back.
• Study and implement analysis of the data over a longer term.
• Discuss implementing this as a regular service
  September 2000/RIPE-37:

Turn project into a regular service.

At the moment it is expected that 1 network engineer (AA) will work full-time on the project. The second author of this document will spend approximately 20% of his time on this project. It is likely that additional expertise or manpower will be needed during the various stages of the development of the service. These will be borrowed from other groups inside the RIPE NCC.

When the project is turned into a regular service, the manpower situation will be reviewed.

Conclusions

In this document, an overview of the RIS project was presented, the implementation has been discussed and a list of open questions has beengiven. The project was presented at the RIPE-34 meeting and feedback from the RIPE-community has been included in this design document. The RIPE NCCwill now start with the implementation of the RIS.

Acknowledgments

The authors would like to thank: Pedro Figueiredo and João Damas (RIPE NCC) for their work on the prototype setup and the chairmen of the RIPE Routing, DB and IPv6 working groups for their input to this proposal.

References

1. John Crain et al., "RIPE NCC Activities & Expenditure 1999'', ripe-186.
2. John Crain et al., "RIPE NCC Activities & Expenditure 2000", ripe-197.
3. Joachim Schmitz and João Luis Silva Damas, "Reality Checking of the RIPE Routing Registry'', ripe-201.
4. Y. Rekhter et al., "A Border Gateway Protocol 4 (BGP-4)'', RFC 1771, see ftp://ftp.ripe.net/rfc/rfc1771.txt
5. T. Bates et al, PRIDE Users Guide, 1994.
6. Craig Labovitz et al, "Internet Routing Instability'',  Proceedings of SIGCOMM'97, September 1997.
7. Henk Uijterwaal, Olaf Kolkman, "Internet Delay Measurements using Test-Traffic, Design Note'', ripe-158.
8. Zebra routing software, see http://www.zebra.org
9. Multi Threaded Routing toolkit MRT, see  http://www.mrtd.net/
10. See: http://www.ripe.net/ris
11. IPMA tools, see: http://www.merit.edu/ipma/tools/
12. NLANR, see: http://www.nlanr.net/
13. SDSC, see: http://www.sdsc.edu/

...updates/second/peer
The slope of the right-hand side of figure 5 suggests a higher rate but
appears to be a statistical fluke.

...routes
Assuming that the router software on the other side does not need any
announcements from ASN 12654.

This document discusses the test traffic project proposed as a new activity in ripe-144 [1]. The document gives an overview of the project, discusses the implementation and mentions a couple of points that have to be addressed in future design documents. This document is intended to solicit input from interested parties.

Table of Contents:

• 1. Introduction
• 2. Measurements
• 3. Experimental Setup
• 4. Presentation of the data
• 5. Verifying the results
• 6. Implementation
• 7. Conclusions
• References

1 Introduction

Figure 1: Overview.

This document describes the design and implementation of the test traffic project. This project has been proposed as a new RIPE NCC activity [1]. The project is similar to the project proposed for a different community [2]. The goal of the project is to do independent measurements of connectivity parameters, such as delays and routing-vectors, in the Internet.

The general idea behind this project is shown in figure 1. The local network of each Internet Service Provider (ISP) can be divided into two parts: an internal part, that deals with the traffic between the machines of the customers in this network, and an external part, that deals with the traffic between this and other service providers. This project focusses on the external networks only.

Two of the parameters that determine the network performance between two providers are: the total bandwidth and the delay. The bandwidth is the number of bits that can be transferred between two providers per unit of time. The delay is defined as the time elapsed between the moment that a packet leaves the network of one provider and the moment that the packets arrives at its destination. Associated with these parameters is a routing-vector, this vector describes how traffic travels from this provider to another.

In order to measure the delays and determine the routing, measurement-boxes will be installed at each participating provider. These boxes collect data. The data is then transferred to a central machine at the RIPE NCC. Here the data is processed and made available to the users of the networks.

The outline of the remainder of this note is as follows: Section 2 discusses the measurements that we plan to use in order to determine the network performance. Section 3 gives an overview of the hardware and software infrastructure needed for these measurements. Section 4 shows how the data can be presented to both experts and casual users of the Internet. An important point is to prove that our measurements actually reflect the performance of the network. Section 5 discusses how we can verify our results. Finally, Section 6 discusses the implementation, milestones and deliverables of this project.

2 Measurements

There are (at least) two ways to monitor the characteristics of the Internet:

• Passive: by monitoring the amount of traffic that passes a certain point and,
• Active: by generating test traffic and measuring how much time it takes to ship the test traffic over the network.

The advantage of an active measurement over a passive measurement, is that for the active method, the test traffic can be generated under well-defined and controlled conditions, whereas the passive measurement depends on the traffic that happens to pass a certain point. Another important advantage of active measurements is that it avoids the privacy concerns associated with passive measurements. As we want to deliver an objective measurements of the performance of the Internet, we have decided to focus on active measurements only.

Figure 2: Experimental Setup

The principle of our measurements is shown in figure 2. A test box is connected 0 hops away from (or, if that is not feasible, as close as possible to) the border router of each participating ISP. By connecting the Test Box 0 hops away from the border router, we exclude effects of the internal network from our measurements.

If the ISP has more than 1 border router, we will either connect the Test Box in such a way that the delays to each border router are similar or we will install multiple test boxes. This way, we avoid a bias for traffic in certain directions in our measurements.

The box at ISP-A generates a pre-defined pattern of test messages. The test messages are send through the border router and the network to a similar box at ISP-B. Both Test Boxes are connected to a clock. By looking at these clocks when a test message is sent and when it arrives, one can determine the delay between these two providers. There are other measurements that one can do with this setup, this will be discussed in section 2.1.

The Test Boxes at both ISP's are identical, thus the process can be reversed to measure the delay between ISP-B and ISP-A. Also, the boxes at both ISP's can be used to send test traffic to similar boxes at other ISP's.

It should be obvious from figure 2 that the clocks at the two ISP's should be synchronised (in other words, the time offset between the two clocks should be known and remain constant) and provide the time with a high accuracy compared to the time difference that we want to measure. This will be discussed in more detail below.

In the extreme case where all O(1000) ISP's in the RIPE NCC (geographical) area participate in the project, the Test Boxes can be receiving test traffic from and sending it to 1000 other boxes, thus testing all 1000² possible connections. However, the topology of the Internet is such that it is likely that one can still do reliable measurements of the performance of the Internet without having to send test traffic for all 1,000,000 possible connections. This will be studied.

Although this method can be used to measure the performance of the internal network of each provider, we want to limit our work to the external networks only. For this reason, the Test Boxes should be located as close as possible to the border routers of the ISP's, in order to eliminate any effects caused by the internal networks.

2.1 Data Collection

There are several possibilities for the test traffic that we can generate:

• One-way: A packet is sent from ISP-A to ISP-B, as described in the previous section.
• Two-way: A packet is sent from ISP-A to ISP-B and then returned to the sender. This principle is used in "ping'' and other tools that determine the round trip time.

  However, assuming that the paths are known, the results obtained with one two-way traffic measurement, will simply be the sum of two the one-way traffic measurements. For this reason, we plan to use two-way traffic only for independent checks of the one-way results.

• Real life applications: A measurement that a user sees in a real application, like fetching a WWW-page or downloading a file by FTP, using a well defined TCP benchmark connection.

In the first instance (phase 1, section 2.1.1), we plan to do one-way and two-way measurements. At a later stage (phase 2, section 2.1.2), this will be expanded to performance measurements for real life applications. Our measurements will, already in phase 1, produce several observables, this will be discussed in section 2.1.3.

2.1.1 Phase 1

The one way test traffic will consist of UDP data packets of 3 different sizes: small (56 bytes, as used by ping and similar tools), medium (576 bytes, the minimum packet size that any router must handle as a single packet) and large (say, 2048 bytes, to see the effect of (possible) splitting data-packets into smaller units and related effects).

The packets will contain:

• The address of the sender.
• A time-stamp that shows when the packet was sent, together with the dispersion of the clock in the sending machine. The dispersion is needed to determine the overall error in the measurement.
• A reference number, in order to keep track of the number of packets send and lost.
• A hop-count (number of routers that the packet passed between sender and receiver).
• Administrative information, such as the version number of the software and a checksum.
• Padding to give the packet the desired size.

The receiving process will remove the padding and then add the following information to the packet:

• The address of the receiver.
• A time-stamp that shows when the packet was received, together with the dispersion of the clock in the receiving machine.

The result is raw-data that contains all information about this particular delay measurement.

The path or routing-vector between ISP-A and ISP-B is defined as the collection of machines and network between the border routers of these two ISP's. This path may vary as a function of time. A tool like "traceroute'' will be used to determine the path for the test traffic at any given time. The path information will be made available.

There are several potential problems while doing these measurements:

• Set-up effects: even for Internet traffic, the routers have to be setup before a connection between two points is established. If this is not taken into account, then the measured delay will be larger than the delay that can be attributed to the network. Two possible solutions to circumvent this problem are:
  • Precede any measurement by a "traceroute". This determines the path and provides the routing information that we are interested in anyway.
  • Run the test more than once and compare the results. If the first result is significantly different from the other measurements, discard it.
  Set-up effects are interesting in themselves and this data will be recorded and studied.
• No connectivity. The fact that there is, contrary to what one expects, no connectivity between two points is interesting information in itself. However, our software should be written such that it will survive this case without operator interference.

For the two-way measurements, we plan to use data-packets that are similar to the one-way measurements. These results will be used for consistency checks only.

2.1.2 Phase 2

In the second phase of the project, we plan to do measurements with:

• TCP-streams.
• Simulations of applications.
• Packet Trains. A packet train is a number of test messages that are sent with very short intervals. They provide a way to study if the packets are delivered out of order and/or merged together into larger packets along the way.

The implementation of this will be discussed in a future design note.

2.1.3 Observables

These measurements will provide several observables:

• Delay Information: The results of the delay measurements can be stored in a N×N matrix D(t,s) where each element D_sd(t,s) represents the time that a packet needed to travel from a source s to a destination d. If there is no connection between two points, then D_sd(t,s) will be .

  The elements of D are a function of the time t, as the network characteristics will change over time, and the packet size s.

  The elements D_sd(t,s) all have an error ðD_sd(t,s) which gives the total error in this delay measurement.

• Routing Vector or Path Information: Each delay measurement is accompanied by a determination of the path between the two locations using a tool like "traceroute''. These results can be written as a N×N matrix P(t) of vectors.

  This matrix provides, assuming that our Test Boxes are installed at a significant fraction of the ISP's, an up-to-data map of the Internet as well as the history of of the network.

  If there is no connectivity between two points i and j, then the element P_ij will be 0. The number of zeroes therefore gives a measure of the total connectivity. Also, the two elements P_ij and P_ji should either both be zero (no connection between these two points) or non-zero. If only one of them is zero, this indicates a network configuration error.

  Finally, the elements P_ij can be used to detect routing loops.

It should be noted that on any particular Test Box, only the results of delay measurements to that box and the path information from that box are available (in other words: the columns of the matrix D and the rows of the matrix P ): the results of a delay measurement between ISP-A and ISP-B will be collected at ISP-B, whereas the traceroute information is only available at ISP-A. It is undesirable to do the traceroute at ISP-B, as the path from A to B is not guaranteed to be the same as the path from B to A.

In order to get the full matrices, the results have to transferred to a central point. This is one of the reasons for collecting all test results on a single machine.

From these observables we can determine if there are trends in the transfer times as a function of time and isolate special or suspicious events. During the initial phase of the project we will concentrate on developing (statistical) tools to analyze the matrices. In a later phase we, or other researchers, may develop tools to correlate the data in the matrices, analyze the effect of the size of the packets or perform analysis of the routing vector matrix itself.

Once enough data has been collected we must be able to define meaningful metrics and we might tune the measurements to optimally sample this metric.

2.2 Frequency of the test traffic

There are 5 basic requirements:

1. The test traffic should be small compared to the load on the connection under consideration. If not, then the test traffic will affect the performance and the measurement becomes useless.
2. The intervals should be small enough to study (interesting) fluctuations in the performance of the network.
3. As the network changes over time, the amount and type of test traffic should be easily configurable.
4. The interval between two measurements should be small enough to detect and eliminate set-up effects.
5. As suggested by the IPPM [2] the measurements should be randomly distributed to prevent synchronization of events due to weak coupling as demonstrated in [3]. The IPPM suggest using a so-called Poisson sampling rate. We will investigate the IPPM suggestion and other possibilities to prevent weak coupling.

The first two requirements are contradictory: smaller time intervals means more test traffic, but more test traffic means a higher load on the network.

For the initial settings, suppose that we generate the following amount of test traffic:

• 1 small packet per minute.
• 1 medium sized packet per minute.
• 1 large packet every 10 minutes. These packets are used to see the effects of splitting large packets into smaller ones, presumably this effect is constant over time.

This then generates a data-volume of approximately 14 bytes/s for each measurements. This number does not include overheads and the data for the "traceroute" program.

If this amount of data is too high, then one might consider increasing the interval between two packets and add a burst mode (a short time with a higher test traffic volume) for occasional measurements with a smaller interval between two test packets.

This leads to a number of questions that have to be answered

1. Can we generate this amount of data without the ISP's objecting? Probably not for 1 connection, but what if we have installed Test Boxes at 10, 100 or even 1000 sites, with 10², 100² or 1000² possible connections?
2. Are we sure that the test traffic does not affect the performance of the network?
3. How do we check that a packet has left the Test Box before the next one is sent out? We do not want packets to sit in a buffer if the local network is at its maximum load.

2.3 Error Analysis, Required Clock Accuracy

In this section, we want to estimate the errors on the final results. As we mentioned before, our plan is to do both one and two way measurements.

One way measurements involve sending a time-stamped message from one machine to another. The second machine compares its arrival time with its local clock and determines the transfer delays from that. This requires that the local clocks are synchronised and provide the time with a high enough accuracy. The synchronisation is the difference between the clock on this machine and an absolute time standard. The accuracy is the error in the time measurement. The errors in synchronisation and accuracy determine the overall error in the delay measurement.

Two way measurements are a bit simpler in this respect: a message is sent to another machine and echoed there. All time measurements can be done on the same machine, so the clock has to be accurate and the the resolution of the clock has to be high enough. The resolution is the smallest time interval that can be measured by this clock.

In both cases, we will be subtracting two times, t₁ and t₂. If we call the total error on these times , then the error in the final result () is equal to:

It should be noted that, even with an high resolution clock, can become relatively large for small time intervals. For example, if we measure a 50 ms time interval with a clock with a resolution of 10 ms, then the error in the result will be 14 ms.

Systematic effects of the equipment on the results have to be studied and should be understood.

3. Experimental Setup

3.1 General Idea

The general setup is shown in figures 1 and 2.

The central machine at the RIPE NCC controls the Test Boxes at the ISP's. The data is collected by the Test Boxes and then transferred to the central machine.

It is planned that the data collected at the local machines will be kept there for a few days, so that it can be viewed and analysed locally. The main processing of the data will be done on the central machine.

Of the order of 1000 ISP's are active in the geographical area serviced by the RIPE NCC. Although we certainly do not foresee that all 1000 ISP's will participate in the project or that test traffic will be sent for all 1000² possible connections, we do plan to design the software such that these numbers can be handled.

3.2 Data Flow Diagram

Figure 3 shows a first version of the Data Flow Diagram (DFD) for this project. This diagram will be the starting point for the software design for the project.

Figure 3: The Data Flow Diagram for this project.

3.3 The Central Machine

The central machine (or machines) performs a number of tasks, including:

• It controls the configuration, which determines which connections are being tested,
• It collects the data from all Test Boxes,
• It acts as the software repository,
• It will be the platform where the software is being developed,
• Finally, this machine will be used for data-analysis.

The main problem for the central machine is the amount of data to be stored.

The results of each delay measurement will consist of the packet sent by one Test Box to another, plus information added by the receiving Test Box such as the arrival time. If we assume that this results in (at most) 100 bytes of data for each delay measurement, then measuring at a rate of once a minute will produce 150 kbyte/day or 55 Mbyte/year of data for each connection.

In addition to that, one also has to store the routing-vector information. For a stable network, the amount of data for this will be less, as one can store a routing-vector with a validity range, rather than the routing vector for each measurement.

So, allowing for short intervals with high rates and a significant amount of routing-vector information, a safe upper limit of the amount of data to be stored is 250 kbyte/day or 90 Mbyte/year of data per connection.

This means that in a setup where O(100) connections are being tested, one needs several Gbytes/year of disk space to store the raw data. If the number of connections that is being tested goes up by an order of magnitude, one will need a tape-unit or similar mass storage device.

In the extreme case, where 1000 providers participate in this project and we test all 1000² possible connections between them, the data volume will be of the order of several Tbyte/year. While this volume is not unmanageable, it will require a tape robot for storage, a major investment, or a significant compression of the raw data before it is stored.

A database program will be used to store the data. This database should be able to handle data volumes up to several Tbyte, distributed over several physical volumes, and read the data in a reasonable amount of time. As the data might be used for several different analysis, it should be possible to generate Data Summary Tapes (DST's), with a subset of the data.

For presenting the data, we need a graphical analysis tool. This tool should be able to plot and histogram data, and, of course, have an interface to the database.

The Test Boxes at the ISP's

3.4.1 Hardware Setup

The Test Boxes consist of an industrial PC, which can be mounted in a 19 inch rack. Inside the box, there will be a CPU, a disk and a high precision clock. The disk should be large enough to store several days worth of data, assuming the 250 kbyte/day/connection, 1 Gbyte should be sufficient. The box should be hooked up to the border router of the ISP.

The machine will be "plug and play", in order to make installation easy and to avoid tampering with the machine by a local ISP. After the box has been installed, all maintenance will be done remotely from the NCC. Only in the case of major hardware failures that cannot be solved remotely, we will need some support from the local ISP's.

The design of the box will be such that the ISP cannot affect the performance of the computer or reconfigure his network such that the results will be affected ("Design the network for the benchmark"). This will include installing machines at the customers of this ISP and do consistency checks.

3.4.2 Software Setup

The software on the machine will read a configuration file that specifies which measurements it has to do, perform the measurements and collect the data.

The machine load should be small, to avoid that the measurements are affected by processes competing for CPU-time. This will also put a limit on the number of connections that can be simultaneously tested.

As the number of machines will be large, the software should be easy to maintain:

• One script should restart all processes after a reboot or power failure.
• The software should survive common errors and restart itself without operator interference.
• An automated procedure, like rdist, will be used keep the software up to date.
• As the machines will be located in the heart of the networks of the ISP's, they should be hacker-proof. In order to accomplish this, the number of accounts on the machines will be limited to the absolute minimum, all software will run in secure shells and no un-encrypted passwords will be sent over the net. Finally, all the software running on the machines will be made available to the participating ISP's, so that they can convince themselves that the Test Boxes do not introduce any security holes.

All the software written for this project as well as the design of the Test Boxes will be made available to the participating ISP's. They can use it for measurements on the internal part of their networks.

3.5 The clock

The clock at the remote machines is the most critical part of the whole project. Before we start to work on any other part of the project, we should prove that a clock with sufficient accuracy can be built.

We aim for a accuracy that is at least 1 order of magnitude better than the smallest delays that will be measured by the box, including drifts. In a typical network environment, the delays will be of the order of 10 ms, this then translates to a required accuracy of 1 ms or less. The overall error in a 10 ms delay measurement will then be of the order of 1.4ms.

We are considering three solutions for this problem:

3.5.1 NTP/Network Time Protocol

This is a software protocol [4] that uses timing servers all over the world. With atomic clocks as references and suitable hardware, accuracies of down to a few hundred ps can be obtained. With off-the-shelf products, the accuracy will be less.

The current list of servers [5] lists primary servers in: France, Germany, Holland, Italy, Norway, Switzerland, Sweden and the UK, and secondary servers in: Portugal, Poland and Slovenia.

This approach uses free software, so the costs for this solution are small, assuming that the internal PC clock in a standard environment is stable enough for our purposes.

A potential problem is that there are large areas without a nearby server. Also, the accuracy might be affected by unstable networks. A test-setup will be used to determine if this solution can provide a stable clock under these circumstances. In these cases, an external reference clock has to be used to obtain the necessary resolution.

3.5.2 Radio Time Servers

This is a system of long-wave radio stations broadcasting the current time. A receiver can collect these timing signals and provide the local time with an accuracy of the order of a few ms. Typical receivers cost of the order of $ 2500.- in 1992 [6]. There are, however, several problems with this approach:

First of all, timing signals are only available in the UK, Germany and surrounding areas. This means that this approach can only provide a clock signal for a small part of the global area covered by the RIPE NCC. For the remaining part, one needs another approach.

Then, reference [6] mentions seasonal effects and other corrections which are needed in order to get the accuracy of a few ms. This implies that the clocks need constant attention in order to keep the high accuracy.

Finally, the costs for each clock is much higher then for a solution that uses GPS receivers (discussed below).

For all these reasons, we will not consider this solution any further.

3.5.3 GPS Receivers

This approach uses the Global Position System (GPS), a satellite navigation system developed by the US Department of Defence (DoD). Receivers collect a timing and position signal from up to 24 satellites. With the system, the time can be obtained with accuracies down to 200 µs.

There are several receivers on the market that can be mounted as an extension card inside a PC. These cards have to be connected to an antenna and can be read out by the PC. The NTP package [4] can then be used to synchronize the internal clock to the global time. The GPS receivers will therefor provide an easy to maintain and reliable clock.

A test showed that an antenna mounted in the window of our offices was able to pick up the signals from several satellites, which is sufficient to provide a timing signal. This antenna has to be located, depending on the type of card and antenna, within 10 to 100m of the receiver. This puts some constraints on the local infrastructure at the ISP's.

Cards typically cost of the order of $1000 [7].

A side-effect of this solution is that we create a network of high-precision clocks (so-called stratum-1 NTP servers) through the area where our Test Boxes are located. These clocks can be used for other purposes. Also, the GPS receiver will provide its global position with an accuracy of about 100m.

3.5.4 Conclusion on clocks

We believe that GPS receivers combined with the NTP software will provide a suitable clock signal for all Test Boxes. However, this is the most critical problem in the project and we should focus on this first.

3.6 Costs

The funding for the prototypes and initial deployment of the test boxes is discussed in [1]. Large scale deployment and maintenance may require additional funding which will be sought separately.

4. Presentation of the data

We realise that the presentation of the measurement results is a delicate matter. We plan to organise this in close coordination with the ISP's concerned and the relevant RIPE Working Groups. The schemes presented below represent a first idea which certainly needs refinement as the project progresses.

4.1 Disclosure of the data

We foresee three different levels of disclosure of the data:

1. Experts: The experts at the NCC will, of course, have access to all data that is collected with our Test Boxes.
2. Participating ISP's: Each ISP that installs a Test Box will get access to all data related to his network (but not to data related to somebody else's network).
3. Non-participating ISP's: They will only have access to global quantities derived from the data.
4. The general public: The general public will have access to global quantities derived from the data, but with a more detailed explanation describing how the data should be interpreted.

During the initial phases of the project, the data will be distributed under a non-disclosure agreement. Both the experts at the RIPE NCC and the participating ISP's can use the raw data for any analysis that they find interesting but both sides agree not to publish any results until the results have been verified (see section 5) and both sides agree that that they are meaningful, correct and can be published. The details of this agreement will be worked out with the participating ISP's.

4.2 Access to the data

There will be several ways in which the data can be accessed:

1. Test-box: The Test Box will have a mechanism that will return the results of the last set of measurements from this box to the ISP where the box is located.
2. Server: All information about an ISP will be available for that ISP only on the central computer for the project at the RIPE NCC.
3. Reports: Finally, we will use automated tools to extract summary plots from the data. These plots will be made available on a regular basis.

   Two different kinds of reports may be made available: Reports with data relevant to a single ISP and reports with data relevant to all ISP's. The plots in the first category will be made to that particular ISP only, whereas the other plots will be made available to the entire Internet community. In the latter case, the plots will be presented in such a way that it is not possible to trace the data back to a particular ISP.

The format in which the data will be presented will be decided in mutual agreement with the participating ISP's.

4.3 Reports

4.3.1 Step 1

In the initial phase, we foresee the following plots:

• Delays between two Test Boxes.
• Length of the routing-vector. This information can be cross-checked against the number of hops that the delay measurement packets saw between sender and receiver.
• Packet loss (# messages received/# messages sent). From the reference numbers in the packets, the receiving machine can determine which packets actually arrived and estimate the packet loss along the way.
• Percentage of the connections working. A connection between two points is considered broken if less than a pre-defined fraction of the test traffic messages sent over that connection actually arrives. From that, we can determine the percentage of the connections in the entire sample that is actually working.
• Time of arrival of the last message from host X. This provides another way to determine if a connection is working and to calculate the fraction of connections that are working.
• Number of changes in the routing vector.
• More plots will be defined as the project progresses.

All these quantities will be plotted as a function of time of the day. In addition, the path information will be made available in some sort of data-base.

4.3.2 Step 2

As soon as the individual plots are understood, we will start generating summary plots. These summary plots will contain information about 1 particular ISP (for example, the average delay to or from this ISP) or even more global quantities (like the average delay on the Internet).

As soon as we are confident that exceptional behavior (for example, the delay between two points suddenly doubles) is an indication of a network problem rather than a problem with our Test Boxes, we can start to generate warning messages to flag these conditions. There are several ways to detect these conditions, including quantities exceeding a certain threshold, test, Kolgomorov tests against known usage patterns, and so on.

4.3.3 The far future

There is much more information that can be extracted from this data-set. For example:

• Delays inside one country or geographical region, or from one region to another.
• Correlations, if the traffic between points A and B is slow, is this correlated with heavy traffic from A to C, or C to B or even C to D?
• Trend analysis, how do the delays develop as a function of time?
• Optimization of the routing in the network.
• Relation between the delays and the geographical distance between two Test Boxes.

These and other subjects will be studied.

5. Verifying the results

5.1 Local Test Bed

Our plan is to first build a test bed consisting of two Test Boxes in a well known setup at the NCC and a third Test Box at a remote ISP. This test bed will be used for:

• Software development.
• Software performance (timings, latencies, machine load, etc).
• Tests of the clock cards and their performance, in particular:
  • Development of the interface for GPS clocks.
  • Measurements of the accuracy, drift and resolutions that can be obtained with both GPS and NTP clocks as well as tests of hardware modifications and filter algorithms to reduce drift and increase the accuracy of the clocks.
  • Measurements of the differences between two identical clocks.
  • Tests of the effects of the environment (temperature!) on the clock.
  • Testing possible locations of the antenna and their effect.
• Tests of the hardware stability.
• Tests of the system maintenance procedures.
• External checks: for example by connecting a scope to two clocks and comparing the clock-pulses with an independent, reliable, source.

In short, with this setup we should convince ourselves that the hardware does what we expect it to do.

5.2 Internal consistency

We will check the internal consistency of the data by comparing one-way and two-way measurements. For example, the sum of the one-way delays in traffic sent from A to B and B to A should, assuming that the paths are the same, be equal to the delay measured with a two-way measurement.

Also, a delay measured for traffic from A to C via B, should be about the same as the sum of the delays for traffic sent from A to B and B to C.

There are more tests like this possible, they should give us a handle on the in-situ performance of the Test Boxes.

5.3 Error Analysis

The treatment of statistical errors is well-known and should not present any problems.

In order to estimate the systematic errors, we will do tests where the clocks are artificially set of by x ms and see if we can detect this from the results.

6 Implementation

Table 1: The time schedule for this project. The milestones are marked with a *.

6.1 Tools

We are in the process of deciding which tools will be used to implement this project.

• Clock: we are currently investigating which cards are available on the market.
• Test-boxes: our plan is to use off-the-shelf industrial PC's running BSD Unix.
• Plot package: Two packages are being considered:
  • The CERN-library tools (which includes a DST-mechanism) and
  • PGPerl, a PERL extension to the PGPlot package.
• Database: For the first prototypes, we plan to use a home-grown database. At a later stage, a commercial product will be considered.

6.2 Time Schedule

The basic approach is that we want to start with a test bed (see section 5). This setup will be built during the summer of 1997. Data taking will start in the late summer.

If the test bed is a success, we will try to find a handful (10...30) ISP's who are interested in having a Test Box installed at their sites. These boxes will be used to test of the order of 100 connections, where the number of 100 is set by the data volume that can be handled without major investments in storage devices. This setup will start running early 1998. Several ISP's, both from the RIPE community and from those participating in the TERENA TF-ETM working group [8], have shown interest in hosting our Test Boxes.

After this setup has proven to be a success, we will develop a plan to include more ISP's and connections.

A more detailed plan of deadlines and milestones can be found in table 1.

7. Conclusions

In this document, we gave an overview of the test traffic project, discussed the implementation and mentioned a number of questions that have to be studied. The project was presented at the RIPE 27 Meeting and the feedback