High Resolution Traffic Measurement

Glenn Mansfield(1), Sandeep Karakala(2), Takeo Saito(1), Norio Shiratori(2)
(1) Cyber Solutions Ltd. Sendai, Japan.
(2) Research Institute of Electrical Communication, Tohoku University, Japan

Information about the network is of paramount importance for network operation and management as well as network research and development. Presently the main mechanisms of obtaining information from the network are

1. Active monitoring: Standard management protocols like SNMP, tools like ping, traceroute etc., are used to obtain operational information from agents in the network.
2. Passive monitoring: Agents tap all packets traversing a network and they hold and process the information.
3. Hybrid monitoring: Agents essentially carry out passive monitoring and collect information about the network. Clients/applications poll the agents for information.

These mechanisms have developed to a great extent and together they provide a good framework for monitoring networks.

With the growth and spread of the network newer applications and requirements are arising, resulting in more demands being made on the nature and quality of information that is available on the network. One such example is the requirement to know the network traffic at higher time resolutions (e.g. millisecond intervals). Such precision measurements are required for better estimation of network traffic characteristics like burstiness, latency, jitter, for analyzing and detecting traffic patterns etc..

Taking traffic dumps and handling it offline to obtain the desired information is a trivial matter and is a regular practice among network researchers. However, catering to the requirement of online precision measurement poses some interesting challenges. Say we want to take online measurements at intervals of T msecs. Polling the sensor at T msec intervals is impractical as the communication overheads will be prohibitive. Fetching the data, in bulk, in a semi-online manner say, every n X T msecs, where n is large enough, has to be considered carefully as there is a cost associated with making data packets large (fragmentation etc.) and, the measurement latency increases too.

At Tohoku University, as part of the the Japan Gigabit Network[JGN] research project [http://www.shiba.tao.go.jp/JGN] of the Ministry of Posts and Telecommunication we are designing and experimenting with the architecture of a Network Information Warehouse [NIWH] service. The NIWH is open and configurable in the sense that users, with the proper access rights can configure a sensor of the NIWH to capture network traffic information. The user then polls the sensor to fetch the desired information. One of the services provided by the NIWH is high resolution traffic measurement.

We have developed a prototype HRTM system which is based on the current standard network management framework. It uses hybrid monitoring. An HRTM MIB is designed. A new datatype is proposed. It is essentially an SMIv2 data type that resolves to the ASN.1 type OCTET STRING and contains several data elements in compressed form.

A prototype of the HRTM system is currently under experimentation on the JGN. Traffic on a Gigabit Network link is tapped into the system. From the traffic, high resolution information is synthesized according to user specified requirements. Users/applications access the synthesized information using standard management protocols. Public domain tools are extensively used for this purpose.

In this paper we first present our investigations to find the lower limit of the polling interval that can be used to collect information from a network. This constrains the resolution that can be obtained using presently available technology. We then present the architecture and design issues of the HRTM system followed by relevant implementation details and an example application of the HRTM. We discuss the results of our experiments using the HRTM the value it adds and the costs it incurs. We also discuss the limits of the resolution that can be attained using this technology within the current networking and management framework.