Skip to main content

IP Streams, Flows and Torrents: Measuring Stream Distributions in Real Time

Nevil Brownlee, The University of Auckland / CAIDA

RTFM (RFCs 2720-2724) considers network traffic as being made up of Flows, which are arbitrary groupings of packets defined only by attributes of their end-points. RTFM flows are also bi-directional, with the user specifying which end-point is the flow's source.

This paper extends RTFM's view of traffic by adding two further concepts, torrents and streams. A 'torrent' is the collection of flows which make up the total traffic on a network link. An RTFM meter may measure all packets in the torrent, or may select only a small subset of them.

'Streams' (which could be referred to as 'bi-directional microflows') are individual IP sessions (e.g. TCP or UDP) between ports on pairs of hosts. An RTFM meter has been extended to maintain queues of streams belonging to currently active flows, so as to produce distributions summarising stream behaviour. This was done so as to implement packet-pair matching for the 'turnaround time' attributes proposed in RFC 2724.

Streams introduce a new dimension into RTFM, making it possible to observe distributions of stream sizes and lifetimes. It also introduces new complexities into the meter's packet matching algorithms.

When using streams in the meter, flows provide a first level of packet matching, selecting the packets of interest. If a flow's direction is specified (e.g. packets from A to B), all its streams will have that direction, so the meter can easily decide whether a packet within a stream is a request (source to destination) or a response. A flow, however, can be very coarse-grained, leaving its direction unspecified To cope with such flows (e.g. all UDP packets), the meter must do bi-directional packet matching for streams.

To associate pairs of packets (e.g. request/response pairs for ICMP echo or for DNS over UDP), the meter maintains a queue of packets for each stream. In its memory management the meter implements a variety of timeout schemes for flows, streams and packet queues; these include fixed times (flows), protocol-based (TCP streams) and dynamic (packet queues). Details of the dynamic timeout algorithm will be discussed.

Stream measurement work using a meter located at SDSC (the San Diego Supercomputer Centre) will be presented. This has been concentrated on three measurement projects:

  • 10s data rates: Data rate distributions for all traffic on a busy Internet link. These give a good overview of the link's behaviour.
  • DNS response: DNS packet matching has been used to investigate response time and loss rates to the root and gTLD nameservers. Plots will be presented showing the behaviour of various roots, and comparing them with their corresponding gTLD servers.
  • Stream sizes and lifetimes: Plots showing how these vary during the day will be presented. Differences between UDP, non-web TCP and web TCP traffic will be discussed.

For packet pair matching to work as expected it is important that the meter sees all the request and response packets within a flow. The meter was used to discover which of the netblocks using the SDSC link met this 'symmetry' condition. A lack of such symmetry can influence stream size distributions; current work on this will be described.