Test Traffic Working Group Minutes from RIPE 50

• content to the Chair of the Working Group.
  
• format to webmaster@ripe.net.

Name abbreviations

AS = Ad Spelt
AT = Alexander Tudor
BN = Brian Nisbet
CP = Christian Panigl
GN = Geert Nijpels
HU = Henk Uijterwaal
LC = Lorenzo Colitti
LD = Luca Deri
NR = Niall O'Reilly
RD = Rickard Dahlstrand
RR = Rainer Rudiger
TS = Thomas Schmid

The minutes of the previous meetings were approved.

Presentations

Henk Uijterwaal: TTM status update

- Internal restructuring done
- DNSMON v1 out, now a self-supporting service. Request list for v2 exists
- Technical improvements: taildaemon bug fixed, timekeeping improvements
- CDMA clocks: looking for a volunteer to help debug problems
- OWAMP: currently planning implementation
- Other work in progress
- Bandwidth, VOIP measurements, v4/v6 comparison
- Look at Neil O'Reilly's ISP quality measurements

Thomas Wana: Improving Time Measurement in TTM

- Evaluated precision of current TTM setup

- Looked at NTP alternatives
- ATOM is a little better than standard PPS_SYNC and will be used
- ntpns only works on FreeBSD 5

- Improved sending timestamping
- Implemented kernel-level timestamping for TTM sender
- Lower offset, lower variation
- Production ready

- Looked at receiver timestamping
- This is the major source of noise in current TTM
- Not easy to fix, depends on FreeBSD architecture
- Proposal: equip testboxes with DAG cards
- Accuracy is extremely good
- Requires calculating delays centrally
- Results on the testboxes themselves would continue using the old numbers
- Would be optional
- A DAG card costs ~800 euros

LD: You should use something like nCap to obtain direct access to the card.

Discussion on Third Party Verified SLAs

AT: It's not clear to the membership what the value proposition of TTM is. The original idea was that ISP would be interested in comparing their performance to others, but it was not until very recently that someone proposed to use TTM in operations.

There is a place for an organisation such as RIPE NCC that could provide an unbiased metric. ISPs or IX operators could then point to the data to show that their service is good.

Points for discussion:

- Does it make sense for RIPE NCC to play the role of a trusted third party?
- If so, what do we want it to measure? Loss, delay, etc. are probably not sufficient.
- What does it take to turn the service into something that's useful for members and their customers?

RD: We must distinguish between several markets: service providers, government, content providers, end-users

TS: Let me add "the press"

AT: The press always turns any measurement into a competition or a race. This is meant to help ISPs. The measurements must be data that the members can point to, not something to pit one ISP against another.

RR: As soon as you measure something, you are comparing

RD: Compare with DNSMON: it can be used to see which of the three competitors for .se is best. They don't like the fact that DNSMON is publicly available. Similarly if TTM data is available that could be used to make ISPs look bad.

AT: We need to allow the ISP to provide info on SLAs

HU: 1. This is not about monitoring the quality of service of one ISP (meaningless since there are different packages, SLAs,
overprovisioning, prices...). What is interesting for a customer is "does ISP X deliver what they promise?"
2. The .se operators might not be happy, but they are committed to provide good service, and if there is an objective tool that shows that they are not then it is interesting both for users and for them themselves.

RD: This is still an obstacle. The test-boxes are placed in the network of the ISPs that they are going to monitor and the ISPs won't want to host them if they fear the results might make them look bad.

AT: Again this is not about ISP x vs ISP y but measuring ISP x against their claims. This is not contentious and is good for the service provider.

BN: It's a fine distinction. Users may say, "Well, I'll move to the ISP who does fulfil their promises"

AT: This is an important concern. But sooner or later users will become more savvy. Do ISPs want to be prepared or not?

BS: Well, for us the point is that we are not a commercial service.

AT: How many people here are ISPs?

Show of hands: 5

AT: So you ISPs: If you went to marketing and asked to do this, would it fly?

AS: In the Netherlands we already have a service that ranks ISPs

TS: You can cheat and apply qos to the addresses that measure quality

AS: We can't: they just buy our products and use them to go to random websites

AT: This is getting close to Neil's talk. Who would be happy with providing such information to their customers?

RR: We do publish information. Actually, we did the measurements to prove to the customer that our service was ok and that the problem was on their side.

AT: So the customer doesn't trust your numbers and wants a third party.

RD: ISP networks are complex. What are we testing? DSL? Distribution? Core?

AT: We don't know. What should we do? You tell us. Those ISPs who say this is not an idea, why would this not be a good idea?

Reaction from the participants: none

AT: So RIPE NCC is a good party to measure this? It would be trustworthy?

GN: we chose RIPE NCC precisely because it's is a trusted third party

RR: The residential customer does not know who RIPE NCC is.

AT: So you would need to build up the brand

HU: You just say "this is a third party"

RR: If we document exactly how the measurement is done, it doesn't matter who is doing it.

AT: So you could base your SLAs on standard numbers that TTM measures.

AT: Second point: we agree that RIPE NCC is a good third party to do this work, but what is this work? What is it that is interesting for ISPs and their customers to measure?

RD: In Sweden we have three types of services that are used to test bandwidth, etc. We could complement that not with a network of measurement points (too expensive to deploy many boxes in Sweden) but by providing a test suite or a tool which could be used by other people

AT: So we could put a test-box at a public exchange point and people can sign up for measurements by providing a pingable address. Would this be acceptable?

LC: How do you know it's not the user's fault if his connection is performing badly?
AT: You don't...

RD: I think we need something intermediate between the current ttm network and this solution. Maybe a 100 euro box that could be deployed where the customers want it? There is no single point in the network that you can use as reference for measurement

HU: You could provide a box at the ISP's edge and measure from there to the Internet and from there to the customers. Internet users installing hardware at home is a logistical nightmare.

AT: We need to walk before we run. First we need to show that there is community interest in a proposal. So is a public exchange point a reasonable place from which to use the TTM metrics as they stand today?

LC: An exchange point LAN doesn't make sense, most of the time they don't even use globally routed addresses.

GN: I talked to people at Euro-IX and there is interest from other IXs to use TBs for multi-location L2 exchange point monitoring. That's a new market.

CP: Operating a reference point at an exchange point is logical, but the exchange points in europe are very neutral with their
customers (the ISPs). Providing data for ISP's would seem to be discrimination between ISPs. Also, if we were a reference point for measuring we would need to answer user requests too.

RD: In Sweden we have a bandwidth measurement project and there is a reference point for that at Netnod.

AT: The customer doesn't care where the box is. He knows that the box is operated by a third party and that his ISP has given him a URL to a page where he can check the measurements. But is an exchange point a good location?

BN: I don't think the ISP would think the IX was taking sides. But the exchange is not always the best egress point in the network from which to measure.

AT: Ok, so to wrap things up:
- We agree that this is an idea that makes sense
- We should continue discussion amongst people who are interested
- Who would like to provide a proposal?

RD, RR, NR volunteer

AT: Ok, so we expect something before next meeting.