Test Traffic Measurements (TTM) Survey 2002
Rene Wilhelm
Matthew Williams
Document ID: ripe-270
Date: 03 March 2003
Table of Contents
1. Introduction
Prompted by the RIPE NCC’s 2002 Members and Stakeholders Survey,
the New Projects group decided to initiate a survey focusing on aspects
of Test Traffic Measurements (TTM) that, due to scope constraints, could
not be a part of the general survey. The objective of the TTM survey was
to shed some light on user experiences in addition to obtaining valuable
input on improvements, direction and potential services.
2. Methodology
The survey encompassed two different methods for gathering information
from respondents. Most respondents provided answers by filling out a questionnaire
and returning it either via normal or electronic mail. However, in three
cases, test-box hosts were interviewed on-site in order to attain a more
comprehensive understanding of how users interact with TTM services in
daily operations. These hosts were arbitrarily chosen out of a group of
candidates within reasonable geographical proximity to the RIPE NCC in
Amsterdam.
We approached all test-box hosts for whom we had complete address information
in early December last year. This meant that a total of 45 users, some
who have more than one test-box, were either contacted in person or sent
a questionnaire via the postal system. Sixteen hosts, nearly 36 percent
of the initial sample, responded to our survey by our deadline in mid-January,
2003.
3. Results
3.1 Usage
We learnt from the on-site interviews that the TTM services were mainly
used internally at Network Operation Centres (NOCs) for investigating
customer complaints and monitoring transit providers. This implied that
other applications for TTM data seemed to be rare.
Our impression was also supported by results in the survey. For instance,
75 percent said that they had never used TTM as a sales tool to acquire
new business, and furthermore, the same number of respondents had never
shown plots to upstream providers or customers when there were connectivity
problems. There appeared to be some interest in integrating TTM into Network
Management Systems, however, the results were not conclusive. Forty percent
of respondents answered that integration would definitely be a major benefit,
while 53 percent said "no" and another seven percent did not
know for sure.
Comments:
"...eases visibility and reaction time..."
"...prefer manual access, enough for now..."
The average test-box host accesses the TTM web site on a weekly or monthly
basis. Delay plots/summaries and the traceroute database appear to be
the most frequently accessed services, while jitter plots and plots-on-demand
are used to a lesser extent. Moreover, with some notable exceptions, the
test-box web interface seems only to be used sporadically or not at all.
Seventy-nine percent, who answered the question: "How often do you
directly access the test-box web interface?", had either used the
web interface twelve times or less per year or never accessed it. The
majority of respondents who had tried it indicated that they had only
investigated either the "inbound delay" and/or "satellite
conditions" features on the test-box. Another service that appears
to be utilised to a lesser extent than expected are the e-mail alarms
trigged when unexpected measurement conditions are detected by TTM. Sixty
percent of respondents indicated that they either always or usually ignore
the notifications, mainly because of the sheer volume of messages generated
per day.
Comments:
"...filter out these emails and check the TTM site myself..."
"...too many alarms that didn't indicate real problem..."
 According to the survey, half of test-box hosts first heard about RIPE
NCC's Test Traffic Measurements at a RIPE Meeting; another 25 percent
found out about TTM on the Internet; and the rest selected other options
such as "RIPE NCC contact", "mailing list" or "training
course". Many respondents, about 38 percent, also indicated that
the RIPE Meetings were the main forum for staying informed of TTM news
and developments. Mailing lists and the web site were other important
information channels for users. Looking at survey results, it was concluded
that the awareness of TTM services among users is very good. Eighty-one
percent expressed that they knew at least four out of six stated services.
When asked about new services 44 and 50 percent, respectively, indicated
that QoS/DiffServ and native IPv6 networks would be deployed within the
short-term future. This was encouraging to the New Projects group as TTM
supports IPv6 measurements since 1 February 2003.
3.2 Ratings
Respondents were asked to rate different variables in regard to the TTM
web site, test-box interface, support and data analysis on a scale from
one to six. In general, the averages of ratings varied from 4.7 to 5.6,
which would have to be interpreted as "very good" to "excellent".
Both support and data analysis were given significantly higher average
scores than the TTM web site or test-box interface, especially support
was rated very highly by users. However, in cases where respondents gave
lower scores, usually around four, more improvements to services were
noted.
We can conclude that "ease of use" and "presentation"
with regard to the TTM web site's appearance as well as the layout of
measurement data are issues that need attention. Other items such as improved
"online help", "features" and "configuration"
options could be added to the test-box interface at a later stage. Please
note that only TTM test-boxes of series C/D have a web interface.
3.3 Statements
As the RIPE NCC has been evaluating the TTM business model for the 2004
Activity Plan, test-box hosts were asked to express their opinions regarding
a number of statements. The statements were:
"The financial cost of having TTM installed at your site has
been a worthwhile investment."
Practically all of the respondents, 88 percent, strongly agreed, agreed,
or slightly agreed that installing a test-box at their site had been a
worthwhile investment. This indicates that, although some elements of
the service need improvements, it is a welcome addition to the set of
tools available to network operators.
"TTM should not change its current business model."
The results were quite conclusive in this case. Seventy-seven percent
of those who answered either strongly agreed, or agreed with the statement.
A number of "non-respondents" said privately that the construction
of a business model was an internal matter for the RIPE NCC to solve on
its own and, therefore, refrained from selecting their preference.
"TTM may raise its service fees by 25% to 50% and my organisation
will continue to use the service."
As one would expect, most test-box hosts indicated that they disagree
with this statement. Two-thirds of those who gave an answer slightly disagreed,
disagreed, or strongly disagreed with a 25-50 percent increase of the
service fee. During our on-site interviews, we did sense a certain amount
of flexibility regarding this issue if TTM would improve or extend its
services.
"The RIPE NCC may outsource TTM to a third party not-for-profit
organisation that continues to provide the service under the same conditions
as today."
The responses from hosts with regard to this statement were mixed and,
therefore, harder to interpret. Fifty-six percent chose an "agree"
option, while the rest have indicated that they disagree to some extent
with above. However, there is an emphasis in the distribution of answers
on "slightly agree", which was the response of 31 percent. It
is clear that TTM users show some flexibility on this point. Their main
concerns regarding a not-for-profit third party are related to the neutrality
of the organisation and its immunity against mergers and acquisitions.
"The RIPE NCC may outsource TTM to a third party commercial
organisation that continues to provide the service under the same conditions
as today."
It became very clear that the above statement was quite unacceptable
to users. Nearly two-thirds of respondents, 62 percent, either disagreed
or strongly disagreed with outsourcing TTM services to a commercial third
party. It can be concluded that this is not a viable route for the future
of TTM.
4. Conclusions
The main points from the survey have been summarised into three categories:
4.1 Strong Points
- Most of the respondents have only praise for the quality of TTM data
and support. New Projects has apparently done well in this aspect as,
in most cases, TTM is a RIPE NCC service that is highly regarded by
users.
- During the on-site interviews, most have emphasised the importance
of the neutrality and objectivity of the RIPE NCC. It is considered
that few other organisations could be trusted to produce unbiased internetwork
performance measurements. This is possibly the strongest argument for
a TTM service provided by the RIPE NCC.
4.2 Improvements Required
- The usability of the TTM web site and presentation of data needs
much attention. It is clear that the structure of our web site needs
to be reconsidered and TTM data made more accessible to users who are
not engineers. For instance, some survey participants suggested producing
the TTM plots in two versions, one for network operators and another
for people with a non-technical background. Other suggestions from respondents
address the lack of customisation possibilities of TTM services such
as controlling the amount of alarms sent to hosts, selecting relevant
measurement relations on delay summaries, and being able to group them
by region or upstream providers.
- Throughout the on-site interviews, we were surprised to learn that
TTM users did not access the web interface features on the C/D series
of the test-boxes as much as expected. However, most hosts did show
an interest in these features once they became aware that the plots
of inbound delays and GPS conditions were in near-real-time. There seem
to be a couple of pitfalls here that should be avoided in the future.
These include:
- Adding features that do not benefit most users instead of focusing
resources on consolidating the existing product.
- Incorrectly assuming that hosts have digested TTM communications
just because a message has been announced on a mailing list or at
a RIPE Meeting.
These situations could be avoided by actively seeking more interaction
with users, i.e. by conducting beta testing programmes, visiting customers
on-site, and doing surveys more frequently. Improved interaction with
customers is definitely needed.
4.3 Features Wanted by Users
As mentioned previously, it is prudent to focus on improving and consolidating
current features before developing new ones. Nevertheless, survey respondents
have provided many interesting suggestions that should be investigated
further in parallel with consolidation. Here are a few examples:
- Bandwidth measurements
- SNMP/NMS integration of alarms
- Internal measurements between PoPs
- Measurements from multiple NICs
- Manipulating the ToS field in the IP header
- Protocol-based performance analysis
5. Lessons Learnt
The outcome of this survey has provided the TTM crew with much food for
thought. The main lesson being learnt is the importance of actively approaching
users for ideas on improving current features before moving on with new
projects. It is imperative for TTM's appeal that the New Projects group
approach its customer base more often in the future. Obtaining feedback
by conducting a survey like this has been a very rewarding experience.
Credits to all who participated.
Many thanks!
| 
you are here: RIPE -> RIPE Document Store
