Benchmarking Methodology WG Minutes

WG Chair: Kevin Dubray

Minutes reported by Kevin Dubray.

The BMWG met at the 43rd IETF on Monday, December 
7th, 1998.  The group consisted of over 40 people.

The chair presented the agenda:

    1. Agenda/Administration
    2. Benchmarking Terminology for Firewall
       Performance.
    3. Terminology for Cell/Call Benchmarking
    4. Presentation: Latency Characterization in
       LAN switching devices.

1. Agenda/Administration

The agenda was approved as presented.  Dubray 
announced that the multicast benchmarking terminology
draft was announced as RFC 2432.  Unfortunately,
the LAN switch benchmarking methodology draft did not
progress enough to merit discussion at this time.
Additionally, the editor of the latency terminology
draft believed that the scope of the work merited
investigation before proceeding further.

With regards to the multicast terminology draft, 
Dubray announced that a volunteer to help edit
the draft was still needed.  Interested parties
should apply.

2. Benchmarking Terminology for Firewall Performance.
    
David Newman was then introduced to lead a discussion
on the Firewall Draft.  David gave a quick overview
of the modifications to the draft.  Notable changes 
include:

    Security Association
    Session
    Session Establishment
    Session Maintenance
    Session Teardown
    Unit of Transfer
    Enhancement of connection-related terms.

David also communicated that some other minor
cleanup had also ocurred.  In discussing some of
the changes, David announced that he added the
notion of security association to address an issue
brought up in earlier discussion: one does not
want to undertake a forwarding rate type of
measurement until "all hands have been shaken."

On the term "session," David was keen to point out
that session is not a measurement itself; rather it
was a list of things that needs to happen before 
measurement can happen.  Likewise, the need to
finish forwarding rate measurements should occur
before session teardown.

In exploring the term "Unit of Transfer," Newman
stated that he felt the alternative term 
"transaction" was too generic.  He also concluded that
other definitions were too narrow.  The suggested term
"Unit of Work" was attractive, but David thought it
didn't reflect the movement of data.  In the UOT
discussion, David thought it was important to 
communicate that while UOT is characteristically broad,
users must keep the comparisons narrow to be useful. 

David closed the summary, saying that he believe the 
draft was ready for WG Last Call.

A question was raised as to the communicated duality 
of term "session," - is it data or a metric?  David 
articulated that he would attempt to clear up the 
ambiguity.

Presuming the inclusion of discussed issues in a
subsequent draft, the chair surveyed the group as 
to whether a WG Last Call should be issued on the
draft.  There was consensus to issue the Last Call.

4. Presentation: Latency Characterization in LAN 
   switching devices.
 
The chair announced that a reordering of the
agenda was going to take place.  John Dawson was
introduced to give his presentation on Latency.
(see slides in the proceeding or the Powerpoint
presentation on the BMWG archive).

Some questions were posed after the presentation.
One person asked whether it was possible for latency
measurements to yield a negative number.  Dawson 
indicated that it was, indeed, possible - especially
in devices with cut-through architectures.  He added
that it was his experience that these architecture 
are not predominant.  He cited that handling mixed
media speeds made cut-through more problematic than
store-and-forward architectures.

Another person asked whether the latency proposals
operated at Layer 2 or Layer 3.  John indicated the
focus was on a packet from one interface to the next.

There was discussion about the notion that 100%
line rate is susceptible in regards to clock
variances.  Someone queried as to whether 100%
is a practical case, anyways.  John didn't believe so
as many protocols require acknowledgement in practice.

Another question was posed as to whether clock 
synchronization was an issue.  John thought is was
primarily an issue when synchronizing the DUT's clock 
with the test gear's clock.

A question asked whether the drift of multiple
streams cancel the net effect of the drift.  Dawson
said no.

Dawson made the recommendation that a review of 
latency related items in existing RFCs may make
make the metrics more meaningful.  More pointedly, 
Dawson stated that he believes that current workding
of the metrics or their related methodology lends to 
novel workarounds that demonstrate something other 
than true latency.

Dawson said he would forward the presentation and 
related paper to the chair for posting on the BMWG 
list.

3. Terminology for Cell/Call Benchmarking

With the editors of the draft not present, Dubray
proceeded to attempt a review of the current draft.

Many of the comments reflected that the metrics
currently presented seemed too link-centric and did
not necessarily offer insight into characterizing 
devices that combined network and link based
forwarding technology such as edge devices.

Another comment suggested that the measuring
buffering ability and impact may be useful.
 
With that Dubray closed the working group session
and stated the WG goals for the next period:

     1. Update and reissue the Firewall draft.
        Announce WG Last Call.
     2. Show demonstrable progress on LAN switch
        methodology document.
     3. Name editor(s) to multicast methodology
        work and produce a first draft.
     4. Advance the Cell/Call benchmarking draft.


** Minutes after the WG adjournment, the Cell/Call
   terminology draft's editors arrived having 
   experienced flight delays.  An auxillary meeting
   was announced and was held on Tuesday, Dec 8.

   The minutes of that meeting follow:

Terminology for Cell/Call Benchmarking

Jeff Dunn and Cynthia Martin gave a quick overview 
of the presentation to meeting attendees.  The 
presentation is to be posted to the list and
in the BMWG minutes.

Following the presentation, some issues were 
discussed.

It was asked, "What is the goal of the draft given 
that other organizations (ITU-T, ATMF, etc.) are
operating in this space.  The editors responded that
they were leveraging existing work, because the
this work helps differentiate the parameters that
may affect the IP context.  For example, CDV 
is a factor that will affect TCP on the user layer.

The authors communicated taking a "layered" approach, 
e.g., how does the service provider affect the 
service user(s).  For example, if you have an error 
at the ATM layer, it has an implication at the AAL 
level, which has implications on the IP layer and 
TCP layer.  The document needs to point out that 
there is correlation to higher layer(s).

Authors also stated their belief that restricting 
the document's scope to ATM may allow for quicker 
prototypical work.

The authors put forward the notion that they would
like to advance the methodological work - if nothing
else, they would like to start discussion on how to 
architect the methodolgy. The goal would be to produce
a usable framework for testing, independent of
implementation.

When queried as to the changes for the next 
version of the terminology draft, the editors
stated that they would like to develop the discussion
section and enhance the measurements offered.