DetNet Working Group                                            J. Joung
Internet-Draft                                      Sangmyung University
Intended status: Informational                                   X. Geng
Expires: 23 April 2025                                            Huawei
                                                                 S. Peng
                                                         ZTE Corporation
                                                               T. Eckert
                                                  Futurewei Technologies
                                                         20 October 2024


                     Dataplane Enhancement Taxonomy
                draft-ietf-detnet-dataplane-taxonomy-02

Abstract

   This draft is to facilitate the understanding of the data plane
   enhancement solutions, which are suggested currently or can be
   suggested in the future, for deterministic networking.  This draft
   provides criteria for classifying data plane solutions.  Examples of
   each category are listed, along with reasons where necessary.
   Strengths and limitations of the categories are described.
   Suitability of the solutions for various services of deterministic
   networking are also briefly mentioned.  Reference topologies for
   evaluation of the solutions are given as well.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 23 April 2025.

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.




Joung, et al.             Expires 23 April 2025                 [Page 1]

Internet-Draft                  taxonomy                    October 2024


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   4
     2.1.  Terms Used in This Document . . . . . . . . . . . . . . .   4
     2.2.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Conventions Used in This Document . . . . . . . . . . . . . .   4
   4.  Taxonomy with Performance . . . . . . . . . . . . . . . . . .   5
     4.1.  Per Hop Dominant Factor for Latency Bound . . . . . . . .   5
   5.  Taxonomy with Functional Characteristics  . . . . . . . . . .   6
     5.1.  Periodicity . . . . . . . . . . . . . . . . . . . . . . .   6
     5.2.  Network Synchronization . . . . . . . . . . . . . . . . .   7
     5.3.  Traffic Granularity . . . . . . . . . . . . . . . . . . .   8
     5.4.  Work Conserving . . . . . . . . . . . . . . . . . . . . .   9
     5.5.  Target Transmission Time  . . . . . . . . . . . . . . . .  10
     5.6.  Service Order . . . . . . . . . . . . . . . . . . . . . .  11
   6.  Reference Topologies  . . . . . . . . . . . . . . . . . . . .  12
     6.1.  Grid  . . . . . . . . . . . . . . . . . . . . . . . . . .  13
       6.1.1.  Network topology  . . . . . . . . . . . . . . . . . .  13
       6.1.2.  Flow characteristics  . . . . . . . . . . . . . . . .  14
       6.1.3.  Flow paths  . . . . . . . . . . . . . . . . . . . . .  15
       6.1.4.  Utilization . . . . . . . . . . . . . . . . . . . . .  16
     6.2.  Hierarchical Ring-Mesh  . . . . . . . . . . . . . . . . .  17
       6.2.1.  Network topology  . . . . . . . . . . . . . . . . . .  17
       6.2.2.  Flow characteristics  . . . . . . . . . . . . . . . .  18
       6.2.3.  Flow paths  . . . . . . . . . . . . . . . . . . . . .  19
       6.2.4.  Utilization . . . . . . . . . . . . . . . . . . . . .  20
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  20
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  20
   9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  20
   10. Contributor . . . . . . . . . . . . . . . . . . . . . . . . .  20
   11. References  . . . . . . . . . . . . . . . . . . . . . . . . .  20
     11.1.  Normative References . . . . . . . . . . . . . . . . . .  20
     11.2.  Informative References . . . . . . . . . . . . . . . . .  20
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  22







Joung, et al.             Expires 23 April 2025                 [Page 2]

Internet-Draft                  taxonomy                    October 2024


1.  Introduction

   This draft is to facilitate the understanding of the data plane
   enhancement solutions, which are suggested currently or can be
   suggested in the future, for deterministic networking.

   An enhancement solution can be a combination of multiple data plane
   functional entities, such as regulators, queues, and schedulers.  A
   solution can also include functional entities across network nodes,
   e.g. traffic enforcement or regulation functions at the edge.  A
   regulator, or equivalently a shaper, is defined as a functional
   entity that makes the arrival process of a flow conform to a
   predefined process.  A packet scheduler, or simply a scheduler, is a
   functional entity that determines when a packet is transmitted.

   We use the term taxonomy as a synonym to the criteria for classifying
   the solutions accordingly.  A category is a subset of solutions
   classified into a single group with a taxonomy.  This draft provides
   several taxonomies and the criteria for classifying data plane
   solutions.  These taxonomies are orthogonal to each other.

   Examples of the categories are listed, along with reasons where
   necessary.  Strengths and limitations of the categories are
   described.

   Suitability of the solutions for various services of deterministic
   networking are also briefly mentioned.  The services can be
   classified according to the flow characteristics and the performance
   requirements.  For example, Requirements for Reliable Wireless
   Industrial Services [I-D.ietf-detnet-raw-industrial-req]
   characterizes the services by the latency bound, the burst size, the
   burst transmission period, the number of nodes, etc.  This document
   adopts this characterization rule, and classifies the services into
   one of tight/loose latency, large/small burst, periodic/non-periodic,
   and large/small scale services.  For example, the display information
   service defined in Section 4.4. of
   [I-D.ietf-detnet-raw-industrial-req] is a loose latency, large burst,
   non-periodic, and small scale service.

   The taxonomies described in this draft can be applied for the
   solutions of other standardization bodies, such as IEEE 802.1 TSN TG.

   In this draft, the candidate solutions currently being proposed in
   DetNet WG are simply listed without any descriptions.  The details of
   the solutions are intentionally omitted.  Interesting readers may
   refer to the corresponding drafts.  When necessary, the solutions
   from IEEE TSN TG or existing popular ones are used as examples to
   better understand the taxonomy and the derived categories.



Joung, et al.             Expires 23 April 2025                 [Page 3]

Internet-Draft                  taxonomy                    October 2024


   The mechanisms raised in the DetNet WG are not entirely new concepts
   but rather variations of existing mechanisms.  These deliberate
   approaches aim to address the scalability requirements defined in
   [I-D.ietf-detnet-scaling-requirements] while ensuring a degree of
   continuity and compatibility with the current practices.  The
   taxonomy in this draft reflects how new mechanisms extend existing
   ones to address scalability challenges.

   For instance, Cycle Specified Queuing and Forwarding (CSQF)
   [I-D.chen-detnet-sr-based-bounded-latency], Tagged Cyclic Queuing and
   Forwarding (TCQF) [I-D.eckert-detnet-tcqf], IEEE 802.1Qdv Enhanced
   CQF (ECQF) are enhancements built upon the foundation of Cyclic
   Queuing and Forwarding (CQF).  Similarly, Work Conserving Stateless
   Core Fair Queuing (C-SCORE) [I-D.joung-detnet-stateless-fair-queuing]
   is an extension of Fair Queuing (FQ).  Timeslot Queuing and
   Forwarding (TQF) [I-D.peng-detnet-packet-timeslot-mechanism] is an
   extension of IEEE 802.1Qbv, also known as Time Aware Shaper (TAS).
   Earliest Deadline First (EDF)
   [I-D.peng-detnet-deadline-based-forwarding] proposed to DetNet WG is
   a variation of the well-known mechanism that has the same name.
   Other well-known mechanisms that could provide bounded latency are
   also covered, for example Deficit Round Robin (DRR) and Asynchronous
   Traffic Shaping (ATS) [IEEE_802.1Qcr].

   Reference topologies (RTs) are also listed in this document.  An RT
   consists of a network topology and flows' characteristics.  The RTs
   listed in this document cover various topologies such as ring, mesh,
   hybrid etc.  The purpose of listing the reference topologies (RTs) is
   to evaluate the dataplane solutions how they perform in real
   networks, in terms of E2E latency bound and jitter bound.

2.  Terminology

2.1.  Terms Used in This Document


2.2.  Abbreviations


3.  Conventions Used in This Document

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.





Joung, et al.             Expires 23 April 2025                 [Page 4]

Internet-Draft                  taxonomy                    October 2024


4.  Taxonomy with Performance

   Taxonomy based on the performance, such as E2E latency bounds and
   jitter bounds, is helpful to understand the solutions.  The
   performance should be exhibited as a mathematical expression with the
   network and traffic parameters.

4.1.  Per Hop Dominant Factor for Latency Bound

   One possible taxonomy would be based on the per hop dominant factor
   for the latency bound.  The dominant factor is defined as the largest
   sum term in the expression, when the network and traffic conditions
   are the worst.  The worst condition typically means high network
   utilization, large packet and burst sizes, and large number of hops.
   Any existing solution can be put into one of three categories.

   Category 1 (Max Packet Length/Service Rate): FQ and its variations
   like C-SCORE fall into this category, where the latency bound is
   primarily influenced by the ratio of a flow's maximum packet size to
   its allocated service rate.  This category emphasizes individual flow
   isolation.  The consequence is that the variation of E2E latency
   bound for a flow is minimized with the other flows' join and leave.
   Therefore, this category performs well with dynamic flows.  This
   category also fits well to services with large bursts, since the
   burst sizes of flows are not the dominant factor of the latency
   bound.

   Category 2 (Sum of Max Packet Lengths/Capacity): Solutions like DRR
   belong here, where the dominant factor is the sum of maximum packet
   lengths of all DetNet flows over the total allocated bandwidth.  This
   category typically has less implementation complexity than Category 1
   but can impact individual flow isolation.  The other flows' max
   packet lengths affect the latency bound, which can be altered as
   flows join and leave.

   Category 3 (Sum of Max Burst Sizes/Capacity): CQF, TAS, their
   variations (including CSQF, TCQF, ECQF, TQF), and EDF fall into this
   category.  The key influence on latency here is the total burst sizes
   of all DetNet flows relative to the network capacity.  This category
   prioritizes bounded latency guarantees but may require tighter burst
   control mechanisms.  Once the burst is controlled, for example by an
   extremely strict regulation, into a packet length level, then this
   category may be indistinguishable with Category 2.  This category
   fits well to the services for static flows with small bursts.







Joung, et al.             Expires 23 April 2025                 [Page 5]

Internet-Draft                  taxonomy                    October 2024


   As an example, assuming the capacities and maximum packet lengths are
   identical in all the links along the path of a flow under
   observation, the E2E latency bound of the flow by FQ is given as the
   following [STILIADIS-LRS].

                      (B-L)/r + H(Lh/Rh + L/r),     (1)

   where B, L, and r are the maximum burst size of, the maximum packet
   length of, and the allocated service rate to the flow, respectively;
   H is the number of hops; Lh and Rh are the maximum packet length and
   the capacity of all the links.

   In this example, the term (Lh/Rh + L/r) can be seen as the per hop
   latency, because the max burst size, B, appears only once.  The
   service rate of a flow, r, is likely to be much less than the link
   capacity, Rh, while the maximum lengths L and Lh would not differ too
   much.  Therefore, the dominant factor here is L/r.

   The dominant factor determines the level of flow isolation, as well
   as the level of E2E latency bound value.

5.  Taxonomy with Functional Characteristics

   Taxonomy based on the functional characteristics is the key to
   understanding the solutions.  The taxonomy listed in this section is
   orthogonal to each other, if not stated explicitly.

5.1.  Periodicity

   If a solution transmits packets in a periodic pattern, in which a
   packet is assigned to a time slot based on a predefined rule and a
   set of consecutive time slots repeated periodically, then the
   solution is periodic.  Otherwise, the solution is non-periodic.

   The set of consecutive time slots are called a period.  Note that
   here we use the term period to avoid confusion with the term cycle
   used in CQF, which is equivalent to the time slot defined in this
   draft.

   According to the above definition, IEEE 802.1Qbv TAS is a periodic
   solution.  A finite Gate Control List (GCL) of TAS contains multiple
   gate control entries.  Each entry represents a time slot with an
   assigned set of flows.  A set of consecutive time slots forming a GCL
   is repeated periodically.  Time slots can be overlapped with each
   other, as in ECQF.

   TAS based solutions and CQF based solutions belong to periodic
   solutions, for example CSQF, TCQF, ECQF, TQF and so on.



Joung, et al.             Expires 23 April 2025                 [Page 6]

Internet-Draft                  taxonomy                    October 2024


   Periodic solutions may fit well to periodic services, and vice versa.

5.2.  Network Synchronization

   According to whether network synchronization is required, a solution
   can be classified as either phase synchronous, frequency synchronous,
   or asynchronous.

   Phase synchronous solutions require network nodes to be both phase
   and frequency synchronized.  These solutions can be called strictly
   synchronous.  TAS and CQF are in this category.

   Frequency synchronous solutions require network nodes to be only
   frequency synchronized.  Such nodes are often called syntonized.  CQF
   variations and TAS variations are in this category, for example CSQF,
   TCQF, ECQF, TQF and so on.

   Asynchronous solutions may also require loose phase and frequency
   synchronizations, for example ATS and EDF.

   In non-synchronized networks, it has been shown that ignoring the
   timing inaccuracies can lead to network instability due to unbounded
   delay in per-flow or interleaved regulators [THOMAS-Sync].  However,
   the level of synchronization required is not high.  The problem can
   be solved by adjusting the regulator parameters conservatively, even
   when loosely synchronized clocks are used.  Thus, the solutions that
   require regulators such as ATS are categorized into asynchronous
   solutions.

   The criteria to distinguish between synchronous and asynchronous
   solutions should be the level of required synchronization precision.
   One indicator suitable to such criteria would be the allowable
   Maximum Time Interval Error (MTIE).  MTIE is usually calculated as
   the difference between the largest and smallest time differences in
   the ensemble of measurements.  With this definition, a device that
   has an arbitrarily large and constant time difference with the
   standard reference has an MTIE value of 0, because MTIE is a measure
   of the evolution of the time difference, not the magnitude of the
   time difference itself.  In this respect, the MTIE statistic is
   really a measure of the frequency offset between the device under
   test and the standard reference.

   Therefore, the allowable MTIE value can be applied equivalently, for
   the precision level evaluation, to both phase synchronous and
   frequency synchronous solutions.






Joung, et al.             Expires 23 April 2025                 [Page 7]

Internet-Draft                  taxonomy                    October 2024


   In a distributed system, typical MTIE can be managed within nano
   second level.  However, the exact value of the allowable MTIE as an
   indicator for synchronous solutions is for further study.  It is
   expected to be within tens of nanoseconds.

   Note that the taxonomy of network synchronization is closely related
   to the taxonomy of periodicity.  However, these two can be used
   independently of each other.

5.3.  Traffic Granularity

   This draft categorizes data plane solutions based on the granularity
   of their traffic control target, which refers to the size and
   specificity of the traffic entity they handle.  Three granularity
   levels exist.

   Flow level: Each packet is controlled based on its specific flow,
   which can be identified usually by the 5-tuple.  Examples include FQ
   and its variations such as C-SCORE, which offer precise service
   differentiation but require potentially complex implementation.

   Flow aggregate level: Flows are grouped by shared characteristics
   like traffic specification, service requirement, or routing path.
   This coarser level simplifies control but may offer less precise
   differentiation.  Examples include interleaved regulators in ATS.

   Class level: Flows are further grouped by similar service
   requirements, regardless of specific path or traffic details.  This
   coarsest level simplifies control and accommodates traffic
   fluctuations but provides the least individual flow differentiation.
   Typically, time or time based information could be used for
   classification, such as in EDF, CQF and its variations.

   For each level solution, packets within the same traffic entity
   receive the same treatment.  For example, if a solution is flow
   aggregate level, then the packets within the same flow aggregate are
   treated identically, regardless of the flows they belong to.

   There are cases in which a single solution consists of multiple
   functional entities that treat packets according to multiple traffic
   entities of different granularities.  In such cases, it is defined
   that the functional entity with the coarsest granularity is dominant,
   thus the whole solution belongs to the coarsest granularity category.

   For example, ATS consists of interleaved regulators (IRs) and a
   strict priority scheduler.  An IR has a queue dedicated to a flow
   aggregate having the same class and the same input port.  The
   regulation function itself is based on a flow.  According to the



Joung, et al.             Expires 23 April 2025                 [Page 8]

Internet-Draft                  taxonomy                    October 2024


   definition above, IR is a flow aggregate level solution.  On the
   other hand, the strict priority scheduler in ATS is class-based.
   Therefore, ATS as a whole is class level.

   A finer granularity level solution has a benefit of a more accurate
   service differentiation among flows.  Its limit is the larger
   implementation complexity.  It fits to services with flows having
   various independent latency bound values.

   Periodic solutions can further be categorized based on the traffic
   granularity.  A time slot can be assigned per flow, per flow
   aggregate, or per class.

   Note that TAS in 802.1Qbv is a scheduling mechanism defined in an
   output port with eight queues.  The queues are controlled by GCL and
   its gate control entries.  Each queue can serve a class.  In an
   entry, queues can be either open or closed.  Thus, TAS can be seen as
   a class level solution.  However, in many cases TAS is understood as
   a scheduling mechanism, where the number of queues are not limited to
   8.  There could be a natural extension, such as TQF, which enables
   Qbv to allocate one queue to each flow or a flow aggregate.

   Finer granularity periodic solutions have more strengths in jitter
   control.  They also fit services with many periodic flows of
   independent period values.

5.4.  Work Conserving

   A work conserving solution never idle when there is a packet to send
   [Fedorova].

   A non-work conserving solution can idle even if there is a packet to
   send in the queue.

   A solution can be a combination of multiple data plane functional
   entities, and each functional entity has its own attribute of work
   conserving or non-work conserving.  A solution is non-work
   conserving, as long as any of the functional entities included in the
   solution has the non-work conserving attribute.

   FIFO, round robin schedulers, FQ and its variations like C-SCORE are
   examples of the work conserving solutions.  TAS, CQF, ATS, and their
   variations are non-work conserving solutions, for example CSQF, TCQF,
   ECQF, TQF and so on.  EDF can be operated either as work conserving
   or non-work conserving.






Joung, et al.             Expires 23 April 2025                 [Page 9]

Internet-Draft                  taxonomy                    October 2024


   Work conserving solutions have strengths in terms of average delay.
   They usually show smaller observed maximum latencies than the
   theoretical latency bound expressions suggest.  They also benefit
   from the statistical multiplexing gain without any wasted capacity,
   thus more room for best effort traffic.

   Non-work conserving solutions have strengths to avoid burst
   accumulation and are also beneficial for jitter control.  The burst
   size of a flow can be kept similar or the same with the initial burst
   size.  Therefore, the buffer size necessary typically is less than
   those in work conserving solutions.  This further makes the latency
   evaluation process simple.

5.5.  Target Transmission Time

   Data plane solutions can be categorized as "on-time" or "in-time"
   based on how closely they adhere to predefined target transmission
   times for packets.

   On-time solutions strive to transmit packets as close as possible to
   their target times without ever exceeding them.  This ensures tight
   control over both latency and jitter, but it can sometimes lead to
   higher average latency.

   In-time solutions allow more flexibility, transmitting packets
   without a specified target transmission time.  FQ and its variations
   are in-time solutions.

   ATS, which includes the interleaved regulator, is an in-time
   solution.  A regulator determines an eligible time for a packet to be
   transmitted.  Packets are always transmitted at or later than their
   eligible times.  An eligible time is not a target transmission time.
   Note that ATS is a non-work conserving but in-time solution.

   TAS, CQF, and their variations are on-time solutions.  A time slot of
   TAS, within which a packet should be transmitted, can be seen as the
   target interval.  EDF can be operated either as in-time or on-time.

   The on-time/in-time taxonomy here is about the scheduling decision,
   which determines when a packet is transmitted.  It is not about the
   consequence of the scheduling, whether the jitter bound is also
   guaranteed or not.

   On-time solutions typically control the jitter as well as latency,
   but suffer from larger average latency.  In-time solutions have
   limitations on controlling jitter.  In-time solutions may have to
   handle the jitter with additional mechanisms.




Joung, et al.             Expires 23 April 2025                [Page 10]

Internet-Draft                  taxonomy                    October 2024


5.6.  Service Order

   Data plane solutions prioritize packets from different flows using
   various decision rules, categorized as follows.

   Rate-based: Packets are ordered based on the allocated service rate
   of their flows or flow aggregates.  Examples include FQ and its
   variations like C-SCORE, and DRR.

   Time-based: Packets are prioritized based on their allowed delay or
   deadline.  Examples are CQF, TAS, their variations, and EDF.

   Arrival-based: Packets are served in the order they arrive.  FIFO is
   an example.

   Priority-based: Packets are ordered based on assigned priorities.

   A solution can determine the service order of the packets from
   different flows, based on a rule which considers the rate allocated
   to a flow or a flow aggregate, the delay a packet is allowed, the
   packet arrival time, or the packet priority.  A rule may also be
   constructed with a combination of these characteristics.  Note that
   the service order within a flow cannot be altered, thus is already
   decided.  We focus only on the service order among packets from
   different flows.

   According to its primary service order decision rule, a solution can
   be categorized into either rate-based, time-based, arrival-based, or
   priority-based.  Any solution can also use the packet arrival time as
   a secondary decision rule.

   Strict priority scheduler uses primarily the priority of a packet.
   It also uses the arrival times among packets of the same priority.
   In this case it is categorized as priority-based.

   ATS has IRs and a strict priority scheduler.  The service order among
   packets at an IR is arrival-based.  The order among packets from
   different input ports are decided at the strict priority scheduler.
   Thus, ATS is priority-based.

   Rate-based solutions have a simple admission condition check process
   that is dependent only on the service rates of flows.  They benefit
   from the "pay burst only once" property, by which the maximum burst
   size of a flow contributes to the E2E latency bound only once,
   without being multiplied by the hop count.  Rate-based solutions
   typically fit well to services with large burst and large scale
   services, without a need for overprovisioning, or additional burst
   control mechanisms.



Joung, et al.             Expires 23 April 2025                [Page 11]

Internet-Draft                  taxonomy                    October 2024


   Time-based solutions have strengths in precise delay control for
   packets or flows.  The services with tight latency, small burst, and
   small scale services may fit this category.

   Priority-based and arrival-based solutions benefit from the
   implementation simplicity.  The latency and jitter differentiation
   among flows can be coarse, however.  The services with loose latency,
   small burst, and non-periodic services may fit this category.

6.  Reference Topologies

   The purpose of listing the reference topologies (RTs) is to evaluate
   the dataplane solutions how they perform in real networks, in terms
   of the E2E latency bound and jitter bound.  It is required to exactly
   calculate the E2E latency bound and jitter bound to any flow, given a
   dataplane solution and its parameter choices in implementation
   practices.

   Additionally, the statistical performance evaluation results such as
   the average E2E latency, or the E2E latency distribution is
   recommended to be given.  The scalability in both the data plane and
   the control plane are also recommended to be demonstrated.  The
   implementation complexity of the dataplane solution, the complexity
   of the admission control procedure, and the slot scheduling
   procedure, in an environment with dynamic flows' join and leave, are
   the recommended performance metrics to be demonstrated.

   An RT consists of a network topology and flows' characteristics.  A
   network topology in this document specifies the abstract locations of
   source, destination, relay nodes and their interconnections.  A flow
   characteristic is composed of its path, requested specifications
   (RSpec), and traffic specifications (TSpec).  The requested
   specification includes the E2E latency and jitter bounds.  The
   traffic specification includes the maximum burst size and the average
   rate, as if they have been shaped by a token bucket.  Alternatively,
   a traffic can be specified by the period, the phase, and the maximum
   burst size.  In this case, the maximum burst is transmitted at a
   certain fixed phase within a period of time.

   By specifying the above information, other parameters such as the
   diameter and the maximum utilization of a network can be derived.

   The RTs listed in this document cover various topologies such as
   ring, mesh, hybrid etc.

   Some aspects of the RTs are derived from use cases, in order to
   reflect the current or future network deployment examples.




Joung, et al.             Expires 23 April 2025                [Page 12]

Internet-Draft                  taxonomy                    October 2024


   Based on the RTs, it is also able to check whether a dataplane
   solution can solve the scalability issues, e.g. those specified in
   [I-D.ietf-detnet-scaling-requirements].  The network diameter and the
   utilization level in RTs are set to examine the scalability.

   The major interest of the deterministic networking is in the worst
   case delay, or equivalently the latency bound.  Note that the
   reference topologies have specified the number of flows and their
   traffic specifications.  There can be two different latency bounds
   with either the current scenario specified in the reference
   topologies, or the possible future scenario when more flows enter,
   thus the network becomes fully utilized.

   A dataplane solution can either prepare the worst scenario from the
   beginning, or adjust as the flows come and go dynamically.  If the
   solution is something flexible and tries to adjust, then the both
   latency bounds can be specified.

6.1.  Grid

6.1.1.  Network topology

   A reference network topology, the grid, is shown in Figure 1.  It
   represents a general network of partial mesh or grid topology,
   without considering a specific use case.  A partial mesh is a common
   topology that can be seen in many real deployments, including
   datacenter networks.

                       Src 1      Src 2      Src 3
                         |          |          |
                      +------+   +------+   +------+
                Dst 1-|Node 1|<--|Node 2|-->|Node 3|-Dst 4
                      +------+   +------+   +------+
                         |          ^          |
                         V          |          V
                      +------+   +------+   +------+
                Dst 2-|Node 4|-->|Node 5|<--|Node 6|-Dst 5
                      +------+   +------+   +------+
                         ^          |          ^
                         |          V          |
                      +------+   +------+   +------+
                Dst 3-|Node 7|<--|Node 8|-->|Node 9|-Dst 6
                      +------+   +------+   +------+
                         |          |          |
                       Src 4      Src 5      Src 6

                          Figure 1: Grid topology




Joung, et al.             Expires 23 April 2025                [Page 13]

Internet-Draft                  taxonomy                    October 2024


   In Figure 1, arrowed links indicate the directions to follow for any
   traffic route.  For example, from Node 2, only Node 1 and Node 3 are
   the next possible route.

   The capacity of all the links in the topology is 1Gbps.  While real
   deployments easily exceed 1Gbps link capacity, this RT represents a
   rather scaled down example in terms of the link capacity and the
   number of nodes.

6.1.2.  Flow characteristics

   In-vehicle network (IVN) is an example network which demands
   deterministic networking.  [Buffered_Network] summarizes the flows
   that require deterministic networking services in IVNs as in Table 1.

   +=========+============+===============+=========+=================+
   |   Flow  |  Maximum   |    Maximum    | Arrival |     Required    |
   |   type  | burst size | Packet length |   rate  | maximum latency |
   +=========+============+===============+=========+=================+
   |  Audio  |   2Kbit    |     2Kbit     | 1.6Mbps |       5ms       |
   +---------+------------+---------------+---------+-----------------+
   |  Video  |  360Kbit   |     12Kbit    |  11Mbps |       10ms      |
   +---------+------------+---------------+---------+-----------------+
   | Command |  2.4Kbit   |    2.4Kbit    | 480Kbps |       5ms       |
   |   and   |            |               |         |                 |
   | Control |            |               |         |                 |
   +---------+------------+---------------+---------+-----------------+

                Table 1: Flow types in in-vehicle networks

   To simplify performance analysis, the flows in Table 1 are abstracted
   as shown in Table 2.  Flows of the same type are aggregated into a
   single flow.  For example, ten command and control (CC) flows that
   share the same E2E path can be considered to be a single type C flow
   as in Table 2.  The maximum burst size and the average rate of a type
   C flow are about ten times those of one CC flow, in this case.  Each
   flow type has specific destination nodes.  For example, type A flows
   are destined only to destination 1 or 6.













Joung, et al.             Expires 23 April 2025                [Page 14]

Internet-Draft                  taxonomy                    October 2024


    +======+============+=========+=========+==========+=============+
    | Flow |  Maximum   | Maximum | Arrival | Required | Destination |
    | type | burst size |  Packet |   rate  | maximum  | in Figure 1 |
    |      |            |  length |         | latency  |             |
    +======+============+=========+=========+==========+=============+
    |  A   |   20Kbit   |  2Kbit  |  20Mbps |   5ms    |  Dst 1, Dst |
    |      |            |         |         |          |      6      |
    +------+------------+---------+---------+----------+-------------+
    |  B   |  4000Kbit  |  10Kbit | 100Mbps |   10ms   |  Dst 3, Dst |
    |      |            |         |         |          |      4      |
    +------+------------+---------+---------+----------+-------------+
    |  C   |   20Kbit   |  2Kbit  |  5Mbps  |   5ms    |  Dst 2, Dst |
    |      |            |         |         |          |      5      |
    +------+------------+---------+---------+----------+-------------+

            Table 2: Flow type used in the reference topology

   A source creates one flow to each destination for a total of 6 flows.
   36 flows are created throughout the network.  Table 2 describes
   characteristics of the three different flow types used in the RT.

6.1.3.  Flow paths

   The links are unidirectional as specified in Figure 1.  All the flows
   must follow the direction of the arrows in every link.  For example,
   a flow from source 1 to destination 5 follows the path of
   Src1-1-4-5-2-3-6-Dst9.

   If there are more than one possible route to the destination, then
   the shortest path is selected.  If there are more than one shortest
   path, then the following rules are applied.

   Note that there are at most two outgoing links from a node to select.
   If both choices give the same distance to the destination, the node
   closer to the destination is selected as the next node.  For example,
   from Src 5 to Dst 4, the selection from node 8 is to node 9, not to
   node 7, because node 9 is closer to Dst 4.  When the above rule does
   not break the tie, i.e. the possible next nodes are within the same
   distance to the destination, then the node closer to the source is
   selected as the next node.  For example, from Src 4 to Dst 5, the
   selection from node 5 is to node 8, not to node 2, because node 8 is
   closer to Src 4.

   The above rules generate a unique route for every source and
   destination pair.






Joung, et al.             Expires 23 April 2025                [Page 15]

Internet-Draft                  taxonomy                    October 2024


   The reason for introducing unidirectional links is to make the
   network diameter large.  With this configuration, the network
   diameter is 7 hops, which is relatively large considering a small
   number of nodes.

   The destination of a flow decides the flow type.  For example, all
   the flows destined to node 1 are of type A.  There are 6 flows for
   each destination.  There are 12 flows for each type.  The flows with
   longest paths within the same flow type are of interest.  Table 3
   shows the path of the flows with longest paths for each flow type.
   For all the flow types, the number of hops in the longest paths is
   the same.  The utilizations may differ for different links.

                   +===========+=======================+
                   | Flow type |      Longest path     |
                   +===========+=======================+
                   |     A     | Src5-8-7-4-5-2-1-Dst1 |
                   +-----------+-----------------------+
                   |     A     | Src2-2-3-6-5-8-9-Dst6 |
                   +-----------+-----------------------+
                   |     B     | Src5-8-9-6-5-2-3-Dst4 |
                   +-----------+-----------------------+
                   |     B     | Src2-2-1-4-5-8-7-Dst3 |
                   +-----------+-----------------------+
                   |     C     | Src3-3-6-5-2-1-4-Dst2 |
                   +-----------+-----------------------+
                   |     C     | Src6-9-6-5-8-7-4-Dst2 |
                   +-----------+-----------------------+
                   |     C     | Src1-1-4-5-2-3-6-Dst5 |
                   +-----------+-----------------------+
                   |     C     | Src4-7-4-5-8-9-6-Dst5 |
                   +-----------+-----------------------+

                       Table 3: Longest path of each
                         flow type in the reference
                                  topology

6.1.4.  Utilization

   Network utilization is defined as the maximum link utilization over
   all the links.  The RT achieves network utilization around 60%.  The
   bottleneck links, e.g. the link 2-3, have one type A flow, five type
   B flows, and two type C flows.  The scalability of a solution can be
   properly evaluated with this topology.







Joung, et al.             Expires 23 April 2025                [Page 16]

Internet-Draft                  taxonomy                    October 2024


6.2.  Hierarchical Ring-Mesh

6.2.1.  Network topology

   Another RT, the hierarchical ring-mesh, is illustrated over Figure 2
   and Figure 3.  The core network of the RT is depicted in Figure 2.  A
   backbone node in the core network can be connected to one or two leaf
   network groups.  A leaf network group consists of multiple leaf
   networks.  The number of leaf networks in a group is a design
   parameter, but is recommended to be from 2 to 10.  A leaf network of
   the RT is depicted in Figure 3.

   This RT represents a wide area network, e.g. a state wide backbone
   network, having multiple subsidiary regional networks.  The core
   network is a partial mesh with unidirectional links.  Some of the
   nodes in the core network have the links to the leaf networks.  A
   leaf network is a unidirectional ring with eight nodes.  One of the
   nodes in the leaf network is linked to the core network.

                       Leaf NW    Leaf NW    Leaf NW
                       group 0    group 1    group 2
                          |          |          |
              Leaf NW  +------+   +------+   +------+ Leaf NW
              group 11-|Node 1|<--|Node 2|-->|Node 3|-group 3
                       +------+   +------+   +------+
                          |          ^          |
                          V          |          V
              Leaf NW  +------+   +------+   +------+ Leaf NW
              group 10-|Node 4|-->|Node 5|<--|Node 6|-group 4
                       +------+   +------+   +------+
                          ^          |          ^
                          |          V          |
              Leaf NW  +------+   +------+   +------+ Leaf NW
              group 9 -|Node 7|<--|Node 8|-->|Node 9|-group 5
                       +------+   +------+   +------+
                          |          |          |
                       Leaf NW    Leaf NW    Leaf NW
                       group 8    group 7    group 6

    Figure 2: Topology of the core network in the hierarchical ring-mesh

   The link capacity within the core network is another flexible design
   parameter.  It can be set such that the maximum link utilization is
   50%~80%.  The capacity of connecting link between a leaf network and
   the core network is 1Gbps.






Joung, et al.             Expires 23 April 2025                [Page 17]

Internet-Draft                  taxonomy                    October 2024


   In Figure 2, arrowed links indicate the directions to follow for any
   traffic route.  For example, from the leaf network group 0 to the
   group 6, the nodes 1, 4, 5, 8, 9 have to be visited.

                                 To/From
                                 Core NW
                                    |
                      +------+   +------+   +------+
             Src/Dest-|Node 7|-->|Node 0|-->|Node 1|-Src/Dest
                      +------+   +------+   +------+
                         ^                      |
                         |                      V
                      +------+              +------+
             Src/Dest-|Node 6|              |Node 2|-Src/Dest
                      +------+              +------+
                         ^                      |
                         |                      V
                      +------+   +------+   +------+
             Src/Dest-|Node 5|<--|Node 4|<--|Node 3|-Src/Dest
                      +------+   +------+   +------+
                                    |
                                 Src/Dest

     Figure 3: Topology of a leaf network in the hierarchical ring-mesh

   The capacity of all the links in the leaf network is 1Gbps.

   In Figure 3, arrowed links indicate the directions to follow for any
   traffic route.  For example, for a flow to travel from the node 1 to
   the core network, every node in the leaf network should be visited.

6.2.2.  Flow characteristics

   In this RT, the flows specified in Table 1 are used again.  There are
   three types of flows, which are the audio, video, and command &
   control.  Let's call these flows A, V, and C.

   According to [Buffered_Network], the proportion of these three types
   in in-vehicle networks are such that A:V:C = 7:7:32.  This RT lets
   seven A, seven V, thirty two C flows share the same path.  Let's
   define these flows collectively as a flow set.  A flow set's total
   arrival rate is 103.56 Mbps.









Joung, et al.             Expires 23 April 2025                [Page 18]

Internet-Draft                  taxonomy                    October 2024


6.2.3.  Flow paths

   Every flow in a leaf network travels from node i to node (i+7)mod8.
   Every flow travels 7 hops before leaving the leaf network.  Exactly a
   single flow set enters into and leaves from the network by a node in
   the leaf network.  The node 0, which is the connecting node to the
   core network, also has one flow set coming in from and another flow
   set going out to the core network.

   The links in a leaf network are unidirectional as indicated by the
   arrows in Figure 3.  All the flows follow the direction of the arrows
   in every link.

   All the links in Figure 3 are passed by seven flow sets.  This is the
   maximum number of flow sets in a link, over all the links.  Any flow
   from a leaf network travels seven hops before leaving.  This is the
   maximum number of hops in the leaf network.

   In the core network, each leaf network group i sends n flow sets to
   the leaf network group (i+6)mod12, where n is the number of leaf
   networks in a leaf network group.  These n flow sets are distributed
   to the destination leaf networks within the group evenly.

   The links in the core network are unidirectional as indicated by the
   arrows in Figure 2.  All the flows follow the direction of the arrows
   in every link.

   At a crossroad, where a flow can choose one of two possible paths,
   the flow always turns left.  For example, a flow from the leaf
   network group 1 travels to nodes 2, 3, 6, 5, 8, and then the leaf
   network group 7.

   The link between the nodes 4 and 5 in Figure 2 are passed by (n times
   six) flow sets.  This is the maximum number of flow sets in a link,
   over all the links.  After joining the core network, a flow from the
   leaf network group 0 travels four hops before leaving.  This is the
   maximum number of hops in the core network.

   The maximum hop count in the leaf and core networks are 7 and 4,
   respectively.  However, there are one hop each to/from the core
   network.  The maximum hop count in this RT is 20.










Joung, et al.             Expires 23 April 2025                [Page 19]

Internet-Draft                  taxonomy                    October 2024


6.2.4.  Utilization

   Network utilization is defined as the maximum link utilization over
   all the links.  A leaf network has the network utilization of (seven
   times 103.56 Mbps) over 1 Gbps, which is around 72.5%.  The
   utilization of the core network can be of any value, because the link
   capacity in the core network can be chosen to be any value greater
   than the (n times six times 103.56 Mbps), where n is the number of
   the leaf networks in a leaf network group.  For example, if n equals
   to 2, then the link capacity is required to be larger than 1242.72
   Mbps.  If the link capacity is 1.5 Gbps, then the utilization is
   around 82.85%.

7.  IANA Considerations

   There might be matters that require IANA considerations associated
   with metadata.  If necessary, relevant text will be added in a later
   version.

8.  Security Considerations

   This section will be described later.

9.  Acknowledgements


10.  Contributor


11.  References

11.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

11.2.  Informative References

   [Buffered_Network]
              Joung, J. and J. Kwon, "Zero jitter for deterministic
              networks without time-synchronization", IEEE Access, vol.
              9, pp. 49398-49414, doi:10.1109/ACCESS.2021.3068515, 2021.



Joung, et al.             Expires 23 April 2025                [Page 20]

Internet-Draft                  taxonomy                    October 2024


   [Fedorova] Fedorova, A., Seltzer, M., and M.D. Smith, "A non-work-
              conserving operating system scheduler for SMT processors",
              In Proceedings of the Workshop on the Interaction between
              Operating Systems and Computer Architecture, vol. 33, p.
              10-17, June 2006.

   [I-D.chen-detnet-sr-based-bounded-latency]
              Chen, M., Geng, X., Li, Z., Joung, J., and J. Ryoo,
              "Segment Routing (SR) Based Bounded Latency", Work in
              Progress, Internet-Draft, draft-chen-detnet-sr-based-
              bounded-latency-03, 7 July 2023,
              <https://datatracker.ietf.org/doc/html/draft-chen-detnet-
              sr-based-bounded-latency-03>.

   [I-D.eckert-detnet-tcqf]
              Eckert, T. T., Li, Y., Bryant, S., Malis, A. G., Ryoo, J.,
              Liu, P., Li, G., Ren, S., and F. Yang, "Deterministic
              Networking (DetNet) Data Plane - Tagged Cyclic Queuing and
              Forwarding (TCQF) for bounded latency with low jitter in
              large scale DetNets", Work in Progress, Internet-Draft,
              draft-eckert-detnet-tcqf-06, 5 July 2024,
              <https://datatracker.ietf.org/doc/html/draft-eckert-
              detnet-tcqf-06>.

   [I-D.ietf-detnet-raw-industrial-req]
              Sofia, R. C., Mendes, P., Bernardos, C. J., and E.
              Schooler, "Requirements for Reliable Wireless Industrial
              Services", Work in Progress, Internet-Draft, draft-ietf-
              detnet-raw-industrial-req-01, 8 July 2024,
              <https://datatracker.ietf.org/doc/html/draft-ietf-detnet-
              raw-industrial-req-01>.

   [I-D.ietf-detnet-scaling-requirements]
              Liu, P., Li, Y., Eckert, T. T., Xiong, Q., Ryoo, J.,
              zhushiyin, and X. Geng, "Requirements for Scaling
              Deterministic Networks", Work in Progress, Internet-Draft,
              draft-ietf-detnet-scaling-requirements-06, 22 May 2024,
              <https://datatracker.ietf.org/doc/html/draft-ietf-detnet-
              scaling-requirements-06>.

   [I-D.joung-detnet-stateless-fair-queuing]
              Joung, J., Ryoo, J., Cheung, T., Li, Y., and P. Liu,
              "Latency Guarantee with Stateless Fair Queuing", Work in
              Progress, Internet-Draft, draft-joung-detnet-stateless-
              fair-queuing-03, 2 July 2024,
              <https://datatracker.ietf.org/doc/html/draft-joung-detnet-
              stateless-fair-queuing-03>.




Joung, et al.             Expires 23 April 2025                [Page 21]

Internet-Draft                  taxonomy                    October 2024


   [I-D.peng-detnet-deadline-based-forwarding]
              Peng, S., Du, Z., Basu, K., cheng, Yang, D., and C. Liu,
              "Deadline Based Deterministic Forwarding", Work in
              Progress, Internet-Draft, draft-peng-detnet-deadline-
              based-forwarding-12, 8 August 2024,
              <https://datatracker.ietf.org/doc/html/draft-peng-detnet-
              deadline-based-forwarding-12>.

   [I-D.peng-detnet-packet-timeslot-mechanism]
              Peng, S., Liu, P., Basu, K., Liu, A., Yang, D., and G.
              Peng, "Timeslot Queueing and Forwarding Mechanism", Work
              in Progress, Internet-Draft, draft-peng-detnet-packet-
              timeslot-mechanism-09, 12 August 2024,
              <https://datatracker.ietf.org/doc/html/draft-peng-detnet-
              packet-timeslot-mechanism-09>.

   [IEEE_802.1Qcr]
              IEEE, "IEEE Standard for Local and metropolitan area
              networks - Bridges and Bridged Networks - Amendment 34:
              Asynchronous Traffic Shaping", IEEE 802.1Qcr-2020,
              DOI 10.1109/IEEESTD.2020.9253013, 6 November 2020,
              <https://doi.org/10.1109/IEEESTD.2020.9253013>.

   [STILIADIS-LRS]
              Stiliadis, D. and A. Anujan, "Latency-rate servers: A
              general model for analysis of traffic scheduling
              algorithms", IEEE/ACM Trans. Networking, vol. 6, no. 5,
              pp. 611-624, 1998.

   [THOMAS-Sync]
              Thomas, L. and J-Y. Le Boudec, "On Time Synchronization
              Issues in Time-Sensitive Networks with Regulators and
              Nonideal Clocks", Proceedings of the ACM on Measurement
              and Analysis of Computing Systems, vol. 4, no. 2, pp.
              1-41, 2020.

Authors' Addresses

   Jinoo Joung
   Sangmyung University
   Email: jjoung@smu.ac.kr


   Xuesong Geng
   Huawei
   Email: gengxuesong@huawei.com





Joung, et al.             Expires 23 April 2025                [Page 22]

Internet-Draft                  taxonomy                    October 2024


   Shaofu Peng
   ZTE Corporation
   Email: peng.shaofu@zte.com.cn


   Toerless Eckert
   Futurewei Technologies
   Email: tte@cs.fau.de











































Joung, et al.             Expires 23 April 2025                [Page 23]