Authors Version-1
Authors Version-1
Authors Version-1
Abstract—Developments such as Industry 4.0, Smart Grids, or Centralized User and Network Configuration (CUC, CNC)
Intelligent Transportation System (ITS) depend on reliable high- serve to manage TSN’s features. Yet, integrating TSN into
performance communications to enable the underlying control existing networks benefits from harnessing established control
algorithms. Nevertheless, in most cases it is not viable to provide
network infrastructures exclusively for mission-critical control planes. Therefore, this work proposes the orchestration of TSN
traffic or just the general use case. Hence, 5G as well as future 6G via Software-Defined Networking (SDN), c.f. Fig. 1. To this
networks entail functionalities such as network slicing to enable end, a SDN controller is enhanced for interfacing with TSN-
the coexistence of mixed-criticality applications on unified net- switches. Thereby, the network topology can be monitored and
works. An approach capable of supporting slicing in the wireline managed centrally. This is evaluated empirically for distributed
domain is Time-Sensitive Networking (TSN). It addresses the
mentioned challenges by enhancing the Ethernet standard with approaches to automatic control of dynamical systems in-
functionalities required for deterministically bounded low laten- cluding communication-demanding Model Predictive Control
cies and preemption of high priority data flows. To facilitate its (MPC). These are deployed on embedded platforms acting as
integration with existing 5G and emerging architectures including agents. A bottleneck in terms of network capacity is introduced
6G and open Radio Access Networks (O-RANs), Software- by forcing traffic flows onto a single link and adding best-effort
Defined Networking (SDN) has emerged for orchestrating the
novel feature set. In this paper, we thus combine TSN and SDN services. This highlights the potential of SDN-driven TSN in
to design an integrated solution. We present experimental results challenging mixed-criticality applications such as those found
on handling communication-demanding control algorithms and within Smart Grids. This work is structured as follows: After
cross traffic simultaneously. Our findings underpin the potential an overview of related works in Sec. II, Sec. III details our
of SDN-driven TSN for mixed-criticality control applications. approach to SDN-driven TSN. Next, Sec. IV discusses the
evaluation scenario, setup and results. Finally, Sec. V draws
I. I NTRODUCTION
conclusions and offers an outlook on future work.
Modern technology increasingly relies on highly auto-
mated systems such as Smart Grids, Intelligent Transportation SDN Controller
TSN
mission-critical control algorithms which in turn depend on Switch PLC
Agents
are neither available nor viable, these requirements have to TSN Switch Substation PLC
traffic which, e.g., can be realized via 5G [2] network slicing Industry 4.0
[3, 4]. Here, as well as for future open Radio Access Network Control
Agent
5G/6G
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses,
including reprinting/republishing this material for advertising or promotional purposes, collecting new collected works
for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
II. R ELATED W ORKS III. S OFTWARE -D EFINED N ETWORK DRIVEN
T IME -S ENSITIVE N ETWORKING
While the orchestration of TSN via SDN has been proposed
Next, we recall the considered distributed algorithms for
in related work, several opportunities for further research
automatic control which are drawn upon as time-critical com-
remain. Existing studies can be grouped into three main
munication tasks. Then we detail TSN and our approach.
categories: Those works primarily concerned with specifying
novel architectures for SDN/TSN integration [9, 10], those A. Distributed Control
centered around analytic or simulative means [11–16] and We design controllers for a set S = {1, . . . , S} of dynamical
approaches build on empirical testing environments [17, 18]. systems. The dynamics of subsystem/agent i ∈ S are
Regarding works belonging to the first group, the Yet Another X
ẋi (t) = fij (xj (t), uj (t)), xi (0) = xi,0 , (1)
Next Generation (YANG) data modeling language as described
i∈S
in an amendment to IEEE 802.1Q for interfacing with TSN
devices via network management protocols such as NETCONF yi (t) = hi (xi (t)), (2)
or RESTCONF is used in [9, 10]. While providing important where xi ∈ Rni is the system state, ui ∈ Rmi is the system
contributions to the overall discussion, such works are limited input, yi (t) ∈ Rpi is the system output, and the functions fij :
in terms of results supporting the proposed approaches. Rnj × Rmj → Rni describe dynamic coupling of agents. We
Other studies are built around analytic models, e.g., presen- apply two different control methods to (1). Firstly, we design
ted by Li et al. [11] for stream reservation and bounded networked output-feedback controllers for linear, decoupled,
E2E latencies in avionics. Further papers rely exclusively single-input-single-output systems. The control law is
on simulation, often lacking real-world traffic models. For X
ui (t) = − lij yj (t),
one, the authors of [12] focus on end-to-end latencies for in-
j∈S
vehicular use, while [13] also studies resource usage efficiency
as a function of guard band size. In [14] an open source where lij are the elements of a coupling Laplacian L ∈ RS×S ,
SDN controller, interfacing with TSN via YANG, is coupled cf. [19]. The matrix L is designed to achieve output synchron-
to a simulation implemented in the OMNeT++ framework for ization, i.e., limt→∞ ∥yi (t) − yj (t)∥ = 0 ∀i ∈ S, ∀j ∈ S. This
solving a linear optimization problem. Their focus is placed control method requires communication coupled subsystems to
on assessing various scheduling schemes and their impact exchange messages once per control interval.
on bandwidth utilization. Traffic classification for accurate As an alternative, we employ Distributed Model Predictive
identification and subsequent prioritization of critical data Control (DMPC) (1) to stabilize the systems at xi = 0 ∀i ∈ S.
flows is discussed in [15]. There, OMNeT++ is harnessed To this end, we discretize (1) to obtain discrete-time dy-
to determine end-to-end latencies achievable by the selected namics fij d
(·). Moreover, we consider compact and convex
approach. Building on this work, the same simulation envir- constraints Ui . In DMPC, the agents at sampling instant
onment is also applied in a proximate study by the same tk , k ∈ N measure their current system state x(tk ) and
researchers in [16]. In an effort to ensure a stable performance cooperatively solve a discrete-time Optimal Control Problem
level in highly dynamic network environments, an SDN-based (OCP) with horizon N of form
approach to path reconfiguration is presented. N −1
1 X X k⊤ N⊤
Finally, research based on measurements performed within min xi Qi xki + uk⊤ k
i Ri ui + xi Pi x N
i (3a)
xi ,ui 2
testing setups or even real-world environments, such as dis- i∈S k=0
by best-effort frame
Northbound Interface (i.e. Representational State Transfer, REST)
t
Network Guard Time-Aware
Guard bands to avoid frames from Monitoring Interval Setup Shaper
Can not be sent intruding on time-critical time slots
Misc. Features Pre-emption Classification
& IEEE 802.1Qbu IEEE 802.1Qbv
Offset [ns]
Offset [µs]
A. Evaluation Scenario & Experimental Setup 0 20
0
The evaluation setup created for the purposes of this work
-200
is shown in Fig. 4. Two NXP LS1021ATSN serve as data -20
plane, i.e., TSN switches, offering a per link data rate of -400 min=-383ns -40 min=-30.8µs
100 Mbit s−1 . They connect to each other, as well as to -60
0 500 1000 1500 2000 0 300 600 900 1200 1500
embedded computers (i.e., Raspberry Pi 3) acting as dis- Time [s] Time [s]
tributed control agents. However, unlike its peers Agent 4
does not run control algorithms, but introduces generic best- Figure 5: Synchronization of the evaluation setup. Software
effort cross traffic into the network, fully saturating the middle based (control agents, left) vs. hardware based PTP (TSN
link. Thereby, a resource conflict is generated on the central switches, right). Note the different orders of magnitude and
link between both TSN switches. Network orchestration is significantly more stable clocks on the right.
performed by an SDN controller. It is a self-developed solution As control applications, one scenario with the output-
used in previous works [25] and based on the open source feedback controller and three scenarios with DMPC are ana-
Floodlight controller [23]. Among its tasks is the configuration lyzed for networks with three agents (ni = 2, mi = pi = 1).
of the time-aware shaper to ensure hard service guarantees for For the output-feedback controller, we consider a network of
control applications in a mixed-criticality environment. The fully coupled undamped oscillators as individual subsystems
various traffic streams’ dissimilar priority levels are encoded (Scenario 1), design L, and choose a sampling time of 15 ms.
into the field offered by Virtual Local Area Networks (VLANs) In case of DMPC, we design controllers with N = 5 and a
to specify service classes and acted upon by the switches. sampling time of 200 ms for a linear chain of masses system
TSN depends on highly synchronized network entities to en- and apply either ADMM (Scenario 2) or ASM (Scenario 3).
able the time slot based concept required for ensuring bounded We use five ADMM iterations per MPC step to solve OCP (3)
latencies. Therefore, we employ the Precision Time Protocol in Scenario 1 and solve the OCP to optimality with ASM in
(PTP) [26] to establish a firm time reference throughout the Scenario 2. Additionally, we design a controller with N = 4
setup. Hence, timing variance at the different devices is min- and a sampling time of 100 ms for a network of three fully
imized, allowing stable operation which in turn provides the coupled nonlinear Van der Pol oscillators with parameters
high confidence levels required for the subsequent evaluation. from [27] and apply dSQP (Scenario 4). In dSQP, we run
Observed measurement results are given in Fig. 5. As clearly 25 inner iterations per MPC step to solve OCP (3). We use
shown, the TSN switches’ hardware assisted PTP implementa- qpOASES [28] to solve the arising optimization problems
tion yields significantly more homogeneous and precise offsets in ADMM, CasADi [29] to compute sensitivities, and the
in contrast to the software solution employed at the agents. Lightweight Communications and Marshalling (LCM) [30]
While hardware PTP shown on the left mostly exhibits delays library to exchange iteration variables between agents.
in a range of approximately ±100 ns with peaks well into
the 300 ns range, software PTP fluctuates primarily between B. Evaluation Results
±20 µs, with outliers below 100 µs. Taking this into account, Fig. 6 shows the measured interframe latencies for the
results presented in the following are rounded to 100 µs. DMPC scenarios. Each of them is analyzed under three dif-
ferent traffic conditions. Black dots in the subgraphs mark the
Agent 1 Time critical control traffic Best-effort traffic Agent 2
respective mean interframe latencies. The left column displays
scenarios with inactive TSN and no cross traffic. Hence, these
distributions represent the ideal traffic patterns generated by
TSN Switch TSN Switch the employed control algorithms. Any deviation thus implies
unwanted interference with application behavior, i.e., as caused
by a lack of control traffic prioritization. Therefore, the ideal
Resource Conflict
result achievable by the presented SDN-driven approach to
TSN is the avoidance of such variations.
Agent 4 Agent 3 The control applications send messages with an interframe
delay of around 1 ms to solve the OCP. Also, messages with
SDN Controller
larger interframe delays occur when waiting for the following
Figure 4: Setup for evaluating SDN-driven TSN comprised of control interval. This applies in case of ADMM and ASM,
two switches orchestrated by an SDN controller. Four agents whose communication behavior features interframe delays just
execute distributed control algorithms and generate best-effort below 200 ms respectively slightly above 190 ms, as depicted
traffic, thereby creating a resource conflict on the central link. in the corresponding rows of the leftmost column in Fig. 6.
No Cross Traffic Cross Traffic Cross Traffic 4 4
No TSN No TSN TSN y1 (t) y1 (t)
y2 (t) y2 (t)
(Chain of Masses) (Chain of Masses)
2 y3 (t) 2 y3 (t)
ADMM
Output y
Output y
0 0
−2 −2
−4 −4
ASM
0 20 40 60 0 20 40 60
Time [s] Time [s]