https://doi.org/10.2352/ISSN.2470-1173.2017.6.

MOBMU-311
© 2017, Society for Imaging Science and Technology

A Study of IoT MQTT Control Packet Behavior and its Effect on

Communication Delays
Brian Bendele, David Akopian; University of Texas at San Antonio; San Antonio, Tx/USA

Abstract Antonio (UTSA) have agreed to test multiple smart grid

An important aspect of enabling Smart Grid and other applications and interoperability demonstrations to better
similar technologies is the development of communication understand and optimize implementations of power systems field
networks that efficiently gather information from remote devices that communicate on a common schematic definition and
locations and reliably deliver the data to a world of connected a scalable publish/subscribe architecture.
devices. In this context, the concept of Internet of Things (IoT) Numerous publish-subscribe communication protocols have
has emerged and led to the development of several lightweight been developed to enable IoT communications with multiple
communications protocols optimized for various environments. detailed surveys performed [5-6]. Many studies have compared
This study will focus on the Message Queue Telemetry Transport these protocols in head to head trials with mixed results [7-11].
(MQTT) protocol as used in a remote sensor network setting with Available protocols tend to excel in divergent, specific situations,
the goal of characterizing delay patterns to improve reliability of with no single protocol considered ideal for every situation [5].
large scale sensor networks in a publish/subscribe Message Queue Telemetry Transport (MQTT) protocol offers a
communication environment. Revealing the connection between publish-subscribe method of communication supported by the
data size, data collection intervals, network traffic, and delay, an TCP/IP protocol [12]. “Due to its light-weight, simple design,
approach for modeling an MQTT network design based on MQTT has become a popular protocol for IoT communications”
experimentation and inspection of behavior at the packet level is [13].
presented, and statistical distributions are explored to develop a Utilizing multiple platform trials, this paper will focus on
method for real-time analysis. detailed packet inspection to characterize delay patterns found
within MQTT enabled, IoT communications. Delay in MQTT
Introduction communications has been studied using a closed private network
Utilities around the nation are increasing their focus on [14]; however, publishing frequency, packet size, and the role
Smart Grid technologies and methodologies to better understand TCP plays in the characterization of delay was not considered. A
and control the flow of energy across the grid. It is needed to correlation analysis of delay and packet loss to Quality of Service
optimize its distribution and predict real time energy needs levels offered by MQTT has been measured with mixed results
through an integrated information network [1]. The goal of using [9]. In a comparison of MQTT to CoAP, it was shown that
IoT in Energy production and distribution chains is to enable MQTT’s performance is dependent on differing network
utilities to monitor operational and environmental variations that conditions when experiencing controlled packet loss [11]. Still,
effect grid viability allowing improved stability, efficiency, and more detailed packet level analysis has not been performed on the
durability [1]. variations of communication delay patterns which may indicate
A few obstacles to the realization of a smart grid network on issues with integrity, and security, as well as provide a
include interoperability and communication across a multitude of generalized observation of the overall health of the connection.
devices and software applications [2]. For this reason, an Open This paper offers multiple novel contributions including: (i)
Field Message Bus (OpenFMB) standardization process is a deep packet inspection of MQTT to better understand the
launched with the goal of permitting devices and systems to easily behavioral relationship to TCP; (ii) integration of a real-time IoT
communicate with one another in a shared contemporary server using Amazon Web Services to deliver brokered
framework that will utilize publisher/subscriber type of IoT communications; (iii) multiple publisher-subscriber nodes
communication and IP networking protocols [3]. OpenFMB operating from geographically separated locations to deliver real
spawned from a recent effort by Duke Energy and several world network traffic conditions; and (iv) a methodology for
working partners to combine IoT technologies with advanced establishing a baseline delay pattern in various environments that
interoperability for the power grid. The result was the could prove useful for monitoring connection health.
development of a framework for a distributed intelligence The subsequent sections of this paper are organized as such:
platform to enable Smart Grid technologies [2]. The Smart Grid A more detailed examination of the MQTT protocol is presented
Interoperability Panel (SGIP) has taken on the responsibility of and directly followed by a description of the experimental setup
further evolving and standardizing the specifications of used to test and evaluate the protocol, including the
OpenFMB to offer power systems field devices the ability to communication architecture of the system. The final sections of
exercise a non-proprietary and standards based architecture this paper present the experimental procedures with results,
consisting of Internet Protocol (IP) networking and IoT conclusions, and future work.
communication protocols [3].
Many utilities and test beds are participating in OpenFMB Message Queue Telemetry Transport
simulations, tests, and interoperability demonstrations including (MQTT) Overview
San Antonio based CPS Energy as noted in their 2014 public The following section briefly examines the basic connection
request for partners to test the “grid of the future” [4]. In and transmission methods of MQTT. All information regarding
coordination with CPS Energy, the Texas Sustainable Energy
Research Institute (TSERI) and the University of Texas San

IS&T International Symposium on Electronic Imaging 2017

120 Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017
 specifics of the protocol were taken directly from the MQTT
Standard, Version 3.1.1.
Why MQTT?
MQTT is described as a lightweight, open, simple
publish/subscribe messaging transport protocol [13]. The
protocol is ideal for use in situations where a code footprint is
small and network bandwidth is scarce [12]. MQTT was chosen
as the test protocol for this study due to its compatibility with the
process of collecting data from large sensor networks, and
organizing the data at one central location. This is an ideal
scenario for the collection of sensor data at solar, wind, and other
remote energy smart grid applications.
The protocol was originally designed to operate over TCP/IP
providing “ordered and lossless bidirectional connections.” [12].
The publish/subscribe method is enabled by a central server, often
called a broker. The broker sorts incoming messages by topic and Figure 1 CONNECT Packet, Variable Header Example
forwards the traffic to subscribers of the topic. MQTT provides resulting in messages not being sent until its predecessor has been
three options for quality of service (QOS), as well as “small acknowledged [12].
transport overhead” [12]. The MQTT protocol follows a basic
flow of communication that only adds minor complexity with Subscribe, Unsubscribe, and Other Processes
changes to QOS. The subscribe side of the protocol operates in a slightly
different manner than that of the publish side. A SUBSCRIBE
Connection Method request contains a fixed control type header with a variable header
To begin an MQTT session one must first establish some that contains desired topic name, wildcards, and QOS level. A
connection parameters with a broker. Every control packet has a SUBSCRIBE request is always followed by a SUBACK that
fixed header that gives instructions to the server to describe what specifies whether a subscription was successful as well as the
the client wants to do, and a variable header (Fig. 1) to stablish granted QOS level [12].
the parameters that characterize the action type allowed by the Other control options include UNSUBSCRIBE which allows
protocol. clients to unsubscribe from topics. This type of request requires
A basic MQTT session connection scenario would follow a the sever to send an unsubscribe acknowledgement
two-packet request/response format. Once a connection is (UNSUBACK), complete any QOS 1 or QOS 2 messages
established a client can then either send a PUBLISH request, a transactions, and then delete the client subscription.
SUBSCRIBE request, or any other control packet. PINGREQ/PINGRESP are packets used to test the liveliness of a
connection, and the DISCONNECT packet is an indication from
Publish Messages and QoS Levels
the client to the server that the client is requesting a clean
A PUBLISH request contains a payload with a topic/label
disconnection [12]. The newest version of the protocol, MQTT
attached to it. This topic is what the broker will use to transfer
v3.1.1 allows for message transport across TLS and Websocket
data to the appropriate destination on the subscription side. If a
protocols as well as the TCP/IP protocol.
QOS level of zero is selected, then the publisher is in “fire and
forget” mode as there will be no acknowledgement packet from
the MQTT server [12]. If the QoS level is greater than 0, the Experiment setup
acknowledgement scenario changes. The Texas Sustainable Energy Research Institute has multiple
QOS level one requires that after the publish message is locations utilizing National Instruments products [15] designed
received and processed, the server will return a PUBACK for wired and wireless sensor networks. This experiment will
message [12]. This is nothing more than an acknowledgement for leverage these pre-existing setups to test data collected from these
the publisher to log that the message was processed [12]. If this locations. The distribution of collected data is handled by a
acknowledgement is not verified, the packet will be resent with modified version of a publishing virtual instrument (VI) currently
the DUP flag set to 1 to indicate that the packet may be a operating in a remote location. An established LabVIEW MQTT
redelivery [12].
For a QOS of level 2, a PUBREC packet is sent back to the
publisher as the second packet in the four-way handshake. with
the PUBREL being the third packet sent back to the server by the
publisher to let the server know the second packet was received
[12]. The PUBCOMP is the fourth and final packet in the QOS 2
protocol exchange. The packet is sent by the server to the
publisher to let the publisher know that the server is ready to
receive the next packet of published data. Figure 2 generalizes
the communication process for QOS levels one and two.
Asynchronous Publication QoS-1
A sliding window can be defined by the designer to permit a
specified number of published packets through to the Server
before the Client receives an acknowledgement. This type of
asynchronous publication behavior results in the possibility of
duplicate packets received after one of its successor messages.
This can be avoided by narrowing the window to size one packet,
Figure 2 QoS Level 1 & 2 Flow Diagram

IS&T International Symposium on Electronic Imaging 2017

Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017 121
 API [16] was used to implement the publisher side of the protocol,
while a Linux based Mosquitto™ Broker [17] was deployed on
an Amazon Web Server [18] to enable live network traffic
conditions. Multiple subscribers were used to collect the data
from the Mosquitto™ Broker. These subscribers reside on
Windows 7 and Windows 10 based systems running a LabVIEW
subscriber VI, while a third publisher/subscriber exists on a Linux
based Raspberry Pi [19] utilizing the Eclipse PAHO project’s,
open source Python version of the MQTT protocol [20]. The
network traffic was monitored at the server/broker, as well as the
publisher, by means of a popular packet capture software package
called Wireshark and Tshark [21, 22]. The collected data was (a)
then imported into R for statistical analysis [23].
LabVIEW Development Environment
LabVIEW is a virtual instrument (VI) engineering workbench
with a visual programming language platform [24]. Publisher and
Subscriber VIs were developed in LabVIEW for use on the
Windows based systems. Implementation fragments are shown in
Fig. 3. The Subscriber developed for this experiment is a simple
state machine VI that contains the following states: CONNECT,
SUBSCRIBE, UNSUBSCRIBE, DISCONNECT. The goal of
this VI was simply to establish a connection with the broker to
simulate typical use in a live environment.
The Publisher developed for this experiment is a modification
of a Publisher currently deployed in the field at a TSERI/CPS
solar energy site. The modifications include the ability to adjust (b)
the byte size of the data to be published, a modification to change
the data publishing interval, the ability to add multiple topics, and
the capability to reset the TCP connection; all while the publisher
is in operation. Comparable to the Subscriber VI, this code is
designed as a simple state machine with the following states:
INITIALIZE, CONNECT, PUBLISH, DISCONNECT, EXIT.
The INITIALIZE stage incorporates user selected session
attributes in preparation for the connection. In the CONNECT
state, a TCP connection is established, followed by the MQTT
CONNECT request. If the CONNECT state is successful, the
code moves into the PUBLISH state. The PUBLISH state
publishes the data to the server under the selected topic. In this (c)
state the user can control the message size, and the interval in
Figure 3 LabVIEW Code Sample of Subscriber State Machine (a),
which the message is published. This allows the evaluation of Publisher User Interface (b), Publisher Code Fragment (c)
delay based on the publishing interval during packet inspection.
From this state the user may also reset the TCP connection. This
is useful in establishing a new initial round trip time which is used
in the delay calculation. This code continually attempts to The default settings were used in this study with the exception of
establish a connection in the event of a disconnect due to network adjustments made to the sliding window size in QoS-1
issues. In the event of a user initiated exit, the code will end the communication.
MQTT session with the broker and signal to terminate the TCP
connection during the DISCONNECT state. The EXIT state exits Ubuntu Server 16.04
the program. Ubuntu Server is a Debian-based Linux operating system
developed to run network servers [25]. This is an open source
LabVIEW MQTT API operating system, and it is offered as an option with Amazon Web
LabVIEW Client API in native LabVIEW is a community Services compute package.
project that implements MQTT v 3.1.1 standard protocol using
the LabVIEW 2013 or later development environment [16]. A Packet Capture Software
thorough examination of this API was performed including Wireshark is an open source network protocol analyzer [21].
header and packet formation and disassembly on the publishing Wireshark provides live capture and offline analysis of the
and subscribing sides. Compatibility tests were performed to TCP/IP protocols as well as support for MQTT traffic. Wireshark
ensure seamless performance with the server/broker. It should be was used on all Windows platforms for capture and analysis.
noted that this API does not provide the method for establishing a TShark offers a network protocol analyzer that allows the
sliding window for QoS > 0. This is instead left up to the same capture ability of Wireshark, but for console based systems
designer/engineer as suggested by the MQTT standard. [22]. Tshark was used on the Ubuntu server to capture network
traffic at the Mosquitto™ Broker. The created pcap files were
Eclipse Mosquitto™ Broker then transferred to a Windows based system for analysis in
Eclipse Mosquitto™ is an open source (EPL/EDL licensed) Wireshark.
message broker that implements the MQTT protocol versions 3.1
and 3.1.1 [17]. This API is installed on the Ubuntu 16.04 Server
to act as the broker between publish and subscribe client requests.

IS&T International Symposium on Electronic Imaging 2017

122 Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017
 System Architecture
The system architecture consists of a collection of hardware
and software that perform two main functions, gathering data and
processing data. A sensor network consisting of the above-
mentioned hardware was placed at a solar array site to collect
relevant data. A MQTT network was also built to process this
data in a manner consistent with live operations. Fig. 5 and Fig. 6
generalize the arrangements of the MQTT network and sensor
array respectively.

MQTT Network
The MQTT network was designed with flexibility in mind.
The layout enabled multiple publisher and subscriber nodes at
Figure 4 TSERI Labs Publish/Subscribe Nodes
geographically separated locations to connect simultaneously to
the AWS EC2 Mosquito Broker. The system operated in a live
Python MQTT network traffic environment to achieve an accurate representation
Python is open source software utilized by the Eclipse PAHO of real world conditions. Wireshark collected TCP and MQTT
project to implement MQTT protocols [20]. In this experiment, traffic at the client nodes, while TSHARK collected the same type
Python code was used to publish and subscribe to topics from a of traffic at the broker.
Raspberry Pi.
Elastic Compute Cloud (EC2)
Amazon Web Services (AWS) offers a cloud based compute
platform called Elastic Compute Cloud (EC2) [18] which is used
as the server for this project. This compute instance was launched
with Ubuntu Sever 16.04 and hosts the Eclipse Mosquitto™
broker used to implement the MQTT v3.1.1 protocol over a TCP
connection.
Windows Platform Systems
Four Windows based systems were used to perform
publishing and subscribing duties. A Windows 7 platform
computer referred to as TSERI Labs is located on campus at
UTSA, while two Windows 10 platform desktops, referred to as
the pink box and the black box, are geographically separated from
the TSERI Lab. A fourth Windows 10 laptop is used to access
the Mosquitto™ Broker for coordinating TShark network traffic
captures. All the Windows based systems contain the LabVIEW
Publisher and Subscriber VIs designed for this experiment, as
well as Wireshark for network traffic analysis.
Raspberry Pi Figure 5 MQTT Network Architecture
A Raspberry Pi is used as a publisher and subscriber [19],
implementing the Python based MQTT API created by the Eclipse
PAHO project [20]. Sensor Network
The sensor network gathered environmental information and
Vaisala Weather Transmitter reported the data to the myRIO. From that point, the publisher
Vaisala WXT520 Weather Transmitter offers six weather software residing on the myRIO organized and published the data
parameters including wind speed and direction, precipitation, to which clients could then subscribe. This data was captured and
atmospheric pressure, temperature and relative humidity [26]. stored for use with this study. The VIs created to run on the
The WXT520 outputs to serial data collected by the myRIO using Windows systems used in this study, accessed the data captured
RS-232 interface. from the sensor network and published to the MQTT Network.
Pyranometer
Kipp and Zonen CMP11 Pyranometer is a “radiometer
designed for measuring short-wave irradiance on a plane surface”
[27]. This pyranometer’s micro-Volt output is amplified to a 4-
20mA signal. Using a translation table, the analog signal is
converted to irradiance prior to publishing. The analog signal is
captured and processed by the myRIO.
myRIO
National Instruments myRIO is an embedded hardware
device with multiple, configurable, digital and analog IO [28].
This device gathers and formats all the sensor network data prior
to publishing. Sample data gathered in the field by this device is
used in the experiment.
Figure 6 Sensor Network Architecture

IS&T International Symposium on Electronic Imaging 2017

Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017 123
 Figure 7 WireShark Display Example

Experiment Dtransmission (sec) = Segment Length(bits)/Rate(bits/sec)

The following experimental procedures were informed by
Equation 2 Transmission Delay Equation
initial observation and packet inspection. The process of
identifying and separating TCP behavior from MQTT behavior recorded by the packet inspection software. This interval
helped establish the conditions under which the delay trials were represents an estimate for the latency of the connection between
performed. the TCPs. From the perspective of the node, Dpropagation can be
approximated by IRTT/2. When MQTT data is published at
Packet Inspection intervals at least two times the IRTT, the publishing node’s TCP
Over the course of this study, packet inspection remained an will forward the single publish request at a rate roughly equivalent
integral process to the development of delay analysis. From to the publishing interval minus the IRTT; however, if the publish
packet inspection, behavioral characteristics of TCP and MQTT interval is reduced below the IRTT threshold, the TCP will begin
can be separated and controlled, permitting a more accurate to queue published MQTT messages and push them as a single
approach to understanding MQTT delay characteristics. The segment causing an increase in the packet length transmitted.
control process developed is detailed throughout the subsequent Figure 8 demonstrates the TCP’s behavior in relation to the IRTT
sections. threshold. An inverse variation begins to takes shape as the
The process of packet inspection was performed in two steps. publish interval breaches the IRTT threshold. This relationship
i. Capture the packets with TShark and/or Wireshark between the publishing interval and the segment length can be
ii. Analyze the packets by means of Wireshark’s user estimated by Equation 3, where k is an integer >= 1.
interface. k-1(IRTT) ∝ k (min [Segment Length])
Capture filters were used to isolate the network traffic to the nodes
of concern. An example of one such filter is as follows: Equation 3 Relationship Below IRTT Threshold

The increase in packet length is limited by an established

tshark -c 10000 -f"(host ip address) and port not 22" Maximum Segment Size (MSS) regulated by the TCP. Once the
MSS is reached, the TCP begins segmenting the packets into
This TShark code will capture 10k packets (-c 10000) of all traffic smaller portions to keep up with the transmission requirements.
from a host defined by an ip address (-f"(host ip address) and port Increased Packet Length has a direct relationship with Dtransmission,
not 22"). Port 22 is ignored to eliminate the capture of SSH
which in turn directly effects Dqueuing (Equation 4). It is important
protocol packets. Inspection of the packets in Wireshark is
to understand and control this behavior to facilitate accurate
performed with display filters, and column filters were used to
MQTT delay calculations without TCP interference while using
configure exported .csv files used for statistical analysis. Figure
7 demonstrates results of a typical capture session.
Dqueuing = Dtransmission * (Length of queue)
Delay in General Terms Equation 4 Queuing Delay
End-to-end delay in a network connection is made up of
various events that can be expressed in typical situations by
Equation 1. For any given node, there exists delay due to
processing, queuing, transmission, and propagation [29].
Processing, queuing, and transmission delays occur at the node,
while propagation delays occur between node. In this study, some
of these variables are found to be controllable at times, while
others are merely defined by a mean or median value through
repeated experimentation.
Dnode = Dprocessing + Dqueuing + Dtransmission + Dpropagation
Equation 1 End to End Delay Equation

Transmission Delay, QoS-0 and QoS-1 with

Asynchronous Publication
Special attention must be paid to transmission delay when
aggregate data from multiple sensor networks is distributed
through a single node. Transmission delay can be defined by
Equation 2 [29]. Publishing intervals can influence transmission
delay inversely, which has a consequential effect on queuing
delays. When the TCP initiates its 3-way handshake between Figure 8 TCP Behavior at IRTT Threshold
communicating nodes, an initial round trip time (IRTT) is

IS&T International Symposium on Electronic Imaging 2017

124 Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017
 Figure 9 Publish Interval vs. TCP Segment Length

QoS-0 or Qos-1 with asynchronous publication. Figure 9 two sections; delay at the node, and round trip time. From the
demonstrates this behavior as the published data was held perspective of the sending node, RTT includes the delay at the
constant at 238 Bytes, while the publishing interval was decreased receiving node as well as the propagation delay to send data and
from 65ms to 5ms. The IRTT was recorded at 62.5ms and the receive the corresponding acknowledgement. Having described
Maximum Segment Size (MSS) negotiated by the TCP was 1460 the publishing behavior for QoS-0 and Qos-1 with a sliding
Bytes. It is shown that the segment length is broken up at 10ms window, the remaining experiments include an examination of
due to the segment exceeding the MSS. IRTT to set a baseline delay pattern, testing the effects of packet
length on delay, and an analysis of delay in the presence of heavy
Measuring Delay network traffic.
Visually, the measured delay involved in the process of
sending and receiving packets at the Client and Server is shown Initial Round Trip Time (IRTT) Test
in Fig. 10. The packet capture software is used will capture two As previously stated, the initial round trip time is recorded
packets per publish interval in normal network conditions. These when the Client establishes a connection with the Server by way
packets include the published data and either the MQTT or TCP of the TCP three-way handshake. Through deep packet
acknowledgement. In the special case of QoS-2 multiple packets inspection, the IRTT has potential to be a good indication of
will be sent in the exchange between the client and server as connection latency and may be used to establish a baseline for
discussed in section 2.3. The calculated delay will be split up into measuring delay shifts during an MQTT session. The IRTT test

Figure 10 Components of Delay in RTT Timestamp

IS&T International Symposium on Electronic Imaging 2017

Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017 125
 Figure 11 IRTT Distributions

performed for this study involved a TCP connection loop in which allows Jumbo frames [33] with a MSS value of 8960 established
a connection was established and then terminated at two second by the Server. Packet inspection revealed that indeed the limits
intervals. Each connection and termination cycle required a 6- per publishing interval rest at roughly 8 Kbytes and breaching this
packet exchange. 10,000 packets were captured, yielding ≈ 1600 limit leads to unpredictable behavior and data loss.
IRTT samples per test. The tests were performed at the TSERI Determining a round trip time for QoS-0 proves difficult due
labs and Pink Box locations to establish reference points for each to the lack of a MQTT acknowledgment from the Server. A TCP
node with the intention of illustrating that the propagation acknowledgment is sent from the Server however, and it provides
difference two geographically separated nodes might experience a close approximation to the round-trip time when the publishing
has a minimal effect on the distribution of the IRTT. Figure 11 interval is greater than two times the IRTT. At publishing
establishes the IRTT distributions for both Clients. Outliers intervals near or below the IRTT threshold, these
totaling less than 10% of the data sets were removed from both acknowledgments become less reliable. The Server, to maximize
distributions using the five number summary method [30]. efficiency, begins to stack acknowledgements which increases the
Though the propagation delay of IRTT is different in time at the difficulty of accurately measuring delays. QoS-1 with
nodes, the distribution over the median yields a similar curve from asynchronous publication can yield similar results below the
both locations; however, the spread of the data collected at the IRTT threshold, though reliable data can be gathered due to the
Pink Box location was slightly larger than that of the TSERI Labs MQTT acknowledgement requirement.
location. A deviation calculation was performed on the data to For this test, quality of service levels examined were
justify the use of the TSERI Labs node over that of the Pink Box published at an interval of 65ms. This interval was chosen for
node for the remaining tests as a more stable connection could reasons previously discussed in sections 5.2.1 and 5.3.1. QoS-0
reduce error in delay interpretation and calculation. Median and QoS-1 with asynchronous publication display the stacking
absolute deviation was used as it is considered a more robust behavior detailed in section 5.2.1 due to contributory increases in
method for handling outliers in a univariate data set [31]. Table both Dtransmission and Dqueuing. In this state, instability increases
1 shows the median absolute deviation along with the median for producing variations in the delay of the system. This pattern
each location. continues until the TSERI Labs MSS is exceeded at which point
the RTT stabilizes due to the delay in the system reaching a level
TABLE I. Robust measure of variability in IRTT of process equilibrium as seen in Figure 12.
QoS-1 with synchronous publication was held to the same
publishing interval of 65ms, though publishing interval in this
Median and Deviation Table of IRTT situation only sets the minimum as the next MQTT packet will
not be published until the previous packet has been
Median Median Absolute acknowledged. This eliminates the variable delay patterns caused
Client IRTT (s) Deviation by increases to Dqueuing since the queue never exceeds one MQTT
Pink Box 0.08911979 0.007440069 packet.
In this circumstance, delay for QoS-1 with synchronous
TSERI Labs 0.06247810 0.000901661
publication remains remarkably stable while the MQTT packet is
kept below the MSS as seen by the RTT distribution in Fig. 13
(top). As the packet length exceeds the MSS, the distribution of
Published Packet Length Trial delay shifts to a less predictable pattern shown in Fig. 13 (bottom).
The packet length test involves changing the published packet This is due in large part to the processing of the packets at the
size from 40 Bytes to roughly 9 KBytes. While the MQTT Server. While the MQTT packet length is held below the TSERI
standard allows control packets of up to 256MB in length, the Lab’s MSS, the TCP and MQTT acknowledgements are
Ethernet standard breaks this down even further to a maximum combined into a single ACK; however, packet inspection reveals
transmission unit (MTU) at 1500 Octets [32]. The AWS Server a separation of the TCP and MQTT acknowledgements when the

IS&T International Symposium on Electronic Imaging 2017

126 Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017
 Figure 12 RTT(s) vs. MQTT Packet Length (bytes)

MQTT packet length exceeds the MSS. This is caused by the measured at the Server. Figure 14 (top) shows the distribution of
increased processing time required by the MQTT broker to the delay before the increased traffic, while Fig. 14 (bottom)
examine and acknowledge the receipt of the larger MQTT displays the shifted delay distribution recorded as the network
packets. In this scenario, the TCP ACK is followed by the MQTT traffic was increased.
PUBACK reducing efficiency at the Server and increasing traffic The results of the network traffic test are promising in that
in the channel. they display a perceived shift in the established delay pattern.
The packet length test revealed a good candidate for Using a combination of MSS load balancing, IRTT, and expected
establishing a predictable delay pattern that can be used to detect deviations, one could conceivably create a reactive monitoring
shifts in delay during live sessions. QoS-1 with synchronous system used to report unexpected behavior based on an expected
publication (a sliding window set to size one) and a MQTT packet distribution.
length set below the smallest established MSS results in a stable
and controllable system that may allow for reactive monitoring of Conclusion and future work
the delay. This paper presents an approach to establish an expected
Network Traffic Trial delay pattern for MQTT based communications which could be
used to enable live reactive monitoring of an MQTT connection.
Network traffic tests were performed using multiple nodes to
Through deep packet inspection the relationships between data
generate traffic to the server. The test involved publishing a fixed
payload to the Server from a single node, and then gradually size, data collection intervals, network traffic, and its effect on
adding traffic to the server. Packets were captured at the Client delay were examined. It was shown that the initial round trip time
(IRTT) established during the TCP three-way hand shake can give
of interest, as well as the Server. This method allows a
a reliable estimation for the propagation delay of the connection,
comparison of delay measured at the Client, to traffic (in Bytes)
which in turn can be used to establish a baseline for expected delay

Figure 13 QoS-1 RTT for Synchronous Publication; Packet Length<MSS (top), Packet Length >MSS (bottom)

IS&T International Symposium on Electronic Imaging 2017

Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017 127
 Figure 14 Measured Delay Shift Due to Network Traffic

in a connection. Through experimentation we demonstrated that [3] OpenFMB. 2016 [cited 2016 02 November]; Available from:
shifts in the distribution of the expected delay are measurable using http://www.sgip.org/openfmb/
the combined application of MSS load balancing, IRTT, and
[4] Idell Hamilton, T. “CPS Energy seeking partners to test “grid of
expected deviations. These shifts may indicate session integrity,
the future” and bring more value to customers.” 2014 [cited 2016
security, as well as generalize the overall health of the connection. 02 November]; Available from:
Use of this proposed method to improve sensor network http://newsroom.cpsenergy.com/grid-of-the-future/.
designs in a publish/subscribe communication environment was a
goal of this project. We hope that this research provides some [5] Al-Fuqaha, A., Mohsen, G., Mohammadi, M., “Internet of Things:
insight into the development of Smart Grid IoT communications A Survey on Enabling Technologies, Protocols, and Applications.”
in situations where MQTT and sensor networks play an important IEEE Communications Surveys & Tutorials, 2015. 17(4): p. 2347-
role. 2376.
Further research into nodal behavior regarding the processing [6] Al-Fuqaha, A., Mohsen, G., Mohammadi, M., "Toward better
of MQTT packets and delay effects caused by transport protocol horizontal integration among IoT services." IEEE Communications
behavior at the node would be a future goal. This requires a more Magazine, 2015. 53(9): p. 72-79.
detailed investigation of individual MQTT packets at both the
Client and the Server nodes. We believe a thorough study on [7] Beckmann, K. and O. Dedi. "sDDS: A portable data distribution
recovering from packet loss, and the impact the recovery has on service implementation for WSN and IoT platforms." in Intelligent
delay and throughput would provide a useful comparison of Solutions in Embedded Systems (WISES), 2015 12th International
MQTT to other IoT protocols for use in large-scale senor Workshop on. 2015.
networks. We would also like to further investigate modeling [8] Fernandes, J.L., Lopes, I.C., Rodrigues, J., Ullah, S., "Performance
ideal, scalable, remote sensor network designs for the MQTT evaluation of RESTful web services and AMQP protocol." 2013
protocol to optimize performance and reduce data loss. Fifth International Conference on Ubiquitous and Future Networks
(ICUFN), 2013: p. 810-815.
Acknowledgment [9] Lee, S., Kim, H., Hong, D., Ju, H., "Correlation analysis of MQTT
This project and the preparation of this paper were funded in part loss and delay according to QoS level." The International
by monies provided by CPS Energy through an agreement with Conference on Information Networking 2013 (ICOIN), 2013: p.
the University of Texas at San Antonio. 714-717.
© CPS Energy and the University of Texas at San Antonio
[10] Luzuriaga, J.E., Perez, M., Boronat, P., Cano, J.C., Calafate, C.,
Support for this project was also provided in part by Texas Manzoni, P., "A comparative evaluation of AMQP and MQTT
Sustainable Energy Research Institute. protocols over unstable and mobile networks." 2015 12th Annual
IEEE Consumer Communications and Networking Conference
(CCNC), 2015: p. 931-936.
References
[1] Goel, S., Bush, S., Bakken, D, “IEEE Vision for Smart Grid [11] Thangavel, D., Ma, X., Valera, A., Tan, H.X., Tan, C.K.Y.,
Communications: 2030 and Beyond.” 2013. "Performance evaluation of MQTT and CoAP via a common
middleware." Intelligent Sensors, Sensor Networks and
[2] Laval, S. and B. Godwin, “Distributed Intelligence Platform (DIP)
Information Processing (ISSNIP), 2014 IEEE Ninth International
Reference Architecture, in Vision Overview”, S. Laval, Editor.
Conference on, 2014: p. 1-6.
2015, Duke Energy.

IS&T International Symposium on Electronic Imaging 2017

128 Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017
 [12] Banks, A. and R. Gupta. MQTT Version 3.1.1. 2014 29 October [23] R Studio. [cited 2016 02 November]; Available from:
[cited 2016 02 November]; Available from: http://docs.oasis- https://www.rstudio.com/.
open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html.
[24] National Instruments. 2016 31 October 2016 [cited 2016 2
[13] OASIS Message Queuing Telemetry Transport (MQTT) TC. November]; Available from:
November 02, 2016 [cited 2016 02 November]; Available from: https://en.wikipedia.org/wiki/National_Instruments.
https://www.oasis-
open.org/committees/tc_home.php?wg_abbrev=mqtt. [25] adconrad. Ubuntu 16.04.1 LTS ReleaseNotes. 2016 [cited 2016 02
November]; Available from:
[14] Ahmed, T., D. Akopian, and R. Vega, "Measurement and https://wiki.ubuntu.com/XenialXerus/ReleaseNotes.
Characterization of Communication Delays for Internet of Things."
2015, UTSA, Texas Sustainable Energy Research [26] Multi-Weather Sensor WXT530. [cited 2016 02 November];
Institute(TSERI). Available from:
http://www.vaisala.com/en/products/multiweathersensors/Pages/W
[15] National Instruments, A Global Leader in Test, Measurement, and XT530.aspx.
Control Solutions. [cited 2016 04 September]; Available from:
http://www.ni.com/en-us.html. [27] CMP Series Pyranometer Instrunction Manual. 2006, KIPP &
ZONEN.
[16] Peter-daq.io. Community: MQTT Client API in native LabVIEW -
National Instruments. 2016 Aug 9, 2016 [cited 2016 05 [28] myRIO - National Instruments. [cited 2016 05 September];
September]; Version 12:[Available from: Available from: http://www.ni.com/myrio/.
https://decibel.ni.com/content/docs/DOC-32539. [29] Network Delays and Losses. [cited 2016 08 November]; Available
[17] An Open Source MQTT v3.1 Broker. 2016 [cited 2016 02 from: http://www.d.umn.edu/~gshute/net/delays-losses.xhtml.
November]; Available from: http://mosquitto.org/. [30] Kerns, G.J., Introduction to Probability and Statistics Using R.
[18] Ami. Elastic Compute Cloud (EC2) Cloud Server & Hosting – [31] Median absolute deviation. [cited 2016 13 November]; Available
AWS. [cited 2016 05 September]; Available from: from: https://en.wikipedia.org/wiki/Median_absolute_deviation.
https://aws.amazon.com/ec2/?nc2=h_m1.
[32] Association, I.S., IEEE Standard for Ethernet, in IEEE Std
[19] Raspberrypi. [cited 2016 04 September]; Available from: 802.3™-2015 (Revision of IEEE Std 802.3-2012). 2015, The
https://www.raspberrypi.org/. Institute of Electrical and Electronics Engineers, Inc.
[20] Paho - Open Source messaging for M2M. [cited 2016 05 [33] Jumbo frame. [cited 2016 18 November]; Available from:
September]; Available from: https://en.wikipedia.org/wiki/Jumbo_frame.
http://www.eclipse.org/paho/downloads.php.
Author Biography
[21] WIRESHARK Go Deep. [cited 2016 05 September]; Available
Brian Bendele received his BFA from Texas State University (2000) and
from: https://www.wireshark.org/.
spent time in Los Angeles working in post-production for film and TV.
[22] tshark - Dump and analyze network traffic. [cited 2016 05 Since receiving his BS in Electrical Engineering from the University of
September]; Available from: https://www.wireshark.org/docs/man- Texas San Antonio (2014) and his MS in Electrical Engineering from the
pages/tshark.html. University of Texas San Antonio (2016) he has worked designing test
equipment for data collection, with a focus on sensor network design,
wireless, and RF communications.

IS&T International Symposium on Electronic Imaging 2017

Mobile Devices and Multimedia: Enabling Technologies, Algorithms, and Applications 2017 129