This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
1

xFogSim: A Distributed Fog Resource

Management Framework for Sustainable IoT
Services
Asad W. Malik, Senior Member, IEEE, Tariq Qayyum, Anis U. Rahman, Muazzam A. Khan,
Osman Khalid, Samee U. Khan, Senior Member, IEEE

Abstract—Streaming large amounts of data to cloud data centers cause computing is investigated for an energy-efficient solution,
network congestion resulting in high network and energy consump- however, the additional delay of gathering data from the IoT
tion. The concept of fog computing is introduced to reduce workload devices adds further communication delays. To cope with
from backbone networks and support delay-sensitive Internet of Things such latency issues and for efficient bulk data management
(IoT) applications. The concept places compute, storage, and network
close to the user, the concept of fog computing is intro-
services closer to the source of the requests. In general fog-based
simulators are used for better understanding and optimum fog resource
duced by Cisco [2]. The novel computing paradigm brings
allocation. Unfortunately, most of the simulators lack core features like computing resources near to the data source, and is used
network delay, latency, packet error rate, energy consumption, and for different innovative services including delay sensitive
distributed fog node management. In this paper, we propose a fog applications [3], [4], for instance to perform data analytics
simulation framework termed as xFogSim to support latency-sensitive at the extremes of the network, necessarily working as a
applications at the fog layer with multi-objective optimization to trade-off continuum between the end users and backend data center.
cost, availability, and performance among the fog federation. Moreover, To handle latency issues and to facilitate compute, stor-
during peak load, the framework provides locality-aware distributed bro- age and networking services, a device can work as a fog
ker node management that enables borrowing resources from nearby
node offering services in a fog computing paradigm. Note
fog locations to meet service and energy requirements. The results show
that the framework is lightweight, configurable, and scalable, capable
that the terms fog computing and edge computing are used
of handling a large number of user requests using dynamic resource interchangeably; however, there are differences between
provisioning across the fog federation. them. Traditional edge computing is referred to as the
cellular base station where the cloud computing capabilities
Index Terms—Energy, fog computing, IoT, mobility models. are provided on a limited scale. However, fog computing is
more decentralized compared to edge computing. Further-
1 I NTRODUCTION more, fog computing is designed to provide interoperability
among devices and computing units; whereas, edge servers
Recent developments in IoT-based applications is driven by are usually the proprietary servers [4]. Another important
the tremendous growth in devices connected to the internet. attribute of fog is its inherent location awareness, and
Notably, the expected number of IoT devices will reach hence, allowing applications being supported by nearby fog
75 billion by the end of 2025 [1]. Therefore, this growth locations. However, the use of fog devices to handle in-
opens up new research challenges in different fields includ- coming computing requests consumes a significant amount
ing smart farming, intelligent transportation systems, smart of energy. Moreover, with the exponential growth in IoT,
cities, smart industries, and smart health management sys- fog devices cannot be energy efficient without sharing the
tems. Most of the challenges are due to the limitation of IoT workload of other fog devices. Therefore, multiple fog loca-
devices in terms of energy, reliability, interoperability, and tions can form a resource federation, often used to handle a
security. Over the years, energy efficiency has got consider- bulk of computing requests. The overall fog communication
able attention in both academia and industry. Initially, cloud model is presented in Figure 1 where connected devices
communicate with designated fog devices that are further
• Asad W. Malik, Tariq Qayyum, Anis U. Rahman and Muazzam A. Khan
are with School of Electrical Engineering and Computer Science (SEECS),
connected to cloud data centers. In this way, fog comput-
National University of Sciences and Technology (NUST), Islamabad, ing provides different levels of computation and storage
Pakistan. services. Therefore, it is specifically designed to provide
support for those end user applications that require low
• Osman Khalid is with COMSATS University Islamabad, Pakistan.
latency responses [5]. By making the resources available
• Samee U. Khan is with Department of Electrical and Computer Engineer- near to the end users improves the overall users’ Quality
ing, Mississippi State University Starkville, MS 39762, USA. of Experience (QoE) and profit for resource providers [6].
Manuscript received May 1, 2020; revised July 11, 2020. With the emerging fog paradigm, there is wide adoption

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
2

Fig. 1: Fog communication model – devices communicate with fog locations offloading work from back-end data centers.
The data center is used to permanently archive data and run more complex algorithms.

of fog/edge simulators to benchmark the performance of a resource table comprising tuples with updated val-
real-world applications under a dynamic simulated envi- ues for performance, cost, and fog resource availability
ronment such as federated resources, energy, and scalability. in terms of energy and workload. This information is
Notably, existing simulators fail to provide a flexible design shared with other brokers for efficient and transparent
that should allow easier integration of new resources to resource sharing in a federated environment.
operate as a federation and management policies. Moreover, (b) Supports both static and mobile nodes. To maintain
the existing simulators provide support for heterogeneous seamless service provision, the framework dynamically
IoT devices to connect to real cloud infrastructure, but at the manages handover mechanism and tracks mobile de-
cost of reduced flexibility demanded by researchers to sim- vices and corresponding hop counts. This allows effi-
ulate complex scenarios to test their proposed algorithms cient result delivery to subscriber devices located any-
before actual deployment. Furthermore, the fog paradigm is where within the network.
specifically designed to support delay-sensitive applications (c) Provides evaluation in terms of resource usage, energy,
with mobile end users. Therefore, the underlying mobility latency, and scalability.
model should support efficient and smooth handover be- (d) Provides GUI environment to facilitate simulating com-
tween base stations while keeping fog resources transparent plex fog scenarios with varying network characteristics.
to the end-users. This is lacking in existing simulators with The rest of the paper is organized as follows. A review on
inadequate support for mobility handover. Many of these existing fog simulators is covered in Section 2. In Section 3
simulators focus on increasing the number of nodes while system model is presented. The proposed framework is pre-
ignoring crucial network characteristics like bit error rate, sented in Section 4. The framework evaluation is presented
packet loss, and bandwidth, thereby making it difficult to in Section 5. Section 6 presents the details of framework soft-
construct a realistic environment for simulation. Moreover, ware. Finally, Section 7 concludes the research contribution.
the fog federated environment for efficient utilization of
available resources is relatively less explored. The existing
simulation frameworks lack the support for fog federation 2 I OT S IMULATORS
with fog nodes acting as autonomous entities, i.e., working Fog computing is an emerging paradigm designed to bring
independently with no collaboration. This support is essen- cloud services closer to end users. With cloud servers con-
tial to optimize the use of fog resource utilization, provide nected to fog servers, the latter monitoring and updating
scalability and reduce energy effectively for the fog devices. data from the servers, in turn, providing services including
In this paper, we propose xFogSim, a simulation frame- compute, storage, and transmission to its users. The area
work for fog resource sharing to meet growing resource is promising with many researchers exploring its benefits
demand with ever-increasing numbers of IoT devices. No- in real-world applications. In this section, we mainly focus
tably, existing simulators provide standalone fog devices on existing fog frameworks designed to support complex
to facilitate end-user requests [7], [8]. However, xFogSim simulation scenarios.
introduces the concept of fog federation for resource sharing SimpleIoTSimulator [9] – a commercial simulator pri-
among fog locations. The proposed framework provides marily designed for IoT environments. The simulator creates
support for latency-sensitive applications at the fog layer test environments consisting of a number of sensors and
with multi-objective optimization to trade-off cost, availabil- gateways. The objective is to allow various gateway ven-
ity, and performance among the fog federation. The main dors to test their equipment in different IoT configurations.
contributions of the proposed framework are as follows: Moreover, the simulator supports a number of common
IoT protocols, including, MQTT, CoAP, MQTT-SN, MQTT-
(a) Provides a fog resource sharing environment using a Broker.
resource allocation algorithm to meet the user service- MobIoTSim [10] – an IoT-based simulator controlling
level agreement (SLA). Here, every broker maintains IoT devices using popular IoT protocols like MQTT and

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
3

TABLE 1: Comparison of different fog simulators

Simulators Platform Open Customized Fog Device Developed Integrates # distributed Energy
independent source mobility federation handover by (IoT/fog/cloud) fog locations
models (R/A/I)∗
SimpleIoTSimulator [9] × × × × I IoT/Cloud - NA
MobIoTSim [10] × × × × A/R IoT/Fog/Cloud - NA
Google cloud IoT [11] Cloud-based × × × × I IoT/Cloud - -
iFogSim [12] × × × A/R IoT/Fog Single
Cooja [13] × × × × A/R IoT Single ×
FogTorch [14] × × × A/R IoT/Fog - NA
EmuFog [15] × × × A/R IoT/Fog - ×
EdgeCloudSim [16] × × A/R Edge/Cloud Single ×
Mobile fog [17] - × × × × A/R IoT/Cloud - ×
FogBus [18] × × × × A/R IoT/Fog/Cloud Single
Virtual Fog [19] × × × × A/R IoT/Fog Limited
FogRoute [20] × × × × × A/R Fog/Cloud - ×
FogBed [21] × × × A/R Fog/Cloud - ×
D2D fogging [22] - - × × × A/R Fog Single
FogNetSim++ [7] × Limited A/R IoT Single User-level
xFogSim [Proposed] A/R IoT/Fog/Cloud Multiple Fog-level
∗
R:Researchers, A:Academics, I:Industry, Single: single fog; Multiple: Distributed fog nodes

HTTP with support for different data interchange formats. environments. The CPU utilization models in the proposed
The simulator uses a cloud-based gateway service for com- simulator limit the maximum number of tasks running on
munication between the IoT devices and the cloud. VMs. Even though the simulator evaluates different edge
iFogSim [12] – a fog-based simulator to handle ap- computing scenarios by providing models for network link-
plication scheduling and resource management across fog ing, mobility, and edge service. However, it does not exhibit
environments. The proposed simulator is an extension of hand-off mechanisms and support for fog federation.
CloudSim and is capable of evaluating the resources in FogBus [23] – a framework to support the integration
terms of energy consumption, network congestion, and of IoT-enabled systems with fog and cloud infrastructures.
latency. The authors present two case studies to test their The framework allows multiple applications to run simul-
simulation framework. The fog devices in the proposed taneously and provides platform independent interfaces for
simulator subscribe to the data coming from IoT sensors IoT applications. The system allows customization, develop-
and generate insights and commands to be forwarded to the ment, and testing of cross platform IoT applications, as well
actuators. However, the simulator does not take into account as allows service providers to manage various resources. To
the fog-federation scenarios. Moreover, the simulator has address data privacy in sensitive applications, the proposed
outdated APIs not compatible with current versions of Java. framework applies data authentication and privacy, and
FogTorch [14] – a fog-based simulator FogTorch to test block chain for data integrity. The proposed framework is
the deployment, execution, and testing of IoT applications limited in mobility handling and handovers. Moreover, the
over fog infrastructures. The framework can help prior eligi- system does not cater the federated fogs environment.
bility checking of an application before deployment in terms Fogbed [21] – an extension of MiniNet framework allow-
of resources and network related constraints. However, the ing the use of Docker containers as virtual nodes. The pro-
proposed framework is limited in a sense that it does not posed system allows real-world emulation of cloud and fog
have support for mobility and handling of multiple fogs. infrastructures by allowing dynamic addition, connection,
EmuFog [15] – a fog emulator developed on top of and removal of containers from the topology. Moreover,
MaxiNet (an extension of network emulator Mininet). The the resources for a container, such as CPU and memory
simulator allows users to design a network topology based can be changed at runtime. The framework is tested for a
on fog infrastructure. The BRITE topology generator is used healthcare monitoring system. The proposed system lacks
to generate the topology. Once the fog topology is generated, the mobility handing, and fog federation.
it is input to MaxiNet for emulations. The framework allows FogNetSim++ [7] – supports heterogeneous devices with
the developers to customize the topology configurations handover mechanism. The devices offload tasks to the
and workloads. The outcome is a group of network nodes nearby fog node. A single broker node manages the fog
and resources that are exported to MaxiNet emulator for nodes and tracks the source devices to deliver results using a
evaluations. The proposed system lacks support for mobility multi-hop model. The single broker node handles peak load;
models and multiple fogs. however, resource sharing among fog devices is not consid-
EdgeCloudSim [8] – an extension of CloudSim for edge ered. Moreover, it is designed using OMNeT++, therefore,

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
4

existing mobility models can be adopted in Fognetsim++. fulfill the SLA. Without loss of generality, we assume that all
Discussion – Many of the aforementioned fog simulators broker nodes are directly connected to one another sharing
are designed to work in a particular scenario, for instance, fog resource information.
efficient network utilization and/or resource management. A mobile device m ∈ N is categorized as a sensor,
Other third-party simulators allow modifications in terms receiver or hybrid. The sensor devices can only generate
of devices, user nodes, and backend servers. However, the data consumed by other receiving devices. The model also
closed simulation environment restricts researchers to incor- supports hybrid devices that can generate and consume
porate and benchmark different algorithms and strategies. data while acting as a relay. The proposed model is required
Furthermore, fog federation and different network-related to support a large number of latency-sensitive applications.
configurations are missing including latency, delay, and A user u ∈ U is categorized as a postpaid or prepaid user.
packet error rate. Therefore, there is a need for a rapidly de- Moreover, a user can use fog nodes by using spot instance
ployable simulator to benchmark algorithms. In this study, requests, dedicated fog nodes, or pay-as-you-go model. In
we propose a federated fog framework where fog broker the spot instance model, the user bids a price (p) with
nodes communicate with one another for task migration starting time for a job j ∈ J . On the availability of fog, the
and handover. Each fog broker being an autonomous entity user computes resources and overall usage trends, broker
manages its fog nodes, and it is also aware of the availability b ∈ B decides to execute the requests. In the dedicated
of other fog nodes and their pricing. Therefore, allowing the model, the user reserves a set of fog nodes by paying on a
broker to sub-lease computing resources from neighboring monthly basis. Lastly, in pay-as-you-go model, users request
fog locations to improve overall resource usage, in turn, fog resources at any time while paying only for the time
meeting SLA requirements. being used by its applications. The pricing model used in
A comparison with existing simulators based on the xFogSim is shown in Table 2.
crucial parameters is covered in Table 1. Here, different
TABLE 2: xFogSim pricing models based on Amazon [24]
simulator attributes like integrated devices, task models,
result handover, platform, mobility models, energy, handoff
Pricing model Description Price
mechanisms, and device mobility are considered. Moreover,
the integration with IoT, fog, and cloud is also listed in Pay-as-you-go Networking/storage/compute $0.0943 per hour
Dedicated Computation unit $2 per hour
the table for comparison. Some of these parameters are Spot instance Storage/compute User bidding
mostly focused on IoT simulators. Moreover, all existing Hybrid model Fluctuates with demand $0.06–1.5 per hour
simulators support a single fog location to simulate task
execution at fog nodes. With limited computing and storage The broker node b ∈ B maintains a multiqueue Q =
capability at the fog devices, they fail to handle a large Qlocal + Qleased to hold incoming job requests J , where
number of requests concurrently. In this scenario, existing Qlocal includes computing requests from the user set U and
simulation frameworks use backend cloud or edge locations Qleased includes compute request shared or bartered among
to address delay-sensitive computing requests. Thus, the broker set B . On broker nodes, we implement an M/M/c
main contribution of xFogSim is the use of fog federation model for fog nodes where an available fog node dequeues a
where resource requests are services across different fog job request j ∈ J from the top of the queue Q. Moreover, we
locations. That is, to enable the sharing, every fog location is assume that the compute jobs are independent and can be
managed through a broker node that acts as an agent to keep scheduled at any fog node. Nevertheless, the broker nodes
track of resource availability in its own and neighboring are capable of managing composite or dependent tasks.
regions, as well as, for timely request placement. The broker Furthermore, computationally expensive jobs fail to execute
communicates with neighboring broker nodes to find the on fog nodes, as they can be of same or different architecture
most suitable resource, keeping in view workload, availabil- providing different computing capacities.
ity, communication delay, and execution cost. Furthermore, Using the Poisson process, the average arrival rate λ for
xFogSim provides a comprehensive platform for researchers k ∈ N mobile devices is given as,
to incorporate new resource sharing algorithms, mobility X
λ= λk . (1)
models, and also customizes the network parameters.
k∈N
A fog location i ∈ L accepts the request as,
3 S YSTEM M ODEL 
1 φi > λ
ϕi = φ i (2)
We consider a connected fog communication model, e.g., λ φi < λ.
Figure 1, as an undirected graph G = (N, E), where node where φi is the maximum queuing capacity at the fog
set N includes user set U , fog set F and broker set B , i.e., location. Using λ and ϕi , we compute the execution rate
N = U ∪ F ∪ B while edge set E represents these links ϑi for fog location i ∈ L as,
between nodes. The N nodes are dispersed across L fog 
locations. Each location l ∈ L comprises f ∈ F fog nodes, λ φi > λ
ϑi = λ · ϕi = (3)
which is managed by a fog broker b ∈ B . More precisely, φi φi < λ.
there is a link eij between fog node i ∈ F and broker j ∈ B Based on queuing theory analysis and Erlang formula, the
if and only if they are within the same fog location l ∈ average waiting time ti for any job at fog location i ∈ L is,
L. The number of fog nodes per location ranges between λ
κ, κρ 1
2 to 12 nodes with the broker responsible for scheduling ti = i
+ . (4)
received requests and coordinating with other brokers to κρi − λ ρi

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
5

where ρi represents the job completion rate at the location. fog resources is computed as one minus the failure proba-
Since fog nodes provide computing service based on a bility Pf of the fog nodes.
request, the quality of service (QoS) is measured in terms of Y
delay. The cost of service delay (cd ) for a job j ∈ J is, A=1− Pf (k). (12)
k
cd = h + cq , (5)
where cq is the queuing cost, and h is the network delay for
the job. The tasks follow the Poisson process model with the
delay measured as,
ϑi
cq = , (6)
λ − ϑqi
where q is the number of computing resources used by user
u ∈ U.
We know that network delay depends on the distance
between nodes [25]. More precisely, in this case between i)
user and broker, ii) between brokers, and iii) broker and fog Fig. 2: Node availability calculation tree
nodes. We express the network cost as a linear function to
The user requests for a resource along with cost, re-
distance given as,
sponse time, and minimum availability. Here, availability
means the time a resource can be accessed without fail-
 
X
h = β1 · db,u + da,b  , (7) ures, which depends upon intermediate hops. As shown
a∈Bleased in Figure 2, nodes of the network are represented in the
form of a tree. With b1 to bn as the broker nodes, s1 to
here β1 is a constant, db,u is the distance between the user sn as the switches, and f1 to fn as the fog nodes. Every
u and its local broker b, and da,b is the distance of the local node having a failure probability such as failure probability
broker b to brokers leasing compute resources. The cost of of fi is represented as Pf (fi ). As an example, in case the
request cr at current fog location li can be stated as, Algorithm 1 selects a solution with f1 , f2 , and f4 fog
nodes to meet the user requirements then the probability
X X
cr = px + py , (8)
x=∀k,k∈li y=∀k,k∈l
/ i
of the selected solution is X = (Pf (f1 ) ∩ Pf (f2 )) ∪ Pf (s1 ),
Y = Pf (f4 ) ∩ Pf (s3 ), and Z = Pf (b1 ). Here X is the failure
where the two terms denote the prices of resources at locally probability of fog nodes connected to switch s1 , Y is the
available fog locations and resources at other fog locations, failure probability of fog nodes connected to switch s2 , and
respectively. Note that the cost can be categorized into two Z is the failure probability of the local broker. The total
types because the price of leased resources is different from failure probability of the selected solution is Q = X ∩ Y ∩ Z ,
that of the available resources. Similarly, in xFogSim, the and hence, the total availability is P = 1 − Q.
availability and performance are stipulated in the SLA.
The performance of a fog node is evaluated using re-
sponse time tr calculated as,
tr = tlocal
r + tleased
r , (9)

where tlocal
r represents the response time of locally avail-
able resources and tleased
r represents the response time of
resources leased from its broker. The response time tlocal
r is
calculated using,
1
tlocal
r = P 1 + β2 · d(b, u) + dh · cc , (10)
tx
r
x=∀k,k∈li

where β2 is a constant, txr is the response time of xth fog

resource, d(b, u) is the distance between user u and its local
broker b, dh is the total hop distance between the broker b
and fog resource x, and cc represents the communication Fig. 3: xFogSim framework- Visual interface showing mobile
cost. Whereas, the response time tleased
r is calculated using, users connected with fog brokers through access point, fog
n locations are managed by fog brokers.
1 X
tleased
r = P 1 + cd , (11)
tx
r i=1
x=∀k,k∈li

here cd is the cost of service delay. 4 P ROPOSED X F OG S IM F RAMEWORK

We know that availability is an important factor besides The concept of fog virtualization is used to provide ser-
performance [26]. In the framework, total availability A of vices to end users in a resource constraint environment,

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
6

providing layers of abstraction between hardware, soft- vary based on current workload. The broker node manages
ware, storage, and network. More precisely, fog computing fog resources either from its pool of fog nodes or leases
comprises specialized fog servers relying on virtualization some from neighboring brokers. It is worthwhile to mention
technologies to provide cloud services to users. Generally, that xFogSim takes communication costs into account when
fog is a framework supporting object, network, and service resources are arranged from other locations, i.e., the broker
virtualization [19]. The main focus of fog is to decouple dif- nodes are aware of the associated costs for using resources
ferent service modules so that resources can be provisioned of neighboring broker locations. The broker periodically
dynamically based on demand. Moreover, the context-aware broadcasts fog advertisements (F A) in its region. The users
feature is adapted to provide services to nearby devices [27]. respond to the advertisement when they require resources.
That is, fog servers are geographically dispersed across Though the advertisement includes the cost but only for
different locations providing services to end-users, often local resources, the user can accept the cost or make a bid.
transparent to the users. Using a simple resource allocation
scheme returns resources with high cost-effectiveness while
maintaining QoS as agreed in the SLA. Here, the resources 4.3 Resource Pricing
are selected based on factors like price cost, time, energy A bidding process similar to the spot instance mechanism
and availability [6], [28]. used by Amazon is used where a bid succeeds based on
xFogSim is designed as an extension of FogNetSim++, the availability of resources [30]. However, in xFogSim,
which is based on OMNeT++ and INET framework. The the resources assigned by the broker after a successful bid
execution model of xFogSim is illustrated in Figure 3, where can be preempted at any time depending upon the local
every broker node with multiple fog nodes is connected to demand. Moreover, on receiving a bid, the broker tries to
mobile users generating computation requests. negotiate with other brokers to find a win-win situation,
i.e., neighboring brokers can barter their resources when
4.1 Core Components requested. Consequently, the broker also maintains a leased
Precisely, the core components of xFogSim design are broker resource table and cost of usage per minute, which the
nodes, fog nodes and end device/users. The main function- broker can adjust later with incoming lease requests. It is
ality of each component is presented as under: noteworthy that all communications and negotiations are
1) Broker – is the core component responsible for manag- completely transparent from the end user devices.
ing user requests, resources allocation and collaboration
among other brokers (B) in a distributed manner. The
broker tracks usage of fog resources and movement of 4.4 Broker-Fog Pairing
mobile users for successful delivery of service. The data In xFogSim, the broker acts as an agent to handle fog
dissemination task becomes more challenging when an node assignment. On receipt of fog resource request from
unplanned data movement occurs, affecting network neighboring brokers, the broker makes its resources avail-
usage. As pointed out earlier, communications between able using a locking mechanism. This is followed by a
the data center and fog server are expensive, and they positive acknowledgment that the resources are available
can choke network bandwidth [29]. However, the as- for reservation. Upon expiration of the interval, the locked
signment of fog node is also considered based on the resources are released. The requesting broker might end
amount of data need to be transferred from one fog up getting positive acknowledgments from multiple neigh-
location to another. The data-intensive tasks have more boring brokers, and hence, selecting the most appropriate
priority over local fog locations. resources based on network parameters such as round trip
2) Fog nodes – are the actual computing resources avail- time (RTT), time to live (TTL), mobile user movement di-
able at different fog locations. Each fog node registers rection and network stability parameters. Here, we assume
itself with a broker node, and starts sharing its resource that only available fog locations respond to the resource
usage price and computing capacity. This information is request. Therefore, overloaded nodes will not participate as
passed on by the broker to other brokers. Note that the candidate locations for offloaded tasks.
framework supports both mobile or static fog nodes, Handover –In order to dynamically manage the user re-
but in this study we only assume static fog nodes. quests and delivery of results after execution, a transpar-
3) End devices – are the user devices including sensors ent handover mechanism is incorporated. Here, we have
uploading tasks and/or seeking computations. The considered the task and result handover mechanism. The
users can avail resources either via dedicated allocation task handover is used to forward the task to the selected
or bidding process. The fog node discovery process fog location either local or remote; whereas, result handover
is based on beacon messages. Here, the fog broker mechanism delivers the result through multi-hops. At re-
broadcasts beacons at regular intervals. These beacon mote execution, the remote broker dispatches the results
messages are used to inform nearby end devices about directly to the requested device; thus, source broker is not
the fog location. Any device interested in utilizing the involved in this process in order to reduce the burden
fog computing capacity responds to the messages, that from the backend network. The entire system manages the
is the device connects to the fog network. handover mechanism, and also track the user movement
through the mobile handoff mechanism. These backend pro-
4.2 Computing Resource Management cesses are totally transparent from the end users. Moreover,
In xFogSim, user access fog resources without knowing the physical location of the fog nodes executing requests is
the location of fog nodes. The cost of using fog resources transparent to the users.

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
7

4.5 Resource Sharing Module Algorithm 2 Calculate availability – CA()

Input F : list of fog nodes
In this subsection, we detail the fog resource allocation Pf (f ): failure probability of fog nodes
(FRA) algorithm to allocate resources according to the SLA. Pf (s): failure probability of switch
The algorithm is executed by each broker using information Pf (b): failure probability of broker
Output A : Total availability of selected fog nodes
about available fog resources at its fog location. Moreover,
1: S ← {φ} . List of switches at fog location l
the broker nodes maintain tables to come up with a task 2: L ← {φ} . List of fog locations
allocation solution that exhibits the highest performance, 3: for each f ∈ F do
lowest cost, and highest availability. 4: s ← f.getP arent();
5: S −1 (s) ← S −1 (s) + 1
Algorithm 1 details the FRA algorithm working in a 6: L−1 (f ) ← L−1 (f ) + 1
distributed manner to satisfy user requests. Every broker 7: end for
node maintains a fog resource table including information 8: X ← 1, Y ← 1
about resource performance, cost, and availability. For every 9: for each l ∈ L do
10: Pf (l) ← 1
request, the broker scans its resource list to find the best pos- 11: for each f ∈ F do
sible solution. Otherwise, the broker looks for cumulative 12: Pf (l) ← Pf (l) × Pf (f )
resources at other fog locations to meet its requirement. For 13: end for
14: X ← X × (Pf (b) + Pf (l) − Pf (b) × Pf (l))
cumulative resource allocation, the broker adds available 15: Y ← Y × (X + Pf (s) − X × Pf (s))
fog resources into its resource candidate table. The algo- 16: end for
rithm, at every step, calculates desired response time tdesired
r 17: A ← 1 − Y
using response times tlocal
r and tleased
r for local and leased
resources, respectively. This is used to build a list of fog
nodes to meet requirements. Moreover, the communication management. Any unsuccessful requests are placed in its
cost is used to filter out unqualified nodes. Similar steps are local queue Qlocal for time interval τlocal for retry. On the
performed for cost and availability. The latter is calculated other hand, if the broker receives a lease request from
using Algorithm 2. The objective is to find solutions with another fog location while no resources are available, it
maximum availability and minimum cost. queues the request into its lease queue Qlease for τlease
interval of time. Upon expiration of τlocal and τlease , the
Algorithm 1 Fog resource allocation (FRA) for broker node broker sends refusal messages against local end device
Input F : list of fog nodes sorted on response time tr requests and neighboring broker lease requests. The broker
Lc : current fog location node supports multiple applications with communication
tdesired
r : desired response time protocols such as TCP, UDP, MQTT, and PingApp. The
pdesired : desired price
Adesired : desired availability
Algorithm 3 explains the working of fog nodes.
Output S : solution for task allocation
1: Pmin ← +∞ Algorithm 3 Fog node execution
2: Amax ← −∞ Input f : fog node; b: local broker
3: S ← {φ} Output Nil
4: for each k ∈ N do
1: τ ← 0
5: if tkr 6 tdesired
r then return F . assign local fog node
2: while true do
6: else . assign remote fog node
3: if f InWait then
tk
7: F ← F + {k : tkr 6 tk
r
desired −1 } 4: j ← Recv() . Receive job
r ···tr
8: if Ω(k) = Lc then tr = tr local 5: o ← Exec(j) . Execute job
9: else tr = tleased 6: Send(o, b) . Send outcome to broker
r
10: end if 7: end if
F ← F − {x x desired } 8: if τ EXPIRES then
11: Q : x ∈ F, tr > tr 9: τ ←0 . Reset timer
12: A←1− Pf (x)
x∈F 10: Send(mHB ,b) . Send heartbeat to broker
13: for each f ∈ F do 11: end if
14: if A > Adesired & pf < pdesired then . matches the 12: end while
desired availability and price.
15: p ← p + pf
16: pmin ← min(p, pmin )
17: Amax ← max(A, Amax )
18: S ←S∪f 5 P ERFORMANCE E VALUATION
19: end if
20: end for The proposed xFogSim framework provides an extensive
21: end for environment to simulate distributed fog applications. In the
22: return S framework, different fog locations can be combined together
to form a large scale computing environment in a federated
As mentioned earlier, a broker node manages all fog manner. Each broker at a fog location can work indepen-
nodes. In case all its fog nodes are engaged, the broker dently, as well as, in collaboration with other brokers.
contacts brokers at a one-hop distance to negotiate for We set up a federated fog simulation environment to
additional resources to complete the request. Similarly, if benchmark the performance of xFogSim. The setup com-
all neighboring fog nodes are occupied, the broker contacts prises different sensors sending data, and users requesting
brokers at a two-hop distance until it finds suitable fog computation resources. The users can be mobile based on
nodes otherwise it retries later. The broker manages multiple different mobility models. At each fog location, there is
queues for the request to perform local and leased resource a broker and a set of fog nodes made available initially.

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
8

The users request computation resources from its respective

Memory usage (MBs)

broker. The resources are requested in the form of million 800
instructions per second (MIPS) and categorized as large-
600
sized tasks (3000 MIPS or greater) and small-sized (500-
2500 MIPS). The broker arranged resources either locally or 400
through other brokers while ensuring the SLA. We evalu- µ=0.5s
ated the framework by varying the request rate (RR) param- 200 µ=1.0s
eter. Other simulation parameters and simulation system µ= 1.5s
specifications are shown in Table 3. 0
0 200 400 600 800 1,000 1,200
No. of users
TABLE 3: Simulation configuration and system specifica-
tion [7] (a) Memory usage

Parameter Value
30
Fog locations 5–10

CPU usage (%)

Fog nodes (f ) 2–4
Fog nodes compute capacity 1000 MIPS 20
Broker(s) 5–10, one per fog location
Cloud data center(s) 1–4
Wireless sensors 10–600 µ=0.5s
10
Wireless access point(s) 5–100 µ=1.0s
µ= 1.5s
Wireless mobility Linear, circular, mass mobility
Device mobility (1–150) Linear 0
0 200 400 600 800 1,000 1,200
Device mobility (151–300) Mass
Device mobility (301–700) Circular No. of users
Device mobility (701–1200) Linear (b) CPU usage
Broker-to-broker link 10 Gbps
Broker-to-data center link 1 Gbps Fig. 4: Average memory and CPU usage with respect to
Mobile device-to-broker link 4 Mbps number of users and task request rates (µ)
Mobile device transmission range 400m
Request arrival rate 0.5s, 1.0s, 1.5s
Heartbeat message rate 5s
Packet error rate 10−1 IP assignment, gateway setup, and GUI rendering. The
delay increased with increasing number of network nodes
Users/sensors nodes application mqttApp
Brokers group Resource request including, sensors, users, brokers and fog nodes as shown
Fog nodes application ComputeApp in Figure 5. OMNeT++ provides a GUI enriched interface
Broker nodes application BaseBrokerApp with Tken module, which can be disabled to reduce this
Task handover true delay. Nevertheless, the module gives detailed simulation
Fog federation true insights at any point of execution. Therefore, we measured
Fog locations per federation All
Resource sharing policy Proposed FRA
framework setup delay including the module.

CPU Intel CoreT M i5-3470 3.2 GHz

Network setup delay (s)

RAM 8 GB 400
OS Ubuntu 16.04.03 LTS
Simulator OMNeT++ 300

200
The computation request along with suggested cost and
response time is received at the B node which traverses into 100
its directory to find the best f node that can meet the user
0
requirements. This is achieved using FRA Algorithm 1. 0 200 400 600 800 1,000 1,200
Resources usage – The performance of the proposed No. of users
framework is evaluated in terms of memory, CPU usage,
and delay in network setup. Figure 4a shows memory usage Fig. 5: One time delay caused during xFogSim bootup
while varying the number of users and RR values. The
generated tasks required different computations. Therefore, Figure 6 shows an end-to-end delay between mobile
with more frequent task rate, more memory was required users and assigned fog resources. This delay includes fog
to hold and execute. A similar trend was observed for CPU assignment either locally or through neighboring broker
usage as shown in Figure 4b when varying the number of nodes. The resource assignment from neighboring locations
users, i.e., frequent task requests consume more CPU cycles. can add further delay to communication. Moreover, fre-
Simulation execution and end-to-end delay – Another quent communication between device and fog location can
important factor considered for evaluation was the time affect the overall performance of the framework.
required to build the framework with a variable number Execution time – Figure 7 shows task execution time by
of users, devices, and fog nodes. The incurred setup de- varying the number of users. Similarly, Figure 7a shows
lay was mainly due to many influencing factors such as the times for small-and large-sized computation requests.
modules initialization, network configuration and setup, Clearly, with large task sizes, the execution time increased.

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
9

mobility, the requests generated in a particular fog location

80 local requests
End-to-end delay (ms) leased requests
varied. Thus, leading to brokers negotiating with neighbor-
60 ing brokers, increasing the queuing delay.

40 100 fog location-1

Queuing time (ms)

fog location-2
20
fog location-3
0
0 20 40 60 50
No. of requests

Fig. 6: Average end-to-end delay between user and fog

nodes via single or multiple hops for both local and leased 0
0 20 40 60 80 100
resource requests Simulation time (s)

Fig. 8: Real-time queuing pattern at different simulation

This was due to limited resources available at fog locations times
compared to cloud data centers. For small sized tasks, the
fog locations can easily facilitate more users with less delay. Cloud support – The proposed framework supports
Figure 7b shows the impact of request arrival rate when integration with backend cloud data centers to handle re-
varying the number of users. With frequent requests, task quests during peak hours. This leads to an additional delay
execution took more time due to faster filling up of queues. in response, even though improving the capacity of the
framework to service many requests. Figure 9 shows the
behavior of the framework in the presence and absence of
Task execution time (ms)

80 small-sized
large-sized cloud support in the simulated environment. In the former
60 case, the framework offloading tasks to cloud data centers.

40
with cloud support
400
20 without cloud support
No. of tasks

0
0 200 400 600 800 1,000 1,200
200
No. of users
(a)

80 0
Task execution time (ms)

µ=0.5s 0 50 100 150 200

µ= 1.0s Simulation time (s)
60 µ= 1.5s
Fig. 9: Workload comparison in real-time for xFogSim with
40 and without cloud support

20 Residual energy – Figure 10 presents the average energy

consumption concerning the simulation time. In traditional
0
0 200 400 600 800 1,000 1,200 cloud computing architecture, fog nodes are used to transfer
No. of users data and tasks to the back end cloud; however, in this
(b)
scenario, the fog nodes are used to execute the incoming
tasks; thus, they consume a significant amount of energy.
Fig. 7: Average task execution time, (a) Average task execu- Moreover in the fog collaborate environment, computa-
tion time with respect to the number of users and task size tional tasks are distributed to other fog locations base on
(small and large) keeping the request rate fixed at 1s, and availability. Thus, the energy consumption at various fog
(b) Average task execution time with respect to the number locations is similar due to load sharing approach. The
of users and request rate (µ) for random sized task requests proposed framework allocates available energy values to
(between 500-3000 MIPS) the brokers in the beginning and then computes residual
energy as they service requests. Figure 10 shows the average
Queue length – Figure 8 shows queue lengths at broker amount of energy available at the brokers as the simulation
nodes at discrete points. The y-axis shows the queuing delay progresses. Initially, the residual energy gradually decreases
and x-axis shows the total simulation time. Initially, all due to the broker nodes being lightly loaded; however, the
queues are empty. With increasing computation requests, energy consumption increases with the increasing workload
the delay increased. Here, all fog locations comprise an at the brokers. Furthermore, Figure 11 shows the energy
equal number of fog nodes but due to the end user/device consumption comparison between the proposed xFogsim

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
10

federated approach and the traditional non-federated sce- tion with nearby broker nodes. This helps sharing resource
nario. The result demonstrates that energy consumption un- information at almost real-time. Moreover, due to the fog
der the non-federated approach varies significantly across federation environment, xFogSim better utilizes resources to
fog locations due to the uneven workload distribution. In provide QoS, even with limited devices at the fog location.
contrast, the federated approach benefits from task sharing
with other fog locations based on their availability, that is FogNetSim++
the energy consumption at each fog location is similar to xFogSim
other fog locations irrespective of the uneven workloads. 200

No. of tasks
Thus, this improves the availability of fog locations when
providing services to end-users.
100
Residual energy (x100 J)

2
0
0 20 40 60 80 100
1 Simulation time (s)

Fig. 12: Request management capacity at broker for xFogSim

0 and FogNetSim++. Broker queue size is set to 100)
0 20 40 60 80 100
Simulation time (s)
5.1 General Discussion
Fig. 10: Average residual energy of broker nodes vs Simula- Resource sharing environment – xFogSim framework pro-
tion time vides a resource sharing environment through the federa-
tion approach. Every broker node schedules tasks to its local
set of fog resources. However, to handle the burst of task
requests, resources at other fog locations can be leased. In
200
Federated Non-federated FogNetSim++, the brokers were capable of executing tasks
Energy (J)

using local fog resources. Moreover, as mentioned in Table 1,

no such fog simulation framework exists that provides fog
100 federation and support resource sharing environment. All
the existing work manages a single fog location.
0 Fog resource allocation strategy – The proposed fog re-
f1 f2 f3 f4 source allocations strategy in xFogSim employs attributes
Fog locations including cost, performance, and availability to schedule
tasks at remote fog locations. The user accesses fog resources
Fig. 11: Residual energy (J) measured at fog locations during without knowing the location of fog nodes. The cost of using
intermediate stage. fog resources vary based on current workload. The broker
node manages fog resources either from its pool of fog nodes
Scalability – Here, we performed comparison between or leases some from neighboring brokers. It is worthwhile
xFogSim and FogNetSim++ in terms of the number of to mention that xFogSim takes communication costs into
requests accepted for execution. The proposed framework account when resources are arranged from other locations,
facilitates requests through fog federation. With every bro- i.e., the broker nodes are aware of the associated costs for
ker node sharing its resource updates with nearby broker using the resources of neighboring broker locations. The
nodes, thereby allowing nodes to lease computation re- broker periodically broadcasts fog advertisements (FA) in
sources, consequently, servicing a large number of users. its region. Users respond to the advertisement when they
When handing over the computation requests, the broker require resources. Though the advertisement includes the
selects the best possible fog resource location to service cost but only for local resources, the user can accept the cost
the requests while keeping intact the agreed QoS. Note or make a bid. The detailed working of the strategy (FRA
that during this servicing of requests, the fog resource algorithm) is discussed in Section 4. In FogNetSim++, due to
locations remain transparent to the end users. Figure 12 non-availability of sharing environment, no such algorithms
shows a comparison between FogNetSim++ and xFogSim and federation environment are presented.
in terms of request management capacity. The broker node Handover mechanism – xFogSim provides fog resource
maintains a centralized queue used by available fog devices sharing across different fog locations using multiple types
to pick a task. For this comparison, the queue size for both of handover mechanisms. To dynamically manage user re-
frameworks is set to hold 100 pending requests at any time. quests and delivery of results after execution, a transparent
After some time, the queues on both frameworks reach their handover mechanism is incorporated. Here, we have con-
capacity due to frequent computing requests received from sidered the task and result handover mechanism. The task
the devices. FogNetSim++ starts dropping the new requests handover is used to forward the task to the selected fog
due to queue limitations. On the contrary, xFogSim accepts location either local or remote; whereas, result handover
new requests and executes them on nearby fog locations. mechanism delivers the result through multi-hops. At re-
The broker node in the latter case shares context informa- mote execution, the remote broker dispatches the results

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
11

directly to the requested device. To reduce the burden from framework in terms of memory usage, CPU usage, network
the backend network, source broker is not involved in the configuration delay, and task execution time. The results
process of remote execution. The entire system manages the show that xFogSim is lightweight and easy to configure on
handover mechanism, and also tracks the user movement commodity hardware. It can handle a large number of user
through the mobile handoff mechanism. These backend pro- requests using dynamic resource provisioning across fog
cesses are totally transparent from the end users. Moreover, locations. In the future, the framework can incorporate VM
the users are not aware of the physical location of fog nodes migration among fog locations while tracking user requests.
executing their requests. On the contrary in FogNetSim++, Moreover, techniques to predict task execution times can be
tasks are only executed using local fog resources with results incorporated to reduce the overall migration overhead.
delivered using the handover mechanism.
Supported mobility models – To handle mobile devices,
various mobility models are incorporated in xFogSim. More- ACKNOWLEDGMENTS
over, xFogSim’s flexible design allows easy integration of The work of Samee U. Khan is supported by (while serving
mobility models that are newly designed or extended from at) the National Science Foundation (NSF). Any opinion,
the existing models, provided that the mobility template findings, and conclusions or recommendations expressed in
defined in OMNeT++ is used. This feature is similar to this material are those of the authors and do not necessarily
the one provided in FogNetSim++. Table 3 only shows the reflect the views of the NSF.
mobility models used during the evaluation of xFogSim.
However, there is a support to add other models to simulate
more realistic and complex scenarios. R EFERENCES
Network parameters – xFogSim provides a federation en-
vironment where the brokers act as agents to negotiate [1] S. Lucero et al., “Iot platforms: enabling the internet of things,”
White paper, 2016.
resources from other fog locations. Evidently, such resource [2] A. Yousefpour, C. Fung, T. Nguyen, K. Kadiyala, F. Jalali, A. Ni-
assignment introduces communication delay. Therefore, the akanlahiji, J. Kong, and J. P. Jue, “All one needs to know about fog
environment allows its users to define different network computing and related edge computing paradigms: A complete
parameters, for instance, bandwidth and packet error rate survey,” J. Syst. Archit., 2019.
[3] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on
for communication links between broker nodes. Moreover, mobile edge computing: The communication perspective,” IEEE
multiple federate grouping strategy can be defined at the Commun. Surv. Tutorials, vol. 19, no. 4, pp. 2322–2358, 2017.
user level, that is fog locations can be a part of multiple [4] A. Munir, P. Kansakar, and S. U. Khan, “Ifciot: Integrated fog
cloud iot: A novel architectural paradigm for the future internet of
federations based on user definition. things.” IEEE Consum. Electron. Mag., vol. 6, no. 3, pp. 74–82, 2017.
Usability/Manageability – xFogSim is an open-source dis- [5] Y. Xiao and M. Krunz, “Distributed optimization for energy-
tributed fog simulation framework providing IoT device efficient fog computing in the tactile internet,” IEEE J. Sel. Areas
mobility and backend connectivity to a data center. More im- Commun., vol. 36, no. 11, pp. 2390–2400, Nov 2018.
[6] L. Ni, J. Zhang, C. Jiang, C. Yan, and K. Yu, “Resource allocation
portantly, the framework allows researchers to incorporate strategy in fog computing based on priced timed petri nets,” IEEE
and analyze different resource allocation algorithms under IoT J., vol. 4, no. 5, pp. 1216–1228, Oct 2017.
fog federation with varying mobility scenarios. Moreover, it [7] T. Qayyum, A. W. Malik, M. A. K. Khattak, O. Khalid, and S. U.
being open source makes it easy to manage and extend to Khan, “Fognetsim++: A toolkit for modeling and simulation of
distributed fog environment,” IEEE Access, vol. 6, pp. 63 570–
meet the growing challenges of the IoT community. 63 583, Oct 2018.
[8] C. Sonmez, A. Ozgovde, and C. Ersoy, “Edgecloudsim: An envi-
ronment for performance evaluation of edge computing systems,”
6 AVAILABILITY in 2017 Second International Conference on Fog and Mobile Edge
xFogSim framework is an open source project anonymously Computing (FMEC), May 2017, pp. 39–44.
available on GitHub at https://github.com/rtqayyum/ [9] “Simplesoft simpleiotsimulator,” http://www.smplsft.com/
SimpleIoTSimulator.html, accessed: 2020-04-10.
exFogSim. [10] T. Pflanzner, A. Kertész, B. Spinnewyn, and S. Latré, “Mobiotsim:
towards a mobile iot device simulator,” in 2016 IEEE 4th Interna-
tional Conference on Future Internet of Things and Cloud Workshops
7 C ONCLUSIONS (FiCloudW). IEEE, 2016, pp. 21–27.
Fog computing is designed to facilitate delay-sensitive ap- [11] “Googlecloudiot,” https://cloud.google.com/solutions/iot/, ac-
cessed: 2020-04-10.
plications, pulling in cloud resources and services to the
[12] H. Gupta, A. Vahid Dastjerdi, S. K. Ghosh, and R. Buyya, “ifogsim:
edge of the user network. Thus, it reduces communication A toolkit for modeling and simulation of resource management
cost, unnecessary delay, bandwidth usage, while increases techniques in the internet of things, edge and fog computing
throughput, and overall performance of real-time appli- environments,” Software: Practice and Experience, vol. 47, no. 9, pp.
1275–1296, 2017.
cations. In this paper, we proposed xFogSim, an exten- [13] “Contiki cooja,” http://www.contiki-os.org/start.html, accessed:
sion of FogNetSim++, to incorporate support for multiple 2020-04-1.
fog locations managed by independent broker nodes. The [14] A. Brogi and S. Forti, “Qos-aware deployment of iot applications
coordination among broker nodes results in an extensive through the fog,” IEEE IoT J., vol. 4, no. 5, pp. 1185–1192, Oct 2017.
[15] R. Mayer, L. Graser, H. Gupta, E. Saurez, and U. Ramachan-
framework to simulate context-aware applications. We also dran, “Emufog: Extensible and scalable emulation of large-scale
proposed a resource allocation algorithm that implements a fog computing infrastructures,” in 2017 IEEE Fog World Congress
trade-off among parameters such as price, response time, (FWC), Oct 2017, pp. 1–6.
communication cost, and availability of resources. More- [16] C. Sonmez, A. Ozgovde, and C. Ersoy, “Edgecloudsim: An envi-
ronment for performance evaluation of edge computing systems,”
over, the framework design is flexible to adopt new coor- in Fog and Mobile Edge Computing (FMEC), 2017 Second International
dination algorithms for experimentation. We evaluated the Conference on. IEEE, 2017, pp. 39–44.

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSUSC.2020.3025021, IEEE
Transactions on Sustainable Computing
12

[17] K. Hong, D. J. Lillethun, U. Ramachandran, B. Ottenwälder, and ANIS U. RAHMAN received Master’s degree
B. Koldehofe, “Mobile fog: a programming model for large-scale in Parallel and Distributed Systems and Ph.D.
applications on the internet of things,” in MCC@SIGCOMM, 2013. in Computer Science from Grenoble University,
[18] S. Tuli, R. Mahmud, S. Tuli, and R. Buyya, “Fogbus: A blockchain- France, in 2009 and 2013. He is currently an
based lightweight framework for edge and fog computing,” J. Syst. Assistant Professor at NUST-SEECS, Pakistan.
Software, 2019. In 2019, he worked as Research Fellow at the
[19] J. Li, J. Jin, D. Yuan, and H. Zhang, “Virtual fog: A virtualization FSKTM, Universiti Malaya, Malaysia. His main
enabled fog computing framework for internet of things,” IEEE research interests include internet of things, sim-
IoT J., vol. 5, no. 1, pp. 121–131, Feb 2018. ulation and modeling, visual perception and cog-
[20] L. Gao, T. H. Luan, S. Yu, W. Zhou, and B. Liu, “Fogroute: Dtn- nition, computer vision and machine learning.
based data dissemination model in fog computing,” IEEE IoT J.,
vol. 4, no. 1, pp. 225–235, Feb 2017.
[21] A. Coutinho, F. Greve, C. Prazeres, and J. Cardoso, “Fogbed: A
rapid-prototyping emulation environment for fog computing,” in
2018 IEEE International Conference on Communications (ICC), May
2018, pp. 1–7. MUAZZAM A. KHAN (Senior Member IEEE)
[22] L. Pu, X. Chen, J. Xu, and X. Fu, “D2d fogging: An energy-efficient is working as tenured Assoc. Prof. at NUST-
and incentive-aware task offloading framework via network- SEECS, Pakistan. He received his PhD under
assisted d2d collaboration,” IEEE J. Sel. Areas Commun., vol. 34, joint program from IIUI and UMKC in 2011.
no. 12, pp. 3887–3901, Dec 2016. He completed his postdocs from University of
[23] S. Tuli, R. Mahmud, S. Tuli, and R. Buyya, “Fogbus: A blockchain- Ulm in 2013 and University of Missouri in 2016.
based lightweight framework for edge and fog computing,” arXiv He worked at Networking and Multimedia Lab,
preprint arXiv:1811.11978, 2018. UMKC as Research Fellow. His research inter-
[24] “Amazonpricing,” https://calculator.aws, accessed: 2020-04-1. ests are wireless sensor networks, body area
[25] X. Meng, V. Pappas, and L. Zhang, “Improving the scalability networks, image compression, image encryption
of data center networks with traffic-aware virtual machine place- and data network security.
ment,” in 2010 Proceedings IEEE INFOCOM, March 2010, pp. 1–9.
[26] W. Dai, L. Qiu, A. Wu, and M. Qiu, “Cloud infrastructure resource
allocation for big data applications,” IEEE Trans. Big Data, vol. 4,
no. 3, pp. 313–324, 2016.
[27] R. Iqbal, T. A. Butt, M. O. Shafique, M. W. A. Talib, and T. Umer, OSMAN KHALID has completed his PhD from
“Context-aware data-driven intelligent framework for fog infras- North Dakota State University, USA and his mas-
tructures in internet of vehicles,” IEEE Access, vol. 6, pp. 58 182– ters from Center for Advanced Studies in En-
58 194, 2018. gineering. He is currently Assistant Professor
[28] H. Zhang, Y. Zhang, Y. Gu, D. Niyato, and Z. Han, “A hierarchical in COMSATS University Islamabad, Abbottabad
game framework for resource management in fog computing,” Campus. His research areas include: Recom-
IEEE Commun. Mag., vol. 55, no. 8, pp. 52–57, Aug 2017. mender Systems, Network Routing Protocols,
[29] N. C. Narendra, K. Koorapati, and V. Ujja, “Towards cloud-based Internet of Things, and Fog Computing. His web-
decentralized storage for internet of things data,” in 2015 IEEE site is: http://osman.pakproject.com
International Conference on Cloud Computing in Emerging Markets
(CCEM), Nov 2015, pp. 160–168.
[30] M. B. Chhetri, A. R. M. Forkan, Q. B. Vo, S. Nepal, and R. Kowal-
czyk, “Towards risk-aware cost-optimal resource allocation for
cloud applications,” in 2019 IEEE International Conference on Ser-
vices Computing (SCC). IEEE, 2019, pp. 210–214. SAMEE U. KHAN received a PhD in 2007 from
the University of Texas. Currently, he is the Head
and James W. Bagley Chair Professor in Elec-
trical and Computer Engineering at Mississippi
State University. Previously, he was the Cluster
Lead for the Computer Systems Research at
Dr. ASAD WAQAR MALIK is an Assistant Pro- the National Science Foundation, and the Walter
fessor at the Department of Computing (DOC), B. Booth Professor at the North Dakota State
NUST School of Electrical Engineering and University. His research interests include opti-
Computer Science (SEECS). Besides, he is also mization, robustness, and security of computer
working as Senior Lecturer at the Department systems. His work has appeared in over 400
of Information Systems, Faculty of Computer publications. He is associate editor-in-chief of the IEEE IT Pro and
Science & Information Technology, University of associate editor of Journal of Parallel and Distributed Computing and
Malaya, Malaysia. His primary area of interest ACM Computing Surveys.
includes distributed simulation, cloud/fog com-
puting, and internet of things.

TARIQ QAYYUM received Bachelor of Science

in Computer Science (BSCS) degree from The
Islamia University of Bahawalpur, Pakistan. He is
perusing Master of science in Information Tech-
nology (MSIT) from National University of Sci-
ences and Technology, Islamabad, Pakistan. His
research interests include cloud computing, fog
computing, IoT, and distributed systems.

2377-3782 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF WESTERN ONTARIO. Downloaded on November 10,2020 at 14:34:35 UTC from IEEE Xplore. Restrictions apply.