Renewable and Sustainable Energy Reviews 167 (2022) 112782

Contents lists available at ScienceDirect

Renewable and Sustainable Energy Reviews

journal homepage: www.elsevier.com/locate/rser

Utilizing power consumption and SLA violations using dynamic VM

consolidation in cloud data centers
Umer Arshad a , Muhammad Aleem b , Gautam Srivastava c,d , Jerry Chun-Wei Lin e ,∗
a
The University of Lahore, Lahore 54000, Pakistan
b
National University of Computer and Emerging Sciences, Islamabad 44000, Pakistan
c
Brandon University, Brandon, MB R7A 6A9, Canada
d
China Medical University, 40402, Taichung, Taiwan
e
Western Norway University of Applied Sciences, 5063, Bergen, Norway

ARTICLE INFO ABSTRACT

Keywords: Cloud Computing services can be accessed anytime, anywhere via the Internet. The overwhelming growth of
Cloud computing cloud data centers over the past decade has increased their costs as energy demands have risen. As a result,
Service level agreement higher carbon dioxide emissions and other greenhouse gasses are putting a strain on our ecosystem. The main
Energy efficiency
objective of this study is to reduce the power consumption in cloud computing with no or negligible trade-offs
VM migrations
in quality of service. This paper presents a new algorithm called the energy efficiency heuristic using virtual
Real-time cloud data centers
machine consolidation to minimize the high energy consumption in the cloud. By setting two thresholds, hosts
are classified into three main classes. The designed model reallocates virtual machines from one physical host
to another to minimize energy consumption. The results of the proposed algorithm have been obtained in
terms of virtual machine migrations, performance degradation caused by migration, service level agreement
violations, and execution time, showing a significant improvement over state-of-the-art techniques.

1. Introduction requirements and the Quality of Service (QoS) between the Cloud users
and the service provider [13].
Many individuals and businesses worldwide use Cloud Computing Given the large cloud data centers of today, colossal energy con-
(CC) [1–3] to obtain storage and computing services accessible via the sumption is one of the main problems [14,15]. A lot of energy is needed
Internet. The general use of cloud computing has increased manifold in a data center to run the cooling systems. In addition, computing,
due to its easy accessibility, excellent scalability, cost efficiency, and storage, and networking equipment consume a lot of power. The en-
reliability [4–6]. Cloud services can be used anytime and anywhere ergy consumed ultimately reflects high CO2 emissions that impact the
via the Internet [7]. Cloud models are divided into three services: biosphere [4]. According to a report by the Natural Resource Defense
Software, Platform, and Infrastructure as a Service [8]. Software-as- Council [5], data centers in the United States of America consume about
a-Service (SaaS) allows users to utilize online software applications 91 billion kWh of electricity, and this number is estimated to increase
over the Internet; an example is Google-Drive, Dropbox, Facebook, etc. to 200 billion kWh by 2030.
Platform-as-a-Service (PaaS) provides a framework for programmers to The increase in electricity consumption also increases the cost of the
develop their customized cloud applications in Microsoft Azure and business model and consequently lowers productivity [16]. According
Google Application Engine. Infrastructure-as-a-Service (IaaS) provides to a study [4], Cloud Computing (CC) consumes more energy than
virtualized computing resources such as Central Processing Unit (CPU), most countries worldwide. To illustrate, if we consider Cloud as
bandwidth, storage, and memory in the form of Virtual Machines a country, it would be the fifth-largest country in terms of energy
(VMs) [9–11]. The benefits associated with each model are unique and consumption [4]. In the cloud, the main components that contribute
different from one another, and the cloud promises to meet the needs to this enormous energy consumption are CPU, memory, networking,
of different types of businesses [12]. The cloud user and provider often storage, cooling and power consumption, etc., as shown in Fig. 1.
agree on specific terms and conditions for the use of cloud services, As highlighted in Fig. 1, there is an urgent need to reduce energy
called a Service Level Agreement (SLA). A SLA often describes the consumption without compromising QoS. Recent advances in hardware

∗ Corresponding author.
E-mail addresses: umer.arshad@cs.uol.edu.pk (U. Arshad), m.aleem@nu.edu.pk (M. Aleem), SRIVASTAVAG@brandonu.ca (G. Srivastava), jerrylin@ieee.org
(J.C.-W. Lin).

https://doi.org/10.1016/j.rser.2022.112782
Received 26 April 2022; Received in revised form 3 July 2022; Accepted 7 July 2022
Available online 27 July 2022
1364-0321/© 2022 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Nomenclature

Abbreviations

EEHVMC Energy Efficiency Heuristic with Virtual

Machine Consolidation
QoS Quality of Service
VM Virtual Machine
VMM Virtual Machine Migration
PDM Performance Degradation caused by VM
migration
SLATAH Service Level Agreement Violations Time
per Active Host
IQR inter-quartile range
HOL Host Over-Loaded Fig. 1. Breakdown of power consumption in data centers [17].
HML Host Medium-Loaded
HUL Host Under-Loaded
MIPS Millions of Instructions Per Second where the classic techniques are slow, or an exact solution is not desir-
MSU Minimum Size Utilization able (considering the employed overhead). The purpose of a heuristic is
CATR Cumulative Available-to-Total Ratio to find a solution in a reasonable time that is good enough to solve the
EVMC Energy-Aware VM Consolidation problem at hand [26]. The heuristic solution is not optimal, however,
SABFD Space Aware Best Fit Decreasing the solution provides approximately the exact answer. Therefore, the
EEOM Energy Efficiency Optimization of Virtual proposed scheme is designed as a heuristic approach to quickly finds
Machine Migrations the placement strategy to conserve energy in the Cloud. Almost 79%
of the consumed power in the cloud comprises CPU and memory;
RALBA Resource Aware Load Balancing Algorithm
hence these two most important aspects are the building blocks of the
EFT Earliest Finish Time
proposed scheme.
ETSA Energy-Efficient Task Scheduling Algorithm
The proposed Energy Efficiency Heuristic using VM Consolidation
MU Maximum Utilization (EEHVMC) reduces power consumption and SLA violations (SLAV). The
MMT Minimum Migration Time MMT main idea is to classify host machines based on CPU and memory usage.
RS Random Selection By setting two thresholds (related to CPU and memory utilization),
MC Maximum Correlation the host machines are categorized into three main classes: Host Over-
DVMC Dynamic Virtual Machine Consolidation Loaded (HOL), Host Medium-Loaded (HML), and Host Under-Loaded
SLAV Service Level Agreement Violations (HUL) machines. As of HOL, we migrate VMs to the HML to minimize
DVFS Dynamic Voltage and Frequency Scaling power consumption in the cloud data centers. In the HML, all VMs are
QoS Quality of Service kept unchanged. From the HUL hosts, the proposed approach reassigns
the VMs to the HML, and the inactive hosts are put into power-saving
SLA Service Level Agreement
mode [27].
CPU Central Processing Unit
The results show that the EEHVMC approach significantly mini-
IaaS Infrastructure-as-a-Service
mizes power consumption and reduces SLAV. In summary, the contri-
PaaS Platform-as-a-Service butions of this work are summarized below:
SaaS Software-as-a-Service
SOTA State-of-the-art • Detail analysis of the literature examines the strengths and weak-
nesses of existing VM consolidation heuristics.
DNN Deep Neural Networks
• A novel scheduling mechanism, EEHVMC consolidates VMs on
CC Cloud Computing
host machines to reduce power consumption, VM migrations, per-
formance degradation, and Service Level Agreement Violations in
the cloud.
• Experimentation and evaluation of the intended approach com-
technology have reduced the energy consumption of a CC system [18–
pared to state-of-the-art VM consolidation heuristics.
20].
One of the fundamental mechanisms for energy conservation is The remainder of the paper is organized as follows: Section 2
Dynamic Voltage and Frequency Scaling (DVFS). This approach auto- discusses related work. Section 3 states the performance model used
matically changes the voltage and frequency to reduce processor heat in this study, SLAV, and PDM. Then, Section 4 presents the system ar-
dissipation and lowers the power consumption. In addition, the reduced chitecture, EEHVMC algorithm, and complexity analysis. Experimental
heat generation allows cooling systems to be turned off, saving more results and discussions are presented in Section 5. Finally, Section 6
energy [21,22]. Another related approach to energy conservation in concludes the paper and explains future work.
cloud computing is Dynamic Virtual Machine Consolidation (DVMC).
The DVMC approach reallocates VMs from one host to another to 2. Related work
reduce the number of active hosts in the data center by putting inactive
hosts into power-saving mode to conserve the power [23–25]. Beloglazov et al. [4] conducted a competitive study to mitigate the
This study presents an energy-conserving framework based on the problems associated with VMM and DVMC. The results state that de-
concept of VM Consolidation. The proposed framework is a heuristic veloping a randomized or adaptive approach is essential to improving
model. A heuristic approach speeds up the process to locate a suitable the performance of optimal deterministic algorithms. In addition, novel
solution. A heuristic is an approach for problem-solving and is effective adaptive heuristics were proposed based on a retrospective study of

2
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

the resource method for VM performance- and energy-efficient consol- The authors aim to reduce power consumption, and SLA violations
idation. The technique significantly reduces energy consumption while in the Space Aware Best Fit Decreasing (SABFD) [21] approach. This
ensuring high compliance with SLA. CPU, memory, network interfaces, approach selects a VM for migration with maximum CPU utilization and
and disk storage are essential for determining to compute node en- places a VM on the host with lower computational power, i.e., Millions
ergy consumption. This approach mitigates energy consumption only of Instructions Per Second (MIPS). It minimizes power consumption
for CPU usage, while memory, network interfaces, and disk storage based only on CPU utilization. The SABFD does not examine the type of
are the components responsible for host power consumption in cloud applications running on the VM and cannot reduce data center power
computing. consumption and minimize SLA.
Dynamic Virtual Machine Consolidation (DVMC) approach [6] re- Buyya et al. [28] proposed an approach to mitigate the problem of
duces SLAV and power consumption in the cloud. This approach is energy consumption in the cloud. This approach helps to reduce data
classified into four main areas: Underload, Overload Detection, VM center energy consumption and enables low-cost cloud production. This
Selection, and Placement. The first part of this approach is to detect paper presents and implements an energy-aware resource allocation al-
whether the host is overloaded or not. If the host is overloaded, gorithm based on a VM consolidation mechanism. Experimental results
depending on the upper threshold, a VM migrates to the other physical show that this technique is efficient compared to other energy-aware
host. The VM selection policies are Minimum Migration Time (MMT), approaches; however, it introduces significant overhead. Moreover, this
Random Selection (RS), Maximum Correlation (MC), and Maximum approach does not consider the application types running on the VMs.
Utilization (MU). All VMs must migrate to another physical host if the Liu et al. [29] presented a VM consolidation approach for a cloud
host is underloaded. The proposed approach does not consider CPU, computing environment. The central concept was to minimize the
memory- and I/O-intensive tasks running on a VM. VMMs and thus minimize energy consumption in the cloud. The main
Energy-Efficient Task Scheduling Algorithm (ETSA) [7] consists of aspects achieved by the proposed approach were: reducing the possi-
three parts: Estimation, Normalization, and, Selection and Execution. bility of host overload, avoiding unwanted VMMs, and consequently
The ETSA approach reduces energy consumption in several ways. The minimizing the total number of VMMs. The proposed method improves
normalization part determines the smallest normalized total value re- the resource utilization of the host machines. It works very well under
sulting from the combination of end time and utilization. The execution different workload traces. Therefore, this approach satisfies the require-
task is assigned to the resources that have the smallest normalized ment of minimizing the cost of data centers for resource providers. This
technique’s prospects are beneficial for service providers and end-users.
total value. The ETSA technique strikes a balance between completion
Uddin et al. [30] correspond to the proposition of a server con-
time and utilization and provides more reliable results. This approach
solidation technique to increase the efficiency of pre-installed server
balances workload and completion time but does not consider SLA.
machines and their utility by shifting them to virtual server machines
In the first phase of the intended approach, scheduling focuses
to promote eco-friendly and energy-efficient cloud data servers. A novel
on VM computational capabilities. The VM with the Earliest Finish
virtualized task scheduling algorithm evenly distributes tasks from
Time (EFT) is selected for job mapping in the second phase [11].
physical server machines to virtual machines. The experimental results
Resource Aware Load Balancing Algorithm (RALBA) improves resource
show that 30% could increase the efficient utilization of resources (on
utilization, reduces makespan, and minimizes execution time. However,
the deployed VMs). Moreover, the study showed that the least amount
RALBA is unsuitable for the dependent task, and quality of service is
of servers used (i.e., up to 50%), resulting in significant energy savings.
not part of this approach.
Energy-Aware VM Consolidation (EVMC) [31] system implements a
Adaptive three-threshold framework energy-aware algorithms cat-
resource parameter-based scheme to regulate overutilized hosts in a vir-
egorize hosts into four categories: less, little, normal, and overloaded
tual cloud environment. The comparisons of VMs and hosts determine
hosts. When a host is overloaded, virtual machines must migrate to the
for analyzing overloaded hosts, while the Cumulative Available-to-
less busy host. All VM remain unchanged if a host is normal and a little
Total Ratio (CATR) uses to determine the underutilized hosts. Transfer
busy. When the host is less active, all VM migrate to a less busy host.
VMs to appropriate hosts; VM placement uses a criterion based on
Zhou et al. [12] only considered the CPU and I/O intensive tasks on normalized resource parameters of hosts and virtual machines. Several
the VM. If the task is CPU intensive, the VM with a maximum CPU tests were performed on many virtual machines using traces from
ratio to memory usage is selected. If the task is I/O-bound, the VM PlanetLab workloads to calculate the performance of VM consolidation.
selection multiplies by the CPU and memory utilization. Zhou et al. The results show that the EVMC approach is on par with other well-
considered the CPU and I/O-intensive tasks but ignored the memory- known methods by improving energy savings, SLA violations, and the
intensive tasks on the overloaded host. This approach minimized power number of VM migrations.
consumption based only on the CPU workload, while CPU and memory The GradeCent algorithm [32] uses the Stochastic Gradient Descent
are the main components responsible for host power consumption in a technique. This technique promotes an upper CPU utilization thresh-
CC data center. old for detecting overloaded hosts using an actual CPU workload. In
The research presented in [13] proposes a scheduling technique for addition, the authors proposed a dynamic VM selection algorithm,
CC systems that is cost-effective and saves energy. While the approach i.e., Minimum Size Utilization (MSU), to select the VMs of an over-
minimizes schedule gaps by performing approximate computations loaded host for VM consolidation. Gradient and MSU maintain the
using per-core DVFS on different multi-core processors, it also accounts tradeoff between minimizing energy consumption and maximizing QoS
for input errors in component tasks. This study aims to maintain among the specified SLA objectives. The proposed algorithms focus on
quality at the desired standard and provide a cost-effective solution that increasing energy saving and violation of SLA by 23% and 27.5% on
provides energy efficiency and timelessness with precision. However, average, respectively, compared to the baseline methods.
this approach increases potential network traffic. In [33], authors present a frequency-aware management technique
Energy Efficiency Optimization of Virtual Machine Migrations for controlling processors’ dynamic and static power in data centers
(EEOM) [15] consists of three steps: VM selection, trigger time, and that operate virtual machines. A frequency-aware model capable of
host location. The EEOM technique migrates some lightly and heavily determining the best frequency ratio for reducing processor energy
loaded VMs to another physical host. The inactive host is put into a usage. The energy consumption of a data center may be enhanced
power-saving mode so that the host’s power consumption can minimize. with this model in place by altering the processor’s rate to meet the
The EEOM approach tries to reduce the number of running hosts but appropriate frequency ratio. This paper devises a management strategy
neglects the remaining factors such as cooling system, network traffic, for intelligently adjusting the frequency ratio to save energy while
and migration costs. compliant with virtual machine frequency requirements. The result

3
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Table 1
Summary of literature review techniques.
Heuristics Strengths Weaknesses
Optimal Online Deterministic Algorithms Minimize the total number of active hosts, live Minimize energy base only CPU utilization whereas neglects
[4] migrations, and performance degradation memory, and disk storage
Dynamic Virtual Machines Consolidation Meet end-to-end performance requirements, CPU, memory and I/O intensive tasks are not considered
Algorithm [6] energy-efficient, cost-effective running on the VM
Energy-Efficient Task Scheduling Improve resource utilization, consider heterogeneous SLA violations and temperature of host increase
Algorithm [7] tasks
Resource Aware Load Balancing Algorithm Improve resource utilization, minimal scheduling SLA is not considered, not efficient for dependent tasks
[11] overhead, reduced makespan
Adaptive Energy-Aware Algorithm [12] Minimize SLA and consider migration overhead They do not consider the total amount of resource cost
An Energy-Efficient, QoS-Aware and Maintain QoS, maximize resource, energy and cost Bandwidth and network traffic is also increasing
Cost-Effective Scheduling [13] savings
Energy Efficiency heuristic of VM CPU and memory are considered for migration Cooling system, network traffic, and migration cost are not
Migrations [15] purpose considered
Space Aware Best Fit Decreasing [21] Minimize resource waste, energy optimize VM placement base on CPU utilization whereas memory is not
considered
Energy-Aware Data Centre Resource CPU and memory are considered energy efficiency Operational cost increase, neglect the application type running
Allocation [28] heuristic in the VMs
Energy-Efficient and QoS DVM Reduce the amount of VMs migrations, low Neglects the essential factors like workload, type of host, and
Consolidation [29] operating costs, meet SLA temperature
Virtualized Task Scheduling Algorithm Reduce time, infrastructure overhead, operational Required load balancing algorithm, including server and
[30] costs network in the data center
Energy-aware VM consolidation [31] Improvement in Quality of Service, Meet SLA Performance degradation
GradeCent algorithm [32] Minimize live migration and execution time Neglects VM placement policy
Frequency-aware DVFS model [33] Energy efficiency of a data center maximize by Decreasing the CPU frequency will reduce the system
adjusting the processor’s frequency performance
Energy Optimization Algorithm [34] better performance in contrast to the interquartile SLA violation
range and local regression algorithms, high
throughput

shows that a modest static power percentage leads to excellent energy- Moreover, this study assumes two important cloud application classes:
saving performance after studying the relationships between frequency memory-intensive and CPU-intensive. The proposed scheduling heuris-
ratio and energy usage. tic considers the most prominent energy-hungry resources and guides
This paper [34] presents an Energy Optimization Algorithm (EOA) appropriate placement/mapping schemes for VMs. The total power
to optimize energy without losing performance. In this technique, consumption of a physical server is composed of two components: 𝑃𝑠
we identify the overload by looking at the whole workload usage of and 𝑃𝑢 , as shown in Eq. (1) [35,36]. 𝑃𝑠 is the fixed power consumption
the data center. With this method, the performance-to-power ratio of the server regardless of whether VMs are operating or not, and 𝑃𝑢 is
increase. Achieve high throughput; virtual machines must migrate less the dynamic power utilized by the VMs running on it.
frequently. This EOA’s primary purpose is to decrease the number of
𝑃(𝑡𝑜𝑡𝑎𝑙) = [𝑃𝑠 + 𝑃𝑢 ], (1)
live migrations while preserving performance.
In summary (see Table 1), most studies [4,6,12,13,21,28–30] target As given in Eq. (2) [36], 𝑃𝑐𝑝𝑢 is the amount of power consumed by
to minimize energy consumption only in terms of CPU, while stor- the CPU in the physical host computer, while 𝑃𝑚𝑒𝑚𝑜𝑟𝑦 is the amount of
age is an important component of energy consumption in cloud-host power consumed by the memory in the physical host machine.
machines. Some of the proposed approaches [7,11,15,31–34] use VM
migration mechanisms from overloaded host machines to underloaded 𝑃(𝑢) = [𝑃𝑐𝑝𝑢 + 𝑃𝑚𝑒𝑚𝑜𝑟𝑦 ] (2)
hosts. However, these approaches do not define appropriate thresholds Several recent studies [37–40] highlight that in a host machine,
(set at runtime) to detect whether host machines are overloaded or almost 70% of the power is consumed when a host is idle compared
underloaded. In addition, existing approaches do not consider the to other fully utilized hosts. This fact justifies that when an inactive
types of applications running on the VMs, which can lead to incorrect host turns off, it saves power significantly, resulting in excellent energy
migration decisions that result in fewer energy savings and more SLA efficiency. Therefore, the energy model [4] used (in our proposed VM
violations. placement or scheduling heuristic) is the energy consumed by both
active and inactive hosts, as shown in Eq. (3) [28].
3. Evaluation models and metrics
𝑃(𝑢) = [𝐾 × 𝑃𝑚𝑎𝑥𝑐𝑝𝑢 + (1 − 𝐾) × 𝑃𝑚𝑎𝑥𝑐𝑝𝑢 × 𝑢]
3.1. Power model
+ [𝐾 × 𝑃𝑚𝑎𝑥𝑚𝑒𝑚𝑜𝑟𝑦 + (1 − 𝐾) × 𝑃𝑚𝑎𝑥𝑚𝑒𝑚𝑜𝑟𝑦 × 𝑢], (3)

Several studies [4,6,13,25] mainly target CPU-intensive tasks to where 𝑃𝑚𝑎𝑥𝑐𝑝𝑢 is the maximum power consumption of a host (tak-
model energy saving in a cloud data center. Memory, networking, ing into account the computationally intensive tasks performed on
bandwidth, cooling systems, storage system, and other specialized com- the CPUs) when it is fully utilized [37,38]. 𝑃𝑚𝑎𝑥𝑚𝑒𝑚𝑜𝑟𝑦 is the max-
puting devices such as GPUs have significant energy requirements in imum power consumption of a host machine (taking into account
CC data centers. Currently, our study targets the most energy-intensive the memory-intensive tasks performed in memory) when it is fully
resources in a cloud data center, namely CPU and memory. CPU utilized [39,40]. There are several studies [37–40] show that a host
and memory consume almost 79% of the power in the cloud data machine, although underutilized, also consumes a significant amount
center [17], as shown in Fig. 1. Therefore, the proposed approach of energy, about 70% compared to a busy host machine. Therefore,
targets CPU and memory most energy-intensive resources in a cloud. the energy model [4] used represents this fact with the term K. The

4
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

remaining host machines (machines that are not idle) are represented
by the term 1-K. The notation u represents the current CPU and memory
utilization of an idle host machine.
However, energy utilization may vary (i.e., increase or decrease) as
CPU and the memory usage of the busy host machines vary. Therefore,
the model emphasizes that CPU and memory utilization is a function of
time and is represented as the term 𝑢 = (𝑡). The busy host machines have
different energy requirements at various execution times, depending on
the percentage of CPU used during each execution. Therefore, the total
energy represented by the term E consumption of a host machine can
be interpreted by the integral power consumption function over some
time, as shown in Eq. (4) [28] of the model used.
∞
𝐸= 𝑃 (𝑢(𝑡))𝑑𝑡 (4)
∫𝑡

3.2. SLA violations

SLA is an agreement between cloud providers and users. This agree-

ment specifies the requirements, price, and QoS between the two
parties [41]. Meeting the quality of service requirements is essential for
cloud computing environments. These are usually formalized in terms
of an SLA that can determine properties such as minimum throughput
or maximum response time provided by the deployed system. The follow-
ing two matrices are used to measure the SLA level in the Infrastructure
as a service cloud model [28]:
(1): As shown in Eq. (5) [42], when an active host is being utilized
100%, then Service Level Agreement Violations Time per Active Host Fig. 2. Cloud computing system architecture [17].
(SLATAH) can be represented as:

1 ∑ 𝑇𝑠𝑖
𝑁
𝑆𝐿𝐴𝑇 𝐴𝐻 = × (5) 4. Proposed energy efficiency heuristic using VM consolidation
𝑁 𝑖=1 𝑇𝑎𝑖 (EEHVMC)
One of the concerning aspects of SLATAH is that when a host serv-
ing an application is fully utilized (i.e., 100%), it limits the application The proposed technique is based on VM consolidation and place-
performance [43]. The Eq. (5) [42] uses N as the number of hosts, 𝑇𝑠𝑖 ment heuristic to conserve energy. Like a typical heuristic based mecha-
is the total time that the host machine i experienced full utilization nism, the proposed Energy Efficiency Heuristic using VM Consolidation
(EEHVMC) employ various methods to produce solutions in a reason-
(i.e., 100%) leading to SLAV, 𝑇𝑎𝑖 is the total time that the host i is served
able time. The EEHVMC approach is designed generically and suits the
in the active state (i.e., serving VMs).
various classical cloud computing framework and the specialized cloud
(2): The Performance Degradation caused by the VM Migration
platforms such as hadoop, spark, etc.
(PDM) as shown in Eq. (6) [42] can be formulated as follows:
Cloud users can access applications anytime, anywhere via the
1 ∑
𝑀
𝐶𝑑𝑗 Internet. System Access Layer acts as an interface between consumers
𝑃 𝐷𝑀 = × (6) and the cloud infrastructure. Energy-Aware Dynamic VM Consolida-
𝑀 𝑗=1 𝐶𝑟𝑗
tion moves VM from one physical host to another host to minimize
M is the number of VMs; 𝐶𝑑𝑗 is the estimate of Performance Degrada- power consumption. Multiple VM can fulfill accepted requests on a
tion of VM j due to migrations (PDM), and 𝐶𝑟𝑗 is the total CPU capacity single machine and dynamically power on and off. Physical Machines
requested by the VM j during its lifetime. The value of 𝐶𝑑𝑗 during the create VM resources using hardware infrastructure to meet service
experiments was estimated to be about 10% of the CPU workload in requests.
Millions Instructions Per Second (MIPS) estimated (during all migra- The EEHVMC approach uses CPU and memory to reduce power
tions of the VM j). Both the SLATAH and PDM metrics are independent consumption and SLAV. By defining two thresholds, 𝑇ℎ𝑖𝑔ℎ and 𝑇𝑙𝑜𝑤 , the
and equally important to characterize the level of SLAV. Therefore, hosts in the data centers classify into three main classes; Host Over-
in this study, we propose a hybrid metric that includes both VMM Loaded (HOL), Host Medium-Loaded (HML), and Host Under-Loaded
and performance degradation as a result of host overloading [44]. The (HUL). First, CPU and memory utilization of the data center’s host
combined metric was presented as service level agreement violations, compared with the defined threshold. If 𝐶𝑈𝐻𝐼 ≥ 𝑇ℎ𝑖𝑔ℎ ∥ 𝑀𝑈𝐻𝐼 ≥ 𝑇ℎ𝑖𝑔ℎ ,
which is calculated as shown in Eq. (7) [28]. then the host is overloaded. Some of the VMs on the HOL should
migrate to the HML to reduce power consumption. If 𝑇ℎ𝑖𝑔ℎ ≤ 𝐶𝑈𝐻𝐼 ≥
𝑆𝐿𝐴𝑉 = 𝑆𝐿𝐴𝑇 𝐴𝐻 × 𝑃 𝐷𝑀 (7) 𝑇𝑙𝑜𝑤 ∥ 𝑇ℎ𝑖𝑔ℎ ≤ 𝑀𝑈𝐻𝐼 ≥ 𝑇𝑙𝑜𝑤 then the hosts are medium loaded and
all VMs remain unchanged. In HUL, the proposed technique collects all
3.3. Performance VM and assigns them to HML to reduce the number of active hosts and
put the remaining inactive hosts to sleep, as shown in Fig. 2.
By comparing the efficiency of the algorithms with the literature,
a new metric can be defined by calculating the product of energy 4.1. An adaptive utilization threshold
consumption along the SLAV. Energy (E) and SLAV, represented in
Eq. (8) [28,45]. Thresholds represent dynamic values affected by the computing
environment (resources in the cloud data center). The inter-quartile
𝐸𝑆𝑉 = 𝐸 × 𝑆𝐿𝐴𝑉 (8) range (IQR) defines the threshold and splits the data set into quartiles.

5
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Table 2 updated accordingly. Similarly, the memory-intensive VMs migrates

EEHVMC flow chart abbreviations.
to HML, which consumes fewer memory-related resources. The host
Variables Description
machine utilization in terms of memory usage is updated for both host
HostList Total number of Host machines involved in the migration, as shown in Fig. 3.
VMList Total number of VM
𝐶𝑈𝐻𝐼 CPU Utilization of 𝑖th Host
𝑀𝑈𝐻𝐼 Memory Utilization of 𝑖th Host
𝐶𝑈𝑣𝑖 CPU Utilization of 𝑖th VM 4.4. Host medium-loaded
𝑀𝑈𝑣𝑖 Memory Utilization of 𝑖th VM
𝑇ℎ𝑖𝑔ℎ High Threshold
𝑇𝑙𝑜𝑤 Low Threshold If 𝑇ℎ𝑖𝑔ℎ ≤ 𝐶𝑈𝐻𝐼 ≥ 𝑇𝑙𝑜𝑤 ∥ 𝑇ℎ𝑖𝑔ℎ ≤ 𝑀𝑈𝐻𝐼 ≥ 𝑇𝑙𝑜𝑤 , then the host is
HOL Host Over-Loaded medium loaded, and all virtual machines remain unchanged.
HML Host Medium-Loaded
HUL Host Under-Loaded
VM𝑖 First VM on a certain Host 4.5. Host under-loaded
VM𝑀 Last VM on a certain Host
CRU𝐻 Current Resource Utilization of Host
LRU𝐻 Less Resource Utilization of Host As shown in Fig. 3, Host Under-Loaded (HUL) moves all VMs to
medium-loaded hosts to reduce the number of active hosts, and inactive
hosts are forced to low-power mode to reduce energy consumption.
The difference between a data set’s upper and lower quartile is the
following step to determine the interquartile range. First, arrange the
4.6. Virtual machine selection and placement
data in ascending order. The second step is calculating the ordered set’s
median (Q2). The third step is to separate the data in half and find
the median of the first half of the ordered set (lower quartile Q1) and In this study, we assume that the workload can be CPU or memory
the median of the second half (upper quartile Q3). The final step is intensive. A task is CPU intensive if its completion depends mainly
𝐼𝑄𝑅 = 𝑈 𝑝𝑝𝑒𝑟𝑄𝑢𝑎𝑟𝑡𝑖𝑙𝑒 − 𝐿𝑜𝑤𝑒𝑟𝑄𝑢𝑎𝑟𝑡𝑖𝑙𝑒. We present a strategy that uses on the use of CPU resources, while a task that spends most of its
two thresholds: an upper threshold and a lower threshold. The lower time interacting with memory (i.e., spends most of its time performing
threshold calculates using the median of the first half of the ordered load/store operations) is said to be memory intensive. In the Scheduling
dataset (host utilization). In contrast, the upper threshold calculates model, the earliest finish time of all most significant jobs determines
using the median of the second half of the ordered data set (host by considering all VMs in the first stage. The second stage selects the
utilization). For example, the proposed approach uses two thresholds VM with the maximum CPU or memory utilization and assigns it to
𝑇ℎ𝑖𝑔ℎ and 𝑇𝑙𝑜𝑤 . When a host observes the CPU or memory utilization, the concerned VM in the host medium load. The ready time of virtual
it compares them to the defined thresholds for that particular data
machines updates after each schedule, and this process repeats until
center. If the threshold 𝑇ℎ𝑖𝑔ℎ exceeds, the proposed algorithm assumes
all jobs execute successfully. We propose a new VMs selection method
that the current host machine is overloaded, which means that VM
named MRCU (Maximum ratio of CPU utilization to memory utiliza-
migrations from that host machine are required. If the threshold is
tion) to select VMs for migration when CPU-intensive tasks overload
below the threshold 𝑇𝑙𝑜𝑤 , the host machine of a particular data center 𝑢 𝑎𝑛𝑑 𝑀 𝑢
is underloaded, so VMs migrate to other hosts (see Table 2). a host. Let the CPU and memory utilization of i VM by 𝐶𝑣𝑚 𝑣𝑚
respectively. Let CPU and memory utilization of any VM e be referred
by 𝐶𝑣𝑚𝑒 𝑎𝑛𝑑 𝑀 𝑒 respectively. The MRCU technique chooses a VM v
4.2. CPU and memory intensive tasks 𝑣𝑚
from the host for migration if it fits the following criteria:
There are two types of applications that run on virtual machines: 𝑢
𝐶𝑣𝑚 𝑒
𝐶𝑣𝑚
CPU and memory-bound applications. If most of an application’s ex- 𝑢
> 𝑒
(9)
𝑀𝑣𝑚 𝑀𝑣𝑚
ecution time is used for compute-intensive operations, it is called a
Eq. (9) [12] shows that the lower 𝑀𝑣𝑚𝑢 value, the higher 𝐶 𝑢 value,
CPU-intensive application (i.e., the CPU resource is used in most cases). 𝑣𝑚
𝑢 / 𝑀 𝑢 value is. As a result, Eq. (9) chooses the VM with
higher 𝐶𝑣𝑚
The VM with the maximum CPU consumption (due to most computa- 𝑣𝑚
𝑢 / 𝑀 𝑢 value to migrate, because higher CPU usage
the highest 𝐶𝑣𝑚
tionally intensive tasks) is selected for migration. Of these selected VMs, 𝑣𝑚
the VM with the ultimate memory consumption pattern is selected first. indicates more energy consumption. When transferring possible VMs,
Transferring a large amount of memory may consume more resources, the MRCU technique considers both the CPU and memory factors.
but due to the host’s memory source limitation, the slower migration For example, if the server is overloaded with CPU -intensive tasks,
will likely result in more delays. For example, the new task might not the power consumption of CPU (s) will account for most of the total
be processed due to insufficient memory, resulting in a higher rate of energy consumption compared to the other components of the host
system violations. Effectively migrating such a VM also frees up more machine. Therefore, the algorithm selects a VM with the highest CPU
memory on the host, which can use to allocate new incoming tasks. value for migration, since a higher CPU workload means higher power
consumption (the objective is to save energy).
4.3. Host over-loaded
Assuming the tasks are memory intensive, the virtual machine with
the highest memory use selects for migration. Although migrating
If 𝐶𝑈𝐻𝐼 ≥ 𝑇ℎ𝑖𝑔ℎ ∥ 𝑀𝑈𝐻𝐼 ≥ 𝑇ℎ𝑖𝑔ℎ , then the host is overloaded. Each
extensive memory data may consume more resources because the host’s
virtual machine in a host is given CPU (𝐶𝑈𝑣𝑖 ) and memory utilization
(𝑀𝑈𝑣𝑖 ), then it is compared to the defined thresholds, e.g., 𝑇ℎ𝑖𝑔ℎ and memory source is overloaded, delayed migration will likely cause more
𝑇𝑙𝑜𝑤 . If 𝐶𝑈𝑣𝑖 ≥ 𝑇ℎ𝑖𝑔ℎ ∥ 𝑀𝑈𝑣𝑖 ≥ 𝑇ℎ𝑖𝑔ℎ , then the VM is overloaded. significant damage. For example, the new task may not be received
The proposed EEHVMC system detects which type of application due to insufficient memory, increasing the rate of system corruption.
consumes the most power. Suppose most of the power consumption Timely migration of such virtual machines can also free up additional
is related to CPU-bound applications (VMs with high CPU utilization). RAM on the host, which can use for the new task. The scheduling model
In that case, the VMs migrates to the HML that consumes fewer CPU shifts the memory usage of the overloaded host and sends these tasks to
resources to reduce the CPU-intensive load from the overloaded and HML, which uses less memory in HML, and then changes the memory
underloaded hosts. After the VMs migrates, the host workloads are resources in the medium-loaded host.

6
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Fig. 3. Energy efficiency heuristic using vm consolidation.

4.7. EEHVMC algorithm Table 3

EEOVMS algorithms abbreviations.

The proposed methodology reduces both power consumption and Variables Description

Service Level Agreement (SLA). The ‘‘vmlist’’ is a collection of VMs Hostlist Total number of Host
in the physical host and the ‘‘hostlist’’ is a list of hosts in the cloud vmlist Total number of VM
VM𝑖 First VM on a certain Host
data center (see Algorithm 1). Thresholds 𝑇ℎ𝑖𝑔ℎ and 𝑇𝑙𝑜𝑤 have been set, VM𝑀 Last VM on a certain Host
which divide the host into three main classes (see Algorithm 1). 𝑐𝑈𝐻𝐼 CPU Utilization of 𝑖th Host
EEHVMC return MigrationMap scheme in which VM placement 𝑚𝑈𝐻𝐼 Memory Utilization of 𝑖th Host
policy scheme is saving. When the VM migrates, it checks the next VM 𝑐𝑈𝑣𝑗 CPU Utilization of 𝑗th VM
𝑚𝑈𝑣𝑗 Memory Utilization of 𝑗th VM
from HOL (lines 10–17, Algorithm 1).
𝑇ℎ𝑖𝑔ℎ High Threshold
If 𝑇ℎ𝑖𝑔ℎ ≤ 𝑐𝑈𝐻𝑖 ≥ 𝑇𝑙𝑜𝑤 ∥ 𝑇ℎ𝑖𝑔ℎ ≤ 𝑚𝑈𝐻𝑖 ≥ 𝑇𝑙𝑜𝑤 , then it will be 𝑇𝑙𝑜𝑤 Low Threshold
considered as HML. In HML, there is no requirement for any migration. vmOverLoaded Virtual Machine Over-Loaded
All of the VMs on that host are left undisturbed (lines 18–19, Algorithm
1).
If both conditions are not met, the host is classified as underutilized
(lines 20–21, Algorithm 1). A loop is executed in HUL to determine the is a HOL (lines 5–6, Algorithm 1). A loop is made in the ‘‘vmlist’’ (line
CPU and memory usage of this VM (lines 22, Algorithm 1). To save 7) and determines CPU and the memory utilization of that VM (lines
energy wasted by inactive hosts in the HUL (lines 23–25, Algorithm 8, Algorithm 1). It is a VM over-loaded (lines 9, Algorithm 1) if CPU
1), the algorithm collects all VM from the HUL and moves them to the or the memory usage is above the 𝑇ℎ𝑖𝑔ℎ values. A migration scheme is
HML to reduce the number of active hosts and shut down the remaining mapped that sends a VM from HOL to HML.
empty hosts. EEHVMC return MigrationMap scheme in which VM
placement policy scheme is saving. When the VM migrates, it checks
the next virtual machines from HUL (lines 26–29, Algorithm 1) (see 4.8. Algorithm complexity
Table 3).
‘‘Hostlist’’ is a collection of hosts stored in variable m, while vm- We store the host and VM list in a separate data structure that takes
list is a collection of VMs stored in variable n (lines 1–2, Algorithm 𝑂(𝑛𝑙𝑜𝑔𝑛) time. We retrieve the virtual machine from the data structure
1). A loop is executed in the variable ‘‘hostlist’’ (line 3, Algorithm 1) and and check its load for each host, which also requires constant time
determines CPU and the memory usage of that host (lines 4, Algorithm complexity. Then we select the VMs stored in the structure to distribute
1). If CPU or the memory usage is greater than the 𝑇ℎ𝑖𝑔ℎ values, then it the load. This also takes 𝑂(𝑛) time, where n is the number of VMs. So

7
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

input : hostlist, vmlist, 𝑇ℎ𝑖𝑔ℎ , 𝑇𝑙𝑜𝑤 Table 4

Configuration of the simulation environment.
output: migration scheme
1 𝑚 ← 𝑔𝑒𝑡𝐶𝑜𝑢𝑛𝑡(ℎ𝑜𝑠𝑡𝑙𝑖𝑠𝑡); Simulator/version CloudSim version 3.0.2
Datasets Synthetic – I [4], GoCJ [47]
2 𝑛 ← 𝑔𝑒𝑡𝐶𝑜𝑢𝑛𝑡(𝑣𝑚𝑙𝑖𝑠𝑡);
Energy optimize algorithms DVFS [13], IQR_MC [4], IQR_MMT [4], MAD_MC
3 for 𝑖 ← 1 to 𝑚 do
[6], MAD_MMT [6], SABFD [21]
4 𝑐𝑈𝐻𝑖 ← 𝑔𝑒𝑡𝑐𝑈𝐻𝑖 (ℎ𝑜𝑠𝑡) and 𝑚𝑈𝐻𝑖 ← 𝑔𝑒𝑡𝑚𝑈𝐻𝑖 (ℎ𝑜𝑠𝑡); Performance parameters Energy consumption, VMs migration, PDM,
5 if 𝑐𝑈𝐻𝑖 ≥ 𝑇ℎ𝑖𝑔ℎ ||𝑚𝑈𝐻𝑖 ≥ 𝑇ℎ𝑖𝑔ℎ then Average SLA, Execution time
6 ℎ𝑜𝑠𝑡𝑂𝑣𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑 ← 𝑔𝑒𝑡𝐻𝑜𝑠𝑡𝑂𝑣𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑; Total cloud host machines 800
7 for 𝑗 ←1 to 𝑛 do Total Virtual Machines 800 heterogeneous VMs
8 𝑐𝑈𝑣𝑗 ← 𝑔𝑒𝑡𝑐𝑈𝑣𝑗 (𝑣𝑚) and 𝑚𝑈𝑣𝑗 ← 𝑔𝑒𝑡𝑚𝑈𝑣𝑗 (𝑣𝑚); Total simulation limit 4800 s
9 if 𝑐𝑈𝑣𝑗 ≥ 𝑇ℎ𝑖𝑔ℎ ||𝑚𝑈𝑣𝑗 ≥ 𝑇ℎ𝑖𝑔ℎ then
10 𝑣𝑚𝑂𝑣𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑 ← 𝑔𝑒𝑡𝑉 𝑚𝑂𝑣𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑;
Table 5
11 𝑣𝑚𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒 ← 𝑔𝑒𝑡𝑉 𝑚𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒(ℎ𝑜𝑠𝑡𝑂𝑣𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑);
Workload characteristics.
12 𝑚𝑖𝑔𝑟𝑎𝑡𝑖𝑜𝑛𝑀𝑎𝑝 ←
Date Number of virtual machines
𝑔𝑒𝑡𝑁𝑒𝑤𝑉 𝑚𝑃 𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡(𝑣𝑚𝑠𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒);
13 return 𝑚𝑖𝑔𝑟𝑎𝑡𝑖𝑜𝑛𝑀𝑎𝑝; 03/March/2011 1052
06/March/2011 898
14 else
09/March/2011 1061
15 𝑣𝑚++; 22/March/2011 1516
16 end 25/March/2011 1078
17 end 03/April/2011 1463
18 else if 𝑇ℎ𝑖𝑔ℎ ≤ 𝑐𝑈𝐻𝑖 ≥ 𝑇𝑙𝑜𝑤 ||𝑇ℎ𝑖𝑔ℎ ≤ 𝑚𝑈𝐻𝑖 ≥ 𝑇𝑙𝑜𝑤 then 09/April/2011 1358
19 ℎ𝑜𝑠𝑡𝑀𝑒𝑑𝑖𝑢𝑚𝐿𝑜𝑎𝑑𝑒𝑑 ← 𝑎𝑑𝑑𝑇 𝑜𝐻𝑜𝑠𝑡𝑀𝑒𝑑𝑖𝑢𝑚𝐿𝑜𝑎𝑑𝑒𝑑; 11/April/2011 1233
12/April/2011 1054
20 else
20/April/2011 1033
21 ℎ𝑜𝑠𝑡𝑈 𝑛𝑑𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑 ← 𝑔𝑒𝑡𝐻𝑜𝑠𝑡𝑈 𝑛𝑑𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑;
22 for 𝑘 ←1 to 𝑛 do
23 𝑐𝑈𝑣𝑘 ← 𝑔𝑒𝑡𝑐𝑈𝑣𝑘 (𝑣𝑚) and 𝑚𝑈𝑣𝑘 ← 𝑔𝑒𝑡𝑚𝑈𝑣𝑘 (𝑣𝑚);
24 𝑣𝑚𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒 ← 𝑔𝑒𝑡𝑉 𝑚𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒(ℎ𝑜𝑠𝑡𝑈 𝑛𝑑𝑒𝑟𝐿𝑜𝑎𝑑𝑒𝑑); 5.1. Experimental setup
25 𝑚𝑖𝑔𝑟𝑎𝑡𝑖𝑜𝑛𝑀𝑎𝑝 ← 𝑔𝑒𝑡𝑁𝑒𝑤𝑉 𝑚𝑃 𝑙𝑎𝑐𝑒𝑚𝑒𝑛𝑡(𝑣𝑚𝑠𝑇 𝑜𝑀𝑖𝑔𝑟𝑎𝑡𝑒);
26 return 𝑚𝑖𝑔𝑟𝑎𝑡𝑖𝑜𝑛𝑀𝑎𝑝;
27 end CloudSim Toolkit [42], a novel simulation framework, was selected
28 end as the simulation platform for the CC environment. In addition, using
29 end CloudSim offers two advantages: It supports on-demand resource pro-
Algorithm 1: Energy efficiency heuristic using vm consolidation visioning and management, as well as virtual environment modeling
and energy-aware simulation, including the ability to simulate service
applications with dynamic workloads [48,49]. Table 4 shows the setup
details for the simulation environment used. The experiments are con-
we check each host from the list of m number of host machines, and ducted with 800 VMs hosted on 800 host machines within a cloud
then select a VM for each host that exceeds the threshold of n number better to understand the concept of VMM and energy efficiency.
of VMs. Thus, the other load steps consume the constant time; hence,
the total complexity of the proposed approach is 𝑂(𝑛𝑙𝑜𝑔𝑛) + 𝑂(𝑚 × 𝑛),
which can be written as 𝑂(𝑚 × 𝑛), where m is the number of hosts and n 5.1.1. Realistic dataset based on PlanetLab
is the number of VMs. If 𝑚 = 𝑛, then the time complexity is 𝑂(𝑛2 ). The PlanetLab dataset [4] enables the behavior modeling of Cloud
system components, including VMs, data centers, and resource provi-
sioning policies.
5. Experimental results and analysis
1. Hosts Features: The extensions used in the current study were
This paper’s approaches belong to heuristics; A heuristic technique built using the CloudSim toolkit (version 3.0.3). The cloud con-
is an approach to problem-solving that uses a practical method or sisted of heterogeneous hosts, some HP ProLiant G4 hosts, and
various shortcuts to produce solutions. State-of-the-art (SOTA) Deep the others HP ProLiant G5 hosts. HP ProLiant G4 hosts consist of
Neural Networks (DNNs) are the best patterns you can use for any 1860 MIPS, 2 CPU cores, and 4 GB of RAM, whereas HP ProLiant
specific task. A DNN can recognize SOTA based on its speed, precision, G5 hosts consist of 2860 MIPS, 2 CPU cores, and 4 Gb of RAM
or interest metric. It is costly to train due to complex data models. based on [4].
Furthermore, deep learning requires expensive GPUs and hundreds of 2. Virtual Machines Features: The functionalities of the VMs are
machines. It expands the users’ cost as the number of VMs migration based on Amazon EC2 instance models [50]. CPU high, large,
rises [46]. The proposed approach EEHVMC minimizes the number of small, and micro instances are the four categories of VMs used.
VMs migration, so the SLAV and cost decrease. These approaches are CPU high instance consists of 2500 MIPS and 870 GB of RAM,
part of the cloud simulator: DVFS, IQR_MC, IQR_MMT, MAD_MC, and whereas CPU large instance consists of 2000 MIPS and 1740 GB
MAD_MMT. The common thing in those approaches is that they only of RAM. Similarly, CPU small instance consists of 1000 MIPS and
use CPU utilization and neglect other parts of power consumption. 1740 GB of RAM, whereas the CPU micro instance consists of
The proposed heuristic approach uses a cloud simulator to minimize 500 MIPS and 613 GB of RAM based on [4].
power consumption. Hosts, MIPS, Cores, RAM, and other aspects of 3. Workload Characteristics: The experiments were carefully con-
the parameter are the same as cloud simulator, and then we compare ducted using an existing system’s workload traces to produce
them with related techniques. The Energy Efficiency Heuristic Virtual more realistic results. Data for the tests came from Planet Lab’s
Machine Consolidation (EEHVMC) testing results were compared to the CoMon project [51]. In addition, thousands of VMs from servers
other VM consolidation strategies such as Energy optimize algorithms & at over 500 sites worldwide use data on CPU consumption.
DVFS [13], IQR_MC [4], IQR_MMT [4], MAD_MC [6], MAD_MMT [6], Table 5 lists the characteristics of the dataset in detail [4].
and SABFD [21].

8
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Table 6 5.3. Realistic PlanetLab dataset

Google cloud jobs dataset.
Type of jobs MI % of Jobs The characteristic of hosts and VMs are specified in host and virtual
Type of jobs MI % of Jobs machines features part. Fig. 4 shows that EEHVMC (24.34 kWh) has the
Small 15k–55k 20
lowest energy usage, followed by DVFS (29.79 kWh), IQR_MC (27.06
Medium 59k–99k 40
Large 101k–135k 30
kWh), IQR_MMT (27.29 kWh), MAD_MC (26.49 kWh), MAD_MMT
Extra-large 150k–337.5k 4 (26.64 kWh), and SABFD (28.38 kWh). The percentage improvement
Huge 525k–900k 6 of the proposed approach EEHVMC compared to the other approaches
followed by DVFS (22.39%), IQR_MC (11.18%), IQR_MMT (12.12%),
MAD_MC (8.83%), MAD_MMT (9.45%), and SABFD (16.50%).
The Dynamic Virtual Machine Consolidation (DVMC) method shows
5.1.2. Gocj dataset
that as the number of VM migrations increases, so does the cost.
The GoCJ dataset [47] contains a variety of jobs. Table 6 shows the
Consequently, we need to reduce the number of VM migrations. DVFS
different categories of tasks in the GoCJ dataset i.e., small, medium, big,
approach automatically changes the voltage and frequency to reduce
extra-large, and huge. It also shows the characteristics and specifications
processor heat generation and lower power consumption. DVFS is a
of the host and VM that run in the GoCJ dataset.
frequency-aware model capable of determining the best frequency ratio
for reducing processor energy usage. DVFS approach does not include
5.1.3. Benchmark heuristics the VM migration process; therefore, as shown in Figs. 5 and 10, their
The following is an overview of the other prominent approaches result is 0, but the power consumption is too much compared to other
used for experimental evaluation. approaches as shown in Figs. 4 and 9. As shown in Fig. 5, the EEHVMC
• DVFS [13]: Dynamic Voltage Frequency Scaling is a method to strategy results in fewer migrations compared to the other related
reduce power consumption by automatically changing the fre- approaches, which are based on migration-based mechanisms. Regard-
quency and voltage; ing VM migration, our approach is better than IQR_MC by 13.13%,
IQR_MMT by 9.67%, MAD_MC by 8.38%, MAD_MMT by 5.98%, and
• IQR_MC [4]: InterQuartile Range is utilized to detect overloading
SABFD by 16.12%.
on the host, and the Maximum Correlation policy is utilized for
Performance Degradation caused by VMs migration (PDM) increases
migration;
if the number of VMs migration rises. Therefore, we need to care about
• IQR_MMT [4]: InterQuartile Range is utilized to detect overloading
it that the migration will remain low, so PDM remains to decrease;
on the host, and the Minimum Migration Time policy is utilized for
the DVFS approach is not part of VM migration, so PDM remains 0,
migration;
as shown in Figs. 6 and 11. The proposed EEHVMC approach reduces
• MAD_MC [6]: Median Absolute Deviation is utilized to indicate
the number of live migrations while lowering PDM. Fig. 6 shows that
overload on the host, and the Maximum Correlation policy is
EEHVMC has the least performance degradation (i.e., 0.14) compared
utilized for migration [44].
with IQR_MC (0.18), IQR_MMT (0.17), MAD_ MC (0.17), MAD_MMT
• MAD_MMT [6]: Median Absolute Deviation is used to identify
(0.16), and SABFD (0.19). Our approach as per PDM is up by 28.57%
overloading on the host, and the Minimum Migration Time policy
than IQR_MC, 21.43% than IQR_MMT, 21.43% than MAD_MC, 14.29%
is utilized for migration purpose;
than MAD_MMT and 35.71% than SABFD.
• SABFD [21]: This method selects a VM for migration that has the Service Level Agreement Violations increase if one parameter, PDM
highest CPU usage and is placed in the host with the fewest MIPS. or SLATAH, grows. Compared to other methods, DVFS does not require
any VMs migration, so as a result,
5.1.4. Performance parameters PDM remains 0, so it does not affect any SLAV process. As shown in
The following performance metrics used to evaluate the outcomes Figs. 7 and 12, the development of DVFS is 0 compared to the other ap-
of the proposed approach: proaches. Fig. 7 illustrates that EEHVMC reduces SLA violations when
compared to other methods. The figure clearly shows that EEHVMC
• Energy Consumption kWh: Data centers are huge buildings
has the lowest SLA violations (i.e., 9.01%) compared with IQR_MC
consisting of many physical machines that store and retrieve data.
(10.23%), IQR_MMT (10.12%), MAD_MC (10.1%), MAD_MMT (10%),
A data center consumes over 91 billion kWh of electricity [5];
and SABFD (10.89%).
• VM Migrations: Transferring a Virtual Machine (VM) from one
DVFS takes less time than other approaches, which is why VM
physical host to another. VM Manager keeps track of VMs in the
migration executes early. But the difference is it takes more power
cloud and their availability;
consumption, as shown in Figs. 4 and 9. Fig. 8 shows that the proposed
• Performance Degradation caused by the Migration: PDM
approach EEHVMC requires less execution time compared to other
refers to the general performance degradation that occurs in VMs
approaches. As per Execution time, our approach is better by 5.284%
as a result of live migrations;
than IQR_MC, 5.186% than IQR_MMT, 4.228% than MAD_MC, 2.700%
• SLA Violations: The final SLAV simplify by lowering one of the than MAD_MMT, and 6.948% than SABFD.
parameters, PDM, or SLATAH;
• Execution Time: Execution time is when a task execution takes 5.4. GoCJ dataset
from start to finish.
The GoCJ dataset comprises different task sizes such as small,
5.2. Experimental results medium, large, extra large, and huge generated using dynamic task length
thresholds. The GoCJ task types [47] are listed in Table 6.
The simulations compare DVFS [13], SABFD [21], and energy- The EEHVMC uses the least amount of power (i.e., 16.23 kWh)
aware strategies (e.g. IQR, MAD) [4,6]. The proposed research is com- followed by DVFS (20.75 kWh), IQR_MC (18.9 kWh), IQR_MMT (18.79
pared with the most popular algorithms, such as IQR, MAD and VM kWh), MAD_MC (17.58 kWh), MAD_MMT (17.23 kWh), and SABFD
selection techniques, MC [4] and MMT [6]. The proposed algorithm, (19.23 kWh), as shown in Fig. 9. With regards to energy consumption,
EEHVMC, checks and calculates the host’s threshold based on CPU and our approach is higher than DVFS by 27.85%, IQR_MC by 16.45%,
memory usage. Compared to the previous techniques, the proposed IQR_MMT by 15.77%, MAD_MC by 8.32%, MAD_MMT by 6.16%, and
algorithm EEHVMC consumes the least amount of energy. SABFD by 18.48%.

9
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Fig. 4. Energy consumption — synthetic dataset.

Fig. 5. Virtual machine migrations — synthetic dataset.

Fig. 6. Performance degradation — synthetic dataset.

Fig. 7. Service level agreement violations — synthetic dataset.

10
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Fig. 8. Execution time — synthetic dataset.

Fig. 9. Energy consumption — GoCJ dataset.

Fig. 10. Virtual machine migrations — GoCJ dataset.

The DVMC technique shows that as the number of VM migrations MAD_MMT (0.16), and SABFD (0.19), as shown in Fig. 11. In connec-
increases, the cost also increases. As shown in Figs. 5 and 10, DVFS tion with PDM our approach is better than 20% by IQR_MC, 13.33%
approach does not include the VM migration process; therefore, their by IQR_MMT, 6.67% by MAD_MC, 6.67% by MAD_MMT and 26.67%
result is 0. EEHVMC requires fewer VM migrations than the other by SABFD.
approaches, as shown in Fig. 10. SLAV remains 0 in DVFS because there is no involvement of Per-
In respect of VM migration improvement percentage of our ap- formance caused by VM Migration (PDM), as shown in Figs. 7 and
proach is 100% by DVFS, 12.98% by IQR_MC, 11.56% by IQR_MMT, 12. Fig. 12 shows that the energy efficiency heuristic using virtual
7.48% by MAD_MC, 3.57% by MAD_MMT, and 16.54% by SABFD. machine consolidation (10.2%) has the lowest SLA violations compared
PDM will increase if the quantity of VMs migration rises. Therefore, to IQR_MC (10.9%), IQR_MMT (10.8%), MAD_ MC (10.6%), MAD_MMT
we want to care approximately that the migration will continue to (10.6%), and SABFD (10.89%).
be low, so PDM stays decrease; the DVFS technique is not always a Compared to other methods, DVFS takes less time to execute. This is
part of VM migration, so PDM remains 0, as proven in Figs. 6 and because DVFS does not provide for migrations, which results in better
11. Performance degradation is mitigated by EEHVMC reducing the execution performance, but also consumes more energy.
number of live migrations. EEHVMC takes less time to execute compared to the other methods.
EEHVMC suffers the least performance degradation (i.e., 0.15) com- The reason is that less PDM and VMM are required, so it runs faster. As
pared to IQR_MC (0.18), IQR_MMT (0.17), MAD_ MC (0.16 kWh), shown in Fig. 13, the proposed approach takes less time to execute than

11
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

Fig. 11. Performance degradation — GoCJ dataset.

Fig. 12. Service level agreement violations — GoCJ dataset.

Fig. 13. Comparison of execution time.

alternative strategies. Regarding execution time, our approach is up by EEHVMC technique requires fewer VM migrations and saves more
9.94% than DVFS, 10.24% than IQR_MC, 6.71% than IQR_MMT, 4.59% energy compared to the related approaches.
than MAD_MC, 0.84% than MAD_MMT, and 12.52% than SABFD. If the number of Virtual Machine Migration (VMM) increases, Per-
formance Degradation caused by VM migration (PDM) will increase.
5.5. Result and discussion The proposed technique has the most negligible performance degrada-
tion compared to other approaches, as shown in Figs. 6 and 11. The
The fundamental concept behind the proposed technique final SLAV simplifies [4] by lowering one of the parameters, PDM or
‘‘EEHVMC’’ is to classify cloud hosts based on CPU and memory usage. SLATAH.
The host classifies into three main classes based on the two thresholds
𝑆𝐿𝐴𝑉 = 𝑆𝐿𝐴𝑇 𝐴𝐻 × 𝑃 𝐷𝑀 (10)
of CPU and memory usage: HOL, HML, and HUL. EEHVMC has the low-
est energy consumption compared to DVFS, IQR_MC, IQR_MMT, MAD_ Moreover, the proposed approach reduces the frequency of migra-
MC, MAD_MMT, and SABFD, as shown in Figs. 4 and 9. According to tions and PDM, resulting in a low SLAV. EEHVMC has the lowest
the DVMC, the VM migration cost increases with the number of VM SLAV compared to the other techniques, as shown in Figs. 7 and 12.
migrations. Therefore, the technique that requires fewer VM migrations Compared to the other methods, DVFS takes less time to execute.
leads to better computational performance. The results presented in The reason is that there is no VMM or PDM, so it runs faster than
the previous section (e.g., Figs. 5 and 10) show that the proposed the different approaches (mentioned in the previous section). Figs. 8

12
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

and 13 show that the proposed approach takes less time to execute References
than the alternative techniques such as IQR_MC, IQR_MMT, MAD_ MC,
MAD_MMT, and SABFD. [1] Lin JCW, Djenouri Y, Srivastava G, Li Y, Yu PS. Scalable mining of high-utility
sequential patterns with three-tier MapReduce model. ACM Trans Knowl Discov
Data 2021;16(3):1–26.
[2] Wu JMT, Srivastava G, Wei M, Yun U, Lin JCW. Fuzzy high-utility pattern mining
6. Conclusions
in parallel and distributed hadoop framework. Inform Sci 2021;553:31–48.
[3] Beloglazov A, Buyya R. Optimal online deterministic algorithms and adaptive
heuristics for energy and performance efficient dynamic consolidation of virtual
People and businesses worldwide use cloud computing to manage
machines in cloud data centers. Concurr Comput: Pract Exper 2012;24:1397–420;
and store data over the Internet. As cloud computing data centers
have become more prevalent, the power consumption of the host and Lin JCW, Djenouri Y, Srivastava G. Efficient closed high-utility pattern fusion
other infrastructures has increased. There is a need to reduce power model in large-scale databases. Inf Fusion 2021;76:122–32.
[4] Beloglazov A, Buyya R. Optimal online deterministic algorithms and adaptive
consumption without compromising the Quality of Service. This paper
heuristics for energy and performance efficient dynamic consolidation of virtual
presents the Energy Efficiency Heuristic with Virtual Machine Consol- machines in cloud data centers. Concurr Comput: Pract Exper 2012;24:1397–420.
idation (EEHVMC), which reduces power consumption while reducing [5] Andrae AS, Edler T. On global electricity usage of communication technology:
SLA violations. The host classifies into three main categories based on trends to 2030. Challenges 2015;6:117–57.
[6] Fard SYZ, Ahmadi MR, Adabi S. A dynamic VM consolidation technique for QoS
the two thresholds: Host Over-Loaded, Host Medium-Loaded, and Host
and energy consumption in cloud environment. J Supercomput 2017;73:4347–68.
Under-Loaded. Over-loaded hosts consume more energy than other hosts [7] Panda SK, Jana PK. An energy-efficient task scheduling algorithm for
in the data center, so specific virtual machines must move from over- heterogeneous cloud computing systems. Cluster Comput 2019;22:509–27.
loaded to medium-loaded hosts. All VMs that move from under-loaded [8] Mhouti AEl, Erradi M, Nasseh A. Using cloud computing services in e-learning
process: Benefits and challenges. Educ Inf Technol 2018;23:893–909.
to medium-loaded and empty hosts are put into power-saving mode to
[9] Choudhary A, Rana S, Matahai KJ. A critical analysis of energy efficient
reduce the number of active hosts. Compared to state-of-the-art, the virtual machine placement techniques and its heuristic in a cloud computing
EEHVMC process minimizes power consumption and SLA violations. environment. Procedia Comput Sci 2016;78:132–8.
CPU and memory are the hosts’ components used to consume power. [10] Masdari M, Nabavi SS, Ahmadi V. An overview of virtual machine placement
schemes in cloud computing. J Netw Comput Appl 2016;66:106–27.
Still, other parts are used to consume energy, like network bandwidth,
[11] Hussain A, Aleem M, Khan A, Iqbal MA, Islam MA. RALBA: a computation-aware
storage, cooling overhead, and power overhead. We will minimize load balancing scheduler for cloud computing. Cluster Comput 2018;21:1667–80.
power consumption by considering these parts’ network bandwidth, [12] Zhou Z, Abawajy J, Chowdhury M, Hu Z, Li K, Cheng H, Alelaiwi AA, Li F.
GPU, storage, cooling overhead, and power overhead in future work. Minimizing SLA violation and power consumption in cloud data centers using
adaptive energy-aware algorithms. Future Gener Comput Syst 2018;86:836–50.
The proposed approach only finds CPU and memory-intensive tasks in
[13] Stavrinides GL, Karatza HD. An energy-efficient, QoS-aware and cost-effective
the virtual machine. There are other tasks as well which are part of scheduling approach for real-time workflow applications in cloud computing
virtual machines, like I/O intensive tasks. Suppose the I/O intensive systems utilizing DVFS and approximate computations. Future Gener Comput
tasks consume more power than the other parts, mainly CPU and Syst 2019;96:216–26.
[14] Khalid YN, Aleem M, Ahmed U, Islam MA, Iqbal MA. Troodon: A machine-
memory. In that case, it will not consider this task. We plan to consider
learning based load-balancing application scheduler for CPU–GPU system. J
I/O intensive tasks to reduce power consumption at the data center Parallel Distrib Comput 2019;132:79–94.
level. Hadoop and Spark are models that use in CC. EEHVMC does not [15] Zhou Z, Yu J, Li F, Yang F. Virtual machine migration algorithm for en-
follow any of these models. In the future, we can use this approach in ergy efficiency heuristic in cloud computing. Concurr Comput: Pract Exper
2018;30:e4942.
Hadoop or the Spark model.
[16] Bui DM, Yoon Y, Huh EN, Jun S, Lee S. Energy efficiency for cloud computing
system based on predictive heuristic. J Parallel Distrib Comput 2017;102:103–14.
[17] Barroso LA, Hölzle U, Ranganathan P. The datacenter as a computer: Designing
CRediT authorship contribution statement warehouse-scale machines. Synth Lect Comput Archit 2018;13:i–189.
[18] Mezmaz M, Melab N, Kessaci Y, Lee YC, Talbi EG, Zomaya AY, Tuyttens D. A
parallel bi-objective hybrid metaheuristic for energy-aware scheduling for cloud
Umer Arshad: Conceptualization, Methodology, Writing – origi- computing systems. J Parallel Distrib Comput 2011;71:1497–508.
nal draft. Muhammad Aleem: Writing – review & editing, Supervi- [19] Cheng C, Li J, Wang Y. An energy-saving task scheduling strategy based
sion. Gautam Srivastava: Investigation. Jerry Chun-Wei Lin: Formal on vacation queuing theory in cloud computing. Tsinghua Sci Technol
2015;20:28–39.
analysis, Writing – review & editing.
[20] Hussain A, Aleem M, Islam MA, Iqbal MA. A rigorous evaluation of
state-of-the-art scheduling algorithms for cloud computing. IEEE Access
2018;6:75033–47.
Declaration of competing interest [21] Wang H, Tianfield H. Energy-aware dynamic virtual machine consolidation for
cloud datacenters. IEEE Access 2018;6:15259–73.
[22] Ilager S, Ramamohanarao K, Buyya R. ETAS: Energy and thermal-aware dynamic
The authors declare the following financial interests/personal re- virtual machine consolidation in cloud data center with proactive hotspot
lationships which may be considered as potential competing inter- mitigation. Concurr Comput: Pract Exper 2019;31:e5221.
ests: Jerry Chun-Wei Lin reports a relationship with Western Norway [23] Lin W, Liang C, Wang JZ, Buyya R. Bandwidth-aware divisible task scheduling
for cloud computing. Softw - Pract Exp 2014;44:163–74.
University of Applied Sciences that includes: employment.
[24] Gawali MB, Shinde SK. Task scheduling and resource allocation in cloud
computing using a heuristic approach. J Cloud Comput 2018;7:1–16.
[25] Mohamadi Bahram Abadi R, Rahmani AM, Alizadeh SH. Server consolidation
Data availability techniques in virtualized data centers of cloud environments: A systematic
literature review. Softw - Pract Exp 2018;48:1688–726.
[26] Shukla K, Nefti-Meziani S, Davis S. A heuristic approach on predictive
Data will be made available on request.
maintenance techniques: Limitations and scope. Adv Mech Eng 2022;14:6.
[27] Sayadnavard MH, Haghighat AT, Rahmani. A reliable energy-aware approach
for dynamic virtual machine consolidation in cloud data centers. J Supercomput
Acknowledgments 2019;75:2126–47.
[28] Buyya R, Beloglazov A, Abawajy J. Energy-efficient management of data center
resources for cloud computing: a vision, architectural elements, and open
This paper is partially supported by the Western Norway University challenges. In: Parallel and distributed processing techniques and applications,
of Applied Sciences, Bergen, Norway. Vol. 106. 2010, p. 116–24.

13
U. Arshad et al. Renewable and Sustainable Energy Reviews 167 (2022) 112782

[29] Liu Y, Sun X, Wei W, Jing W. Enhancing energy-efficient and QoS dynamic [41] Piraghaj SF, Dastjerdi AV, Calheiros RN, Buyya R. ContainerCloudSim: An
virtual machine consolidation method in cloud environment. IEEE Access environment for modeling and simulation of containers in cloud data centers.
2018;6:31224–35. Softw - Pract Exp 2017;47:505–21.
[30] Uddin M, Hamdi M, Alghamdi A, Alrizq M, Memon MS, Abdelhaq M, Al- [42] Calheiros RN, Ranjan R, Beloglazov A, De Rose CA, Buyya R. CloudSim: a toolkit
saqour R. Server consolidation: A technique to enhance cloud data center
for modeling and simulation of cloud computing environments and evaluation
power efficiency and overall cost of ownership. Int J Distrib Sens Netw
of resource provisioning algorithms. Softw - Pract Exp 2011;41:23–50.
2021;17:1550147721997218.
[43] Hussain A, Aleem M, Iqbal MA, Islam MA. Investigation of cloud schedul-
[31] Khan MA. An efficient energy-aware approach for dynamic VM consolidation on
cloud platforms. Cluster Comput 2021;21:1–18. ing algorithms for resource utilization using cloudsim. Comput Inform
[32] Yadav R, Zhang W, Li K, Liu C, Laghari AA. Managing overloaded hosts for 2019;38:525–54.
energy-efficiency in cloud data centers. Cluster Comput 2021;33:1–15. [44] Alsbatin L, Öz G, Ulusoy AH. A novel physical machine overload detection
[33] Mao J, Peng X, Cao T, Bhattacharya T, Qin X. A frequency-aware management algorithm combined with quiescing for dynamic virtual machine consolidation
strategy for virtual machines in DVFS-enabled clouds. Sustain Comput: Inform in cloud data centers. Int Arab J Inf Technol 2020;17:358–66.
Syst 2022;33:100643. [45] Khan AA, Zakarya M, Khan R. Energy-aware dynamic resource management in
[34] Kanagasubaraja S, Hema M, Valarmathi K, Kumar N, Kumar BPM, Balaji N. elastic cloud datacenters. Simul Model Pract Theory 2019;92:82–99.
Energy optimization algorithm to reduce power consumption in cloud data
[46] Saxena D, Singh AK. OFP-TM: an online VM failure prediction and toler-
center. In: International conference on advances in computing, communication
ance model towards high availability of cloud computing environments. J
and applied informatics: informatics and systems, Vol. 66. 2022, p. 1–8.
[35] Zolfaghari R, Rahmani AM. Virtual machine consolidation in cloud computing Supercomput 2022;78:8003–24.
systems: Challenges and future trends. Wirel Pers Commun 2020;115:2289–326. [47] Hussain A, Aleem M. GoCJ: Google cloud jobs dataset for distributed and cloud
[36] Gu C, Huang H, Jia X. Power metering for virtual machine in cloud computing infrastructures. Data 2018;3:38.
computing-challenges and opportunities. IEEE Access 2014;2:1106–16. [48] Ibrahim M, Iqbal MA, Aleem M, Islam MA. SIM-cumulus: An academic cloud
[37] Gandhi A, Harchol-Balter M, Das R, Lefurgy C. Optimal power allocation in for the provisioning of network-simulation-as-a-service (NSaaS). IEEE Access
server farms. ACM SIGMETRICS Perform Eval Rev 2009;37:157–68. 2018;6:27313–23.
[38] Kusic D, Kephart JO, Hanson JE, Kandasamy N, Jiang G. Power and performance [49] Zolfaghari R, Sahafi A, Rahmani AM, Rezaei R. An energy-aware virtual machines
management of virtualized computing environments via lookahead control.
consolidation method for cloud computing: Simulation and verification. Softw -
Cluster Comput 2009;11:1–15.
Pract Exp 2021;12:157–62.
[39] Raghavendra R, Ranganathan P, Talwar V, Wang Z, Zhu X. No power struggles:
[50] Iqbal MA, Aleem M, Ibrahim M, Anwar S, Islam MA. Amazon cloud comput-
coordinated multi-level power management for the data center. In: Proceedings
of the 13th international conference on Architectural support for programming ing platform EC2 and VANET simulations. Int J Ad Hoc Ubiquitous Comput
languages and operating systems, Vol. 9. 2008, p. 48–59. 2019;30:127–36.
[40] Verma A, Ahuja P, Neogi A. pMapper: power and migration cost aware [51] Shirvani MH, Rahmani AM, Sahafi A. A survey study on virtual machine
application placement in virtualized systems. In: ACM/IFIP/USENIX international migration and server consolidation techniques in DVFS-enabled cloud datacenter:
conference on distributed systems platforms and open distributed processing, Vol. taxonomy and challenges. J King Saud Univ-Comput Inf Sci 2020;32:267–86.
18. 2008, p. 243–64.

14