IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO.

9, SEPTEMBER 2021 6539

EDL-COVID: Ensemble Deep Learning for

COVID-19 Case Detection From Chest
X-Ray Images
Shanjiang Tang , Chunjiang Wang , Jiangtian Nie , Neeraj Kumar , Senior Member, IEEE,
Yang Zhang , Member, IEEE, Zehui Xiong , Member, IEEE, and Ahmed Barnawi

Abstract—Effective screening of COVID-19 cases has model is generated by combining multiple snapshot mod-
been becoming extremely important to mitigate and stop els of COVID-Net, which has pioneered in an open-sourced
the quick spread of the disease during the current period of COVID-19 case detection method with deep neural network
COVID-19 pandemic worldwide. In this article, we consider processed chest X-ray images, by employing a proposed
radiology examination of using chest X-ray images, which weighted averaging ensembling method that is aware of
is among the effective screening approaches for COVID-19 different sensitivities of deep learning models on different
case detection. Given deep learning is an effective tool classes types. Experimental results show that EDL-COVID
and framework for image analysis, there have been lots offers promising results for COVID-19 case detection with
of studies for COVID-19 case detection by training deep an accuracy of 95%, better than COVID-Net of 93.3%.
learning models with X-ray images. Although some of them Index Terms—Covid-19, chest X-ray images, deep learn-
report good prediction results, their proposed deep learn- ing, EDL-COVID, ensemble learning.
ing models might suffer from overfitting, high variance, and
generalization errors caused by noise and a limited number
of datasets. Considering ensemble learning can overcome
I. INTRODUCTION
the shortcomings of deep learning by making predictions HE novel Coronavirus Disease 2019 (COVID-19), as an
with multiple models instead of a single model, we propose
EDL-COVID, an ensemble deep learning model employ-
ing deep learning and ensemble learning. The EDL-COVID
T unprecedented infectious and dangerous disease around
the world, is caused by SARS-CoV-2, a severe acute respiratory
syndrome coronavirus 2, which has not been ever found in
humans before Dec 2019 [1]. There is a rapid person-to-person
Manuscript received November 7, 2020; revised December 23, 2020 coronavirus transmission between two people in close contact
and January 20, 2021; accepted January 20, 2021. Date of publication via aerosols or small droplets created by talking, coughing,
February 8, 2021; date of current version June 16, 2021. This work and sneezing. Once infected, people tend to have the following
was supported in part by the National the Natural Science Foundation
of China under Grant 61972277, in part by Tianjin Natural Science common symptoms after several days, including fever, cough,
Foundation under Grant 18JCZDJC30800, in part by National Natu- taste/small loss, and shortness of breath. As of 10th January
ral Science Foundation of China under Grant 62071343 and Grant 2020, there are still more than 90 million active infected cases,
51609195, and in part by the Open Research Project of the State Key
Laboratory of Industrial Control Technology, Zhejiang University, China and 1.94 million people have died worldwide.
under Grant ICT20044. Paper no. TII-20-5118. (Corresponding author: To stop the fast spread of COVID-19, an important task is to
Jiangtian Nie.) find out infected people via effective screening such that they
Shanjiang Tang and Chunjiang Wang are with the College of Intelli-
gence, and Computing, Tianjin University, Tianjin 300072, China (e-mail: can be isolated and received immediate treatment. So far, the
tashj@tju.edu.cn; tju_wcj@tju.edu.cn). most commonly used screening approach for COVID-19 case
Jiangtian Nie is with the School of Computer Science and Engineer- detection is to take a reverse transcription polymerase chain
ing, Nanyang Technological University, Singapore 639798, Singapore
(e-mail: jnie001@e.ntu.edu.sg). reaction (RT-PCR) test over a sample of nasopharyngeal exudate
Neeraj Kumar is with the Department of Computer Science and collected from suspectable people for the qualitative detection
Engineering, Thapar Institute of Engineering and Technology, Patiala, of nucleic acid from SARS-CoV-2 attributes to its merits as a
Punjab 147004, India, with the School of Computer Science, University
of Petroleum and Energy Studies, Dehradun, Uttarakhand, and also with simple but specific qualitative assay [6], [30]. Although RT-PCR
the Department of Computer Science and Information Engineering, Asia testing has been recognized as a “gold standard” for infected case
University, Taichung 41354, Taiwan (e-mail: neeraj.kumar@thapar.edu). discovery of the disease, there are still several issues about it.
Yang Zhang is with the School of Computer Science and Technol-
ogy, Technology University of Wuhan, Wuhan 430063, China (e-mail: First, the sensitivity of its detection results is highly variable,
yangzhang@whut.edu.cn). which can generate false-negative and false-positive results
Zehui Xiong is with the Pillar of Information Systems Technology according to a recent study by Tahamtan et al. [34]. Second,
and Design, Singapore University of Technology and Design, Singapore
639798, Singapore (e-mail: zehui_xiong@sutd.edu.sg). due to the quick spread of COVID-19, there are not sufficient
Ahmed Barnawi is with the Computing and IT, King Abdulaziz Uni- PCR reagent kits to satisfy the overwhelming screening demand
versity, Jeddah 21589, Saudi Arabia (e-mail: ambarnawi@kau.edu.sa). especially in poor and heavily affected areas [35].
Color versions of one or more figures in this article are available at
https://doi.org/10.1109/TII.2021.3057683. In addition to RT-PCR, radiography examination is an alterna-
Digital Object Identifier 10.1109/TII.2021.3057683 tive effective screening method for fast detection of COVID-19,
© IEEE 2021. This article is free to access and download, along with rights for full text and data mining, re-use and analysis.

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
6540 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 9, SEPTEMBER 2021

where the chest X-ray (CXR) and CT images are performed of a good ensemble deep learning model depends on both the
and analyzed by radiologists to judge whether a suspectable accuracy of each individual model and the diversity among these
person has been infected or not by SARS-CoV-2 [3], [36]. Recent deep learning models [13].
studies have observed the abnormal features in radiography To reduce the computation cost as well as training time,
images of COVID-19 cases and it has been widely used in instead of training multiple different deep neural networks, we
China at the earlier stage of the global outbreak [15], [16], [38]. consider an alternative approach of generating multiple deep
Although CT scan has a higher sensitivity to pulmonary diseases, learning models using a single deep neural networks. Specif-
there are several limitations for its practical uses in COVID-19 ically, we execute a single training run with multiple model
case detection at a larger scale, including nonportability, long- snapshots first, and then ensemble the prediction results for
time scanning, and the risk of exposing the hospital staff. In a final prediction. However, there is a big problem with this
comparison to CT scans, CXR imaging is portable, faster, more approach that the produced multiple models can be quite similar
readily available, and can be performed within an isolated room to each other since they have the same network architecture
while offering an acceptable accuracy in COVID-19 case detec- as well as training initialization, violating the model diversity
tion [29]. Due to these benefits, many recent studies [7], [18], requirement of ensemble learning. Using these similar models
[29], [36] have now focused on CXR image analysis for COVID- tends to result in similar predictions and prediction errors,
19 case detection. Particularly, there are some studies [25] indicating that the combination of these models cannot offer
suggesting to take portable CXR imaging as a reliance method much benefit. To resolve it, one effective approach is to make an
for COVID-19 case detection with the quick spread of the aggressive learning rate change during a single neural network
pandemic. training process, which can force the exploration of different
While CXR imaging is very fast, it needs expert radiologists model weights and produce a diversity of multiple snapshot
to make judgment for COVID-19 case detection manually, models [22].
which requires professional knowledge and is a time-consuming Specifically, our proposed EDL-COVID is based on COVID-
process. Meanwhile, the number of radiologists is much fewer Net [36], which is state-of-the-art open-sourced deep CNN for
than that of people under detection. An artificial intelligence COVID-19 case detection from CXR images. By using COVID-
(AI)-aided diagnostic system is thus needed to assist radiologists Net network architecture as well as its datasets of COVIDx [2],
to make screening of COVID-19 cases in a more rapid and we first train multiple snapshot deep learning models with a
accurate way, otherwise it is prone to occur that infected people cosine annealing learning rate schedule, for which the learning
cannot be detected and quarantined as soon as possible and in rate fluctuates significantly in that it starts high and drops to
turn cannot receive treatment timely [24], [36]. a minimum value close to zero rapidly before going up to the
Essentially, COVID-19 case detection with CXR images maximum value again [22]. Next, the ensemble deep learning
is a classification problem in the machine learning domain, model of EDL-COVID is developed by combining these models
for which convolutional neural network (CNNs) enabled data- with a proposed model ensembling approach called weighted
driven deep learning methods have demonstrated promising averaging ensembling (WAE), which is based on two observa-
performance [21]. As such, many recent studies [6], [24], [26], tions that 1) there are different sensitivities for different classes
[36], are available for attempting to train new deep learning types of an individual deep learning model, and 2) different
models for infected case detection with CXR images by either deep learning models have different sensitivities for each class
reusing or modifying existing deep neural networks atop of type. WAE is based on the assumption that for a class type, a
collected CXR images datasets. However, due to the noise and model with a higher sensitivity should contribute more to the
limited training data size in practice, a deep learning model final ensembling result by estimating its weight proportional
might suffer from overfitting, high variance, and generalization to its sensitivity. Finally, the experimental results show that
errors although some studies in their articles report much high EDL-COVID achieves promising results for case detection from
prediction accuracy for their proposed deep learning models CXR images with 95.0% accuracy, 96.0% sensitivity, and 94.1%
with their own datasets. PPV for the COVID-19 class type.
To alleviate it, instead of using a single model, ensemble In summary, the following contributions are made in this
learning that combines multiple models together with a proper article.
strategy (e.g., random forest [10], boosting [32], stacking [12]) 1) We propose a weighted average ensembling (WAE) en-
is assumed to be an effective technique. It is able to not only sembling strategy with the awareness of varied class-level
reduce the variance and generalization errors of predictions but accuracies for different machine learning models.
also can yield a better result than any single model [28]. 2) We propose a snapshot ensemble deep learning model
In this article, to take advantage of both deep learning and called EDL-COVID based on the state-of-the-art deep
ensemble learning, we propose EDL-COVID, an ensemble deep learning architecture of COVID-Net.
learning model for detecting COVID-19 cases with CXR im- 3) We evaluate EDL-COVID experimentally, showing the
ages. There are several challenging issues for ensemble deep promising results of EDL-COVID.
learning model development. First, it needs to have multiple The rest of this article is organized as follows. Section II
deep learning models for combination, whereas training a deep provides a background of ensemble deep learning and COVID-
model generally takes a significant amount of computational Net. In Section III, related work is reviewed. The proposed
cost and a long time to converge. Moreover, the development EDL-COVID model is introduced in Section IV. Experimental

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
TANG et al.: EDL-COVID: ENSEMBLE DEEP LEARNING FOR COVID-19 CASES DETECTION FROM CHEST X-RAY IMAGES 6541

evaluation of the proposed model is provided in Section V.

Finally, Section VI concludes the article.

II. BACKGROUND
For the sake of better understanding the EDL-COVID model,
we give a description of ensemble deep learning and COVID-
Net network that EDL-COVID is built on for COVID-19 case
detection.

A. Ensemble Deep Learning

As a powerful machine learning technique, deep learning is
widely applied to many studies, e.g., computer vision, speech Fig. 1. CNN architecture of COVID-Net [36].
recognition, medical image analysis, drug design, etc [8], [23]. It
contains a multilayer neural network that can progressively ex-
tract higher level features from raw data such as images and pro- human-machine collaborative manner with the following char-
duce prediction outputs based on those features. A deep learning acteristics. First, it exploits a design pattern of lightweight resid-
model computation consists of two stages, namely training and ual projection-expansion-projection-extension (PEPX) heavily,
inference. Training is an iterative and stochastic computation for which makes up of first-stage projection, expansion, depth-wise
generating a model based on training data. Several arguments representation, second-stage projection, and extension. Specifi-
need to be initialized before training computation, including cally, the First-Stage Projection is a 1 × 1 convolution to project
learning rate, epoch number, and batch size, where different input features from a high dimension to a lower one. Expan-
configurations tend to result in models with different accuracies. sion is a reverse procedure that expands features to a higher
Inference is a prediction process with the trained deep learning dimension with a 1 × 1 convolution. Dep-wise Representation
model. Some popular deep learning architectures are available, is responsible for learning spatial features to minimize com-
e.g., CNN and recurrent neural networks (RNN). In comparison, putational complexity while keeping representational capacity
CNN is the most popular one, which is good for applications via a 3 × 3 depth-wise convolutions. Second-stage Projection
such as image detection/recognition, image classification, and projects features back to a lower dimension with a 1 × 1 convo-
medical image analysis [5]. lutions. Finally, Extension extends channel dimensionality to a
With the existence of the noise in the training data and the higher one to produce the final features with 1 × 1 convolutions.
randomness in the deep learning algorithm, however, it tends to Second, it contains long-range connectivities at different places
suffer from high variance problem and generalization error [19]. of the network architecture. Third, the COVID-Net network is
Although there are some commonly used techniques such as data considerably diverse in order to realize a strong representational
augmentation and regularization [9], the problems are still not capacity for COVID-19 case detection. All of these features
well addressed for deep learning models. enable COVID-Net to achieve a strong representational capacity
To overcome these problems for a single machine learning while keeping computational complexity reduced. It offers a
model, ensemble learning is assumed to be an effective ap- COVIDx dataset for training, which classifies people into the
proach [28]. It is a hybrid learning paradigm that is able to following cases, i.e., normal, pneumonia, as well as COVID-19
produce more accurate and robust prediction results than a single as Fig. 2 shows. By using the COVIDx dataset [2], it is re-
model by combining multiple machine learning models intel- ported for COVID-Net that it can achieve an accuracy of 93.3%,
ligently. Various ensemble strategies are available, including and positive predictive value (PPV) of 90.5%, 91.3%, 98.9%,
averaging, random forest [10], boosting [32], and stacking [12]. respectively, for normal, pneumonia, and COVID-19.
To make the ensemble effective, we must make sure that multiple
models for combination is diverse. In the proposed work, we III. RELATED WORK
study ensemble deep learning for COVID-19 case detection
by combining multiple deep learning models via an ensemble To reduce the burden of detection from radiography images
strategy. (e.g., CT and CXR images) for radiologists, there have been
several AI-aided detection systems proposed, which are based
on deep learning [20], [31], [37]. Due to several benefits of CXR
B. Covid-Net
imaging including portability, availability, accessibility and fast
COVID-Net [36] is by far the state-of-the-art deep convolu- checking compared to CT imaging, CXR images are becoming a
tion neural network for CXR image processing based COVID-19 popular and commonly utilized data source for COVID-19 case
case detection.1 Fig. 1 presents its network architecture, which detection.
is a carefully designed model for COVID-19 case detection in a Chowdhury et al. [11] made a model training comparison of
different deep learning networks including AlexNet, ResNet18,
DenseNet201, and SqueezeNet to classify two classes (i.e.,
1 [Online]. Available: https://github.com/lindawangg/COVID-Net COVID-19 and Normal) for CXR images, concluding that

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
6542 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 9, SEPTEMBER 2021

Fig. 2. CXR images for normal and illness people from COVIDx datasets [2], which categorizes CXR images into classes of: (a) Normal case,
(b) pneumonia case, and (c) COVID-19 case.

SqueezeNet outperforms other neural networks. Instead, Narin IV. EDL-COVID: ENSEMBLE DEEP LEARNING MODEL FOR
et al. [26] made a comparison of different CNN models (e.g., COVID-19 CASE DETECTION
ResNet-50, Inception V3, and Inception-ResNetV2) trained
In this section, we introduce EDL-COVID, a snapshot ensem-
on CXR images for COVID-19 case detection, showing that
ble deep learning model based on COVID-Net. As illustrated
ResNet-50 outperforms the other two models with 98% accu-
in Fig. 3, the overall training flow for EDL-COVID consists
racy. Farooq et al. [14] provided a COVID-ResNet by fine tuning
of two phases, namely, snapshot model training and model
a pretrained ResNet-50 architecture with a reported accuracy of
ensembling. The snapshot model training phase is responsible
96.23%. Wang et al. [36] made a tailored and first open-sourced
for producing multiple model snapshots (Section IV-A), which
deep neural network called COVID-Net for COVID-19 case
are then combined together for a final prediction in the model
detection with CXR images. Tulin et al. [27] trained a deep
ensembling phase (Section IV-B). The detailed implementation
learning scheme named DarkCovidNet, which detects COVID-
of EDL-COVID can be found in Appendix A.
19 cases based on a number of only 125 CXR images. Maghdid
et al. [24] built deep learning models with pretrained AlexNet
based on its own established dataset of X-ray images and CT A. Snapshot Model Training
images for COVID-19 case detection. Alom et al. [6] proposed a To enable deep learning ensembling, there is a need to have
multitask deep learning system called COVID_MTNet for case multiple pretrained deep learning models. However, the model
detection by considering both CXR and CT images together. training process generally takes hours and a large amount of
HSMA_WOA [4] hybrids the novel Slime mould algorithm computing resources, making deep learning ensembling become
together with whale optimization algorithm to address the CXR a time-consuming and heavy computation process. To alleviate
based image segmentation issue for detecting COVID-19 cases. it, we can instead train multiple model snapshots of a deep
In contrast, we focus on the COVID-19 case detection and learning network for ensembling during a single training run,
HSMA_WOA is complementary to our article. rather than training multiple models from different deep learning
While these studies seem to offer pretty good results, the networks separately. In this article, we choose COVID-Net as
reliability of their proposed deep learning models can be ques- the candidate to generate multiple model snapshots in terms of
tioned due to a serious bias problem for the COVID-19 dataset its promising performance for its CXR image based COVID-19
collected from a small sized group of COVID-19 cases [33]. case detection, as well as public accessibility for its source code.
Moreover, since the training processes of deep learning models To make model ensembling effective in practice, there is a
rely on stochastic algorithms, they are sensitive to the specifies diversity requirement for multiple deep learning models that
of training data and can produce different weights each time they have different distributions of prediction errors. However, a
are trained. Hence, the prediction result of a single deep learning limitation of such a model snapshot approach of a single deep
model is prone to suffer from high variance and generalization learning network training is that the generated model snapshots
errors due to the noise and a limited amount of COVID-19 tend to be similar, which can produce similar predictions and
datasets collected by far. Fortunately, these issues can be allevi- prediction errors. To address it, a commonly used approach is to
ated with the adoption of the ensemble learning technique. An take an aggressive learning rate schedule during a single training
ensemble deep learning model called EDL-COVID is proposed run that makes large changes of model weights and in turn the
in this article by combining multiple snapshot deep learning nature of model snapshots [22].
models trained from state-of-the-art neural network of COVID- We take the cosine annealing learning rate schedule proposed
Net with a proposed WAE approach (see Section IV-B) that is by Loshchilov et al. [22] to change the learning rate aggressively
aware of different sensitivities for different models on each class but systematically to generate different model weights over
type. training epochs, by allowing the learning rate to start high and

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
TANG et al.: EDL-COVID: ENSEMBLE DEEP LEARNING FOR COVID-19 CASES DETECTION FROM CHEST X-RAY IMAGES 6543

Fig. 3. Overall flow for EDL-COVID ensemble model training. It consists of two phases, namely, snapshot model training, and model ensembling.
We propose the WAE approach for model ensembling as described in Algorithm 1.

for model ensemble. For snapshot ensemble learning, averaging

is a popular ensembling strategy [17]. For an input sample, it
simply averages the class probabilities for each class from all
models, respectively. Let M be the number of classifiers (i.e.,
deep learning models). Let pm,k (di ) be the class probability of
the kth class output by the mth classifier with respect

to the input
M
p (d )
sample di . Then the average class probability is m=1 Mm,k i ,
for k ∈ [1, K] of the input sample di , with K denoting the
number of classes.
The traditional averaging ensembling strategy is implicitly
based on the assumption that all models have the same weights.
Fig. 4. Variations of learning rates and corresponding learning losses However, there are two key observations as follows.
under cosine annealing learning rate schedule over training epochs. The 1) For a deep learning model, the testing accuracies for
aggressive change of learning rate produces different model weights different classes are generally different.
with significantly different training loss, making a set of diverse models.
2) Different deep learning models tend to have different ac-
curacies for each class. It indicates that we cannot simply
treat each model equally during the model ensembling.
decrease to a minimum value near zero at a relatively rapid Based on the two observations above, we instead propose
speed before being increased again to the maximum based on a WAE approach for snapshot model ensemble as depicted in
the following formula Algorithm 1. Let ai,j be the test accuracy of the ith model for
    the jth class over the test data of the CXR images dataset. Let
α0 πmod(t − 1, T /M )
α(t) = cos +1 (1) wi,j denote the normalized weight of the ith model for the jth
2 T /M  a
class. Then we have wi,j = M i,ja (Line 7). For each input
m=1 m,j
where α0 is the maximum learning rate, α(t) is the learning sample di , we first get the output of every class probability
rate at epoch t and, M denotes the number of cycles, and T pm,k (di ) from the mth model for ∀m ∈ M (Line 10–12). Then
is the overall number of epochs. Then each time a new model we can estimate the class probability pk (di ) of EDL-COVID by
snapshot is produced after training a network for M cycles. summing up the weighted class probabilities of all models (Line
Specifically, we train multiple model snapshots by initializing 13–14) for ∀k ∈ K. Finally, we can get the predicted class by
α0 = 0.002, T = 50, and M = 6 for COVID-Net network with returning the class index with the maximum class probability
COVIDx dataset. Then 6 model snapshots would be produced for each input sample (Line 15).
from the COVID-Net network. Fig. 4 shows the aggressive vari-
ations of learning rates and corresponding training losses with
V. EXPERIMENTAL EVALUATION
the cosine annealing learning rate schedule during the model
training process. We can see that such a significant variation of We have implemented EDL-COVID atop of TensorFlow
learning rate can produce different model weights by observing based on COVID-Net, and evaluated EDL-COVID using CXR
the big changes of intermediate training losses. images of the COVIDx dataset in a GPU machine.

B. Model Ensembling A. Experimental Setup

After generating multiple model snapshots in the first phase, GPU Machine. We train and evaluate our proposed model on
we now come to the model ensembling phase for building EDL- a GPU machine consisting of two Intel Xeon E5-2640 CPUs,
COVID by combining these models as illustrated in Fig. 3. As we 256 GB of RAM, and two NVIDIA Tesla P100 GPUs. It runs
discussed in Section II-A, there are many ensembling strategies on CentOS 7.7 OS with TensorFlow 2.0.0 installed for deep

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
6544 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 9, SEPTEMBER 2021

Algorithm 1: Pseudo Code of the Proposed WAE Approach.

1: Input:
2: M : the number of classifiers (i.e., deep learning
models).
3: N : the size of CXR images dataset.
4: K : the number of classes for CXR images dataset.
5: ai,j : the test accuracy of the ith model for the jth
class.
6: di : the ith input sample of CXR images dataset.
7: wi,j : the normalized weight of the ith model for
WAE over the jth class prediction, where
a Fig. 5. Overall prediction accuracy.
wi,j = M i,ja .
m=1 m,j
8: Output:
9: c(di ) : the predicted class index for the ith input TABLE I
SENSITIVITIES OF DIFFERENT MODELS ON EACH CLASS (E.G., NORMAL,
sample with EDL-COVID. PNEUMONIA, AND COVID-19), WHERE THE BEST RESULTS ARE
10: for i = 1 to N do HIGHLIGHTED IN BOLD
11: for m = 1 to M do
12: Get the output (i.e., class probability vector
pm (di ) = {pm,k (di )} for all K classes where
1 ≤ k ≤ K) of the mth model w.r.t. the input
sample di .
13: end for
14: for k = 1 to K do
15: pk (di ) = M m=1 pm,k (di ) · wi,k .
16: end for
17: c(di ) = arg max1≤k≤K {pk (di )}.
18: Get the class index with the maximum class B. Performance Evaluation for Individual Model
probability for the ith input sample. Performance metrics of each individual model are evaluated
19: end for in the following aspects.
Accuracy Evaluation. We have trained six deep learn-
ing models (denoted as COVID-Net-M1, COVID-Net-M2, · · ·,
learning model training and inference, where cuDNN is enabled COVID-Net-M6) with COVID-Net network architecture on top
to speedup the training computation on a GPU device. of COVIDx dataset. Fig. 5 presents the prediction accuracy for
Dataset. We take the latest COVIDx dataset proposed all models. It shows that the ensembling approach of our pro-
by Wang et al. [2] for model training and evaluation in posed EDL-COVID model can make it effectively outperform
our experiment. It totally contains 15477 CXR images from the other six deep learning models by about 0.3%.
13870 cases by 20th June 2020, comprising of 6053 Pneu- However, in practice, we cannot simply make a judgment that
monia, 8851 Normal, and 573 COVID-19 cases images. The a model with a higher accuracy must work better than a model
COVIDx dataset is collected from five different data sources, with a lower accuracy for a multiclass problem. We need to
namely, ActualMed COVID-19 dataset,2 COVID-19 image further take a look at the other two important class-level metrics,
data collection,3 COVID-19 radiography database,4 COVID-19 namely, sensitivity and positive predictive value (PPV) as well.
CXR dataset initiative,5 as well as RSNA pneumonia detec- Sensitivity Evaluation. In medical analysis, the sensitivity
tion challenge.6 In our experiment below, we take 13898 CXR of a disease can be interpreted as the proportion of people with
images from the COVIDx dataset for model training and the a certain disease that has been successfully identified. Taking
remaining 1579 CXR images for testing. Specifically, for train- COVID-19 for example, achieving a high sensitivity is quite
ing data, it consists of 7966 Normal, 5459 Pneumonia, and 473 important since we do not want to omit any affected people
COVID-19 CXR images. In contrast, there are 885 normal, 594 during our COVID-19 testing, otherwise the affected people
pneumonia, and 100 COVID-19 CXR images for testing data. that have been omitted cannot receive immediate treatment,
and meanwhile, they can affect other people. Table I gives a
2 [Online]. Available: https://github.com/agchung/Actualmed-COVID- sensitivity analysis for each model with respect to each class
chestxray-dataset
3 [Online]. Available: https://github.com/ieee8023/covid-chestxray-dataset type. We have the following observations. It is seldom to have
4 [Online]. Available: https://kaggle.com/tawsifurrahman/covid19- a model that works best for all three classes. For example,
radiography-database COVID-Net-M5 has the highest sensitivity in Normal class but
5 [Online]. Available: https://github.com/agchung/Figure1-COVID- not for two other class types. In comparison, EDL-COVID has
chestxray-dataset
6 [Online]. Available: https://kaggle.com/c/rsna-pneumonia-detection- the highest sensitivities for both Pneumonia and COVID-19
challenge classes although its sensitivity of Normal class is not the best

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
TANG et al.: EDL-COVID: ENSEMBLE DEEP LEARNING FOR COVID-19 CASES DETECTION FROM CHEST X-RAY IMAGES 6545

TABLE II
PPV OF DIFFERENT MODELS ON EACH CLASS (E.G., NORMAL, PNEUMONIA,
AND COVID-19), WHERE THE BEST RESULTS ARE HIGHLIGHTED IN BOLD

Fig. 8. ROC curves of EDL-COVID prediction on COVIDx test dataset

with respect to each class type.

Fig. 6. Estimated class-level weights for six deep learning models

needed by WAE in model ensembling.

Fig. 9. Execution time for EDL-COVID under different numbers of

testing CXR images.

In summary, we can conclude that, although no model outper-

forms others on all metrics for all classes types, for COVID-19
case detection, our proposed EDL-COVID is the best choice
since it outperforms other models on the accuracy, sensitivity,
and PPV for COVID-19 class type.

C. EDL-COVID Evaluation Results

Fig. 7. Confusion matrix for EDL-COVID on COVIDx dataset that
consists of 100 COVID-19, 594 pneumonia, and 885 normal images.
In this section, we evaluate the EDL-COVID model from the
The red box is the true prediction, and the light blue box is the false following perspectives.
prediction. Weights Estimation for WAE. Recall in Section IV-B that we
proposed a model ensembling strategy called WAE to combine
multiple deep learning models with the awareness of different
across all models. From a practical point of view, there is no accuracies on different class types for different models. We first
doubt to consider EDL-COVID since a high sensitive screening need to obtain weights for all deep learning models on each
for infectious diseases such as COVID-19 is very important. class type for WAE. Fig. 6 illustrates the estimated class-level
PPV Evaluation. PPV denotes the probability of positive weights for all deep learning models, which is based on the
results that are true positive results in diagnostic tests according estimated sensitivity result on each class type as shown in Table I.
to (4). If this value is small, it means that there are many false We can see that different classes tend to have different weights
positives and a follow-up testing is needed for any positive for different deep learning models. For example, for COVID-
result with a more reliable result. For COVID-19 screening, if a Net-M6, it has the largest weight for the COVID-19 class but
model’s PPV is small, we cannot make a judgment that a person relatively low weight for the Pneumonia class.
with a positive testing result is the true COVID-19 case and more Predicted Results. The confusion matrix for the proposed
accurate testing is needed for positive testing results. Table II EDL-COVID is presented as in Fig. 7, which analyzes the
gives a PPV analysis for each model on each class type. Still, COVIDx dataset, including CXR images of 100 COVID-19
no model performs the best for all class types. COVID-Net-M6 cases, 594 pneumonia cases, and 885 normal cases. For COVID-
has the highest PPV on pneumonia class, and our EDL-COVID 19 testing, only four out of 100 CXR images of COVID-19 are
achieves the highest PPVs for Normal and COVID-19 classes. not screened out, and six out of 1579 CXR images are mistakenly

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
6546 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 9, SEPTEMBER 2021

Fig. 10. Core source codes of snapshot models training for EDL-COVID.

considered as COVID-19, indicating that the error ratio is minor WAE ensembling approach that is aware of different sensitivities
compared to the total number of CXR images for EDL-COVID. for different deep learning models on different classes types. Ex-
To show the detection capability of EDL-COVID, we draw periments on a COVIDx test data of 1579 CXR images show that
ROC curves for EDL-COVID prediction on COVIDx test dataset EDL-COVID can detect COVID-19 cases with good promising
with respect to each class type in Fig. 8. Larger area of ROC results of 96% sensitivity and 94.1% PPVs, outperforming each
area is an indication of better prediction ability. We can see individual deep learning model. We hope our AI-based screening
that the ROC area for each class under EDL-COVID is much approach can aid radiologists to accelerate the screening of
closer to the maximum value of one, indicating that our proposed COVID-19 cases while guaranteeing a high accuracy in the
EDL-COVID has a good prediction capability for each class type current COVID-19 pandemic around the world.
in practice. Yet, current work is heavily dependent on work by Wang
Model Execution Time Evaluation. Fig. 9 illustrates the et al. [36] on COVID-Net network architecture and COVIDx
execution time for EDL-COVID to handle different numbers dataset. There are several future work that can be done to enhance
of testing CXR images. We can see that a linear relationship is EDL-COVID for practical use. First, the number of COVID-19
demonstrated between the data size and the total execution time, CXR images is still relatively small compared to CXR images
indicating that our proposed EDL-COVID is scalable. Moreover, of other classes for the COVIDx dataset, indicating that we
we also present the average execution time for processing a cannot simply make a judgment that the currently trained model
CXR image. Interestingly, the average execution time drops snapshots from COVID-Net architecture continue to work well
significantly we increase the number of CXR images at the with a high accuracy for unseen COVID-19 CXR images. To
beginning and later becomes smooth when the total number improve EDL-COVID, we need to retrain model snapshots of
of images is larger than 1000. This is because EDL-COVID COVID-Net whenever new CXR images are available. Besides
takes time to load the six models during its computing process snapshot models, we can also incorporate some other publicly
(i.e., overhead), which cannot be ignored when the workload available good deep learning models into EDL-COVID for better
size is small. However, it becomes relatively small for such an performance. Second, we would plan to extend EDL-COVID
overhead when the workload size becomes large, making the for other COVID-19 applications such as risk stratification
average execution time per image becomes smaller first and later for COVID-19 cases survival analysis, risk status analysis of
keep stable as observed. COVID-19 cases, which are important for patient hospitalization
and care planning.
VI. CONCLUSION
In this article, to overcome several problems of overfitting,
high variance, and generalization errors of an individual deep APPENDIX A
learning model for improved performance on COVID-19 case THE IMPLEMENTATION OF EDL-COVID
detection, we proposed EDL-COVID, an ensemble deep learn- As discussed in Section IV, EDL-COVID consists of two
ing model called for COVID-19 case detection from CXR parts, i.e., snapshot model training and model ensembling. In
images on the basis of the open-sourced network architecture this section, we present its key implementation codes for the
called COVID-Net. We first generated multiple model snapshots sake of a better understanding of EDL-COVID.
by training the COVID-Net network on top of COVIDx CXR Fig. 10 shows the snapshot model training codes for EDL-
datasets, followed by ensembling these models with a proposed COVID. To output a set of diverse snapshot models, we leverage

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
TANG et al.: EDL-COVID: ENSEMBLE DEEP LEARNING FOR COVID-19 CASES DETECTION FROM CHEST X-RAY IMAGES 6547

Fig. 11. Core source codes of snapshot models ensembling for EDL-COVID.

cosine annealing learning rate schedule to dynamically esti- [6] M. Z. Alom, M. Rahman, M. S. Nasrin, T. M. Taha, and V. K. Asari,
mate and adjust learning rate aggressively and systematically COVID_MTNet: Covid-19 detection with multi-task deep learning ap-
proaches, 2020, arXiv:2004.03747.
(Line 26), which is implemented in CosineAnnealingLearn- [7] A. Borghesi and R. Maroldi, “COVID-19 outbreak in italy: Experi-
ingRateSchedule()(Line 2–22). Next, we load the training and mental chest x-ray scoring system for quantifying and monitoring dis-
testing data (Line 27–46). After that, we start to train models ease progression,” La Radiologia Med., vol. 125, no. 5, pp. 509–513,
2020.
with provided COVID-Net network architecture (Line 47–101). [8] Z. Cao, T. Liao, W. Song, Z. Chen, and C. Li, “Detecting the shuttlecock for
Typically, at the beginning of each epoch, we adjust the learning a badminton robot: A yolo based approach,” Expert Syst. Appl., vol. 164,
rate dynamically (Line 78). We spill the intermediate snapshot 2020, Art. no. 113833.
[9] S. Chen, E. Dobriban, and J. H. Lee, “Invariance reduces variance:
model into disks at every epochs_per_cycle configured by users Understanding data augmentation in deep learning and beyond,” 2019,
(Line 99–101). arXiv:1907.10905.
Fig. 11 presents the model ensemble implementation for [10] Y.-Y. Cheng, P. P. Chan, and Z.-W. Qiu, “Random forest based ensemble
system for short term load forecasting,” in Proc. Int. Conf. Mach. Learn.
EDL-COVID. We first use test data to estimate the class-level Cybernet., 2012, pp. 52–56.
accuracy for each snapshot model trained at the previous stage [11] M. E. Chowdhury et al., “Can AI help in screening Viral and COVID-19
with a prediction_accuracy function (Line 2–14), whose im- pneumonia?,” 2020, arXiv:2003.13145.
[12] F. Divina, A. Gilson, F. Goméz-Vela, M. García Torres, and J. F. Tor-
plementation is given in Line 21–52. With the class-level ac- res, “Stacking ensemble learning for short-term electricity consumption
curacies of snapshot models on each class, we next move to forecasting,” Energies, vol. 11, no. 4, 2018, Art. no. 949.
estimate the class-level weights for each model (Line 16–18) [13] F. A. Faria, J. A. Dos Santos, A. Rocha, and R. d. S. Torres, “A framework
for selection and fusion of pattern classifiers in multimedia recognition,”
with class_weight_estimate function, whose implementation is Pattern Recognit. Lett., vol. 39, pp. 52–64, 2014.
given in Line 53–70. Finally, we perform a model ensemble [14] M. Farooq and A. Hafeez, “COVID-ResNet: A deep learning framework
for all snapshot models with WAE strategy (Line 19–22) via for screening of COVID19 from radiographs,” 2020, arXiv:2003.14395.
[15] W.-J. Guan et al., “Clinical characteristics of coronavirus disease 2019
wae_estimate function, which are detailed in Line 71–103. in China,” New England J. Med., vol. 382, no. 18, pp. 1708–1720,
2020.
REFERENCES [16] C. Huang et al., “Clinical features of patients infected with 2019 novel
coronavirus in Wuhan, China,” Lancet, vol. 395, no. 10223, pp. 497–506,
[1] Coronavirus Disease 2019. 2020. [Online]. Available: https://en. 2020.
wikipedia.org/wiki/Coronavirus_disease_2019 [17] G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger,
[2] Covidx Dataset. 2020. [Online]. Available: https://github.com/ “Snapshot ensembles: Train 1, get M for free,” 2017, arXiv:1704.00109.
lindawangg/COVID-Net/blob/master/docs/COVIDx.md [18] A. Jacobi, M. Chung, A. Bernheim, and C. Eber, “Portable chest x-ray in
[3] M. Abdel-Basset, V. Chang, H. Hawash, R. K. Chakrabortty, and M. Ryan, coronavirus disease-19 (COVID-19): A pictorial review,” Clin. Imag., vol.
“FSS-2019-nCov: A deep learning architecture for semi-supervised few- 64, pp. 35–42, 2020.
shot segmentation of covid-19 infection,” Knowl.-Based Syst. vol. 212, [19] D. Jakubovitz, R. Giryes, and M. R. Rodrigues, “Generalization error in
2021, Art. no. 106647. deep learning,” in Compressed Sensing and Its Applications, Springer,
[4] M. Abdel-Basset, V. Chang, and R. Mohamed, “HSMA_WOA: A hybrid 2019, pp. 153–193.
novel slime mould algorithm with whale optimization algorithm for tack- [20] A. Kumar, P. K. Gupta, and A. Srivastava, “A review of modern technolo-
ling the image segmentation problem of chest x-ray images,” Appl. Soft gies for tackling COVID-19 pandemic,” Diabetes Metabolic Syndrome
Comput., vol. 95, 2020, Art. no. 106642. Clin. Res. Rev., vol. 14, no. 4, pp. 569–573, 2020.
[5] S. Albawi, T. A. Mohammed, and S. Al-Zawi, “Understanding of a [21] P. Lakhani and B. Sundaram, “Deep learning at chest radiography: Au-
convolutional neural network,” in Proc. Int. Conf. Eng. Technol., 2017, tomated classification of pulmonary tuberculosis by using convolutional
pp. 1–6. neural networks,” Radiology, vol. 284, no. 2, pp. 574–582, 2017.

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
6548 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 17, NO. 9, SEPTEMBER 2021

[22] I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with Chunjiang Wang received the B.S. degree in
warm restarts,” 2016, arXiv:1608.03983. network engineering from the School of Com-
[23] C. Ma, Z. Liu, Z. Cao, W. Song, J. Zhang, and W. Zeng, “Cost-sensitive puter Science and Technology, Tiangong Uni-
deep forest for price prediction,” Pattern Recognit., vol. 107, 2020, versity, Tianjin, China, in 2019. He is currently
Art. no. 107499. working toward the M.S. degree in computer sci-
[24] H. S. Maghdid, A. T. Asaad, K. Z. Ghafoor, A. S. Sadiq, and M. K. Khan, ence and technology with the College of Intelli-
“Diagnosing COVID-19 pneumonia from x-ray and CT images using deep gence and Computing, Tianjin University (TJU),
learning and transfer learning algorithms,” 2020, arXiv:2004.00038. China.
[25] A. Nair et al., “A British Society of Thoracic Imaging statement: Consider- His research interests include deep learning,
ations in designing local imaging diagnostic algorithms for the COVID-19 video analysis, cloud computing, and parallel
pandemic,” Clin. Radiol., vol. 75, no. 5, pp. 329–334, 2020. computing.
[26] A. Narin, C. Kaya, and Z. Pamuk, “Automatic detection of coronavirus
disease (COVID-19) using x-ray images and deep convolutional neural
networks,” 2020, arXiv:2003.10849.
[27] T. Ozturk, M. Talo, E. A. Yildirim, U. B. Baloglu, O. Yildirim, and
U. R. Acharya, “Automated detection of COVID-19 cases using deep
neural networks with x-ray images,” Comput. Biol. Med., vol. 121,
2020, Art. no. 103792. Jiangtian Nie received the B.S. degree in elec-
[28] R. Polikar, “Ensemble learning,” in Ensemble Machine Learning, Boston, tronic and information engineering from the
MA: Springer, 2012, pp. 1–34. Huazhong University of Science and Technol-
[29] S. Rajaraman, J. Siegelman, P. O. Alderson, L. S. Folio, L. R. Folio, and ogy, Wuhan, China, in 2016. She is currently
S. K. Antani, “Iteratively pruned deep learning ensembles for COVID-19 working toward the Ph.D. degree with ERI@N,
detection in chest x-rays,” 2020, arXiv:2004.08379. Interdisciplinary Graduate School, Nanyang
[30] M. Shen et al., “Recent advances and perspectives of nucleic acid detection Technological University, Singapore.
for coronavirus, J. Pharmaceut. Anal., vol. 10, no. 2, pp. 97–101, 2020. Her research interests include incentive
[31] F. Shi et al., “Review of artificial intelligence techniques in imaging mechanism design in crowdsensing and game
data acquisition, segmentation and diagnosis for COVID-19,” IEEE Rev. theory.
Biomed. Eng., vol. 14, pp. 4–15, 2021, doi: 10.1109/RBME.2020.2987975.
[32] V. Svetnik, T. Wang, C. Tong, A. Liaw, R. P. Sheridan, and Q. Song,
“Boosting: An ensemble learning tool for compound classification and
QSAR modeling, J. Chem. Inf. Model., vol. 45, no. 3, pp. 786–799, 2005.
[33] S. Tabik et al.“COVIDGR dataset and COVID-SDNet methodol-
ogy for predicting COVID-19 based on chest x-ray images,” 2020, Neeraj Kumar is currently a Professor with
arXiv:2006.01409. the Department of Computer Science and En-
[34] A. Tahamtan and A. Ardebili, “Real-time RT-PCR in COVID-19 detection: gineering, Thapar Institute of Engineering and
Issues affecting the results,” Expert Rev. Mol. Diagn., vol. 10, no. 5, Technology, Patiala, India. He has authored or
pp. 453–454, 2020. coauthored more than 400 technical research
[35] B. Udugama et al., “Diagnosing COVID-19: The disease and tools for papers in top-cited journals and conferences
detection,” ACS Nano, vol. 14, no. 4,pp. 3822–3835, 2020. which are cited more than 15 000 times from
[36] L. Wang and A. Wong, “COVID-Net: A tailored deep convolutional neural well-known researchers across the globe with
network design for detection of COVID-19 cases from chest x-ray images,” current h-index of 68 in Google scholar. His
2020, arXiv:2003.09871. research areas are green computing and net-
[37] L. Wynants et al., “Prediction models for diagnosis and prognosis of work management, IoT, big data analytics, deep
COVID-19 infection: Systematic review and critical appraisal,” Brit. Med. learning, and cyber-security.
J., vol. 369, 2020, Art. no. m1328. Prof. Kumar is currently the Editor for ACM Computing Survey, IEEE
[38] S. Zhou, Y. Wang, T. Zhu, and L. Xia, “CT features of coronavirus disease TRANSACTIONS ON SUSTAINABLE COMPUTING, IEEE SYSTEMS JOURNAL,
2019 (COVID-19) pneumonia in 62 patients in Wuhan, China,” Amer. J. IEEE Network Magazine, IEEE Communication Magazine, Elsevier
Roentgenol., vol. 214, no. 6, pp. 1287–1294, 2020. Journal of Networks and Computer Applications, Elsevier Computer
Communication, and Wiley International Journal of Communication Sys-
tems. He is also the TPC Chair and member for various International
conferences such as IEEE MASS 2020, IEEE MSN2020. He was the
recipient of the Best Papers Award from IEEE SYSTEMS JOURNAL and
IEEE ICC 2018, Kansas-city in 2018. He was also the recipient of the
Best Researcher award from parent organization every year from last
eight consecutive years.
Shanjiang Tang received the B.S. degree in
software engineering and the M.S. degree in
computer science and technology from Tianjin
University (TJU), China, in 2008 and 2011, re-
spectively, and the Ph.D. degree in computer Yang Zhang received the B.Eng. degree
engineering from the School of Computer Engi- in aeronautics science and engineering,
neering, Nanyang Technological University, Sin- B.Eng. (Minor) degree in applied mathe-
gapore, in 2015. matics, and M.Eng. degree in computer
He is currently an Associate Professor with science and technology from the Beijing
the College of Intelligence and Computing, TJU. University of Aeronautics and Astronautics
He has authored or coauthored many papers (Beihang Univ.), Beijing, China, in 2008,
in IEEE TRANSACTIONS ON CLOUD COMPUTING, IEEE TRANSACTIONS ON 2010, and 2011, respectively, and the Ph.D.
PARALLEL AND DISTRIBUTED SYSTEMS, IEEE TRANSACTIONS ON KNOWL- degree in computer engineering from Nanyang
EDGE AND DATA ENGINEERING, IEEE TRANSACTIONS ON SERVICES COM- Technological University, Singapore, in 2015.
PUTING, ACM/IEEE Supercomputing Conference, International Confer- He is currently an Associate Professor with
ence on Supercomputing, IEEE International Symposium on High Per- the Wuhan University of Technology, China.
formance Distributed Computing, etc. His research interests include Dr. Zhang is currently an Associate Editor for the EURASIP Journal on
parallel computing, cloud computing, big data analysis, and machine Wireless Communications and Networking, and a Technical Committee
learning. Member of Computer Communications.

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.
TANG et al.: EDL-COVID: ENSEMBLE DEEP LEARNING FOR COVID-19 CASES DETECTION FROM CHEST X-RAY IMAGES 6549

Zehui Xiong received the B.S. degree in elec- Ahmed Barnawi received the M.Sc. degree
tronic and information engineering and the from UMIST, U.K., in 2001, and the Ph.D. de-
Ph.D. degree in computer science and tech- gree from the University of Bradford, U.K., in
nology from Nanyang Technological University, 2005.
Singapore. He is currently a Professor with the Faculty of
He is currently an Assistant Professor with Computing and IT with King Abdulaziz Univer-
the Pillar of Information Systems Technology sity, Jeddah, Saudi Arabia. He is the Managing
and Design, Singapore University of Technol- Director of the KAU Cloud computing and Big
ogy and Design, Singapore. He has authored Data Research Group. His research interests in-
or coauthored more than 90 research papers in clude big data, cloud computing, and advanced
leading journals and flagship conferences and mobile robotic applications. He has authored or
four of them are ESI Highly Cited Papers. His research interests include coauthored more than 100 papers in peer-reviewed journals.
wireless communications, network games and economics, blockchain,
and edge intelligence.
Dr. Xiong was the recipient of five Best Paper Awards in interna-
tional conferences and technical committees, and the Chinese Govern-
ment Award for Outstanding Students Abroad in 2019, and the NTU
SCSE Best Ph.D. Thesis Runner-Up Award in 2020. He is currently
the Editor or Guest Editor for many leading journals including the IEEE
TRANSACTIONS.

Authorized licensed use limited to: IEEE Xplore. Downloaded on November 27,2022 at 14:01:34 UTC from IEEE Xplore. Restrictions apply.