Detection of Lung Opacity and Treatment Planning With Three-Channel Fusion CNN Model
Detection of Lung Opacity and Treatment Planning With Three-Channel Fusion CNN Model
Detection of Lung Opacity and Treatment Planning With Three-Channel Fusion CNN Model
https://doi.org/10.1007/s13369-023-07843-4
Abstract
Lung opacities are extremely important for physicians to monitor and can have irreversible consequences for patients if
misdiagnosed or confused with other findings. Therefore, long-term monitoring of the regions of lung opacity is recommended
by physicians. Tracking the regional dimensions of images and classifying differences from other lung cases can provide
significant ease to physicians. Deep learning methods can be easily used for the detection, classification, and segmentation
of lung opacity. In this study, a three-channel fusion CNN model is applied to effectively detect lung opacity on a balanced
dataset compiled from public datasets. The MobileNetV2 architecture is used in the first channel, the InceptionV3 model in
the second channel, and the VGG19 architecture in the third channel. The ResNet architecture is used for feature transfer
from the previous layer to the current layer. In addition to being easy to implement, the proposed approach can also provide
significant cost and time advantages to physicians. Our accuracy values for two, three, four, and five classes on the newly
compiled dataset for lung opacity classifications are found to be 92.52%, 92.44%, 87.12%, and 91.71%, respectively.
Keywords Lung opacity detection · Deep learning · CNN · Three-channel fusion CNN model
123
Arabian Journal for Science and Engineering
and encourage physicians to pay more attention to these areas tioned. The biggest difference of the dataset from other
before the disease reaches the pneumonia stage [12]. datasets is that it has a higher number of data in the lung
The main contributions of this paper are as follows: opacity, pneumonia, COVID-19, and normal categories.
Also, the maximum number of images possible have been
categorized for the tuberculosis class.
• Deep learning models have been successfully tested on • The three-channel fusion CNN model has been used for
X-ray images. An artificial intelligence system that helps the first time in the classification of diseases with lung
physicians diagnose lung opacity disease through lung images.
images has been developed. • The three-channel fusion CNN model has been proposed
• A dataset with five different classes has been created from as a new architecture that is easy to implement and has
datasets accepted in the literature. This dataset includes shown success in multi-class problems.
the lung opacity, pneumonia, COVID-19, tuberculosis,
and normal classes.
• The dataset is different from those in the literature in The remainder of this manuscript is organized as follows:
terms of the number and consistency of the data. It is a Section 2 introduces a literature review of lung opac-
unique dataset created for the five different classes men- ity diagnosis. Section 3 describes the materials and methods
123
Arabian Journal for Science and Engineering
implemented in this study. Section 4 presents the experimen- rithms in a 1-dimensional vector of images, which produced
tal results and a comparison with the results in the literature. more representable features for individual diseases.
Section 5 discusses conclusions and future work. Li et al. [14] proposed the Cov-Net model for the detec-
tion of four-class (lung opacity, COVID-19, viral pneumonia,
and normal) radiological images. A modified residual net-
work with asymmetric convolution and embedded attention
mechanism was used as a backbone of the feature extractor
2 Literature Review for accurate detection of classes.
Mergen et al. [15] used deep learning methods for detect-
Sirazitdinov et al. [12] proposed an ensemble model consist- ing lung abnormalities. First, multi-scale deep reinforcement
ing of two convolutional neural networks, Mask R-CNN and learning was used for detecting anatomical landmarks. A
RetinaNet, for the localization and detection of lung opac- DenseUNet was trained for lung opacity segmentation.
ity and other pneumonia. Because of the study on 26684 Rahman et al. [16] used five image enhancement tech-
datasets, a recall value of 0.793 was obtained. niques to increase the accuracy of disease diagnosis on a
Senan et al. [13] used two deep learning models, AlexNet three-class (lung opacity, COVID-19, and healthy) 18479
and ResNet-50, to diagnose X-ray datasets created from mul- chest X-ray dataset. They then proposed a new UNet model
tiple sources. Enhanced features extracted from CNN models for lung segmentation. Six different pre-trained CNNs and
were then combined with traditional GLCM and LBP algo-
123
Arabian Journal for Science and Engineering
Table 2 Features used for classification tasks a shallow CNN model were examined on both normal and
Model Three-channel fusion CNN segmented images.
Muhammad et al. [17] successfully applied deep learning
Image dimensions 224 × 224 with CNNs to a five-class (lung opacity, bacterial pneumonia,
Training options Adam viral pneumonia, COVID-19, and normal) dataset to increase
Max epochs 100 diagnostic accuracy. To augment data, due to the lack of X-ray
Activation function LeakyReLU& Softmax images, they proposed a self-augmentation mechanism using
Initial learning rate 0.0001 reconstruction independent component analysis (RICA).
Filter size 5×5
Maximum pooling 2×2
Batch size 32 3 Materials and Methods
Execution environment GPU
The motivation behind the diagnosis and segmentation study
Table 3 Average accuracy of lung opacity disease on chest X-rays is to help physi-
Class Accuracy (%)
values for different classes cians identify and follow the progression of the disease by
Five-class 91.71 using a deep learning technique-developed system to detect
Four-class 87.12 the lung opacity condition. The three-channel fusion CNN
Three-class 92.44 model has been used as a deep learning modeling algo-
Two-class 92.52 rithm to extract the most important distinguishing features
from the X-ray images. The images included in the lung
opacity class have also been segmented using python’s preex-
123
Arabian Journal for Science and Engineering
123
Arabian Journal for Science and Engineering
123
Arabian Journal for Science and Engineering
isting libraries (OpenCV, matplotlib) to mask the image. The architectures available in TensorFlow have been tried in order
obtained mask images have been stored on a web server, and and the ones with the best results have been selected.) The
it has been planned for the physicians to interpret the differ- MobileNetV2 architecture has been used in the first chan-
ence between the new X-ray image of the patient taken after a nel, the InceptionV3 architecture in the second channel, and
certain period (between 1 year and 3 years) has passed. The the VGG19 architecture in the third channel. MobileNetV2
training results will be incorporated into the system using CNN is widely used in image classification and segmenta-
transfer learning to segment the lung opacity class in the tion processes [21, 22]. MobileNetV2 architecture works by
model. The flowchart of the study is shown in Fig. 2. reducing the size and complexity of the network in terms
of the number of parameters. For this reason, it is devel-
oped and preferred for efficiency. InceptionV3 is a modified
3.1 Model Description version of the inception family with some improvements
including LabelSmoothing, 7x7 convolutions as well. It is
The proposed CNN model is designed as a three-channel mostly used in image analysis and object detection problems
model. Classic fusion architecture models are used with two, [23]. VGG19 architecture is a deep neural network with mul-
three, and four channels [18, 19]. The basic idea of the tiple layers of convolution. It is useful due to its simplicity,
fusion architecture is to provide the input image multiple as it is composed of 3x3 convolutional layers stacked on top
times in multiple stages in order to extract more features of each other with increasing depth levels. To reduce the vol-
[20]. However, repeatedly providing the same image on dif- ume size, maximum pooling layers are used [24]. In each of
ferent channels can cause inconsistency in extracting more the three channels, the transfer of features from the previous
features. Therefore, the three channels in our proposed model layer to the current layer has been supported using ResNet
have been implemented with classical CNN models that have architecture.
been successful in classification problems. (In this stage, the
123
Arabian Journal for Science and Engineering
During the transfer to the fully connected layer, the ated by Cohen et al. were also added to the pool of data
features from the three channels are combined with a con- [26]. The publicly available dataset created by Chowdhury
catenated layer and transferred to the output layer. Then, lung et al. was also examined and added to the data pool [16, 27].
opacity is detected with a multi-class classifier using the soft- Finally, the dataset created by Tawsifur et al. was added to
max function. Finally, the mask extraction process from the the data pool [28].
images belonging to the lung opacity class is performed with These are the final classes in the five-class categorization:
the help of the OpenCV and matplotlib libraries and saved lung opacity, pneumonia, COVID-19, tuberculosis, and nor-
on web servers. The architecture of the proposed model is mal (healthy). The number of images in each class included in
shown in Fig. 3. the dataset is shown in Table 1. The created dataset has been
made publicly available for the knowledge of researchers
[29]. More detailed descriptions of the dataset and a link to the
3.2 Dataset dataset can be found at: https://github.com/turkfuat/covid19-
pneumonia-dataset. Sample images used in the study are
The compiled dataset is a comprehensive version of the shown in Fig. 4.
data commonly used in literature. The number of images
belonging to classes that are difficult to learn (such as
pneumonia and lung opacity) has been kept as high as 3.3 Image Preprocessing
possible. First, the publicly available dataset created by
Deb and Jha [25] was examined and categorized. Then, As the chest X-ray images are obtained from different
images from the COVID-19 Grand Challenge dataset were sources, each image has different sizes, different contrasts,
included in these categorized groups (https://cxr-covid19. and different light reflections. Hence, the imaging intensity
grand-challenge.org/Dataset/). Images from the dataset cre- of each image is different. In addition, due to the lack of
123
Arabian Journal for Science and Engineering
a certain standard in X-ray imaging and other reasons such using TensorFlow libraries [32, 33]. Some examples of seg-
as patient movements, noise occurs on the images. In noisy mentation are shown in Fig. 5.
images, the disease diagnostic accuracy of algorithms can be
reduced [30]. For this reason, preprocessing algorithms are
applied to the images in the dataset [31]. For this purpose,
the OpenCV Library in Python is used. If the pixel intensity 3.5 Evaluation Metrics
is less than the specified threshold value, the pixel is set to 0
(black) to prevent it from participating in the computations. Lung opacity detection is a classification task; therefore, the
The average filter is applied to enhance the images. The con- most fundamental metric that can be selected is the confusion
trast of each image is increased to expand the density range. matrix. The confusion matrix technique evaluates the accu-
All images are resized to a standard size of 224x224 pixels racy and performance of the classification algorithm. If the
for deep learning models. images in the classes of the dataset do not show a balanced
distribution, measuring the classification accuracy alone may
not be sufficient and may give misleading results [13, 34]. In
3.4 Lung Opacity Mask Extraction Process this study, the performance metrics calculated for the dataset
used are defined as accuracy, recall, precision, and F1 score.
After a lung opacity class is detected, the images in this class Accuracy is a measure of how well the algorithm is able
are masked in order to be compared with images obtained to correctly predict the class of a given sample. It is calcu-
within a range of one to three years. These masks will be used lated by dividing the number of correctly classified samples
to compare with new images obtained when physicians are by the total number of predictions made. In other words, it
called for control. For this purpose, the images are first con- represents the proportion of the total number of predictions
verted to DICOM format and then the lungs are segmented that the classifier got right.
123
Arabian Journal for Science and Engineering
123
Arabian Journal for Science and Engineering
Accuracy (%)
Table 4 Recall/ precision/ F1 score values for classifications
Class Recall Precision F1 score Samples
91.71
82.50
93.72
87.12
92.44
76.32
92.52
89.6
83.8
Five-class classification
Lung opacity 0.83 0.73 0.78 902
Normal 0.95 0.88 0.91 1679
COVID-19 0.96 0.95 0.95 1738
ResNet-50+SAM+BiLSTM
Three-channel fusion CNN
Three-channel fusion CNN
Three-channel fusion CNN
Three-channel fusion CNN
RetinaNet + Mask R-CNN
Pneumonia 0.90 0.99 0.94 3011
Tuberculosis 0.94 0.95 0.94 448
Four-class classification
Lung opacity 0.63 0.69 0.66 902
ResNet-50
Technique
Normal 0.96 0.86 0.91 1679
Cov-Net
AlexNet
Pneumonia 0.88 0.92 0.90 3011
Tuberculosis 0.95 0.94 0.95 448
Three-class classification
Lung opacity 0.84 0.91 0.88 902
Proposed model
Proposed model
Proposed model
Proposed model
123
Arabian Journal for Science and Engineering
123
Arabian Journal for Science and Engineering
19. Li, M.-A.; Han, J.-F.; Yang, J.-F.: Automatic feature extraction and 29. Fuat, T.; Kökver, Y.: Application with deep learning models for
fusion recognition of motor imagery EEG using multilevel multi- covid-19 diagnosis. Sakarya Univ. J. Comput. Inf. Sci. 5(2), 169–
scale CNN. Med. Biol. Eng. Comput. 59(10), 2037–2050 (2021) 180 (2022)
20. Türk, F.; Lüy, M.; Barışçı, N.: Kidney and renal tumor segmentation 30. Ünver, H.M.; Kökver, Y.; Duman, E.; Erdem, O.A.: Statistical edge
using a hybrid v-net-based model. Mathematics 8(10), 1772 (2020) detection and circular hough transform for optic disk localization.
21. Kadam, K.D.; Ahirrao, S.; Kotecha, K.: Efficient approach towards Appl. Sci. 9(2), 350 (2019)
detection and identification of copy move and image splicing forg- 31. Duman, E.; Erdem, O.A.: Anomaly detection in videos using
eries using mask r-cnn with mobilenet v1. Comput. Intell. Neurosci. optical flow and convolutional autoencoder. IEEE Access 7,
2022 (2022) 183914–183923 (2019)
22. Howard, A.G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; 32. Hofmanninger, J.; Prayer, F.; Pan, J.; Röhrich, S.; Prosch, H.;
Weyand, T.; Andreetto, M.; Adam, H.: Mobilenets: Efficient con- Langs, G.: Automatic lung segmentation in routine imaging is pri-
volutional neural networks for mobile vision applications. (2017) marily a data diversity problem, not a methodology problem. Eur.
arXiv preprint arXiv:1704.04861 Radiol. Exp. 4(1), 1–13 (2020)
23. Shadin, N.S.; Sanjana, S.; Lisa, N.J.: Covid-19 diagnosis from 33. Turk, F.; LÜY, M.; BARIŞÇI, N.; Yalcinkaya, F.: Kidney tumor
chest x-ray images using convolutional neural network (CNN) segmentation using two-stage bottleneck block architecture. Intell.
and inceptionv3. In: 2021 International Conference on Informa- Autom. Soft Comput. 33(1) (2022)
tion Technology (ICIT), pp. 799–804 (2021). IEEE 34. Ali, K.; Shaikh, Z.A.; Khan, A.A.; Laghari, A.A.: Multiclass skin
24. Mateen, M.; Wen, J.; Song, S.; Huang, Z.: Fundus image classifi- cancer classification using efficientnets–a first step towards pre-
cation using VGG-19 architecture with PCA and SVD. Symmetry venting skin cancer. Neurosci. Inf. , 100034 (2021)
11(1), 1 (2018)
25. Deb, S.D.; Jha, R.K.: Covid-19 detection from chest x-ray images
using ensemble of CNN models. In: 2020 International Conference
Springer Nature or its licensor (e.g. a society or other partner) holds
on Power, Instrumentation, Control and Computing (PICC), pp. 1–
exclusive rights to this article under a publishing agreement with the
5 (2020). IEEE
author(s) or other rightsholder(s); author self-archiving of the accepted
26. Cohen, J.P.; Morrison, P.; Dao, L.; Roth, K.; Duong, T.Q.; Ghas-
manuscript version of this article is solely governed by the terms of such
semi, M.: Covid-19 image data collection: prospective predictions
publishing agreement and applicable law.
are the future (2020) arXiv preprint arXiv:2006.11988
27. Chowdhury, M.E.; Rahman, T.; Khandakar, A.; Mazhar, R.; Kadir,
M.A.; Mahbub, Z.B.; Islam, K.R.; Khan, M.S.; Iqbal, A.; Al Emadi,
N.; et al.: Can AI help in screening viral and covid-19 pneumonia?
IEEE Access 8, 132665–132676 (2020)
28. Rahman, T.; Khandakar, A.; Kadir, M.A.; Islam, K.R.; Islam, K.F.;
Mazhar, R.; Hamid, T.; Islam, M.T.; Kashem, S.; Mahbub, Z.B.;
et al.: Reliable tuberculosis detection using chest x-ray with deep
learning, segmentation and visualization. IEEE Access 8, 191586–
191601 (2020)
123