Kussul Et Al 2017 GRSL DeepLearning
Kussul Et Al 2017 GRSL DeepLearning
Kussul Et Al 2017 GRSL DeepLearning
net/publication/315939269
Deep Learning Classification of Land Cover and Crop Types Using Remote
Sensing Data
CITATIONS READS
1,323 23,269
4 authors:
Some of the authors of this publication are also working on these related projects:
Geospatial models and information technologies of satellite monitoring of smart city problems View project
Amazon and GEO Project "Methodology for SDGs indicators assessment" View project
All content following this page was uploaded by Andrey Yu. Shelestov on 07 November 2017.
Abstract— Deep learning (DL) is a powerful state-of-the-art multisources data fusion techniques [4], [5]. Land cover and
technique for image processing including remote sensing (RS) crop type maps are one of the most essential inputs when
images. This letter describes a multilevel DL architecture that dealing with environmental and agriculture monitoring tasks
targets land cover and crop type classification from multitempo-
ral multisource satellite imagery. The pillars of the architecture [6]–[8]. Multitemporal multisource satellite imagery is usually
are unsupervised neural network (NN) that is used for optical required in order to capture specific crop growth stages and
imagery segmentation and missing data restoration due to clouds thus being able to discriminate different crop types. For exam-
and shadows, and an ensemble of supervised NNs. As basic ple, multispectral optical imagery only might not be enough
supervised NN architecture, we use a traditional fully connected to discriminate summer crops in a complex and heterogeneous
multilayer perceptron (MLP) and the most commonly used
approach in RS community random forest, and compare them environment. For this, SAR-derived information adds an added
with convolutional NNs (CNNs). Experiments are carried out value that allows discrimination of particular crop types [9],
for the joint experiment of crop assessment and monitoring test [10].
site in Ukraine for classification of crops in a heterogeneous A comprehensive study on the state-of-the-art supervised
environment using nineteen multitemporal scenes acquired by pixel-based methods for land cover mapping was performed
Landsat-8 and Sentinel-1A RS satellites. The architecture with
an ensemble of CNNs outperforms the one with MLPs allowing by Khatami et al. [11]. They found that support vector
us to better discriminate certain summer crop types, in particular machine (SVM) was the most efficient for most applications
maize and soybeans, and yielding the target accuracies more than with an overall accuracy (OA) of about 75%. The second
85% for all major crops (wheat, maize, sunflower, soybeans, and method with approximately the same efficiency (74% of OA)
sugar beet). was a neural network (NN)-based classifier. In that study,
Index Terms— Agriculture, convolutional neural classification was done only for a single date image. At the
networks (CNNs), crop classification, deep learning (DL), same time, SVM is too much resource consuming to be
joint experiment of crop assessment and monitoring (JECAM), used for big data applications and large area classification
Landsat-8, remote sensing (RS), Sentinel-1, TensorFlow, Ukraine.
problems. Another popular approach in the RS domain is the
random forest (RF)-based approach [12]. However, multiple
I. I NTRODUCTION features should be engineered to feed the RF classifier for the
efficient use.
T HE last several years and onward could be called the
years of Big Free Data in remote sensing (RS). During the
2013–2016 period, several optical and synthetic aperture radar
Over the past few years, the most popular and efficient
approaches for multisensor and multitemporal land cover
(SAR) RS satellites were launched with high spatial resolution classification are ensemble-based [13]–[16] and deep learn-
(10–30 m), in particular Sentinel-1A/B and Sentinel-2A within ing (DL) [17]–[20]. These techniques are found to outperform
the European Copernicus program [1], [2], and Landsat-8 the SVM [21]–[23]. DL is a powerful machine learning
within the Landsat Project, a joint initiative between the U.S. methodology for solving a wide range of tasks arising in image
Geological Survey (USGS) and the National Aeronautics and processing, computer vision, signal processing, and natural
Space Administration [3]. These data sets are freely available language processing [24]. The main idea is to simulate the
on operational basis. This opens unprecedented opportunities human vision to deal with big data problem, use all the data
for a wide range of preoperational and operational applications available and provide the semantic information at the output.
in the environment and agricultural domains taking advantage Plenty of models, frameworks and benchmark databases of ref-
of high temporal resolution data sets and advances in the erence imagery are available for image classification domain.
Over past years, more and more studies have been using DL
Manuscript received February 17, 2017; accepted March 6, 2017. for processing of RS imagery [25], [26]. DL proved to be
N. Kussul and M. Lavreniuk are with the Department of Space Information efficient for processing both optical (hyperspectral and mul-
Technologies and Systems, Space Research Institute, National Academy
of Sciences of Ukraine and SSA Ukraine, 03680 Kyiv, Ukraine (e-mail: tispectral imagery) and radar images, in extracting different
inform@ikd.kiev.ua; nataliia.kussul@gmail.com; nick_93@ukr.net). land cover types such as road extraction, buildings extrac-
S. Skakun is with the Department of Geographical Sciences, University of tion [17], [27], [28]. In terms of particular DL architectures,
Maryland, College Park, MD 20742 USA (e-mail: skakun@umd.edu).
A. Shelestov is with the Department of Information Security, National convolutional NNs (CNNs), deep autoencoders, deep belief
Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute,” networks, and recurrent NN with long short-term memory
03056 Kyiv, Ukraine (e-mail: andrii.shelestov@gmail.com). model have already been explored for RS tasks [17], [28]–[31].
Color versions of one or more of the figures in this letter are available
online at http://ieeexplore.ieee.org. It should be noted that most studies with DL for RS utilize
Digital Object Identifier 10.1109/LGRS.2017.2681128 a single date image for classification purposes, e.g., land
1545-598X © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
TABLE I TABLE II
D ATES OF A CQUISITION OF L ANDSAT-8 AND S ENTINEL -1A N UMBER AND A REA OF P OLYGONS C OLLECTED D URING
FOR THE K YIV R EGION IN 2015 G ROUND S URVEYS FOR THE K YIV R EGION IN 2015
RF build a global transformation of features. The 2-D CNNs [20] N. Kussul, A. Shelestov, R. Basarab, S. Skakun, O. Kussul, and
outperformed the 1-D CNNs, but some small objects in the M. Lavrenyuk, “Geospatial intelligence and data fusion techniques
for sustainable development problems,” in Proc. ICTERI, 2015,
final classification map provided by 2-D CNNs were smoothed pp. 196–203.
and misclassified. [21] J. Ding, B. Chen, H. Liu, and M. Huang, “Convolutional neural
network with data augmentation for SAR target recognition,”
R EFERENCES IEEE Geosci. Remote Sens. Lett., vol. 13, no. 3, pp. 364–368,
Mar. 2016.
[1] M. Drusch et al., “Sentinel-2: ESA’s optical high-resolution mission [22] F. J. Huang and Y. LeCun, “Large-scale learning with SVM and
for GMES operational services,” Remote Sens. Environ., vol. 120, convolutional for generic object categorization,” in Proc. IEEE
pp. 25–36, May 2012. Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Jun. 2006,
[2] R. Torres et al., “GMES Sentinel-1 mission,” Remote Sens. Environ., pp. 284–291.
vol. 120, pp. 9–24, May 2012. [23] T. Ishii, R. Nakamura, H. Nakada, Y. Mochizuki, and H. Ishikawa,
[3] D. P. Roy et al., “Landsat-8: Science and product vision for terrestrial “Surface object recognition with CNN and SVM in Landsat 8 images,”
global change research,” Remote Sens. Environ., vol. 145, pp. 154–172, in Proc. 14th IAPR Int. Conf. Mach. Vis. Appl. (MVA), May 2015,
Apr. 2014. pp. 341–344.
[4] J. Zhang, “Multi-source remote sensing data fusion: Status and trends,” [24] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
Int. J. Image Data Fusion, vol. 1, no. 1, pp. 5–24, Nov. 2010. pp. 436–444, May 2015.
[5] M. D. Mura, S. Prasad, F. Pacifici, P. Gamba, J. Chanussot, and [25] F. Zhang, B. Du, and L. Zhang, “Saliency-guided unsupervised feature
J. A. Benediktsson, “Challenges and opportunities of multimodality learning for scene classification,” IEEE Trans. Geosci. Remote Sens.,
and data fusion in remote sensing,” Proc. IEEE, vol. 103, no. 9, vol. 53, no. 4, pp. 2175–2184, Apr. 2015.
pp. 1585–1601, Sep. 2015. [26] F. Zhang, B. Du, and L. Zhang, “Scene classification via a gradi-
[6] A. Kolotii et al., “Comparison of biophysical and satellite pre- ent boosting random convolutional network framework,” IEEE Trans.
dictors for wheat yield forecasting in Ukraine,” Int. Arch. Pho- Geosci. Remote Sens., vol. 54, no. 3, pp. 1793–1802, Mar. 2016.
togramm., Remote Sens. Spatial Inf. Sci., vol. 40, no. 7, p. 35, 2015, [27] V. Mnih and G. E. Hinton, “Learning to detect roads in high-
doi: 10.5194/isprsarchives-XL-7-W3-39-2015. resolution aerial images,” in Proc. Eur. Conf. Comput. Vis., 2010,
[7] F. Kogan et al., “Winter wheat yield forecasting: A comparative analysis pp. 210–223.
of results of regression and biophysical models,” J. Autom. Inf. Sci., [28] J. Geng, J. Fan, H. Wang, X. Ma, B. Li, and F. Chen, “High-resolution
vol. 45, no. 6, pp. 68–81, 2013. SAR image classification via deep convolutional autoencoders,” IEEE
[8] F. Kogan et al., “Winter wheat yield forecasting in Ukraine based on Trans. Geosci. Remote Sens., vol. 12, no. 11, pp. 2351–2355,
Earth observation, meteorological data and biophysical models,” Int. Nov. 2015.
J. Appl. Earth Observat. Geoinf., vol. 23, pp. 192–203, Aug. 2013. [29] Y. Chen, X. Zhao, and X. Jia, “Spectral–spatial classification of hyper-
[9] H. McNairn, A. Kross, D. Lapen, R. Caves, and J. Shang, “Early season spectral data based on deep belief network,” IEEE J. Sel. Topics Appl.
monitoring of corn and soybeans with TerraSAR-X and RADARSAT-2,” Earth Observ. Remote Sens., vol. 8, no. 6, pp. 2381–2392, Jun. 2015.
Int. J. Appl. Earth Observat. Geoinf., vol. 28, pp. 252–259, May 2014. [30] H. Liang and Q. Li, “Hyperspectral imagery classification using sparse
[10] S. Skakun, N. Kussul, A. Y. Shelestov, M. Lavreniuk, and O. Kussul, representations of convolutional neural network features,” Remote Sens.,
“Efficiency assessment of multitemporal C-band Radarsat-2 intensity vol. 8, no. 2, p. 99, 2016.
and Landsat-8 surface reflectance satellite imagery for crop classification [31] H. Lyu, H. Lu, and L. Mou, “Learning a transferable change rule from
in Ukraine,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., a recurrent neural network for land cover change detection,” Remote
vol. 9, no. 8, pp. 3712–3719, Aug. 2016. Sens., vol. 8, no. 6, p. 506, 2016.
[11] R. Khatami, G. Mountrakis, and S. V. Stehman, “A meta-analysis of [32] E. Vermote, C. Justice, M. Claverie, and B. Franch, “Preliminary
remote sensing research on supervised pixel-based land-cover image analysis of the performance of the Landsat 8/OLI land surface
classification processes: General guidelines for practitioners and future reflectance product,” Remote Sens. Environ., vol. 185, pp. 46–56, 2016,
research,” Remote Sens. Environ., vol. 177, pp. 89–100, May 2016. doi: 10.1016/j.rse.2016.04.008.
[12] P. O. Gislason, J. A. Benediktsson, and J. R. Sveinsson, “Random Forests [33] Z. Zhu, S. Wang, and C. E. Woodcock, “Improvement and expansion
for land cover classification,” Pattern Recognit. Lett., vol. 27, no. 4, of the Fmask algorithm: Cloud, cloud shadow, and snow detection
pp. 294–300, 2006. for Landsats 4–7, 8, and Sentinel 2 images,” Remote Sens. Environ.,
[13] M. Han, X. Zhu, and W. Yao, “Remote sensing image classification vol. 159, pp. 269–277, Mar. 2015.
based on neural network ensemble algorithm,” Neurocomputing, vol. 78, [34] F. Waldner et al., “Towards a set of agrosystem-specific cropland map-
no. 1, pp. 133–138, 2012. ping methods to address the global cropland diversity,” Int. J. Remote
[14] X. Huang and L. Zhang, “An SVM ensemble approach combining Sens., vol. 37, no. 14, pp. 3196–3231, 2016.
spectral, structural, and semantic features for the classification of high- [35] S. V. Skakun and R. M. Basarab, “Reconstruction of missing data in
resolution remotely sensed imagery,” IEEE Trans. Geosci. Remote Sens., time-series of optical satellite images using self-organizing Kohonen
vol. 51, no. 1, pp. 257–272, Jan. 2013. maps,” J. Autom. Inform. Sci., vol. 46, no. 12, pp. 19–26, 2014.
[15] M. S. Lavreniuk et al., “Large-scale classification of land cover [36] N. Kussul, S. Skakun, A. Shelestov, M. Lavreniuk, B. Yailymov, and
using retrospective satellite data,” Cybern. Syst. Anal., vol. 52, no. 1, O. Kussul, “Regional scale crop mapping using multi-temporal satellite
pp. 127–138, 2016. imagery,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., vol. 40,
[16] N. Kussul, G. Lemoine, F. J. Gallego, S. V. Skakun, M. Lavreniuk, and no. 7, pp. 45–52, 2015.
A. Y. Shelestov, “Parcel-based crop classification in Ukraine using [37] W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional
Landsat-8 data and Sentinel-1A data,” IEEE J. Sel. Topics Appl. Earth neural networks for hyperspectral image classification,” J. Sens., to be
Observ. Remote Sens., vol. 9, no. 6, pp. 2500–2508, Jan. 2016. published. [Online]. Available: http://dx.doi.org/10.1155/2015/258619
[17] Y. Chen, Z. Lin, X. Zhao, G. Wang, and Y. Gu, “Deep learning-based [38] D. P. Kingma and J. Ba. (Dec. 2014). “Adam: A method for stochastic
classification of hyperspectral data,” IEEE J. Sel. Topics Appl. Earth optimization.” [Online]. Available: https://arxiv.org/abs/1412.6980
Observ. Remote Sens., vol. 7, no. 6, pp. 2094–2107, Jun. 2014. [39] M. Abadi et al. (Mar. 2016). “TensorFlow: Large-scale machine
[18] W. Zhao and S. Du, “Learning multiscale and deep representations for learning on heterogeneous distributed systems.” [Online]. Available:
classifying remotely sensed imagery,” ISPRS J. Photogramm. Remote https://arxiv.org/abs/1603.04467
Sens., vol. 113, pp. 155–165, Mar. 2016. [40] F. J. Gallego, N. Kussul, S. Skakun, O. Kravchenko, A. Shelestov, and
[19] N. Kussul, N. Lavreniuk, A. Shelestov, B. Yailymov, and I. Butko, “Land O. Kussul, “Efficiency assessment of using satellite data for crop area
cover changes analysis based on deep machine learning technique,” estimation in Ukraine,” Int. J. Appl. Earth Observat. Geoinf., vol. 29,
J. Autom. Inf. Sci., vol. 48, no. 5, pp. 42–54, 2016. pp. 22–30, Jun. 2014.