Eur. Phys. J. Appl. Phys.

99, 28 (2024)
© N. Brun et al., Published by EDP Sciences, 2024
THE EUROPEAN
https://doi.org/10.1051/epjap/2024240025 PHYSICAL JOURNAL
APPLIED PHYSICS

ORIGINAL ARTICLE

EELS hyperspectral images unmixing using autoencoders

Nathalie Brun * , Guillaume Lambert, and Laura Bocher
Université Paris-Saclay, CNRS, Laboratoire de Physique des Solides, 91405 Orsay, France

Received: 20 February 2024 / Accepted: 17 September 2024

Abstract. Spatially resolved Electron Energy-Loss Spectroscopy conducted in a Scanning Transmission

Electron Microscope enables the acquisition of hyperspectral images. Spectral unmixing is the process of
decomposing each spectrum of a hyperspectral image into a combination of representative spectra
(endmembers) corresponding to compounds present in the sample along with their local proportions
(abundances). Spectral unmixing is a complex task, and various methods have been developed in different
communities using hyperspectral images. However, none of these methods fully satisfy the spatially resolved
Electron Energy-Loss Spectroscopy requirements. Recent advancements in remote sensing, which focus on Deep
Learning techniques, have the potential to meet these requirements, particularly Autoencoders. As the Neural
Networks used are usually shallow it would be more appropriate to use the term “representation learning”. In this
study, the performance of these methods using autoencoders for spectral unmixing is evaluated, and their results
are compared with traditional methods. Synthetic hyperspectral images have been created to quantitatively
assess the outcomes of the unmixing process using specific metrics. The methods are subsequently applied to a
series of experimental data. The findings demonstrate the promising potential of autoencoders as a tool for
Electron Energy-Loss Spectroscopy hyperspectral images unmixing, marking a starting point for exploring more
sophisticated Neural Networks.
Keywords: Electron energy-loss spectroscopy (EELS) / spectral imaging / spectrum image /
hyperspectral unmixing / autoencoder / deep learning

1 Introduction vectors lie very close to a low-dimensional subspace. One

limitation of PCA involves the non-physical characteristics
Continuous improvements in Scanning Transmission of the components extracted, which makes a physical
Electron Microscopes (STEM) and Electron Energy-Loss interpretation tricky. Although expressed over the same
Spectroscopy (EELS) have allowed the acquisition of spectral range, these components are not spectra, strictly
hyperspectral images (HSIs), (also known as spectral speaking. Thus, more processes beyond PCA are necessary
images
SI), with typical sizes of several tens of thousands to provide a complete data processing result, i.e., a set of
of pixels and around one thousand energy channels. reference spectra and corresponding maps that can be used
The fine structure of the characteristic edges provides to support an interpretation.
access to the bonding environment and electronic structure An HSI is usually processed through the traditional
of the elements constitutive of the sample. To interpret the background subtraction (BS) method and signal
data from a materials science point of view, the integration. The requirement separates a characteristic
characteristic components and maps from the HSI need edge (for example, Co-L2,3 edge) from its underlying
to be extracted. This extraction can be accomplished by background. The background is approximated to a
processing each spectrum individually, for example, by power law energy dependence AE -r, E being the energy
subtracting the background and adding the characteristic loss and A, r, two parameters to be measured over a
signal corresponding to a given edge. However, looking at fitting region immediately preceding the edge. Once the
the HSI as a whole and taking a statistical view of the data background has been removed under the characteristic
is more efficient as well as more systematic and relevant edge, the signal is integrated over the energy window of
regarding results [1]. interest.
One type of data processing application in STEM- Despite its simplicity, this procedure has some draw-
EELS involves dimensionality reduction with Principal backs. For example, the fitting window and the integration
Components Analysis (PCA). The information contained window have to be carefully chosen, and these choices may
in the HSI can be reduced to a few components, as spectral introduce a user bias. Another issue is that the procedure
cannot be applied when two edges overlap [2] or the same
element is present with different electronic structures
* e-mail: nathalie.brun@universite-paris-saclay.fr (valence, coordination) that we want to distinguish [3].
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
2 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

Consequently, it seems more efficient to fully exploit encoding-decoding architecture. AEs can be used in
the low dimensionality structure of the HSI used in PCA unsupervised way comparing the reconstructed spectrum
and perform a form of data analysis that directly provides to the original spectrum for each pixel. Moreover, with a
the desired result, as spectral unmixing (SU) algorithms specific AE architecture, it is possible to perform spectral
can extract significant components of the sample and unmixing, and several algorithms have been proposed
compute associated maps [2,3]. The spectrum collected at [12–25]. EndNet [17] is based on a two-staged AE
an individual pixel is usually a mixture of the signatures of network with additional layers and a particular loss
the different atoms interacting with the beam. Mixed pixels function. DAEN [18–20] is an AE consisting of two parts:
occur if the spatial resolution is low or if different a stacked AE for initialisation and a variational AE for
compounds are present in the sample thickness intersected unmixing.
by the electron probe (e.g., particles in a matrix, diffusion The case of non-linearity is addressed by adding a non-
at an interface, an atomic column with different elements), linear component into the layers of the architecture
leading to an impure spectrum. Many techniques have been [12,14,15]. In work [14], these networks are improved by
suggested to unmix the impure spectrum and recover the incorporating the spatial structure of the data using a
pure signals corresponding to the individual components of 3D-Convolutional Neural Network (CNN). New works
the sample. have combined this spectral-spatial information with
A standard technique is Linear Mixing Model architectures designed to cope with the endmember
(LMM), which assumes that an individual spectrum is variability [26,27]. An adaptation of the architecture used
a linear combination of pure spectra [4]. In the case of in [13] is presented in [16].
EELS spectroscopy, a pure spectrum can correspond to An occurrence of unmixing AE appeared in [22] and was
one element or an element with a specific structural and developed in [23]. The work [24], using an architecture
electronic environment. For example, in [5], one seeks to inspired by multitask learning, operates on image patches
separate the signal of Fe in a six-fold (octahedron) and Fe (a small, fixed-size sub-region or block of an image) instead
in a five-fold (distorted tetragonal pyramid) oxygen of single pixels to utilise the spatial structure. CNN is used
coordination. A pure spectrum can also contain two in [25] to capture the spatial correlations existing in HSIs.
different elemental thresholds, as in [2]: one pure Recently, a transformer network has been combined
spectrum with both Ti-L2,3 and O-K and another with recently with a convolutional AE to capture the interaction
Sn-M and O-K. between image patches [28].
While the pixel size for an EELS SI is typically 0.05 nm This article does not include a complete list of
for atomic resolution, at a completely different scale (about references, as the number of studies devoted to AEs
1 m per pixel), remote sensing (use of satellite- or aircraft- applied to SU has increased rapidly in recent years. Only
based sensor technologies to detect and classify objects on some of the previously described codes are publicly
Earth) produces HSIs with a data structure identical to available to perform unmixing with AEs, although, a
STEM-EELS SI. Due to the importance of military, series of codes have recently been made available to the
intelligence, commercial, economic, planning, and human- community [29].
itarian applications, numerous frameworks have been To evaluate the performance of these methods as applied
developed to analyse vast quantities of data [4]. The to STEM-EELS HSIs, state-of-the-art and often quoted
STEM-EELS community can thus benefit from these models that are among the publicly available ones were
results. selected, including uDAS [21], deep AE unmixing (DAEU)
Many recent publications have discussed novel Deep [23], multitask AE (MTAEU) [24] and CNN AE (CNNAEU)
Learning techniques [6–9] and applied them to processing [25]. These algorithms are presented in Section 2.2. The
remote sensing data. performances of these algorithms are compared to those of
Applied to grayscale, colour, or hyperspectral images, conventional unmixing algorithms currently used in the
deep learning methods leverage extensive datasets to STEM-EELS community, such as Independent Component
achieve high accuracy in various tasks. But recent Analysis (ICA), Non-Negative Matrix Factorization (NMF)
advancements employ architectures such as U-Net which (as implemented in the popular toolbox Hyperspy [30]),
can achieve high performance even when trained with a few Vertex Component Analysis (VCA) [31] that appears at the
dozen images, making it highly effective for applications moment as the most versatile algorithm to perform spectral
with limited data availability [10]. unmixing, and BLU [2,32], which is a Bayesian algorithm
Indeed, in the case of hyperspectral remote sensing that estimates the endmembers and the abundances jointly
images, access is only available for individual images. The in a single step.
subsequent dataset consists of a single HSI, where each Representation learning algorithms need to be verified
pixel represents an item (or a group of pixels for methods before they can replace the traditional SU techniques. They
that incorporate the spatial structure of the HSI). The nevertheless hold the potential to improve the quality of
training is then performed on the dataset defined by all the the results, as well as the execution time. A neural network
pixels of the HSI. Thus, there is no need to rely on an entire can be long to train but, its inference is high-speed if
library of HSI. applied to different data sets, such as a series of HSIs
Some interesting results have been obtained for acquired on the same sample, or similar samples during an
denoising and classification in remote sensing [11]. acquisition session on a given microscope. It is essential to
In particular, autoencoders (AE), a type of neural compare the performance of the different algorithms
network architecture, are based on the principle of an quantitatively.
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 3

Fig. 1. (a) The three components set used to create a synthetic spectrum image
they have been extracted from an experimental
dataset, (b)–(d) 64  64 maps obtained with the chequerboard model (8  8 blocks); each of these maps is associated with one
component.

Synthetic datasets were generated using the method model. Two sets of endmembers were used, one with three
described in [33] to provide this quantitative assessment. endmembers and the other with four endmembers.
These algorithms were then applied to an experimental Endmembers were extracted from the experimental data-
dataset. As no ground truth (GT) is available for this set of Section 4 for the three components HSI (Fig. 1) and
dataset, only a qualitative evaluation was performed using obtained from data described in [2,34] for the four
the chemical maps obtained by the usual BS method. components HSI (Fig. 2).
The remainder of the paper is organised as follows. The three components set has 200 energy channels, and
Section 2.1 describes the synthetic datasets and metrics used the four components set has 903 energy channels. For the
to quantitatively evaluate the different unmixing algo- component referenced as “Pt”, no Pt edge is present in this
rithms. Section 2.2 briefly presents the different algorithms energy range, as the Pt-M edge energy is 2122 eV. The Pt
used and Section 3.1 the results obtained for the synthetic component here is only the background in the part of the
datasets. Section 3.2 applies the same algorithms to real SI sample containing Pt.
datasets. Finally, Section 4 is the conclusion. For the maps, images of 64  64 pixels were divided into
Some data and code will be available after publication units of 8  8 small blocks, as described in [33]. Each block
at: https://github.com/NathalieBrun/EELS-hyperspec was randomly assigned to one of the endmembers. A k  k
tral-images-unmixing-using-autoencoders. averaging filter then degraded the resulting image to create
mixed pixels, with k = 9, with abundances respecting the
2 Materials and methods sum-to-one rule (Fig. 1).
The maps were combined with the components following
2.1 Synthetic datasets and metrics the LMM to create a synthetic SI. Finally, Poissonian noise
was applied to the SI using the in-built Hyperspy method.
The performances of different state-of-the-art unmixing Poissonian noise was chosen in preference to Gaussian noise
methods were compared with those of Representation because the increasingly frequent use of direct detection
Learning based methods using synthetic data. The cameras provides data degraded by this type of noise, as is
synthetic data was generated with the linear mixture the case for experimental data from Section 3.2.
4 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

2.2 Unmixing algorithms

Before proceeding to unmixing, the number of endmembers

has to be estimated, as this is a parameter used by the
unmixing AE. Although various algorithms already exist
for this estimation [35], the process remains a delicate task.
The estimation is all the more difficult as the definition of
endmembers is subjective and depends on the degree of
precision to be attained. For example, if the Co in the
sample had different degrees of oxidation, it is not clear if it
would be necessary to distinguish metallic Co from Co2+
and Co3+. The number of endmembers also depends on the
level of information sought and the experimental con-
ditions of data acquisition (energy resolution for EELS
spectroscopy). Estimating the number of endmembers is
particularly difficult in remote sensing, where the sensor
can uncover unknown target sources that cannot be
identified a priori. However, this estimation process is
different in microscopy, where the global composition of
Fig. 2. The four components set used to create a synthetic
spectrum image. They have been obtained by fitting spectra
the sample, which is a manufactured material, is well
extracted from the dataset used in [2,30]. known. On the other hand, including more endmembers
will not immediately improve the quality of unmixing
results. Thus, in reference [29], increasing the number of
For quantitative comparison unmixing results from the endmembers for the Urban image unmixing degrades the
synthetic data were evaluated using two metrics, Spectral results for most tested AEs.
Angle Distance (SAD) and Normalised Mean Squared In this study, the number of endmembers for the
Error (NMSE). SAD, a widely used metric in HSI analysis, synthetic images was the same as the number used to
was used to measure the similarity between the extracted generate the data. As the sample composition was known
endmember and the true endmember (GT). for the experimental data, this physical information was
SAD is expressed as: part of the unmixing process. Thus each element present

 in the sample, i.e., Ru, Co and Pt, corresponded to one

1 aT a^ spectral signature and then to one endmember. This
SADða; a^ Þ ¼ cos ð1Þ
kak2 k^
a k2 choice of the number of endmembers implied that any
spectral variation occurring in the data was assimilated to
with a the spectral vector corresponding to the true noise.
endmember (GT), â its estimation by the Neural Network, The methods compared in this work are listed in
aT is a transposed, and k.k2 is the ℓ2 norm. Table 1.
NMSE measured the performance of abundance Although many spectral unmixing algorithms exist,
estimation defined as: unmixing is not frequently used in the STEM-EELS
domain. The algorithms most used are NMF and ICA
as
  ^ 2 implemented in the Toolbox Hyperspy
and VCA.
^ ¼ kZ
ZkF
NMSE Z; Z ð2Þ Hyperspy uses the Scikit-learn version of the NMF
kZk2F
algorithm, sklearn.decomposition.NMF [36]. NMF decom-
with Z the true abundance of a given endmember (it is a poses the data matrix X by resolving the optimisation
matrix) and Ẑ its estimation by the Neural Network, k.kF problem:
being the Frobenius norm defined as:
minðLðX; WHÞÞ with W; H > 0: ð4Þ
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
X
u
kZkF ¼ u jZ ij j2 ð3Þ
u X being the data, L a loss function measuring the
t1  i  n similarity between X and the decomposition, and W and
1  j  m H the matrices resulting from the decomposition.
A regularisation term can be added to the loss function.
with n, m being the size of the spatial dimensions of the Furthermore, the initialisation mode can be changed, as
data cube and |Zi,j| the absolute value of matrix element Zi,j. well as the distance. The default values provided in
SAD and NMSE are frequently used in the literature on Hyperspy were kept (Frobenius norm, no regularisation,
hyperspectral unmixing [14,27,29]. initialisation by Non-negative Double Singular Value
For each set of endmembers 20 different series of three Decomposition).
(respectively four) chequerboard images were generated ICA decomposes X, assuming that the components are
using the method described previously. Thus, for each statistically independent [37,38]. Independency is a strong
algorithm tested, a mean and a standard deviation were assumption whose relevance has been questioned in the
obtained for each evaluation by a given metric. remote sensing community [31,38]. It is a widely used
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 5

Table 1. Methods used in this work (adapted from [29]).

Name Architecture Implementation

NMF Non-negative Matrix Factorisation Hyperspy [29]
ICA Independent Component Analysis Hyperspy [29]
VCA30 Geometrical based approach Code available on the web
BLU31 Statistical Bayesian Code available on the web
DAEU22 Deep AE Code available on the web
MTAEU23 Deep encoder utilising MTL Code available on the web
CNNAEU24 Fully CNN encoder and decoder Code available on the web
uDAS20 Untied denoising AE with sparsity Code available on the web

Fig. 3. Basic principle of an autoencoder. The hidden layer is a bottleneck that forces a compressed input representation. The input
example is a 2D image from the MNIST dataset [42]. The 2D 28  28 pixels image is processed as a 784 vector. In our case, the input is a
spectrum.

method in STEM-EELS, probably due to its implementa- sensing community and is often used as a reference to
tion from the first versions of Hyperspy. Hyperspy uses, by evaluate new unmixing algorithms. It has already been
default, the Scikit-learn implementation, (https://scikit- used to unmix STEM-EELS SI [2,40,41].
learn.org/stable/modules/generated/sklearn.decomposi BLU is a fully Bayesian algorithm which uses a Gibbs
tion.FastICA.html#sklearn.decomposition.FastICA). This sampler algorithm to solve the unmixing problem without
implementation is based on [39]. This default version was requiring the presence of pure pixels in SI [32]. Its
used in the present work. performance for EELS HSIs has been evaluated in [2].
VCA is one of the most advanced convex geometry- AEs can be used for unsupervised learning technique
based endmember detection methods. It is based on using neural networks to learn a latent space representa-
successive projections on hyperplanes [31]. This algorithm tion of the input. In this work, we used only AEs with small
assumes the presence of at least one pure pixel for each number of layers (typically less than four), and they belong
component in the data. If there is no pure pixel, it uses the to the domain of representation learning rather than
highest quality pixel that is available. Although the pure traditional Deep Learning. The part of the network that
pixel condition is not always verified in STEM-EELS HSIs, compresses the input into the latent representation is called
this algorithm is fast and computationally relatively light. the encoder. The part that reconstructs the input from the
VCA has been implemented in Python (https://github. latent representation is the decoder. The AE architecture
com/Laadr/VCA) and Matlab (http://www.lx.it.pt/ imposes a bottleneck that forces a compressed input
∼bioucas/code.htm). It is commonly used in the remote representation (Fig. 3).
6 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

The AE is trained by using the input as target data, Table 2. Detail of layers of the encoder in DAEU network
meaning that the AE learns to reconstruct the original [23]. B is the number of channels of the input (spectrum). R
input. The decoder part of the AE aims to reconstruct the is the number of units of the latent, hidden layer, i.e., the
input from the latent space representation. By limiting the number of components to unmix. The utility layer
decoder to one layer, it has been shown that the activations performs an operation not specific to a neural network;
of the last layer of the encoder correspond to the in particular, the utility layer does not change the number
abundances and the weights of the decoder to the of units.
endmembers [23]. The encoder converts the input spectra
to the corresponding abundance vectors, i.e., the output of Layer# Layer type Activation Units#
the hidden layer. The decoder reconstructs the input from
the compressed representation with the weights in the last 1 Input – B
linear layer interpreted as the endmember matrix [29]. The 2 Dense LReLU 9R
action of the last layer of the decoder can be written as: 3 Dense LReLU 6R
4 Dense LReLU 3R
^ p ¼ W ðLÞ aðL
1Þ
x ð5Þ 5 Dense LReLU R
where x ^ p is the output of the network (reconstructed 6 Batch Normalization Utility R
spectrum), i.e., an estimation of the input xp, a(L-1) are the 7 Dynamical Soft LReLU R
activations of the previous layer, W(L) are the weights of Thresholding
the output layer, L being the total number of layers, B the 8 ASC enforcing Utility R
number of bands and R the number of endmembers. 9 Gaussian Dropout Utility R
a(L
1) is of dimension R1 and W(L) is a BR matrix,
which has to be interpreted as abundances and endmem-
bers for a given input. The weights are fixed once the
network is trained, and the endmembers are determined for branches again after this shared layer, and each autoencoder
the whole dataset. The activations are dependent on the performs dimensional reduction. The activations for each
input (pixel) analyzed. autoencoder become the abundance fractions. However,
According to this principle, the decoder must be a single all autoencoders use the same decoder to discover the
layer, and this simple structure might affect the perfor- same endmembers. The loss function is the SAD. MTAEU
mance of the AE; however, the experiments show that this perform significantly better than DAEU and uDAS on
AE performs well in unmixing the remote sensing data. remote sensing data sets.
Although several articles have proposed neural networks to The CNNAEU [25] employs 2-D convolutional layers
achieve unmixing, the corresponding code is only some- with LeakyReLU activation, Batch Normalization, and
times published in parallel. This lack of information is Spatial Dropout. A Softmax function is applied to enforce
detrimental, as not only is re-implementing the code time- the sum-to-one constraint on the abundances. The output
consuming, but many implementation details, such as of this layer can be interpreted as the abundance fractions.
utility layers and hyperparameters values, are not specified The final layer is a linear decoder layer that reconstructs
in publications on the subject, while modifying these the input patch. The loss function used is the SAD.
features can significantly alter the results. Recent efforts CNNAEU is claimed to perform particularly well for
have started to mitigate this issue, and several codes are endmembers extraction.
available on the web, e.g. uDAS on GitHub (https:// Implementations of DAEU (see Tab. 2), MTAEU and
github.com/aicip/uDAS). Although linked to the field of CNNAEU have been recently made available (https://
Deep learning by its keywords, uDAS is a shallow AE with github.com/burknipalsson/hu_autoencoders) with the
only one encoding layer. Its architecture makes it corresponding publication [29].
somewhat close to a conventional optimisation method The models for all the AEs evaluated in this work were
for an inverse problem, with an alternating optimisation of those available online without changing the architecture or
the encoder and the decoder [21]. the hyperparameters.
DAEU [23] comprises an encoder with 4 Dense layers
with Leaky ReLU activation functions, followed by utility 3 Results and discussion
layers (Batch Normalization, enforcing ASC SumToOne
layer, Gaussian Dropout layer). The LMM constraints the 3.1 Unmixing of synthetic data
architecture of the decoder which is a single layer.
Objective function is spectral angle distance (SAD). This The results of the different unmixing algorithms are shown
method performs better than the comparison methods in Figures 4 and 5.
(VCA, different NMF-based methods) for tested remote uDAS is the algorithm that gives the best results for
sensing data sets. both sets of synthetic data, three and four components. In
To take the spatial structure of the HSI into account, contrast, ICA and NMF give the worst results. The others
MTAEU [24] uses multiple parallel branches of unmixing (VCA, BLU, DAEU, MTAEU, CNNAEU) produce
autoencoders (AEs), each tasked with unmixing a whole intermediate-quality results. uDAS has the particularity
patch. The inputs (patches) are concatenated and of including denoising and regularisation constraints
connected into a large shared hidden layer. The architecture (ℓ2,1 on endmembers), which may explain the good results
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 7

Fig. 4. Performance of different algorithms for abundances maps estimation with a log scale. ‘BS’ is the background subtraction
method.

Fig. 5. Performance of different algorithms for endmembers extraction (low score is better).

obtained. It should be noted that the chequerboard No significant improvement is observed in MTAEU and
structure of synthetic images creates a non-zero proportion CNNAEU compared to DAEU, despite their higher
of pure pixels, which favours the use of VCA and uDAS (as complexity in accounting for the HSI spatial correlations
it is initialised with VCA). between pixels. These methods utilize this spatial informa-
The results are qualitatively the same for ranking the tion by treating the input as a patch (a patch refers to a
algorithms in terms of performance for the two types of small block of pixels, typically a 3  3 square with 9 pixels).
synthetic data: three components with 200 energy channels Thus, while patch-based methods usually outperform
(Fig. 1) and four components with 903 energy channels conventional methods in image analysis, in this case spatial
(Fig. 2). Moreover, the spectra chosen to build the data are structure does not have an impact on the quality of
significantly different, with M and L edges (Fig. 1) versus K unmixing for STEM-EELS HSIs.
edges (Fig. 2), which does not influence the results The higher number of hyperparameters related to a
significantly. complex architecture may require adjustment to the
The SAD metric is scale-invariant and solely takes into characteristics of the STEM-EELS HSIs, i.e., more energy
account the shape of an extracted component. Its absolute channels and fewer pixels.
amplitude can be very different from the reference The computation times required by each algorithm are
endmember without affecting the result. In contrast, the reported in Table 3 (3 GHz Intel® CoreTMi7-1185G7

NMSE metric will calculate a significant error for a except for CNNAEU, which has been trained on a
calculated abundance map that does not respect the sum- computer with a GPU NVIDIA Quadro RTX4000 8Go
to-one rule. NMF and ICA methods do not use this (7.5 Cuda score)). The complexity of VCA, ICA, and NMF
constraint, so they get a high error. is lower than those of DAEU, BLU and uDAS.
8 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

Table 3. Execution time (expressed in s) for different As the Ru is a delayed M-edge (279 eV for M4,5), it was
algorithms on synthetic data. For the AE, the time is the challenging to determine the maps using the BS and
training time. characteristic signal integration method used in EELS.
Indeed, in EELS the typical method for obtaining an
Unmixing 3 Components-200 4 Components-900 elemental map involves fitting the background with a
method channels SI channels SI power law over a specified energy interval and then
subtracting this background. Finally, the signal is summed
NMF 2.2 5.2 in a second energy window. This process is usually
ICA 0.1 0.1 performed using a dedicated software such as Gatan
VCA-FCLS 0.1 0.1 DigitalMicrograph, as illustrated in Figure 7. In the case of
BLU 659 2055 Ru, we are unable to satisfactorily model the background
DAEU 84 122 under the Ru M4,5 edge (Fig. 7a), leading to non-physical
negative intensity in some parts of the image (Fig. 7e).
MTAEU 192 612 Therefore, using an unmixing method for this type of data
CNNAEU 491 1726 was interesting. Nevertheless, the M2,3 edge of Ru was used
uDAS 72 1021 to obtain intensity maps to compare with the maps obtained
by unmixing. (Fig. 7d), as the M2,3 edge is detectable with
the direct detection camera and the background can be
successfully fit in this energy range (Fig. 7b).
As the acquisition of the data is relatively fast, around The profiles in Figure 7 are obtained by summing three
10 minutes for core loss data, even less with the new lines of pixels corresponding to 3.6 nm width at the top of
generation of direct detection detectors, the microscope the HSI. There is an artefact with a small non-zero
user might want to process the data quickly, whether done intensity outside the Ru layer, depending on the pre-edge
after or online during the experiment on the microscope. energy window selection.
During the acquisition time, using basic neural networks The Pt-M edge is in a high-energy range (2122 eV), so
could allow the training to be carried out with a first data this edge was not used in the unmixing process.
cube (or a previous one in the case of a series of As it is a real sample, there is no available GT, so it was
experiments). Then one could apply the trained network impossible to compute metrics for experimental data; then,
to the following acquisitions, reducing the execution time the evaluation was qualitative. However, the profiles
to 0.1 seconds. The experimental conditions of STEM- obtained by unmixing were compared with those obtained
EELS are thus particularly well adapted to using a neural by the BS method. The following results are obtained with
network because these networks allow exploiting several the processing of data of SI1, which is a restricted area of
HSIs acquired under the same conditions. This situation 60  75 pixels. The extracted components are presented in
differs from the case of HSIs acquired in remote sensing, Figure 8.
where the cases presented in the literature correspond to The Co component is well-extracted in all cases. The
the exploitation of a single HSI. Even if the performance of components obtained by VCA, BLU and uDAS are very
AEs for spectral unmixing is currently limited, their use similar, probably because uDAS and BLU are initialised by
remains interesting in STEM-EELS because of their speed VCA.
in inference. Ru component extraction is more challenging, and there
is still a Co signal in all components except in the component
3.2 Experimental dataset extracted by DAEU. VCA relies on the pure pixel hypothesis,
and there is probably no pure pixel corresponding to Ru.
The different algorithms were applied to a HSI acquired on A remarkable result is that the DAEU neural network not
a Pt/Co/Ru/Pt multilayer [43]. These heterostructures only manages to remove the Co but obtains a component
were investigated regarding their magnetic properties, i.e., close to the reference edge obtained in [44]. Despite a very low
Dzyaloshinskii-Moriya interaction, at metallic interfaces. dispersion (about 3.3 eV/channel), some fine structure is
The nominal stacking corresponds to: Si/SiO2/Ta(10 nm)/ present on the Ru-M edge. Although they have much more
Pt(8 nm)/Co(1.7 nm)/Ru(0.5 nm)/Pt(3 nm). In these complex architectures, MTAEU and CNNAEU do not
samples, the Ru layer and its top and bottom interfaces, manage to extract the Ru component more satisfactorily
i.e., Pt/Ru and Ru/Co, respectively, can have an impact on than DAEU in the case of SI1. The resulting maps are
the local magnetic properties of the stacking. Therefore, presented in Figure 9.
characterising this layer and the corresponding interfaces is As we do not have ground truth for the experimental
essential. In particular, the Ru can diffuse into the Co layer, data, we relied on the maps obtained through continuous
and it is thus necessary to establish a profile for Ru. background subtraction and summation of the character-
The data were acquired on a USTEM Nion microscope istic signal (BS) to assess the quality of the maps calculated
operated at 100 kV using a Medipix3 detector (Merlin EM by the unmixing methods. However, it did not seem feasible
Quantum Detector) with a 50 ms dwell time. The HSI is to use these BS maps for a quantitative evaluation, so our
60  75 pixels  200 energy channels (Fig. 6). A pixel discussion of the results is based on a visual comparison.
represents 0.12  0.12 nm and an energy channel 3.33 eV. The maps obtained with NMF are visually the closest to
Data were corrected for gain before any advanced those obtained by BS. The maps obtained by VCA, BLU
processing. and uDAS are satisfactory. DAEU, MTAEU and
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 9

Fig. 6. HAADF image corresponding to the region where HSI acquisition was conducted.

Fig. 7. Using the M4,5 edge (a) to map the Ru (c) produces negative values for intensities, as can be seen in the resulting profile (e). The
M2,3 edge (b) gives better results (d). On the profile (f) there is an artefact with a small non-zero intensity outside the Ru layer.

CNNAEU give very contrasting maps with a steep obtained by summing three rows from the Ru maps, in the
interface between the Co and Ru layers, which does not same way as the profiles presented in Figure 7. The Ru
correspond to the physical reality. For DAEU and profile presents an asymmetry with diffusion in the Co
MTAEU, the abundances are close to either 0 or 1; this layer.
problem has been reported in the literature for this class of If the unmixing is correctly performed, the shape of the
AEs [24]. Despite their complexity, MTAEU and profile should be close to those obtained by the BS
CNNAEU do not perform better than DAEU on the method. The comparison of abundance profiles is a
experimental data. It might be necessary to adjust some criterion that is not, to our knowledge, used in remote
hyperparameters, which is a complex task, as the networks sensing, where one relies solely on metrics and the visual
are trained on the reconstruction quality rather than on the comparison of maps to evaluate the performance of
quality of the unmixing. different methods. As was the case for the maps, the
To obtain a more precise estimation of the quality of the profile obtained by NMF corresponds to the profile
unmixing, the profiles obtained by the different unmixing obtained by BS and appears close to physical reality. The
methods are presented in Figure 10. The profiles were profiles obtained by VCA and BLU are also satisfying.
10 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

Fig. 8. Components obtained by the different unmixing algorithms tested on SI1. Each component corresponds to a unique edge,
Ru-M and Co-L2,3.

Fig. 9. Maps obtained by the different unmixing algorithms tested on SI1. As each component corresponds to a unique edge, they are
elementary maps.

They are close to zero, far from the layer on the left part of considered. Moreover, the synthetic chequerboard maps
the layer, while it was not possible to eliminate the signal contain a certain proportion of pure pixels for each
by subtracting the background in front of the threshold. component. In contrast, in the experimental situation,
However, an anomaly is observed on the right side of the one of the components (Ru) corresponds to almost no pure
profile. Some degree of spectral variability (caused, for pixels. To elaborate synthetic data closer to reality would
example, by variations in thickness) could explain why the be necessary to obtain a relevant evaluation of the
algorithms have difficulty representing the data set with efficiency of the unmixing algorithms.
only three components.
uDAS somehow reproduces the asymmetry of the
profile but shows a non-zero intensity away from the Ru 4 Conclusions
layer. DAEU, MTAEU and CNNAEU fail to reproduce the
asymmetry of the profile; thus, the weight of the Ru This work demonstrates that AEs give interesting results for
component falls to 0 in the region where it is mixed with Co. spectral unmixing. In particular, suitable extraction of the
Therefore, the improved extraction of the Ru component Ru component can be obtained despite the absence of pure
by DAEU did not result in a satisfactory map. pixels for this element in the experimental data. Moreover,
The results obtained on the synthetic data are not fully the organisation of the STEM-EELS experiments makes
applicable to the experimental data. Several limitations of them well adapted to Representation Learning: the network
synthetic SI can be identified. Firstly, these images are is trained on the first set of data (first acquired HSI) and then
constructed very simply; spectral variability is not the weights are applied to the data acquired subsequently
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 11

Fig. 10. Profiles through Ru maps presented in Figure 9. Intensities values (arbitrary units) have been re-scaled to match the ‘BS’ profile.

while benefiting from a swift execution time. This procedure data. However, they output worse results on our STEM-
can also apply if a series of very similar samples is studied (for EELS data. One hypothesis is that this failure is due to the
example, in the case of Pt/Co/Ru sample by varying the specific shape of the EELS spectra with a strong signal
thickness of the Ru layer). represented by the continuous background and relatively
More complex neural networks such as CNNAEU and weak superimposed specific signals. Adapting either the
MTAEU, which are efficient according to the literature in hyperparameters (batch size, number of hidden units...) or
remote sensing [29], should be able to handle STEM-EELS the architecture would probably be necessary.
12 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024)

On the other hand, the results obtained on the 5. S. Turner, R. Egoavil, M. Batuk et al., Site-specific mapping
experimental data are not as good as expected from the first of transition metal-oxygen coordination in complex oxides,
tests on the synthetic data. The model used to create synthetic Appl. Phys. Lett. 101, 241910 (2012), https://doi.org/
HSI is too simple, and introducing a degree of spectral 10.1063/1.4770512
variability would be helpful, for example by using a variational 6. L. Ma, Y. Liu, X. Zhang, Y. Ye, G. Yin, B.A. Johnson, Deep
AE (VAE) [29] which encodes the input as a distribution. learning in remote sensing applications: a meta-analysis and
Researchers continue to make progress on hyper- review, ISPRS J. Photogramm. Remote Sens. 152, 166
spectral unmixing by Neural Networks. Such progress (2019), https://doi.org/10.1016/j.isprsjprs.2019.04.015
can be achieved through cooperative work in the field, in 7. L. Zhang, L. Zhang, B. Du, Deep learning for remote sensing
particular by allowing open access to the codes used [11]. data: a technical tutorial on the state of the art, IEEE Geosci.
Remote Sens. Mag. 4, 22 (2016), https://doi.org/10.1109/
Acknowledgments MGRS.2016.2540798
8. A. Signoroni, M. Savardi, A. Baronio, S. Benini, Deep
We thank André Thiaville (LPS, Orsay), William Legrand, Nicolas learning meets hyperspectral image analysis: a multidisci-
Reyren and Vincent Cros (UMP CNRS/Thales, Palaiseau) for the plinary review, J. Imag. 5, 52 (2019), https://doi.org/
Pt/Co/Ru data. We are grateful to Calvin Peck from Academic 10.3390/jimaging5050052
Writing Center (U. Paris Saclay) for his patience in editing the
manuscript. 9. X.X. Zhu, D. Tuia, L. Mou et al., Deep learning in remote
sensing: a review, IEEE Geosci. Remote Sens. Mag. 5, 8
(2017), https://doi.org/10.1109/MGRS.2017.2762307
Funding
10. O. Ronneberger, P. Fischer, T. Brox, U-Net: convolutional
networks for biomedical image segmentation, in Medical
This project has been funded in part by the National Agency for
Research under the program of future investment TEMPOS- Image Computing and Computer-Assisted Intervention

CHROMATEM (reference no. ANR-10-EQPX-50) and by the MICCAI 2015. MICCAI 2015. Lecture Notes in Computer
European Union’s Horizon 2020 research and innovation program Science, edited by N. Navab, J. Hornegger, W. Wells, A.
under grant agreement No. 823717 (ESTEEM3). Frangi (Springer, Cham, 2015), vol 9351. https://doi.org/
10.1007/978-3-319-24574-4_28
Conflicts of interest 11. N. Audebert, B. Le Saux, S. Lefèvre, Deep learning for
classification of hyperspectral data: a comparative review,
The authors have nothing to disclose. IEEE Geosci. Remote Sens. Mag. 7, 159 (2019), https://doi.
org/10.1109/MGRS.2019.2912563
Data availability statement 12. M. Wang, M. Zhao, J. Chen, S. Rahardja, Nonlinear
unmixing of hyperspectral data via deep autoencoder
Data are available on request. The codes used are the ones cited in networks, IEEE Geosci. Remote Sens. Lett. 16, 1467
the references and are available on line. (2019), https://doi.org/10.1109/LGRS.2019.2900733
13. M.Zhao,M.Wang,J.Chen,S.Rahardja,HyperspectralUnmixing
Author contribution statement via Deep Autoencoder Networks for a Generalized Linear-
Mixture/Nonlinear-Fluctuation Model, arXiv:1904.13017,
Methodology, N. B., L. B. and G. L.; Data Acquisition, L. B.; https://doi.org/10.48550/arXiv.1904.13017
Data Analysis, N. B., L. B. and G. L.; Writing
Original Draft
Preparation, N. B.; Writing
Review & Editing, N.B., L. B.. 14. M. Zhao, M. Wang, J. Chen, S. Rahardja, Hyperspectral
unmixing for additive nonlinear models with a 3-D-CNN
autoencoder network, IEEE Trans. Geosci. Remote Sens. 60,
References 1 (2021), https://doi.org/10.1109/TGRS.2021.3098745
15. S. Shi, M. Zhao, L. Zhang, J. Chen, Variational autoencoders
1. N. Bonnet, N. Brun, C. Colliex, Extracting information from for hyperspectral unmixing with endmember variability, in
sequences of spatially resolved EELS spectra using multivar- ICASS P 2021–2021 IEEE International Conference on
iate statistical analysis, Ultramicroscopy 77, 97 (1999), Acoustics, Speech and Signal Processing (ICASSP) (Tor-
https://doi.org/10.1016/S0304-3991(99)00042-X onto, ON, Canada, 2021), p. 1875, https://doi.org/10.1109/
2. F. de la Peña, M.-H. Berger, J.-F. Hochepied, F. Dynys, O. ICASSP39728.2021.9414940.
Stephan, M. Walls, Mapping titanium and tin oxide phases 16. H. Li, R.A. Borsoi, T. Imbiriba, P. Closas, J.C. Bermudez, D.
using EELS: an application of independent component Erdogmus, Model-based deep autoencoder networks for
analysis, Ultramicroscopy 111, 169 (2011), https://doi.org/ nonlinear hyperspectral unmixing, IEEE Geosci. Remote
10.1016/j.ultramic.2010.10.001 Sens. Lett. 19, 1 (2021), https://doi.org/10.1109/
3. N. Dobigeon, N. Brun, Spectral mixture analysis of EELS LGRS.2021.3075138
spectrum-images, Ultramicroscopy 120, 25 (2012), https:// 17. S. Ozkan, B. Kaya, G.B. Akar, Endnet: Sparse autoencoder
doi.org/10.1016/j.ultramic.2012.05.006 network for endmember extraction and hyperspectral
4. J.M. Bioucas-Dias, A. Plaza, N. Dobigeon et al., Hyper- unmixing, IEEE Trans. Geosci. Remote Sens. 57, 482
spectral unmixing overview: Geometrical, statistical, and (2018), https://doi.org/10.1109/TGRS.2018.2856929
sparse regression-based approaches, IEEE J. Sel. Top. Appl. 18. Y. Su, A. Marinoni, J. Li, J. Plaza, P. Gamba, Stacked
Earth Observ. Remote Sens. 5, 354 (2012), https://doi.org/ nonnegative sparse autoencoders for robust hyperspectral
10.1109/JSTARS.2012.2194696 unmixing, IEEE Geosci. Remote Sens. Lett. 15, 1427 (2018),
https://doi.org/10.1109/LGRS.2018.2841400
 N. Brun et al.: Eur. Phys. J. Appl. Phys. 99, 28 (2024) 13

19. Y. Su, J. Li, A. Plaza, A. Marinoni, P. Gamba, S. 31. J.M.P. Nascimento, J.M.B. Dias, Vertex component analysis: a
Chakravortty, DAEN: deep autoencoder networks for fast algorithm to unmix hyperspectral data, IEEE Trans.
hyperspectral unmixing, IEEE Trans. Geosci. Remote Sens. Geosci. Remote Sens. 43, 898 (2005), https://doi.org/10.1109/
57, 4309 (2019), https://doi.org/10.1109/TGRS.2018. TGRS.2005.844293
2890633 32. N. Dobigeon, S. Moussaoui, M. Coulon, J.-Y. Tourneret, A.O.
20. S. Zhang, Y. Su, X. Xu, J. Li, C. Deng, A. Plaza, Recent Hero, Joint Bayesian endmember extraction and linear unmixing
advances in hyperspectral unmixing using sparse techniques for hyperspectral imagery, IEEE Trans. Signal Process. 57, 4355
and deep learning, in Hyperspectral Image Analysis, edited (2009), https://doi.org/10.1109/TSP.2009.2025797
by S. Prasad, J. Chanussot (Springer, 2020), p. 377, https:// 33. L. Miao, H. Qi, Endmember extraction from highly mixed
doi.org/10.1007/978-3-030-38617-7_13 data using minimum volume constrained nonnegative matrix
21. Y. Qu, H. Qi, uDAS: An untied denoising autoencoder with factorization, IEEE Trans. Geosci. Remote Sens. 45, 765
sparsity for spectral unmixing, IEEE Trans. Geosci. Remote (2007), https://doi.org/10.1109/TGRS.2006.888466
Sens. 57, 1698 (2018), https://doi.org/10.1109/ 34. R. Arenal, F. de la Peña, O. Stéphan et al., Extending the
TGRS.2018.2868690 analysis of EELS spectrum-imaging data, from elemental to
22. F. Palsson, J. Sigurdsson, J.R. Sveinsson, M.O. Ulfarsson, bond mapping in complex nanostructures, Ultramicroscopy
Neural network hyperspectral unmixing with spectral 109, 32 (2008), https://doi.org/10.1016/j.ultramic.2008.07.005
information divergence objective, in 2017 IEEE International 35. C.I. Chang, Q. Du, Estimation of number of spectrally
Geoscience and Remote Sensing Symposium (IGARSS) distinct signal sources in hyperspectral imagery, IEEE Trans.
(Fort Worth, TX, USA, 2017), p. 755, https://doi.org/ Geosci. Remote Sens. 42, 608 (2004), https://doi.org/
10.1109/IGARSS.2017.8127062. 10.1109/TGRS.2003.819189
23. B. Palsson, J. Sigurdsson, J.R. Sveinsson, M. O. Ulfarsson, 36. F. Pedregosa, G. Varoquaux, A. Gramfort et al., Scikit-learn:
Hyperspectral unmixing using a neural network autoencoder, machine learning in Python, J. Mach. Learn. Res. 12, 2825
IEEE Access 6, 25646 (2018), https://doi.org/10.1109/ (2011)
ACCESS.2018.2818280 37. N. Bonnet, D. Nuzillard, Independent component analysis: a
24. B. Palsson, J.R. Sveinsson, M.O. Ulfarsson, Spectral-spatial new possibility for analysing series of electron energy loss
hyperspectral unmixing using multitask learning, IEEE spectra, Ultramicroscopy 102, 327 (2005), https://doi.org/
Access 7, 148861 (2019), https://doi.org/10.1109/AC 10.1016/j.ultramic.2004.11.003
CESS.2019.2944072 38. J.M.P. Nascimento, J.M.B. Dias, Independent component
25. B. Palsson, M.O. Ulfarsson, J. R. Sveinsson, Convolutional analysis applied to unmixing hyperspectral data, Proc. SPIE
autoencoder for spectral-spatial hyperspectral unmixing, 5238, 306 (2004), https://doi.org/10.1117/12.510652
IEEE Trans. Geosci. Remote Sens. 59, 535 (2021), https:// 39. A. Hyvärinen, E. Oja, Independent component analysis:
doi.org/10.1109/TGRS.2020.2992743 algorithms and applications, Neural Netw. 13, 411 (2000),
26. M. Zhao, S. Shi, J. Chen, N. Dobigeon, A 3-d-cnnframework https://doi.org/10.1016/S0893-6080(00)00026-5
for hyperspectral unmixing with spectral variability, IEEE 40. I. Palacio, A. Celis, M.N. Nair et al., Atomic structure of
Trans. Geosci. Remote Sens. 60, 1 (2022), https://doi.org/ epitaxial graphene sidewall nanoribbons: flat graphene,
10.1109/TGRS.2022.3141387 miniribbons, and the confinement gap, Nano Lett. 15, 182
27. S. Shi, L. Zhang, Y. Altmann, J. Chen, Deep generative (2015), https://doi.org/10.1021/nl503352v
model for spatial-spectral unmixing with multiple endmem- 41. M. Duchamp, M. Lachmann, C. Boothroyd et al., Composi-
ber priors, IEEE Trans. Geosci. Remote Sens. 60, 1 (2022), tional study of defects in microcrystalline silicon solar cells
https://doi.org/10.1109/TGRS.2022.3168712 using spectral decomposition in the scanning transmission
28. P. Ghosh, S.K. Roy, B. Koirala, B. Rasti, P. Scheunders, electron microscope, Appl. Phys. Lett. 102, 133902 (2013),
Hyperspectral unmixing using transformer network, IEEE https://doi.org/10.1063/1.4800569
Trans. Geosci. Remote Sens. 60, 1 (2022), https://doi.org/ 42. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based
10.1109/TGRS.2022.3196057 learning applied to document recognition, Proc. IEEE 86,
29. B. Palsson, J.R. Sveinsson, M.O. Ulfarsson, Blind hyper- 2278 (1998), https://doi.org/10.1109/5.726791
spectral unmixing using autoencoders: a critical compari- 43. W. Legrand, Y. Sassi, F. Ajejas et al., Spatial extent of the
son, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 15, dzyaloshinskii-moriya interaction at metallic interfaces,
1340 (2022), https://doi.org/10.1109/JSTARS.2021. Phys. Rev. Mater. 6, 024408 (2022), https://doi.org/
3140154 10.1103/PhysRevMaterials.6.024408
30. HyperSpy: Multi-dimensional data analysis toolbox — 44. D. Muller, Ruthenium Bulk M2,3 and M4,5, Appl. Phys.
HyperSpy, https://doi.org/10.5281/zenodo.592838. Avail- Group @ Cornell. [Online] Available: https://muller.re
able: https://hyperspy.org/index.html (visited on 09/28/ search.engineering.cornell.edu/spectra/ruthenium-bulk-
2022) m23-and-m45/ (visited on 02/24/2023)

Cite this article as: Nathalie Brun, Guillaume Lambert, Laura Bocher, EELS hyperspectral images unmixing using autoencoders,
Eur. Phys. J. Appl. Phys. 99, 28 (2024)