1

Text-Independent Speaker Verification Using Long

Short-Term Memory Networks
∗
Aryan Mobiny∗ , Mohammad Najarian†
Department of Electrical and Computer Engineering, University of Houston
Email: amobiny@uh.edu
† Department of Industrial Engineering, University of Houston
Email: mnajarian@uh.edu
arXiv:1805.00604v3 [eess.AS] 7 Sep 2018

Abstract—In this paper, an architecture based on Long on the created background model, the new speakers
Short-Term Memory Networks has been proposed for the will be enrolled in creating the speaker model. Tech-
text-independent scenario which is aimed to capture the nically, the speakers’ models are generated using
temporal speaker-related information by operating over
the universal background model. In the evaluation
traditional speech features. For speaker verification, at
first, a background model must be created for speaker phase, the test utterances will be compared to the
representation. Then, in enrollment stage, the speaker speaker models for further identification or verifica-
models will be created based on the enrollment utterances. tion.
For this work, the model will be trained in an end-to-end Recently, by the success of deep learning in
fashion to combine the first two stages. The main goal applications such as in biomedical purposes [1],
of end-to-end training is the model being optimized to be
[2], automatic speech recognition, image recogni-
consistent with the speaker verification protocol. The end-
to-end training jointly learns the background and speaker tion and network sparsity [3]–[6], the DNN-based
models by creating the representation space. The LSTM approaches have also been proposed for Speaker
architecture is trained to create a discrimination space Recognition (SR) [7], [8].
for validating the match and non-match pairs for speaker The traditional speaker verification models such
verification. The proposed architecture demonstrate its as Gaussian Mixture Model-Universal Background
superiority in the text-independent compared to other
Model (GMM-UBM) [9] and i-vector [10] have
traditional methods.
been the state-of-the-art for long. The drawback
of these approaches is the employed unsupervised
I. I NTRODUCTION fashion that does not optimize them for verification
setup. Recently, supervised methods proposed for
The main goal of Speaker Verification (SV) is model adaptation to speaker verification such as the
the process of verifying a query sample belonging one presented in [11] and PLDA-based i-vectors
to a speaker utterance by comparing to the exist- model [12]. Convolutional Neural Networks (CNNs)
ing speaker models. Speaker verification is usually has also been used for speech recognition and
split into two text-independent and text-dependant speaker verification [8], [13] inspired by their their
categories. Text-dependent includes the scenario in superior power for action recognition [14] and scene
which all the speakers are uttering the same phrase understanding [15]. Capsule networks introduced
while in text-independent no prior information is by Hinton et al. [16] has shown quite remarkable
considered for what the speakers are saying. The performance in different tasks [17], [18], and
later setting is much more challenging as it can demonstrated the potential and power to be used
contain numerous variations for non-speaker in- for similar purposes.
formation that can be misleading while extracting In the present work, we propose the use of
solely speaker information is desired. LSTMs by using MFCCs1 speech features for di-
The speaker verification, in general, consists of rectly capturing the temporal information of the
three stages: Training, enrollment, and evaluation. In speaker-related information rather than dealing with
training, the universal background model is trained
using the gallery of speakers. In enrollment, based 1
Mel Frequency Cepstral Coefficients
 2

non-speaker information which plays no role for utterances. From this point, different approaches
speaker verification. have been proposed on how to integrate these enroll-
ment features for creating the speaker model. The
II. R ELATED WORKS tradition one is aggregating the representations by
There is a huge literature on speaker verification. averaging the outputs of the DNN which is called
However, we only focus on the research efforts d-vector system [8], [19].
which are based on deep learning deep learning.
One of the traditional successful works in speaker C. Evaluation
verification is the use of Locally Connected Net- For evaluation, the test utterance is the input of
works (LCNs) [19] for the text-dependent scenario. the network and the output is the utterance represen-
Deep networks have also been used as feature tative. The output representative will be compared to
extractor for representing speaker models [20], [21]. different speaker model and the verification criterion
We investigate LSTMs in an end-to-end fashion will be some similarity function. For evaluation
for speaker verification. As Convolutional Neural purposes, the traditional Equal Error Rate (EER)
Networks [22] have successfully been used for will often be used which is the operating point in
the speech recognition [23] some works use their that false reject rate and false accept rate are equal.
architecture for speaker verification [7], [24]. The
most similar work to ours is [20] in which they IV. M ODEL
use LSTMs for the text-dependent setting. On the
contrary, we use LSTMs for the text-independent The main goal is to implement LSTMs on top of
scenario which is a more challenging one. speech extracted features. The input to the model
as well as the architecture itself is explained in the
following subsections.
III. S PEAKER V ERIFICATION U SING D EEP
N EURAL N ETWORKS
A. Input
Here, we explain the speaker verification phases
using deep learning. In different works, these The raw signal is extracted and 25ms windows
steps have been adopted regarding the procedure with %60 overlapping are used for the generation of
proposed by their research efforts such as i- the spectrogram as depicted in Fig. 1. By selecting
vector [10], [25], d-vector system [8]. 1-second of the sound stream, 40 log-energy of filter
banks per window and performing mean and vari-
ance normalization, a feature window of 40 × 100
is generated for each 1-second utterance. Before
A. Development
feature extraction, voice activity detection has been
In the development stage which also called done over the raw input for eliminating the silence.
training, the speaker utterances are used for The derivative feature has not been used as using
background model generation which ideally them did not make any improvement considering
should be a universal model for speaker model the empirical evaluations. For feature extraction, we
representation. DNNs are employed due to their used SpeechPy library [26].
power for feature extraction. By using deep models,
the feature learning will be done for creating an
output space which represents the speaker in a
universal model.

B. Enrollment
In this phase, a model must be created for each
speaker. For each speaker, by collecting the spoken
utterances and feeding to the trained network, dif- Fig. 1. The feature extraction from the raw signal.
ferent output features will be generated for speaker
 3

B. Architecture there is no indication to the one-to-one speaker com-

The architecture that we use a long short-term parison for being consistent to speaker verification
memory recurrent neural network (LSTM) [27], mode.
[28] with a single output for decision making. We To consider this condition, we use the Siamese
input fixed-length sequences although LSTMs are architecture to satisfy the verification purpose which
not limited by this constraint. Only the last hidden has been proposed in [29] and employed in different
state of the LSTM model is used for decision applications [30]–[32]. As we mentioned before, the
making using the loss function. The LSTM that we Softmax optimization will be used for initialization
use has two layers with 300 nodes each (Fig. 2). and the obtained weights will be used for fine-
tuning.
The Siamese architecture consists of two identical
networks with weight sharing. The goal is to create a
shared feature subspace which is aimed at discrim-
ination between genuine and impostor pairs. The
main idea is that when two elements of an input pair
are from the same identity, their output distances
should be close and far away, otherwise. For this
objective, the training loss will be contrastive cost
function. The aim of contrastive loss CW (X, Y )
is the minimization of the loss in both scenarios
of having genuine and impostor pairs, with the
following definition:
N
1 X
Fig. 2. The siamese architecture built based on two LSTM layers
CW (X, Y ) = CW (Yj , (X1 , X2 )j ), (3)
N j=1
with weight sharing.
where N indicates the training samples, j is the sam-
ple index and CW (Yi , (Xp1 , Xp2 )i ) will be defined
C. Verification Setup as follows:
A usual method which has been used in many
other works [19], is training the network using the CW (Yi , (X1 , X2 )j ) = Y ∗ Cgen (DW (X1 , X2 )j )
Softmax loss function for the auxiliary classification + (1 − Y ) ∗ Cimp (DW (X1 , X2 )j ) + λ||W ||2
task and then use the extracted features for the (4)
main verification purpose. A reasonable argument
in which the last term is the regularization. Cgen
about this approach is that the Softmax criterion
and Cimp will be defined as the functions of
is not in align with the verification protocol due
DW (X1 , X2 ) by the following equations:
to optimizing for identification of individuals and
not the one-vs-one comparison. Technically, the (
Softmax optimization criterion is as below: Cgen (DW (X1 , X2 ) = 21 DW (X1 , X2 )2
Cimp (DW (X1 , X2 ) = 12 max{0, (M − DW (X1 , X2 ))}2
exSpeaker (5)
softmax(x)Speaker =P xDevSpk (1) in which M is the margin.
DevSpk e
(
xSpeaker = WSpeaker × y + b V. E XPERIMENTS
(2)
TensorFLow has been used as the deep learning
xDevSpk = WDevSpk × y + b
library [33]. For the development phase, we used
in which Speaker and DevSpk denote the sample data augmentation by randomly sampling the 1-
speaker and an identity from speaker development second audio sample for each person at a time.
set, respectively. As it is clear from the criterion, Batch normalization has also been used for avoiding
 4

possible gradient explotion [34]. It’s been shown can be seen in Table I, our proposed architecture
that effective pair selection can drastically improve outperforms the other methods.
the verification accuracy [35]. Speaker verification
is performed using the protocol consistent with [36] TABLE I
for which the name identities start with E will be T HE ARCHITECTURE USED FOR VERIFICATION PURPOSE .
used for evaluation.
Model EER
Algorithm 1: The utilized pair selection algo-
GMM-UBM [9] 27.1
rithm for selecting the main contributing impos- I-vectors [10] 24.7
tor pairs I-vectors [10] + PLDA [37] 23.5
Update: Freeze weights! LSTM [ours] 22.9
Evaluate: Input data and get output distance
vector;
Search: Return max and min distances for
match pairs : max gen & min gen;
C. Effect of Utterance Duration
Thresholding: Calculate th = th0 × max gen
min gen
;
One one the main advantage of the baseline meth-
while impostor pair do
ods such as [10] is their ability to capture robust
if imp > max gen + th then
speaker characteristics through long utterances. As
discard;
demonstrated in Fig. 3, our proposed method out-
else
performs the others for short utterances considering
feed the pair;
we used 1-second utterances. However, it is worth
to have a fair comparison for longer utterances as
well. In order to have a one-to-one comparison,
we modified our architecture to feed and train the
A. Baselines system on longer utterances. In all experiments,
We compare our method with different base- the duration of utterances utilized for development,
line methods. The GMM-UBM method [9] if the enrollment, and evaluation are the same.
first candidate. The MFCCs features with 40 co-
efficients are extracted and used. The Universal
Background Model (UBM) is trained using 1024
mixture components. The I-Vector model [10], with
and without Probabilistic Linear Discriminant Anal-
ysis (PLDA) [37], has also been implemented as the
baseline.
The other baseline is the use of DNNs with
locally-connected layers as proposed in [19]. In
the d-vector system, after development phase, the
d-vectors extracted from the enrollment utterances
will be aggregated to each other for generating the
final representation. Finally, in the evaluation stage,
the similarity function determines the closest d-
Fig. 3. The effect of the utterance duration (EER).
vector of the test utterances to the speaker models.
As can be observed in Fig. 3, the superiority
B. Comparison to Different Methods of our method is only in short utterances and in
Here we compare the baseline approaches with longer utterances, the traditional baseline methods
the proposed model as provided in Table I. We such as [10], still are the winners and LSTMs
utilized the architecture and the setup as discussed fails to capture effectively inter- and inter-speaker
in Section IV-B and Section IV-C, respectively. As variations.
 5

VI. C ONCLUSION [10] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouel-

let, “Front-end factor analysis for speaker verification,” IEEE
In this work, an end-to-end model based on Transactions on Audio, Speech, and Language Processing,
LSTMs has been proposed for text-independent vol. 19, no. 4, pp. 788–798, 2011.
speaker verification. It was shown that the model [11] W. M. Campbell, D. E. Sturim, and D. A. Reynolds, “Support
vector machines using gmm supervectors for speaker verifica-
provided promising results for capturing the tem- tion,” IEEE signal processing letters, vol. 13, no. 5, pp. 308–
poral information in addition to capture the within- 311, 2006.
speaker information. The proposed LSTM architec- [12] D. Garcia-Romero and C. Y. Espy-Wilson, “Analysis of i-vector
ture has directly been used on the speech features length normalization in speaker recognition systems,” in Twelfth
Annual Conference of the International Speech Communication
extracted from speaker utterances for modeling the Association, 2011.
spatiotemporal information. One the observed traces [13] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn,
is the superiority of traditional methods on longer and D. Yu, “Convolutional neural networks for speech recogni-
tion,” IEEE/ACM Transactions on audio, speech, and language
utterances for more robust speaker modeling. More processing, vol. 22, no. 10, pp. 1533–1545, 2014.
rigorous studies are needed to investigate the rea- [14] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural
soning behind the failure of LSTMs to capture long networks for human action recognition,” IEEE transactions
on pattern analysis and machine intelligence, vol. 35, no. 1,
dependencies for speaker related characteristics. Ad- pp. 221–231, 2013.
ditionally, it is expected that the combination of [15] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri,
traditional models with long short-term memory ar- “Learning spatiotemporal features with 3d convolutional net-
chitectures may improve the accuracy by capturing works,” in Computer Vision (ICCV), 2015 IEEE International
Conference on, pp. 4489–4497, IEEE, 2015.
the long-term dependencies in a more effective way. [16] S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing be-
The main advantage of the proposed approach is tween capsules,” in Advances in Neural Information Processing
its ability to capture informative features in short Systems, pp. 3856–3866, 2017.
utterances. [17] A. Mobiny and H. Van Nguyen, “Fast capsnet for lung cancer
screening,” arXiv preprint arXiv:1806.07416, 2018.
[18] A. Jaiswal, W. AbdAlmageed, and P. Natarajan, “Capsule-
R EFERENCES gan: Generative adversarial capsule network,” arXiv preprint
[1] W. Shen, M. Zhou, F. Yang, C. Yang, and J. Tian, “Multi-scale arXiv:1802.06167, 2018.
convolutional neural networks for lung nodule classification,” in [19] Y.-h. Chen, I. Lopez-Moreno, T. N. Sainath, M. Visontai,
International Conference on Information Processing in Medical R. Alvarez, and C. Parada, “Locally-connected and convolu-
Imaging, pp. 588–599, Springer, 2015. tional neural networks for small footprint speaker recognition,”
[2] A. Mobiny, S. Moulik, I. Gurcan, T. Shah, and H. Van Nguyen, in Sixteenth Annual Conference of the International Speech
“Lung cancer screening using adaptive memory-augmented Communication Association, 2015.
recurrent networks,” arXiv preprint arXiv:1710.05719, 2017. [20] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-
[3] K. Simonyan and A. Zisserman, “Very deep convolutional end text-dependent speaker verification,” in Acoustics, Speech
networks for large-scale image recognition,” arXiv preprint and Signal Processing (ICASSP), 2016 IEEE International
arXiv:1409.1556, 2014. Conference on, pp. 5115–5119, IEEE, 2016.
[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classi- [21] C. Zhang and K. Koishida, “End-to-end text-independent
fication with deep convolutional neural networks,” in Advances speaker verification with triplet loss on short utterances,” in
in neural information processing systems, pp. 1097–1105, 2012. Proc. of Interspeech, 2017.
[5] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, [22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-
N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, based learning applied to document recognition,” Proceedings
et al., “Deep neural networks for acoustic modeling in speech of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
recognition: The shared views of four research groups,” IEEE
[23] T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ram-
Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
abhadran, “Deep convolutional neural networks for lvcsr,” in
[6] A. Torfi and R. A. Shirvani, “Attention-based guided struc-
Acoustics, speech and signal processing (ICASSP), 2013 IEEE
tured sparsity of deep neural networks,” arXiv preprint
international conference on, pp. 8614–8618, IEEE, 2013.
arXiv:1802.09902, 2018.
[24] F. Richardson, D. Reynolds, and N. Dehak, “Deep neural
[7] Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel
network approaches to speaker and language recognition,” IEEE
scheme for speaker recognition using a phonetically-aware deep
Signal Processing Letters, vol. 22, no. 10, pp. 1671–1675, 2015.
neural network,” in Acoustics, Speech and Signal Processing
(ICASSP), 2014 IEEE International Conference on, pp. 1695– [25] P. Kenny, G. Boulianne, P. Ouellet, and P. Dumouchel, “Joint
1699, IEEE, 2014. factor analysis versus eigenchannels in speaker recognition,”
[8] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and IEEE Transactions on Audio, Speech, and Language Process-
J. Gonzalez-Dominguez, “Deep neural networks for small foot- ing, vol. 15, no. 4, pp. 1435–1447, 2007.
print text-dependent speaker verification,” in Acoustics, Speech [26] A. Torfi, “Speechpy-a library for speech processing and recog-
and Signal Processing (ICASSP), 2014 IEEE International nition,” arXiv preprint arXiv:1803.01094, 2018.
Conference on, pp. 4052–4056, IEEE, 2014. [27] S. Hochreiter and J. Schmidhuber, “Long short-term memory,”
[9] D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, “Speaker Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
verification using adapted gaussian mixture models,” Digital [28] H. Sak, A. Senior, and F. Beaufays, “Long short-term memory
signal processing, vol. 10, no. 1-3, pp. 19–41, 2000. recurrent neural network architectures for large scale acoustic
 6

modeling,” in Fifteenth annual conference of the international

speech communication association, 2014.
[29] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity
metric discriminatively, with application to face verification,” in
Computer Vision and Pattern Recognition, 2005. CVPR 2005.
IEEE Computer Society Conference on, vol. 1, pp. 539–546,
IEEE, 2005.
[30] X. Sun, A. Torfi, and N. Nasrabadi, “Deep siamese convo-
lutional neural networks for identical twins and look-alike
identification,” Deep Learning in Biometrics, p. 65, 2018.
[31] R. R. Varior, M. Haloi, and G. Wang, “Gated siamese convolu-
tional neural network architecture for human re-identification,”
in European Conference on Computer Vision, pp. 791–808,
Springer, 2016.
[32] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural net-
works for one-shot image recognition,” in ICML Deep Learning
Workshop, vol. 2, 2015.
[33] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro,
G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat,
I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefow-
icz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga,
S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens,
B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke,
V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg,
M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale
machine learning on heterogeneous systems,” 2015. Software
available from tensorflow.org.
[34] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating
deep network training by reducing internal covariate shift,” in
International conference on machine learning, pp. 448–456,
2015.
[35] Y.-R. Lin, Y. Chi, S. Zhu, H. Sundaram, and B. L. Tseng,
“Facetnet: a framework for analyzing communities and their
evolutions in dynamic networks,” in Proceedings of the 17th
international conference on World Wide Web, pp. 685–694,
ACM, 2008.
[36] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-
scale speaker identification dataset,” in INTERSPEECH, 2017.
[37] P. Kenny, “Bayesian speaker verification with heavy-tailed
priors.,” in Odyssey, p. 14, 2010.