Skip to main content

Advertisement

Springer Nature Link
Log in

Representation ensemble learning applied to facial expression recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This work introduces the representation ensemble learning algorithm, a novel approach for generating diverse unsupervised representations rooted in the principles of self-taught learning. The ensemble comprises convolutional autoencoders (CAEs) learned in an unsupervised manner, fostering diversity via a loss function designed to penalize similar CAEs’ latent representations. We employ support vector machines, bagging, and random forest as primary classification methods for the final classification step. Additionally, we incorporate KnoraU, a well-established technique used to dynamically select competent classifiers based on a test sample. We evaluate various fusion strategies, including sum, product, and stacking, to comprehensively assess the ensemble’s performance. A robust experimental protocol considering the facial expression recognition problem shows that the proposed approach based on self-taught learning surpasses the accuracy of fine-tuned convolutional neural network (CNN) models. In terms of accuracy, the proposed method is up to 9.9 and 6.3 percentage points better than the CNN-based models fine-tuned for JAFFE and CK+ datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availibility statement

The datasets used in the experiments are available to the scientific community upon request and sign of a proper responsibility agreement.

Notes

  1. All trained models are available for research purposes at [the hyperlink will be inserted in the final version]

References

  1. Yeung MK (2022) A systematic review and meta-analysis of facial emotion recognition in autism spectrum disorder: The specificity of deficits and the role of task characteristics. Neuroscience & Biobehavioral Reviews 133:104518

    Article  MATH  Google Scholar 

  2. Vehlen A, Kellner A, Normann C, Heinrichs M, Domes G (2023) Reduced eye gaze during facial emotion recognition in chronic depression: Effects of intranasal oxytocin. Journal of Psychiatric Research 159:50–56

    Article  Google Scholar 

  3. Qiao Y, Zeng K, Xu L, Yin X (2016) A smartphone-based driver fatigue detection using fusion of multiple real-time facial features. In: 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC), pp. 230–235. IEEE

  4. Li S, Deng W (2020) Deep facial expression recognition: A survey. IEEE transactions on affective computing 13(3):1195–1215

    Article  MathSciNet  MATH  Google Scholar 

  5. Rathour N, Singh R, Gehlot A, Akram SV, Thakur AK, Kumar A (2022) The decadal perspective of facial emotion processing and recognition: A survey. Displays, 102330

  6. Ganaie MA, Hu M, Malik A, Tanveer M, Suganthan P (2022) Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence 115:105151

    Article  Google Scholar 

  7. Raina R, Battle A, Lee H, Packer B, Ng AY (2007) Self-taught learning: transfer learning from unlabeled data. In: Proceedings of the 24th International Conference on Machine Learning, pp. 759–766

  8. Bengio Y, Bastien F, Bergeron A, Boulanger-Lewandowski N, Chherawala Y, Cisse M, Côté M, Erhan D, Eustache J, Glorot X, et al. (2010) Deep self-taught learning for handwritten character recognition. In: NIPS* 2010 Deep Learning and Unsupervised Feature Learning Workshop

  9. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern recognition 29(1):51–59

    Article  MATH  Google Scholar 

  10. Zavaschi TH, Britto AS Jr, Oliveira LE, Koerich AL (2013) Fusion of feature sets and classifiers for facial expression recognition. Expert Systems with Applications 40(2):646–655

    Article  MATH  Google Scholar 

  11. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International journal of computer vision 60:91–110

    Article  MATH  Google Scholar 

  12. Haralick RM, Shanmugam K, Dinstein IH (1973) Textural features for image classification. IEEE Transactions on systems, man, and cybernetics 6:610–621

    Article  MATH  Google Scholar 

  13. Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Speeded-up robust features (surf). Computer vision and image understanding 110(3):346–359

    Article  Google Scholar 

  14. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893. Ieee

  15. Canal FZ, Müller TR, Matias JC, Scotton GG, Sa Junior AR, Pozzebon E, Sobieranski AC (2022) A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences 582:593–617

    Article  Google Scholar 

  16. Feng S, Yu H, Duarte MF (2020) Autoencoder based sample selection for self-taught learning. Knowledge-Based Systems 192:105343

    Article  MATH  Google Scholar 

  17. Allognon SOC, Britto AdS, Koerich AL (2020) Continuous emotion recognition via deep convolutional autoencoder and support vector regressor. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE

  18. Delazeri BR, Vera LL, Barddal JP, Koerich AL, et al (2022) Evaluation of self-taught learning-based representations for facial emotion recognition. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE

  19. Renda A, Barsacchi M, Bechini A, Marcelloni F (2019) Comparing ensemble strategies for deep learning: An application to facial expression recognition. Expert Systems with Applications 136:1–11

    Article  MATH  Google Scholar 

  20. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognitive Computation 9(5):597–610

    Article  MATH  Google Scholar 

  21. Li R, Ren C, Zhang X, Hu B (2022) A novel ensemble learning method using multiple objective particle swarm optimization for subject-independent eeg-based emotion recognition. Computers in biology and medicine 140:105080

    Article  MATH  Google Scholar 

  22. Dhankhar P (2019) Resnet-50 and vgg-16 for recognizing facial emotions. International Journal of Innovations in Engineering and Technology (IJIET) 13(4):126–130

    MATH  Google Scholar 

  23. Chowdary MK, Nguyen TN, Hemanth DJ (2023) Deep learning-based facial emotion recognition for human-computer interaction applications. Neural Computing and Applications 35(32):23311–23328

    Article  Google Scholar 

  24. Akhand M, Roy S, Siddique N, Kamal MAS, Shimamura T (2021) Facial emotion recognition using transfer learning in the deep cnn. Electronics 10(9):1036

    Article  Google Scholar 

  25. Lee H, Grosse R, Ranganath R, Ng AY (2011) Unsupervised learning of hierarchical representations with convolutional deep belief networks. Communications of the ACM 54(10):95–103

    Article  MATH  Google Scholar 

  26. Markov K, Matsui T (2012) Music genre classification using self-taught learning via sparse coding. In: 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1929–1932. IEEE

  27. Hu H, Phan N, Chun SA, Geller J, Vo H, Ye X, Jin R, Ding K, Kenne D, Dou D (2019) An insight analysis and detection of drug-abuse risk behavior on twitter with self-taught deep learning. Computational Social Networks 6(1):1–19

    Article  Google Scholar 

  28. Qureshi AS, Khan A, Shamim N, Durad MH (2020) Intrusion detection using deep sparse auto-encoder and self-taught learning. Neural Computing and Applications 32:3135–3147

    Article  Google Scholar 

  29. Li S, Li K, Fu Y (2017) Self-taught low-rank coding for visual learning. IEEE transactions on neural networks and learning systems 29(3):645–656

    Article  MathSciNet  MATH  Google Scholar 

  30. Ramamurthy SR, Ghosh I, Gangopadhyay A, Galik E, Roy N (2022) Star-lite: A light-weight scalable self-taught learning framework for older adults’ activity recognition. Pervasive and Mobile Computing 87:101698

    Article  Google Scholar 

  31. Germani E, Fromont E, Maumet C (2023) On the benefits of self-taught learning for brain decoding. GigaScience 12:029

    MATH  Google Scholar 

  32. He P, Jia P, Qiao S, Duan S (2017) Self-taught learning based on sparse autoencoder for e-nose in wound infection detection. Sensors 17(10):2279

    Article  MATH  Google Scholar 

  33. Liu L, Wei W, Chow K-H, Loper M, Gursoy E, Truex S, Wu Y (2019) Deep neural network ensembles against deception: Ensemble diversity, accuracy and robustness. In: 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems (MASS), pp. 274–282. IEEE

  34. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, et al (2011) Scikit-learn: Machine learning in python. Journal of machine learning research 12(Oct), 2825–2830

  35. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205. IEEE

  36. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 Ieee Computer Society Conference on Computer Vision and Pattern Recognition-workshops, pp. 94–101. IEEE

  37. Viola P, Jones MJ (2004) Robust real-time face detection. International journal of computer vision 57:137–154

    Article  MATH  Google Scholar 

  38. Doi E, Inui T, Lee T-W, Wachtler T, Sejnowski TJ (2003) Spatiochromatic receptive field properties derived from information-theoretic analyses of cone mosaic responses to natural scenes. Neural computation 15(2):397–417

    Article  MATH  Google Scholar 

  39. Uetz R, Behnke S (2009) Large-scale object recognition with cuda-accelerated hierarchical neural networks. In: 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems, vol. 1, pp. 536–541. IEEE

  40. Huang GB, Mattar M, Berg T, Learned-Miller E (2008) Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In: Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition

  41. Alzubaidi L, Zhang J, Humaidi AJ, Al-Dujaili A, Duan Y, Al-Shamma O, Santamaría J, Fadhel MA, Al-Amidie M, Farhan L (2021) Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions. Journal of big Data 8:1–74

    Article  MATH  Google Scholar 

  42. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25

  43. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  44. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9

  45. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778

  46. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708

  47. Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR

  48. Xu M, Cheng W, Zhao Q, Ma L, Xu F (2015) Facial expression recognition based on transfer learning from deep convolutional networks. In: 2015 11th International Conference on Natural Computation (ICNC), pp. 702–708. IEEE

  49. Zhang J, Li W, Ogunbona P, Xu D (2019) Recent advances in transfer learning for cross-dataset visual recognition: A problem-oriented perspective. ACM Computing Surveys (CSUR) 52(1):1–38

    Article  MATH  Google Scholar 

  50. Peng M, Wu Z, Zhang Z, Chen T (2018) From macro to micro expression recognition: Deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 657–661. IEEE

  51. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  MATH  Google Scholar 

  52. Kola DGR, Samayamantula SK (2021) Facial expression recognition using singular values and wavelet-based lgc-hd operator. IET Biometrics 10(2):207–218

    Article  Google Scholar 

  53. Kartheek MN, Prasad MV, Bhukya R (2023) Radial mesh pattern: a handcrafted feature descriptor for facial expression recognition. Journal of Ambient Intelligence and Humanized Computing 14(3):1619–1631

    Article  MATH  Google Scholar 

  54. Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kranthi Kumar D (2019) Regional adaptive affinitive patterns (radap) with logical operators for facial expression recognition. IET Image Processing 13(5):850–861

    Article  Google Scholar 

  55. Du L, Hu H (2019) Weighted patch-based manifold regularization dictionary pair learning model for facial expression recognition using iterative optimization classification strategy. Computer Vision and Image Understanding 186:13–24

    Article  MATH  Google Scholar 

  56. Wu B-F, Lin C-H (2018) Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE access 6:12451–12461

    Article  Google Scholar 

  57. Lee SH, Baddar WJ, Ro YM (2016) Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recognition 54:52–67

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We would like to thank the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq, grants 306878/2022-4, 406030/2023-5 and 441610/2023-4) and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for co-financing this research.

Author information

Author notes

  1. Bruna Rossetto Delazeri, Andre Gustavo Hochuli, Jean Paul Barddal, Alessandro Lameiras Koerich and Alceu de Souza Britto Jr these authors contributed equally to this work.

Authors and Affiliations

  1. Graduate Program in Informatics (PPGIA), Pontifícia Universidade Católica do Paraná (PUCPR), Imaculada Conceição, Curitiba, Paraná, 80215-901, Brazil

    Bruna Rossetto Delazeri, Andre Gustavo Hochuli, Jean Paul Barddal & Alceu de Souza Britto Jr.

  2. Department of Software and IT Engineering, École de Technologie Supérieure (ÉTS) - University of Québec, 1100 Notre-Dame St W, Montreal, Quebec, H3C 1K3, Canada

    Alessandro Lameiras Koerich

  3. Postgraduate Program in Applied Computing, State University of Ponta Grossa (UEPG), Av. General Carlos Cavalcanti, Ponta Grossa, Paraná, 84010-330, Brazil

    Alceu de Souza Britto Jr.

Authors
  1. Bruna Rossetto Delazeri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andre Gustavo Hochuli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean Paul Barddal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alessandro Lameiras Koerich
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alceu de Souza Britto Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have participated in the conception, design, analysis, and interpretation of the data, drafting the article or revising it, and approving the final version.

Corresponding author

Correspondence to Bruna Rossetto Delazeri.

Ethics declarations

Conflict of interest

The authors certify that they have no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Delazeri, B.R., Hochuli, A.G., Barddal, J.P. et al. Representation ensemble learning applied to facial expression recognition. Neural Comput & Applic 37, 417–438 (2025). https://doi.org/10.1007/s00521-024-10556-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-024-10556-w

Keywords

Search

Navigation