Abstract
Recent advancements in AI and deep learning have created a growing demand for artificial agents capable of performing tasks within increasingly complex environments. To address the challenges associated with continuous learning constraints and knowledge capacity in this context, cognitive architectures inspired by human cognition have gained significance. This study contributes to existing research by introducing a cognitive-attentional system employing a constructive neural network-based learning approach for continuous acquisition of procedural knowledge. We replace an incremental tabular Reinforcement Learning algorithm with a constructive neural network deep reinforcement learning mechanism for continuous sensorimotor knowledge acquisition, thereby enhancing the overall learning capacity. The primary emphasis of this modification centers on optimizing memory utilization and reducing training time. Our study presents a learning strategy that amalgamates deep reinforcement learning with procedural learning, mirroring the incremental learning process observed in human sensorimotor development. This approach is embedded within the CONAIM cognitive-attentional architecture, leveraging the cognitive tools of CST. The proposed learning mechanism allows the model to dynamically create and modify elements in its procedural memory, facilitating the reuse of previously acquired functions and procedures. Additionally, it equips the model with the capability to combine learned elements to effectively adapt to complex scenarios. A constructive neural network was employed, initiating with an initial hidden layer comprising one neuron. However, it possesses the capacity to adapt its internal architecture in response to its performance in procedural and sensorimotor learning tasks, inserting new hidden layers or neurons. Experimentation conducted through simulations involving a humanoid robot demonstrates the successful resolution of tasks that were previously unsolved through incremental knowledge acquisition. Throughout the training phase, the constructive agent achieved a minimum of 40% greater rewards and executed 8% more actions when compared to other agents. In the subsequent testing phase, the constructive agent exhibited a 15% increase in the number of actions performed in contrast to its counterparts.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
Code Availability
The code used to implement the architectures is available at: https://github.com/CST-Group/cst. We have made the results obtained in training available at the repository: https://drive.google.com/drive/folders/10_kaTVhvIupoStY3rVcMKOuaxXsp5_N3.
References
Lungarella, M., Sandini, G., Pfeifer, R.: Developmental robotics: A survey. Connect. Sci. 15(4), 151–190 (2003). https://doi.org/10.1080/09540090310001655110
Cangelosi, A., Schlesinger, M.: Developmental Robotics: From Babies to Robots. The MIT Press, Cambridge, MA (2014). https://mitpress.mit.edu/books/developmental-robotics
Sun, R.: Desiderata for cognitive architectures. Philos. Psychol. 17(3), 341–373 (2004). https://doi.org/10.1080/0951508042000286721
Franklin, S.: The mind according to lida — a brief account, 20 (2012)
Laird, J.E.: Extending the soar cognitive architecture. In: Proceedings of the first AGI conference - frontiers in artificial intelligence and applications, pp. 224–235. Artificial General Intelligence by Wang, P., Goertzel, B., Franklin S, Amsterdam, Netherlands (2008). https://www.researchgate.net/publication/221328941_Extending_the_Soar_Cognitive_Architecture
Thomson, R., Bennati, S., Lebiere, C.: Extending the influence of contextual information in act-r using buffer decay. In: Proceedings of the annual meeting of the cognitive science society, p. 6 (2014). www.researchgate.net/publication/268520256_Extending_the_Influence_of_Contextual_Information_in_ACT-R_using_Buffer_Decay
Reggia, J.A.: The rise of machine consciousness: studying consciousness with computational models. Neural Netw. 44 (2013)
Simões, A.S.: Um modelo cognitivo baseado na atenção para consciência em robôs inteligentes. Universidade Estadual Paulista (Tese de Livre-Docência) (2015)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature (2015). https://doi.org/10.1038/nature14539
Rossi, L.d.L.: Aprendizado sensório-motor em robôs cognitivos utilizando modelo cst-conaim. Master’s thesis, Universidade Estadual Paulista "Júlio de Mesquita Filho" (Unesp) - Instituto de Ciência e Tecnologia de Sorocaba (2021). https://repositorio.unesp.br/handle/11449/214316
Rossi, L.L., Berto, L.M., Rohmer, E., Costa, P.D.P., Simoes, A.S., Gudwin, R.R., Colombini, E.L.: Incremental procedural and sensorimotor learning in cognitive humanoid robots. (Submitted to) IEEE Trans. Cogn. Dev. Syst. (2023)
Silva Simoes, A.d., Colombini, E.L., Costa Ribeiro, C.H.: CONAIM: A Conscious Attention-Based Integrated Model for Human-Like Robots. IEEE Syst. J. 11(3), 1296–1307 (2017). https://doi.org/10.1109/JSYST.2015.2498542
Paraense, A.L.O., Raizer, K., Paula, S.M., Gudwin, R.R., Rohmer, E.: The cognitive systems toolkit and the cst reference cognitive architecture. Biologically Inspired Cogn. Architectures 17, 32–48 (2016). https://doi.org/10.1016/j.bica.2016.07.005
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge, Massachussetts (1998)
Piaget, J.: The Origins of Intelligence in Children. The origins of intelligence in children. W Norton and Co, New York, NY, US (1952). https://doi.org/10.1037/11494-000
Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: A brief survey 34(6), 26–38 https://doi.org/10.1109/MSP.2017.2743240 . Accessed 2023-01-25
Fan, J., Wang, Z., Xie, Y., Yang, Z.: A theoretical analysis of deep q-learning. Proceedings of the 2nd Conference on Learning for Dynamics and Control, PMLR 120, 486–489 (2020)
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G., Agapiou, J., Leibo, J., Gruslys, A.: Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). (2018) https://doi.org/10.1609/aaai.v32i1.11757
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. DeepMind Technologies (2013)
Ravichandiran, S.: Hands-on Reinforcement Learning with Python: Master Reinforcement and Deep Reinforcement Learning Using OpenAI Gym and TensorFlow. Packt Publishing Ltd, United Kingdom (2018)
Dash, C.S.K., Behera, A.K., Dehuri, S., Cho, S.-B.: Radial basis function neural networks: a topical state-of-the-art survey. Open Computer Science 6(1), 33–63 (2016). https://doi.org/10.1515/comp-2016-0005
Que, Q., Belkin, M.: Back to the future: Radial basis function networks revisited. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR 51, 1375-1383, 1375–1383 (2016)
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., Tuytelaars, T.: A continual learning survey: Defying forgetting in classification tasks. IEEE Trans. Pattern Anal. Mach. Intell. 44(7), 3366–3385 (2022). https://doi.org/10.1109/TPAMI.2021.3057446
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Netw. 113, 54–71 (2019)
Gudwin, R.R., Raizer, K., Paraense, A.L.O., Paula, S.M., Figueiredo, V.A., Castro, E.C., Fróes, E., Gibaut, W.S.P.: CST - The Cognitive Systems Toolkit (2013). https://www.cst.fee.unicamp.br
Konduit: Deeplearning4j (2022). https://deeplearning4j.konduit.ai
Paraense, A.L.O., Raizer, K., Gudwin, R.R.: A machine consciousness approach to urban traffic control. Biologically Inspired Cognitive Architectures 15, 61–73 (2016). https://doi.org/10.1016/j.bica.2015.10.001
Gama, F., Shcherban, M., Rolf, M., Hoffmann, M.: Goal-directed tactile exploration for body model learning through self-touch on a humanoid robot. IEEE Transactions on Cognitive and Developmental Systems (2021). https://doi.org/10.1109/TCDS.2021.3104881
Stoytchev, A.: Some basic principles of developmental robotics. IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT 1 (2009) https://doi.org/10.1109/TAMD.2009.2029989
Vernon, D., Hofsten, C., Fadiga, L.: Desiderata for developmental cognitive architectures. Biologically Inspired Cognitive Architectures 18, 116–127 (2016). https://doi.org/10.1016/j.bica.2016.10.004
Piaget, J.: Science of Education and the Psychology of the Child. Penguin Books, London, United Kingdom. (1971). https://doi.org/10.1037/11494-000
Armstrong, K.H., Ogg, J., A., S.-W., A., W.: Early Child Development Theories. Evidence-Based Interventions for Children with Challenging Behavior. Springer, New York (2014)
Piaget, J.: Seis Estudos da Psicologia. Forense Universitária, ??? (1999)
Cook, J.L., Cook, G.: Child Development. London: Pearson education: Allyn and Bacon., London, United Kingdom (2005)
Lefrançoi, G.R.: Teorias da Aprendizagem. Cengage Learning, Boston, MA, USA (2008)
Woolfolk, A.E., Winne, P.H., Perry, N.E.: Educational Psychology. Pearson Education Canada Inc, Allyn and Bacon, Inc Needham Height, MA, Canada (2010)
Wood, K.C., Smith, H., Grossniklaus, D.: Piaget’s stages of cognitive development. University of Georgia, Department of Educational Psychology and Instructional Technology (2011)
Papalia, D.E., Feldman, R.D.: Desenvolvimento Humano. vol. 12. Mcgraw Hill - Artmed, Brazil. (2013)
Berto, L.M.: Exploring cognitive functions in robotics. Master’s thesis, Univeristy of Campinas (2020). https://doi.org/10.47749/T/UNICAMP.2020.1129257 . http://repositorio.unicamp.br/Acervo/Detalhe/1129257
Berto, L.M., Rossi, L.L., ROHMER, E., COSTA, P.D.P., Simoes, A.S., Gudwin, R.R., COLOMBINI, E.L.: Piagetian experiments to devrobotics. Cogn. Syst. Res. J. (2023). https://doi.org/10.1016/j.cogsys.2023.101170
Michalec, D.: In: Goldstein, S., Naglieri, J.A. (eds.) Bayley Scales of Infant Development: Third Edition, pp. 215–215. Springer, Boston, MA (2011). https://doi.org/10.1007/978-0-387-79061-9_295
Boyd, D., Bee, H.: A Criança em Crescimento. ARTMED, Brazil (2011)
Guerin, F., Kruger, N., Kraft, D.: A Survey of the Ontogeny of Tool Use: From Sensorimotor Experience to Planning. IEEE Trans. Auton. Ment. Dev. 5(1), 18–45 (2013). https://doi.org/10.1109/TAMD.2012.2209879
Arrabales, R.: ConsScale - A Machine Consciousness Scale (2020). https://www.conscious-robots.com/consscale/index.html
Harari, Y.N.: Sapiens - Uma Breve História da Humanidade. Harper, New York, NY, US (2011)
Harris, C.L.: Language and cognition. In: L., N. (ed.) Encyclopedia of Cognitive Science, pp. 1–6. Wiley, ??? (2006). https://doi.org/10.1002/0470018860.s00559
Kotseruba, I., Tsotsos, J.K.: 40 years of cognitive architectures: core cognitive abilities and practical applications. Artif. Intell. Rev. 53 (2020)
Meeden, L.A., Blank, D.S.: Introduction to developmental robotics. Connect. Sci. 18(2), 93–96 (2006). https://doi.org/10.1080/09540090600806631
Anderson, J., Lebiere, C.: The newell test for a theory of mind. Carnegie Mellon University (1998)
Laird, J.: The Soar Cognitive Architecture. MIT Press, Cambridge, Mass.; London, England (2012)
Laird, J.E., Kinkade, K.R., Mohan, S., Xu, J.Z.: Cognitive robotics using the soar cognitive architecture. Cogn. Robot. AAAI Tech, Rep (2012)
Mininger, A., Laird, J.: Interactively learning strategies for handling references to unseen or unknown objects. Adv. Cogn, Syst (2016)
Licato, J., Marton, N., Dong, B., Sun, R., Bringsjord, S.: Modeling the creation and development of cause-effect pairs for explanation generation in a cognitive architecture. In: International workshop on artificial intelligence and cognition, Torino, Italy, pp. 29–39 (2015)
Madl, T., Franklin, S., Chen, K., Trappl, R., Montaldi, D.: Exploring the structure of spatial representations. PLoS ONE (2016). https://doi.org/10.1371/journal.pone.0157343
Colombini, E.L.: An attentional model for intelligent robotics agents. Instituto Tecnológico de Aeronáutica, São José dos Campos (PhD Thesis) (2014)
Colombini, E.L., Simões, A.d.S., Ribeiro, C.H.C.: An attentional model for autonomous mobile robots. IEEE Syst. J. 11(3), 1308–1319 (2017) https://doi.org/10.1109/JSYST.2015.2499304
Simões, A.d.S., Colombini, E.L., Ribeiro, C.H.C.: Conaim: A conscious attention-based integrated model for human-like robots. IEEE Syst. J. (2016) https://doi.org/10.1109/JSYST.2015.2498542
Baars, J.B.: A Cognitive Theory of Consciousness. Cambridge University Press, United Kingdom (1998)
Bengio, Y.: The consciousness prior. Université de Montréal, Mila (2017)
Kelleher, J.D.: Deep Learning. MIT press, Cambridge, MA (2019)
Buhmann, M.D.: Radial basis functions. Acta numerica 9, 1–38 (2000)
Asadi, K., Parikh, N., Parr, R.E., Konidaris, G.D., Littman, M.L.: Deep radial-basis value functions for continuous control. Proceedings of the AAAI Conference on Artificial Intelligence 35(8), 6696–6704 (2021). https://doi.org/10.1609/aaai.v35i8.16828
Shannon, J., Grzes, M.: Reinforcement learning using augmented neural networks. University of Kent, UK, School of Computing (2018)
Ouyang, Y., He, W., Li, X.: Reinforcement learning control of a single-link flexible robotic manipulator. Institution of Engineering and Technology (IET) - Control Theory and Applications (2017). https://doi.org/10.1049/iet-cta.2016.1540
Almeida, A.C.: Sistema de memórias complementares e neurogênese aplicado á problemnas de aprendizado por reforço contínuo. Centro Universitário FEI (PhD. dissertation) (2022)
Lin, S., Zeng, J., Zhang, X.: Constructive neural network learning. IEEE Trans. Cybern. 4, 221–232 (2018). https://doi.org/10.1109/tcyb.2017.2771463
Parekh, R., Yang, J., Honavar, V.: Constructive neural-network learning algorithms for pattern classification 11(2), 436–451 https://doi.org/10.1109/72.839013 . 330. Accessed 20 Jan 2023
Han, H.-G., Zhang, S., Qiao, J.-F.: An adaptive growing and pruning algorithm for designing recurrent neural network. Neurocomputing 242, 51–62 (2017)
Luque-Baena, R.M., Urda, D., Subirats, J.L., Franco, L., Jerez, J.M.: Application of genetic algorithms and constructive neural networks for the analysis of microarray cancer data. Theor. Biol. Med. Model. 11(1), 1–18 (2014)
Qiao, J., Li, F., Han, H., Li, W.: Constructive algorithm for fully connected cascade feedforward neural networks. Neurocomputing 182, 154–164 (2016)
Von Zuben, F.J.: Modelos paramétricos e nâo-paramétricos de redes neurais artificiais e aplicações. Faculdade de Engenharia Elétrica da Unicamp - FEE-Unicamp - (PhD. dissertation) (1996)
Hajirahimi, Z., Khashei, M.: Hybrid structures in time series modeling and forecasting: A review. Eng. Appl. Artif. Intell. 86, 83–106 (2019)
Muzhou, H., Taohua, L., Yunlei, Y., Hao, Z., Hongjuan, L., Xiugui, Y., Xinge, L.: A new hybrid constructive neural network method for impacting and its application on tungsten price prediction. Appl. Intell. 47(1), 28–43 (2017)
Sun, R., Wilson, N., Lynch, M.: CLARION Cognitive Architecture Project (2019). http://www.clarioncognitivearchitecture.com/
Sun, R., Franklin, S.: Computational models of consciousness. Cambridge handbook of consciousness. Cambridge: Cambridge University Press., 151–174 (2007)
Snaider, J., McCall, R., Strain, S., Franklin, S.: The LIDA Tutorial Version 1.0 (2012). http://ccrg.cs.memphis.edu/assets/framework/The-LIDA-Tutorial.pdf
Snaider, J., McCall, R., Strain, S., Franklin, S.: LIDA Software Framework (2012). http://ccrg.cs.memphis.edu/framework.html
Laird, J.E., Congdon, C.B., Assanie, M., Derbinsky, N., Xu, J.: Manual Soar User Version 9.6.0 (2017). https://soar.eecs.umich.edu/downloads/SoarManual.pdf
Laird, J.E., Rosenbloom, P.S., Newell, A.: Towards chunking as a general learning mechanism. In: AAAI (1984). https://www.semanticscholar.org/paper/Towards-Chunking-as-a-General-Learning-Mechanism-Laird-Rosenbloom/9bf4cd11fc0053337fca66b29831e0db3922ce2b
Metta, G., Sandini, G., Vernon, D., Natale, L., Nori, F.: The iCub humanoid robot: an open platform for research in embodied cognition. In: Proceedings of the 8th workshop on performance metrics for intelligent Systems - PerMIS ’08, p. 50. ACM Press, Gaithersburg, Maryland (2008). https://doi.org/10.1145/1774674.1774683. http://portal.acm.org/citation.cfm?doid=1774674.1774683. Accessed 23 Aug 2018
Metta, G., Natale, L., Nori, F., Sandini, G., Vernon, D., Fadiga, L., Hofsten, C., Rosander, K., Lopes, M., Santos-Victor, J., Bernardino, A., Montesano, L.: The iCub humanoid robot: An open-systems platform for research in cognitive development. Neural Netw. 23(8–9), 1125–1134 (2010). https://doi.org/10.1016/j.neunet.2010.08.010. Accessed 23 Aug 2018
Fachantidis, A., Di Nuovo, A., Cangelosi, A., Vlahavas, I.: Model-based reinforcement learning for humanoids: A study on forming rewards with the iCub platform. In: 2013 IEEE Symposium on Computational Intelligence, Cognitive Algorithms, Mind, And Brain (CCMB), pp. 87–93. IEEE, Singapore (2013). https://doi.org/10.1109/CCMB.2013.6609170 . http://ieeexplore.ieee.org/document/6609170/. Accessed 23 Aug 2018
Tutsoy, O., Barkana, D.E.: Model free adaptive control of the under-actuated robot manipulator with the chaotic dynamics. ISA Transactions 118 (2021) https://doi.org/10.1016/j.isatra.2021.02.006
Tulving, E.: How many memory systems are there? Am. Psychol. 60, 385–398 (1985)
Chenatti, S.F., Previato, G., Tomazela, R., Kopp, V.G., Begazo, M.F.T., Salaro, L.G., Rohmer, E., Colombini, E.L., Simoes, A.d.S.: Larocs+unesp team description paper for the ieee humanoid racing 2018. In: Proceedings of the IEEE Humanoid Racing 2018 (2018). http://sistemaolimpo.org/midias/uploads/e6d5d0a492bae57e6aed09c99f816152.pdf
Rohmer, E., Singh, S.P.N., Freese, M.: V-rep: a versatile and scalable robot simulation framework. In: Proc. of The international conference on intelligent robots and systems (IROS) (2013). https://ieeexplore.ieee.org/document/6696520
Rohmer, E., Freese, M.: Robot Simulator CoppeliaSim (2019). https://www.coppeliarobotics.com/
Nussenbaum, K., Hartley, C.A.: Reinforcement learning across development: What insights can we draw from a decade of research? Dev. Cogn. Neurosci. 40, 100733 (2019). https://doi.org/10.1016/j.dcn.2019.100733
Acknowledgements
E. L. Colombini is partially funded by CNPq PQ-2 grant (315468/2021-1), A. S. Simões is partially funded by CNPq PQ-2 grant (312323/2022-0), R.R. Gudwin is partially funded by CEPID/BRAINN (Proc. FAPESP 2013/07559-3), L. L. Rossi is funded by MCTI/Softex.
Funding
This project was supported by the Ministry of Science, Technology, and Innovation of Brazil, with resources granted by Federal Law 8.248 of October 23, 1991, under the PPI-Softex. The project was coordinated by Softex and published as Intelligent Agents for Mobile Platforms based on Cognitive Architecture technology [01245.013778/2020-21].
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. L. L. Rossi implemented the agents, performed the experiments and analysis. E. L. Colombini, R. R. Gudwin, A. S. Simões, and P. P. Costa supervised the development of the agents and the obtained results. All authors reviewed and approved the final version of the manuscript.
Corresponding authors
Ethics declarations
Conflicts of interest
Not applicable.
Ethical standard
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
de Lellis Rossi, L., Rohmer, E., Dornhofer Paro Costa, P. et al. A Procedural Constructive Learning Mechanism with Deep Reinforcement Learning for Cognitive Agents. J Intell Robot Syst 110, 38 (2024). https://doi.org/10.1007/s10846-024-02064-9
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-024-02064-9