Gamification Increases Completion Rates in Massive Open Online Courses

International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Gamification Increases Completion Rates in Massive Open Online Courses Krzysztof Nesterowicz, National University of Public Service, Hungary* https://orcid.org/0000-0001-5384-8206 Ulkar Bayramova, Education Quality Assurance Agency (TKTA) Under the Ministry of Education, Azerbaijan Seyed-Mohammad Fereshtehnejad, Karolinska Institutet, Sweden Ana Scarlat, Maastricht School of Management, Romania Anthony Ash, Royal Society of the Arts, UK Anna Maria Augustyn, Independent Researcher, Poland Tamás Szádeczky, Budapest University of Technology and Economics, Hungary ABSTRACT Massive open online courses (MOOCs) aim at unlimited participation and open access via the web. There are concerns about the actual value of such courses. This is predominantly due to higher dropout rates. According to studies, only 7-13% go on to complete these courses. The high dropout rate in MOOCs is a challenge for education providers. This paper aims to explore reasons for high dropout rates within MOOCs and how they can be minimized. With this in mind, two research questions have been set for this study: 1) Why do MOOC participants not complete their courses? 2) How can the course completion rate be increased? Implementation of the strategies investigated in this paper can increase completion rates in MOOCs. In conclusion, after analyzing the collected data, the final results have shown that gamification increased the completion rate of MOOCs. KEywORdS Continuing Education, Distance Education, Dropout Rate, E-Learning, Lifelong Learning, MOOC, Online Education, Online Learning INTROdUCTION Among the various e-learning courses offered, one option is the MOOC model, which is “a course aimed at unlimited participation and open access via the web” (Kaplan & Haenlein, 2016) “with a publicly shared curriculum and open-ended outcomes” (McAuley et al., 2010). In addition to traditional course materials, such as filmed lectures, texts and problems, many MOOCs provide interactive user forums to support community interactions among students, professors and teaching assistants. MOOCs integrate the connectivity of social networking, the facilitation of an acknowledged expert in a field of study and a collection of freely accessible online resources. MOOCs build on the active engagement of several hundred to several thousand attendees who self-organize their participation with regards to learning goals, prior knowledge and skills as well as common interests (McAuley et al., 2010). Figure 1 shows basic characteristics of a MOOC. DOI: 10.4018/IJICTE.294447 *Corresponding Author This article published as an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) which permits unrestricted use, distribution, and production in any medium, provided the author of the original work and original publication source are properly credited. 1 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Figure 1. Meaning of Massive Open Online Courses (Plourde, 2013) MOOCs are often offered through virtual education platforms that have been custom built for the provision of such courses, such as Udacity, edX and Coursera (Ong & Grigoryan, 2015; Pang et al., 2014; Yuan & Powell, 2013). Yuan & Powell and Daniel et al. are two studies, among others, which view the MOOC phenomenon as the outcome of both the techno-media convergence process as well as the massification of tertiary education (Yuan & Powell, 2013; Daniel et al., 2015). MOOCs are viewed by some more as an opportunity for public institutions in the education sector with smaller budgets and less as a threat, alluding to the access advantages that such courses could bring to certain groups in society, such as retirees or employees looking for professional development (Ong & Grigoryan, 2015). This view of MOOCs as an opportunity to advance lifelong learning is equally held by official European bodies, viewing them as agents of change in higher education (De Freitas et al., 2015; European Commission, 2013; European Parliament, 2015). For the purpose of this literature review, the researchers have focused on MOOCs and their completion rate. The authors have set two research questions. Firstly, why do MOOC participants not complete their courses? Secondly, how can the course completion rate be increased? BACKGROUNd Higher education institutions have become more receptive to integrating new technologies into their teaching and learning processes over the last decade. One of these new technologies has been MOOCs (Costa et al., 2018). Figure 2 illustrates the rapid growth of MOOCs from 2012 to the end of 2018 (Shah, 2018). Taking the perspective of the supply side, Hollands and Tirthali looked into why institutions offered MOOCs, with a qualitative study of 83 interviews with leaders of 29 US institutions. They identified 6 main objectives (Hollands & Tirthali, 2014): 1. 2. 3. 4. 5. 6. 2 expanding the institutional scope and attracting a larger number of students (size), building and maintaining their brand (prestige), improving their finances by reducing costs or increasing income, improving their educational results, innovating in teaching and learning and conducting research on teaching and learning processes. International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Figure 2. Growth of MOOCs from 2012 until the end of 2018 (Shah, 2018) MOOCs by nature have some common characteristics: short videos, quizzes, peer base and/ or self-assignment and online forums (Glance et al., 2013), yet there are pedagogical differences in courses even in the same platform (Bali, 2014). Offering or participating in a MOOC has benefits for each party; however, concerns are arising on the real value behind MOOCs. This is predominantly due to higher dropout rates. Usually, only a 7-13% pass rate or sometimes even less than that go on to complete these courses (Jordan, 2014). A Stanford study investigated different engagement levels of the participants from three different MOOCs and found that there were typically four different types of MOOC learners: Completing, Auditing, Disengaging and Sampling Learners (Kizilcec et al., 2013): • • • • Completing MOOC Learners who completed the majority of the assessments offered in the class. Auditing MOOC Learners who only watch video lectures. Disengaging MOOC Learners who did assessments at the beginning but disengaged in the first three weeks of the course. Sampling MOOC Learners who watched video lectures for only one or two assessment periods. MOOCs High dropout Rate Reich and Ruipérez-Valiente attempt to explain why MOOCs mainly failed to achieve their stated goal of revolutionizing education, prompting the major MOOC providers to shift their attention to a more conventional role of assisting universities in bringing their academic programs online. What the authors add to the understanding of the MOOC landscape is an analysis of data from all MIT and Harvard University courses taught from 2012 to 2018 through edX, which quantitatively backs up what has been suspected. The data covers 5.63 million learners from 12.67 million course registrations (Reich, Ruipérez-Valiente, 2019). First, even as supporters pointed out that many people attended MOOCs for knowledge or skill development rather than a certification, one of the major knocks against MOOCs from the start was the low rate at which learners finished the courses. Reich and Ruipérez-Valiente show that completion rates in MIT and Harvard MOOCs did not increase but fell from 2013-14 to 2017-18 for three cohorts: 1) all participants; 2) those with a stated intention to complete; 3) those who paid to take verified courses. Figure 3 shows the completion rates for the three aforementioned groups: the rate for all course participants, for all learners who indicated in the survey that they intended to complete a course, and for all learners who paid for a verified track (Reich, Ruipérez-Valiente, 2019). 3 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Figure 3. Percentage of course completion by year and cohort of learners (n = 5.63 mln learners) (Reich, Ruipérez-Valiente., 2019) Among all MOOC participants, 3.13% completed their courses in 2017-18, down from about 4% from the two previous years and nearly 6% in 2014-15. Among the “verified” students, 46% completed in 2017-18, compared to 56% in 2016-17 and about 50% the two previous years (Reich & Ruipérez-Valiente, 2019). Sanchez-Gordon, Calle-Jimenez and Luján-Mora in “Relevance of MOOCs for Training of Public Sector Employees” describe three challenges that need to be addressed for the successful implementation of MOOCs in education: enrollment, completion rate and web accessibility (Sanchez-Gordon et al., 2015). In the study, the authors focus on the rate of completion. Completion Rate Completion rate is defined as the proportion of enrolled participants who earn a certificate of completion. The average MOOC completion rate is around 13% (Jordan, 2014). Since there are often several thousand registrants in a MOOC, this average completion rate still translates to a high number of participants completing the course. Nevertheless, to adequately interpret these massive enrollment numbers, it is important to consider emerging behaviors in MOOC registrants (Table 1). The following strategies may maximize the completion rate in MOOCs (Kizilcec et al., 2013): • • • • • • Working adults have difficulty following an 8 to 12 weeks course, which is the norm for university-led MOOCs. Reducing the length of the MOOCs to between 2 and 6 weeks will increase completion rates (Pappano, 2012). Keep the weekly time commitment in the range of 2 to 6 hours. Provide Internet access so the course can be taken at work. Design a clear syllabus. Create a social learning community. Assessing the performance of participants can reflect the efforts and contribution of civil servants, therefore creating for them the motivation to improve and participate in the training (Tien Vi, 2019). In this study, the authors focus on gamification as a strategy to increase the completion rate. 4 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Table 1. Different strategies of MOOC participants (Milligan et al., 2013; Hill, 2013) Strategy Characteristics No-shows enrol but never log in once the course opens; these can be as much as 50% of enrollment Lurkers enrol but just to observe or sample a few items at most; many of these participants do not complete week 1 Drop-ins become active only for a selected topic within the course but do not intend to complete the entire course; some of these are focused attendees who use a MOOC to meet external goals Passive participants view the course as content to consume and expect to be taught; these students typically watch videos, perhaps take quizzes but tend not to take part in activities or class discussions Active participants these whose intention is to fully participate and complete a MOOC Gamification in MOOCs The term gamification itself is quite recent. In 2002, Nick Pelling, a British game developer, coined and used it to describe his idea of enhancing the enjoyability and the speed of “electronic transactions” with “game-like accelerated user interface design” (Nepal et al., 2015). An often cited definition of gamification was elaborated by Deterding et al. in 2011 who referred to it as “the use of game design elements in non-game contexts” (Deterding et al., 2011). According to Nah et al., the most used game design elements in education are Points, Levels, Badges, Leaderboards, Prizes and Rewards, Progress Bars, Storylines and Feedback (Nah et al., 2014). Gamified learning environments are considered to be the next competitive key value in higher education institutions (HEIs) (Niman, 2014). Through the use of game mechanisms, gamification techniques can improve participant motivation and engagement, commitment, and loyalty among students, leading to a higher number of proactive participants (Gené et al., 2014). Few research studies have examined the experiences and effects of gamification techniques in MOOCs as quality innovative learning. Freire et al. (Freire et al., 2014) and Romero and Usart (Romero & Usart, 2013), discuss some MOOC experiences using Serious Games as integrated activities. METHOdOLOGy This paper focuses on the reasons for the dropout rates in MOOCs and how they can be addressed. The authors have set two research questions. Firstly, why do MOOC participants not complete their courses? Secondly, how can the course completion rate be increased? The research hypothesis is that gamification can significantly increase MOOC completion rates. The authors have worked with secondary data and a literature review based on peer-reviewed articles from research databases predominantly from Scopus and Web of Science. Secondary data is data that was collected by others for another primary purpose. During the secondary research, authors may draw data from government documents, scientific papers, statistical databases and other sources (Panchenko & Samovilova, 2020). Over 70 peer-reviewed papers have been collected and data from them was extracted and analyzed. The chosen articles were retrieved by searching for the combination of the following keywords: “Gamification” AND “MOOC” OR “MOOCs” OR “Massive Open Online Course” OR “Massive Open Online Courses”. A chi-square statistic was implemented to check the research hypothesis. The chi-square statistic is a way to show a relationship between two categorical variables, in this case, MOOCs with gamification vs. MOOCs without gamification. 5 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 RESULTS The collected data have been analyzed thematically and the success factors have been grouped in the table below (Table 2). When two or more sources are compared and contrasted—again, even if representing qualitative, quantitative, or mixed research—then cross-case qualitative analyses are justified (Onwuegbuzie et al, 2012). Based on this, the researchers grouped all data collected in the table and compared how MOOCs completion rate increased. In the first column there are written titles of investigated courses; in the second column number of registered students; in the third column number of completed registrants; the fourth column presents the completion rate, the fifth one indicates if gamification was implemented; the sixth one presents the year when the course took place and in the final column the source of the data is inserted. Statistical Analysis The authors pooled data from all studies summarized in Table 2 into a meta-Chi square 2*2 table (Table 3). In total, 123,453 participants underwent MOOC with gamification, out of which 13.7% completed the MOOC. On the other hand, a total of 490,686 learners underwent MOOC without gamification where only 1.7% of them completed the MOOC. Results from the meta Chi-squared test (Table 3) demonstrated that gamification significantly increases the rate of MOOC completion with a p-value < 0.00001. The chi-square statistic is 36072.5132. The p-value is < 0.00001. Significant at p < .05. The chisquare statistic with Yates correction is 36069.4748. The p-value is < 0.00001. Significant at p < .05. dISCUSSION In comparison to traditional face-to-face education as well as distance education – where students often have to meet certain admission requirements and primarily follow full educational programs – a MOOC is a relatively short course (generally 5–12 weeks) which is accessible anytime, anywhere, and to anyone. It is therefore recognized that it should not be compared to a traditional learning context for completion and dropout rates (Huin et al., 2016; Walji et al., 2016). Dropout rates have been long researched and studied in academia. One particular piece of research by Tinto differentiated between two levels of dropouts: 1) those who leave a single educational institution without an end qualification; 2) those who attend several educational establishments and leave without an end qualification from any of them (Tinto, 1975). He proposed a model for explaining student dropout that includes a combination of individual and organizational variables influencing dropout. In 1986 this theoretical model was taken and applied by Sweet in a study based on a distance education context. Furthermore, Garrison argued that research on dropouts in distance education was too focused on understanding and predicting but without taking into consideration the very nature of distance education. To this end, Garrison put forward a recommendation to focus on the student’s perspectives and developing situation specific models and theories before trying to generalize (Garrison, 1987). This is also in line with recommendations by Tinto (1975): ‘A […] more important limitation […] is the tendency to ignore the perspective of the individual’ (Tinto, 1975). This paper assumes that the research of Tinto from 1975 on dropout rates at higher education institutions is relevant to this present day study on MOOCs. In his research, Tinto points out that the individual motivation of course participants is crucial in regard to course dropout rates. The authors have noticed that this observation by Tinto is applicable to dropout rates in MOOCs. However, unlike the courses, Tinto observed, with many MOOCs there is no institution or representative of the institution compelling participants to complete the course: completion stems solely from the participants’ self-motivation. It has been observed that participants focus more on taking as much from the MOOC as they need rather than completing the course (Henderikx et al., 2017). 6 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Table 2. Overview of completion rate in MOOCs with and without gamification MOOC 1. “Energy saving” Number of registrants Number of participants who completed the course Completion rate Gamification Year of starting the course Reference 12,929 2019 15.62% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 2. “Distribution of electrical energy” 5,549 639 11,52% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 3. “Smart Grid: Electrical networks of the future” 6,608 821 12,42% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 4. “Smart Grid: Technical fundamentals” 5,498 743 13,51% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 5. “Electric power transmission” 5,961 1,074 18,02% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 6. “Conventional, clean energy, and its technology” 18,693 2,770 14,82% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 7. “Electric power: Concepts and principles” 15,978 1,807 11,31% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 8. “Energy: Past, present, and future” 13,224 2,106 15,93% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 9. “Carbon markets” 6,710 910 13,56% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 10. “Energy markets” 10,255 846 8,25% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 11. “The new electricity industry in Mexico” 8,975 1,224 13,64% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 12. “Energy reform and its opportunities” 12,744 1,928 15,13% yes 2017-2018 Romero-Rodriguez, Ramírez-Montoya, Gonzalez, 2019 13. “6.002x: Circuits & Electronics” 154,763 7,157 4.63% no 2012 Vaibhav, Gupta, 2014 14. “8.02x Electricity and Magnetism” about 40,000 1,721 4.3% no 2013 Vaibhav, Gupta, 2014 15. “Information Theory” 10,953 15 0.14% no 2014 Lyu, Chan, Yeung, 2018 16. “Introduction to Entrepreneurship” 45 12 26.67% yes 2013 Romero, Usart, 2013 17. “Bioelectricity: A Quantitative Approach” 12,725 313 2.46% no 2012 Belanger, Thornton, 2013 18. “Gratis Online Lernen. GOL-2014” 1,003 176 17.54% no 2014 Khalil, Ebner, Admiraal, 2017 19. “Gratis Online Lernen. GOL-2015” 476 94 19.74% no 2015 Khalil, Ebner, Admiraal, 2017 20. “Gratis Online Lernen. GOL-2016” 284 74 26.05% yes 2016 Khalil, Ebner, Admiraal, 2017 21. “First-Year Composition 2.0” 21,934 238 1.09% no 2013 Georgia Institute of Technology course via Coursera, 2013 22. “A History of the World since 1300” 83,000 605 0.73% no 2012 Princeton University course via Coursera, 2012 23. “Technicity” 21,000 400 1.9% no 2013 Ohio State University course via Coursera, 2013 24. “Generating the Wealth of Nations” 28,922 500 1.73% no 2013 University of Melbourne course via Coursera, 2013 continued on following page 7 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Table 2. Continue MOOC Number of registrants Number of participants who completed the course Completion rate Gamification Year of starting the course Reference 25. “Writing II Rhetorical Composing” 30,000 500 1.67% no 2013 Ohio State University course via Coursera, 2013 26. “Introduction to Sociology” 40,000 1,283 1.21% no 2012 Princeton University course via Coursera, 2012 27. “E-learning and Digital Cultures” 42,844 1,719 4.01% no 2013 University of Edinburgh course via Coursera, 2013 28. “Surviving Disruptive Technologies” 16,000 700 4.38% no 2013 University of Maryland College Park course via Coursera, 2013 29. “ICT in Primary Education: Transforming children’s learning across the curriculum” 9,000 315 3.5% no 2014 University of London course via Coursera, 2014 Table 3. Results from the Chi-squared test MOOCs Completed MOOCs Not Completed Marginal Row Totals With Gamification 16973 (5101.03) [27630.4] 106480 (118351.97) [1190.89] 123453 Without Gamification 8403 (20274.97) [6951.61] 482283 (470411.03) [299.62] 490686 Marginal Column Totals 25376 588763 614139 (Grand Total) A 14-week course called “6.002x: Circuits and Electronics” offered in 2012 by the Massachusetts Institute of Technology registered an enrollment of 154,763 students and only 7,157 (4.62%) of them fully completed the course (Romero-Rodriguez et al., 2019). Another example is the course “Information Theory” designed by the Chinese University of Hong Kong that registered 10,953 participants in 2014 and only 0.14% of the total completed it (Lyu et al., 2018). The high dropout rate in most MOOCs is the fundamental challenge faced by online education providers (Mamman et al., 2017). As per Saxena and Mishra, gamification supports the development of students’ motivational, cognitive, social, and emotional outlook (Saxena & Mishra, 2021). Romero-Rodriguez et al. in their study focused on the impact of gamification on the completion rate of MOOCs. They assumed that gamification, like using a system of badges, points, dashboards, challenges and leader boards, will create competition among the participants and will influence the creation of learning communities. They compared results of completion rates between MOOCs with gamification (14.43%) and MOOCs without gamification (6.16%) (Romero-Rodriguez et al., 2019). The positive impact of gamification on MOOC use has also been proven in a recent study by Aparicio et al. Their conclusions show that the factors which directly influence individual impact are use, user satisfaction and gamification. Furthermore, gamification was found to have a significant impact as a moderator between individual and organizational factors (Aparicio et al., 2019). Last but not least, MOOCs have turned out to be useful on a bigger scale in the time of the Covid-19 pandemic when on-site learning has been significantly limited worldwide. “Enrolment at 8 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Coursera has skyrocketed and was 640% higher from mid-March to mid-April 2020 than during the same period last year, growing from 1.6 to 10.3 million” (Shah, 2020). CONCLUSION The application of gamification in MOOCs opens up new learning opportunities, motivating learners to complete courses without any dropouts (Gené et al., 2014). This research has explored the implementation of gamification in MOOCs leads to increasing completion rates. Also, other recent studies prove that the implementation of gamification significantly reduces the dropout rate in MOOCs (Deterding et al., 2011; Romero-Rodriguez et al., 2019; Aparicio et al., 2019). Besides, assessing the performance of participants can reflect the efforts and contribution of learners, therefore helping to instil the motivation to improve and participate in the training (Tien Vi, 2019). Some people assume that MOOCs will not transform higher education. Rather they will provide new support for specific niches within already existing education systems. New education technologies are rarely, perhaps never, disruptive; rather they are domesticated by existing cultures and systems (Cuban, 1986). According to the findings, 123,453 attendees enrolled in a MOOC with gamification, of which 13.7% finished the course, while 490,686 participants took a MOOC without gamification, of which just 1.7% completed the course. Results from the data analysis demonstrated that gamification significantly increases the rate of MOOC completion. In conclusion, the authors recommend implementing gamification in MOOCs in order to increase the rate of completion. LIMITATIONS OF THE STUdy The study was limited by the number of investigated MOOCs (n = 29). Also, the completion rate was compared among MOOCs with and without gamification. Other factors, for example, individual reasons for dropouts among participants were not investigated due to limited access to such data. It was also not possible to study the reasons for dropouts due to any socio-cultural issues or language limitations of enrolled learners. dISCLOSURE STATEMENT No potential conflict of interest was reported by the authors. 9 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 REFERENCES Aparicio, M., Oliveira, T., Bação, F., & Painho, M. (2019). Gamification: A key determinant of massive open online course (MOOC) success. Information & Management, 56(1), 39–54. doi:10.1016/j.im.2018.06.003 Bali, M. (2014). MOOC Pedagogy: Gleaning Good Practice from Existing MOOCs. MERLOT. Journal of Online Teaching and Learning, 10(1), 44–57. Belanger, Y., & Thornton, J. (2013). Bioelectricity: A quantitative approach: Duke University’s first MOOC. Duke University. https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161/6216/Duke_Bioelectricity_ MOOC_Fall2012.pdf?sequence=1 Costa, C., Teixeira, L., & Alvelos, H. (2018). Exploring the Usage of MOOCs in Higher Education Institutions: Characterization of the Most Used Platforms. International Journal of Information and Communication Technology Education, 14(4), 1–17. doi:10.4018/IJICTE.2018100101 Cuban, L. (1986). Teachers and Machines. Teachers College Press. Daniel, J., Vázquez, E., & Gisbert, M. (2015). The Future of MOOCs: Adaptive Learning or Business Model? RUSC Universities and Knowledge Society Journal, 12(1), 64–74. 10.7238/rusc.v12i1.2475 De Freitas, S. I., Morgan, J., & Gibson, D. (2015). Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision. British Journal of Educational Technology, 46(3), 455–471. doi:10.1111/bjet.12268 Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From Game Design Elements to Gamefulness: Defining Gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, 9-15. 10.1145/2181037.2181040 European Commission. (2013). Opening up Education: Innovative teaching and learning for all through new Technologies and Open Educational Resources. Author. European Parliament. (2015). Internationalisation of Higher Education. Author. Freire, M., Del Blanco, A., & Fernández-Manjón, B. (2014). Serious games as edX MOOC activities. Proceedings of Global Engineering Education Conference (EDUCON), 867-871. Garrison, D. R. (1987). Researching dropout in distance education. Distance Education, 8(1), 95–101. doi:10.1080/0158791870080107 Gené, O. B., Núñez, M. M., & Blanco, Á. F. (2014). Gamification in MOOC: challenges, opportunities and proposals for advancing MOOC model. In Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM ’14). Association for Computing Machinery. 10.1145/2669711.2669902 Glance, D. G., Forsey, M., & Riley, M. (2013). The pedagogical foundations of massive open online courses. First Monday, 18(5). Advance online publication. doi:10.5210/fm.v18i5.4350 Henderikx, M. A., Kreijns, K., & Kalz, M. (2017). Refining success and dropout in massive open online courses based on the intention–behavior gap. Distance Education, 38(3), 353–368. doi:10.1080/01587919.2017.1369006 Hill, P. (2013). Emerging Student Patterns in MOOCs: A Graphical View. Academic Press. Hollands, F. M., & Tirthali, D. (2014). Why Do Institutions Offer MOOCs? Online Learning: Official. Journal of the Online Learning Consortium, 18(3), 1–20. Huin, L., Bergheaud, Y., Caron, P. A., Codina, A., & Disson, E. (2016). Measuring completion and dropout in MOOCs: A learner-centered model. In Proceedings of the European MOOC Stakeholder Summit 2016. Norderstedt: Books on Demand GmbH. Jordan, K. (2014). Initial Trends in Enrolment and Completion of Massive Open Online Courses. International Review of Research in Open and Distributed Learning, 15(1), 133–160. doi:10.19173/irrodl.v15i1.1651 Kaplan, A. M., & Haenlein, M. (2016). Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the Cookie Monster. Business Horizons, 59(4), 441–450. doi:10.1016/j.bushor.2016.03.008 10 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Khalil, M., Ebner, M., & Admiraal, W. (2017). How Can Gamification Improve MOOC Student Engagement? Proceedings of European Conference on Game Based Learning, 819-828. Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK ’13), 170-179. 10.1145/2460296.2460330 Lyu, C. X., Chan, R. Y. Y., & Yeung, R. W. (2018). Promoting student completion in a MOOC on information theory. In Proceedings of the IEEE Frontiers in Education Conference (FIE). IEEE., doi:10.1109/FIE.2018.8658570. Mamman, B., Yusof, A., Abuhassna, H. M., Aly, H., Al-Ahmadi, T., Atan, N. A., Harun, J., Said, M. N. H. B. M., Ismail, Z., Yahaya, N., & Ibrahim, F. K. (2017). Design and Learning Strategies Applied in Mooc: A MetaAnalysis. Sains Humanika, 9(1-4), 71–76. 10.11113/sh.v9n1-4.1127 McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC Model for Digital Practice. Academic Press. Milligan, C., Littlejohn, A., & Margaryan, A. (2013). Patterns of Engagement in Connectivist MOOCs. Journal of Online Learning and Teaching / MERLOT, 9(2), 149–159. Nah, F. F., Zeng, Q., Telaprolu, V. R., Ayyappa, A. P., & Eschenbrenner, B. (2014). Gamification of Education: A Review of Literature. International Conference on HCI in Business, 401–409. 10.1007/978-3-319-07293-7_39 Nepal, S., Paris, C., & Bista, S. (2015). Gamification on the Social Web. In Social Media for Government Services. Springer International Publishing. 10.1007/978-3-319-27237-5_10 Niman, N. B. (2014). The Gamification of Higher Education. In Developing a Game-Based Business Strategy in a Disrupted Marketplace. Palgrave Macmillan US. 10.1057/9781137331465 Ong, B. S., & Grigoryan, A. (2015). MOOCs and Universities: Competitors or Partners? International Journal of Information and Education Technology (IJIET), 5(5), 373–376. doi:10.7763/IJIET.2015.V5.533 Onwuegbuzie, A. J., Leech, N. L., & Collins, K. M. (2012). Qualitative Analysis Techniques for the Review of the Literature. Qualitative Report, 17(28), 1–28. doi:10.46743/2160-3715/2012.1754 Panchenko, L., & Samovilova, N. (2020). Secondary data analysis for educational research: opportunities for PhD students. Proceedings of The International Conference on History, Theory and Methodology of Learning (ICHTML 2020), 75. 10.1051/shsconf/20207504005 Pang, Y., Wang, T., & Wang, N. (2014). MOOC’ Data from Providers. Proceedings of Enterprise Systems Conference (ES), 87-90. Pappano, L. (2012). The Year of the MOOC. New York Times. Peters, O. (1971). Theoretical aspects of correspondence instruction. In O. MacKenzie & E. L. Christensen (Eds.), The changing world of correspondence study: International readings (pp. 223–228). Pennsylvania State University Press. Plourde, M. (2013). MOOC, every letter is negotiable. Licensed CCBY. Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC Pivot. Science, 363(6423), 130–131. doi:10.1126/ science.aav7958 PubMed Romero, M., & Usart, M. (2013). Serious Games Integration in an Entrepreneurship Massive Online Open Course (MOOC). In Serious Games Development and Applications. Springer Berlin Heidelberg. Romero-Rodriguez, L. M., Ramírez-Montoya, M.-S., & Gonzalez, J. R. V. (2019). Gamification in MOOCs: Engagement Application Test in Energy Sustainability Courses. IEEE Access : Practical Innovations, Open Solutions, 7, 32093–32101. doi:10.1109/ACCESS.2019.2903230 Sanchez-Gordon, S., Calle-Jimenez, T., & Luján-Mora, S. (2015). Relevance of MOOCs for Training of Public Sector Employees. In Proceedings of 2015 International Conference on Information Technology Based Higher Education and Training (ITHET). IEEE., doi:10.1109/ITHET.2015.7218016. 11 International Journal of Information and Communication Technology Education Volume 18 • Issue 1 Saxena, M., & Mishra, D. K. (2021). Gamification and Gen Z in Higher Education: A Systematic Review of Literature. International Journal of Information and Communication Technology Education, 17(4), 1–22. Advance online publication. doi:10.4018/IJICTE.20211001.oa10 Shah, D. (2018). By The Numbers: MOOCs in 2018. Class Central. Shah, D. (2020). Highlights from Coursera Partners Conference 2020. https://www.classcentral.com/report/ coursera-conference-2020-highlights/ Tien, Vi. C., (2019). Training and Retraining Civil Servants of Home Affairs in Vietnam in Response to the Requirements of International Integration. International Journal of Human Resource Studies, 9(1), 126-135. https://www.macrothink.org/journal/index.php/ijhrs/article/download/14085/11271 Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125. doi:10.3102/00346543045001089 Walji, S., Deacon, A., Small, J., & Czerniewicz, L. (2016). Learning through engagement: MOOCs as an emergent form of provision. Distance Education, 37(2), 208–223. doi:10.1080/01587919.2016.1184400 Yuan, L., Powell, S. (2013). MOOCs and Open Education: Implications for Higher Education. eLearning Papers, In-depth, 33(2), 1-7. Krzysztof Nesterowicz completed his MSc from the Faculty of Pharmacy at the Jagiellonian University in Poland. Starting his PhD in 2008, his research lies in the effectiveness of e-learning compared to on-site learning. He is continuing his research at the Ludovika - University of Public Service, Faculty of Public Administration Sciences in Hungary. His research interests are e-learning and neuroscience. Ulkar Bayramova has been working as an advisor at the Education Quality Agency (TKTA) under the Ministry of Education of the Republic of Azerbaijan. She worked as a Director of Teaching and Learning Center and also as an instructor at Khazar University. She is a PhD candidate in Pedagogy. She has a BA and MA in educational sciences from Lankaran State University, Azerbaijan. She was a distance education student at Tallinn University as well as at the University of Turku and in the USA (Internet Society Next Generation Leaders course). She was an exchange PhD student at Tallinn University, Estonia for 2 years. She has been involved in ISOC and ICANN activities and since 2007 she has been working in different NATO, FP7, Tempus, Erasmus Mundus and Erasmus+ projects as an international, senior and junior expert. She is also an International Counselor in Mediation. Seyed-Mohammad Fereshtehnejad completed his Neuroscience PhD at Karolinska Institutet, Sweden in 2015, then performed 2 years of postdoctoral research fellowship at McGill University, Montreal, Canada. He also earned a Master’s degree in Medical Education with extensive experience in biostatistical methods and analyses. Apart from the field of neuroscience, he has participated in several research projects on educational methods in medicine and pharmacy. He is currently a Neurology resident physician at the University of Ottawa, Canada. Ana Scarlat completed her MBA at the Maastricht School of Management in the Netherlands. As a manager with more than 10 years experience, she is currently at an e-commerce company. She also has extensive experience of building and leading IT teams and is interested in finding new ways of improving employee hard and soft skills. While having teams in multiple locations, it is vital to find ways of disseminating the information via the Internet, thus giving the same opportunities of learning to everybody, no matter their location. Anthony Ash studied language and linguistics at university, later on specialising in language education. After a decade of working in education, particularly focused on the effectiveness of technology in relation to classroom methodology, he was awarded a Fellowship from the Royal Society of the Arts. He has since gone on to work in the field of localization, where he now manages a team dedicated to localization and is exploring the use of technology, training and language in that field. Anna Maria Augustyn graduated in applied social sciences from Warsaw University. She is an experienced consultant working in the field of evaluation and capacity building of actors undertaking transformative and innovation projects. Throughout her career, she conducted various assignments for a diverse range of clients spanning international agencies, such as the EU and UN, national public sector, NGOs, corporate and SMEs. She was also in charge of e-learning projects targeting international audiences. Tamás Szádeczky is an associate professor of Budapest University of Technology and Economics. He has been working in the field of information security audit, consultation and training since 2003. He is also a lecturer and researcher on the topic for more than a decade in multiple universities in three countries. He also deals with cybersecurity research at NATO SPS Independent Scientific Evaluation Group. 12