Using Facebook Metrics to Measure Student Engagement in Moodle

IJODeL, Vol. 2, No. 2, (December 2016) Using Facebook Metrics to Measure Student Engagement in Moodle Leonardo Magno Senior Editor, Publishing Team, Asian Development Bank, lmagno@gmail.com Abstract Facebook uses engagement rates as a metric for social media managers and marketers to measure how effectively a page, a post, a comment, a brand, or a topic is able to engage the target audience. Studies have proposed the use of these metrics for other types of online measurements in applications and industries other than social media. This concept paper proposes the use of a modified Facebook engagement rate formula to (i) measure students’ engagement with forum posts using virtual learning environments such as Moodle and (ii) get data from students regarding which topics or posts engage them more in the learning process. While virtual learning environments have their own logs and learning analytics software, and studies have explored the use of Facebook as a discrete and supplementary platform to traditional learning management systems, this paper fills the research gap by applying Facebook’s engagement rate formula to Moodle itself. Applying an existing social media audience engagement formula to an online course may provide instructors and educational institutions valuable feedback on what types of posts, comments, or topics make students interact and engage more in online learning sessions. Such an exercise may lead to new metrics which distance learning institutions could use to calibrate their content with the objective of improving students’ interaction and engagement with instructors, fellow students, and the online course in general. Keywords: Facebook, Moodle, social media, engagement rate, learning analytics Introduction Virtual learning environments like Moodle and Blackboard—sometimes referred to as online course platforms or learning management systems—come with their own analytics and reporting tools either as built-in features or as plug-in software. Alternatively, instructors also use external software such as Microsoft Excel to derive learning analytics for their courses (Konstantinidis and Grafton 2013). Data logs have also been studied in an effort to help instructors interpret Moodle usage data and analyze student behavior in online courses (Casey et al. 2010). Meanwhile, studies have explored the use of Facebook to supplement or complement learning analytics gathered from data logs of online courses (Junco 2012). Junco examined the relationship between the frequency of Facebook use, participation in Facebook activities, and student engagement. He concludes: “Higher education administrators, faculty and staff have an opportunity to help students use Facebook in ways that are beneficial to their engagement and, by extension, to their overall academic experience. Given that Facebook continues to be popular among college students, and that universities are interested in engaging and retaining students, it is important for those _____________________ DISCLAIMER: This is a conceptual paper. Those who are interested to test it may communicate with the author. Using Facebook Metrics to Measure Student Engagement in Moodle 39 IJODeL, Vol. 2, No. 2, (December 2016) working in higher education to … design and support interventions that meet students where they are in order to help them get to where they are going.” Studies have also examined the use of course-specific Facebook pages that had been integrated into several university courses (Irwin et al. 2012). Irwin et al. evaluated the efficacy of these pages as a course learning tool using student perceptions. Their findings indicate that “Facebook may be an appropriate addition to traditional e-learning tools, providing an integration of technology that is well received and used by today’s students with an application that can be adapted to deliver content in a similar way to conventional learning management systems.” However, while Facebook has long been used as a complementary yet separate learning platform side-by-side with Moodle, the use of Facebook’s own calculations for engagement are not used to analyze data generated by logs from learning management systems. Social media such as Facebook and Twitter allow users not only to interact with one another but also to express their approval or disapproval of posts, content, or pages they find in their social networks. To derive meaningful information from such data, Facebook has developed its own metrics to gauge the efficacy of content based on the interactions such content receives. The primary metric Facebook uses is called the “engagement rate,” or “the percentage of people who saw a post that reacted to, shared, clicked or commented on it.” The engagement rate becomes useful when content creators want to measure the level of engagement and interaction a piece of content receives from the audience through comments, “Likes,” and “Shares.” This thus becomes an indicator of impact the content had on the audience.1 Facebook’s post engagement rate is calculated thus:2 PER=(CL+C+S+L) R Where: PER = Post engagement rate CL = Clicks C = Comments S = Shares L = Likes R = Reach (# of people who saw the post) The Facebook post engagement rate numerator thus becomes the number of “Clicks,” “Comments,” “Shares,” and “Likes” on a particular Facebook post. The denominator becomes the number of people that particular post reached, or the number of people who actually saw the post (as opposed to counting the total number of “Fans” or “Friends,” some of whom may not have even seen the post).3 1 Facebook. (2016). Help Center: How is engagement rate defined? Retrieved 20 June 2016 from https://www.facebook. com/help/178043462360087 2 Smitha, N. (2016). Simply Measured. Facebook Metrics Defined: Engagement Rate. Retrieved 20 June 2016 from http://simplymeasured.com/blog/facebook-metrics-defined-engagement-rate/#sm.0001srq731egmefaxeo1fnrz0fhib 3 Gupta, S. (2015). Contify Musings. An Effort to Solve Paradox of Facebook Engagement Rate. Retrieved 10 June 2016 from http://www.contify.com/blog/2015/10/01/effort-solve-paradox-facebook-engagement-rate/ 40 Leonardo Magno IJODeL, Vol. 2, No. 2, (December 2016) Studies have also used social media data to derive meaningful information in applications other than determining engagement rate. For example, Hall compares a scientist or academic’s social media profile against the citation indexes of their scholarly work (Hall 2014). To help quantify this, Hall proposed the creation of the “Kardashian Index” or “K-Index,” a measurement of “discrepancy between a scientist’s social media profile and publication record based on the direct comparison of numbers of citations and Twitter followers.” Hall believes that Hollywood and television celebrity Kim Kardashian is famous simply for being famous, and that the same phenomenon might apply within the scientific community—that scientists and academics are renowned simply because they are already renowned. The K-Index is a calculation of an academic’s number of Twitter followers divided by the number of followers that particular academic is expected to have based on his or her citation record. Hall plotted the number of scientists’ and academics’ Twitter followers against the number of scientific citations they have received to calculate their K-Index. The study found that some scientists and academics had inflated numbers of Twitter followers versus the number of citations their works have received. A high K-index may be interpreted as undue scientific fame while a low K-index suggests that a scientist is being undervalued. According to Hall, a researcher, scientist, or academic whose K-index is greater than 5 can be considered a “Science Kardashian.” There is also a study by Asur and Huberman who created an equation in an attempt to predict which movie would become a box office hit using Twitter comments (Asur and Huberman 2010). They derived the Tweet rate by dividing the number of Tweets about a movie by the number of hours such a movie was being discussed, and factored in subjectivity by dividing positive and negative Tweets by the number of neutral Tweets. Edwards et al. also studied the use of aggregated social media metrics like the online service called Klout which measures social media influence through the Klout Score (Edwards et al. 2013). Klout uses more than 400 of what it calls “signals” from eight different social networks to update a person’s Klout Score on a daily basis. The Klout Score is calculated through these signals that are derived from combinations of attributes, such as the ratio of reactions a person generates compared to the amount of content that person shares. Klout explains it thus: “For example, generating 100 retweets from 10 tweets will contribute more to your Score than generating 100 retweets from 1,000 tweets. We also consider factors such as how selective the people who interact with your content are. The more a person likes and retweets in a given day, the less each of those individual interactions. Additionally, we value the engagement you drive from unique individuals. One-hundred retweets from 100 different people contribute more to your Score than do 100 retweets from a single person.”4 After measuring multiple pieces of data from several social networks, Klout then applies the data to the Klout Score algorithm, and then shows the resulting number on the person’s Klout profile. The higher a person’s Klout Score, the harder it becomes to increase. The Klout Score is the accumulation of a person’s influence across all of his or her social network memberships, not the average. Adding social media networks adds to the person’s ability to share expertise, and that helps increase the Klout Score. 5 4 Klout. (2015). The Klout Score. Retrieved 14 June 2016 from https://klout.com/corp/score Ibid. 5 Using Facebook Metrics to Measure Student Engagement in Moodle 41 IJODeL, Vol. 2, No. 2, (December 2016) In short, social media metrics have been used to measure and even predict phenomena in other industries apart from social media, such as the next box office hit, a person’s online influence, or an academic’s online fame. Perhaps social media metrics could also be used in the distance learning sector, specifically to measure student engagement in online courses. This concept paper proposes the use of social media engagement calculations, Facebook engagement rates in particular, to determine student’s engagement with content found in Moodle. Moodle data logs can be used to derive student engagement rates patterned after Facebook’s engagement rate calculations. Why use social media metrics as a feedback mechanism and data collection vehicle? Studies have shown that it has become an effective platform for both purposes. Asur and Huberman concluded thus: “We show that social media feeds can be effective indicators of real-world performance.” (Asur and Huberman 2013). Leskovec et al. noted that with proper research, one can build models to aggregate the opinions of the collective population and gain useful insights into their behavior, while predicting future trends (Leskovec, et al. 2006). Such a practice using social media could be useful in certain industries, and as concluded by Huberman et al. in their study using Twitter to analyze consumer behavior, gathering information on how people converse regarding particular products can be helpful when designing marketing and advertising campaigns (Huberman et al. 2009). Or, in the case of Hall’s study, to determine the popularity of scientists and academics based on their Twitter followers and number of citations (Hall 2014). Esteves in 2012 explored the use of Facebook side-by-side with a Moodle course on web design and publishing at the University of the Philippines Open University (Esteves 2012). Her objective was to see if Facebook would augment the learning process and increase student engagement. However, in her case study, Facebook and the Moodle course were treated as two separate platforms. This concept paper deviates from Esteves’ approach by integrating Facebook-like functionalities into Moodle and by using Facebook-like metrics to measure the engagement rate of forum posts in Moodle courses. This paper proposes the use of Moodle data logs as primary research data that would be fed into Facebook’s engagement rate calculation. While Moodle data mining is already being done to study student behavior (Casey et al. 2010), feeding such data into a social media calculation has not yet been fully explored. Marrying the two (Moodle data logs and Facebook’s engagement rate formula) may produce meaningful data for the improvement of learning management system courseware engagement using metrics for a current tool in which students are already active and adept—social media (Facebook in particular). While there are data logs that can be mined, Facebook’s engagement rate formula to measure student interaction in Moodle has not yet been utilized. This leads us to the question: What results would we get if Facebook’s engagement rate equation were used to analyze student interaction with Moodle forum posts? 42 Leonardo Magno IJODeL, Vol. 2, No. 2, (December 2016) Objectives The objectives and methodologies of this concept paper are similar to those of the previously mentioned studies, namely (i) aggregating data using social media formulas, (ii) gaining insights and measuring content interaction from such social media inputs, and (iii) predicting a trend or a behavior using the social media data collected. In this case, the trend or behavior being predicted is the type of content that would effectively engage students enrolled in online learning courses, specifically Moodle. This paper proposes the: • modification of Facebook’s engagement rate calculation for the purpose of this study, • modification of Moodle to incorporate Facebook-like features, • mining of Moodle data logs specific to the factors needed in the modified engagement rate calculation, • application of the gathered data to the modified equation, and • use of the resulting engagement rate to measure student engagement with Moodle posts. Such an exercise may allow us to determine if: • Facebook’s engagement rate calculation can be modified to measure student engagement in posts found in online courses; • data logs from existing learning management systems can be applied to the modified engagement rate calculation; and • such a measurement could predict student interaction with or active engagement in future course topics, posts, or activities. Ultimately, such an exercise could help educational institutions improve course and content design by discovering which types of content achieve higher student engagement rates in learning management systems. This exercise is not meant to measure student performance (grades) in the course. It is intended to measure only student engagement in certain topics or posts in the course. Methodology As shown earlier, Facebook’s engagement rate is calculated thus:6 PER=(CL+C+S+L) R The post engagement rate shows how the audience interacts with a certain post on Facebook. The audience performs such interaction in four ways. First, they click on a post. Second, they might comment on that post. Third, they might “Share” that post. Finally, they might “Like” that post. 7 Smitha, N. (2016). Simply Measured. Facebook Metrics Defined: Engagement Rate. Retrieved 20 June 2016 from http://simplymeasured.com/blog/facebook-metrics-defined-engagement-rate/#sm.0001srq731egmefaxeo1fnrz0fhib 7 Center for Rural Enterprise Management. (2016). Facebook Analytics Tips. Kansas State University. Retrieved 6 July 2016 from http://www.k-state.edu/cree/Facebookanalytics.html 6 Using Facebook Metrics to Measure Student Engagement in Moodle 43 IJODeL, Vol. 2, No. 2, (December 2016) Moodle has log and usage reports that track student activities similar to the numerators indicated in the above formula (CL = Clicks, C = Comments, and L = Likes) except for “Shares” as public external “sharing” of content is limited in self-contained online courses (virtual classroom content is usually confined to class members). Moreover, there is no “Like” feature in Moodle, but there are third-party widgets that allow administrators to add a “thumbs-up” button which students can click on to “like” a forum post. Facebook itself has a “Like” widget code that can be added to Moodle. 8 While there is a “Rating” feature in Moodle which allows students to “rate” a forum post, the use of a “Like” or “thumbs-up” button similar to that of Facebook is proposed as an indicator of a student’s approval of a forum post. This method is proposed in lieu of the “Rating” feature to more closely mirror Facebook’s engagement rate formula to see its applicability in Moodle. Meanwhile, the denominator of the Facebook engagement rate formula is “R” or “Reach,” the number of people who saw the post. Again, this is not the total number of Facebook “Fans” or “Friends” as it is possible that not all of those people actually saw the post. Similarly for Moodle, “R” would not be represented by the total number of students in the class or course but rather the unique number of students who actually saw the Moodle forum post. “Reach” alone, or counting the number of students who saw the Moodle forum post, will not give an indicator of whether or not that particular post engaged the student. The interaction of the student with that post is also important (clicking, commenting, or liking). This interaction divided by the number of people who saw the post would show if the audience is engaging with the content. 9 The modified Moodle post engagement rate calculation could thus be expressed as: MPER=(CL+C+L) R Where: MPER = Moodle post engagement rate CL = Clicks C = Comments L = Likes R = Reach (# of people who saw the post) To get the engagement rate of a particular forum post in Moodle, we need to get (i) the number of clicks generated by that post, or how many times that forum post was clicked on or visited by a member of the course (note that it is possible that students may click that post again or revisit that post); (ii) the number of comments generated by that post (multiple comments from a single student may occur); and (iii) the number of “Likes” generated by that post (after installing the “thumbs-up” widget, students can only “Like” a post once). 8 University of the Arts London. (2013) Moodle: Add a Facebook Like etc. Retrieved 5 July 2016 from http:// elearningsupport.myblog.arts.ac.uk/files/2013/05/Create-a-Facebook-Like-box.pdf 9 Center for Rural Enterprise Management. (2016). Facebook Analytics Tips. Kansas State University. Retrieved 6 July 2016 from http://www.k-state.edu/cree/Facebookanalytics.html 44 Leonardo Magno IJODeL, Vol. 2, No. 2, (December 2016) Let’s take the hypothetical example of 30 students in a Moodle course (maximum possible “Reach” or “R” would thus be 30), where two forum posts—Post A and Post B—were made by two different students. However, let’s say only a third or 10 members of the class became interested in Post A and actually clicked on and saw the post. In contrast, Post B gained more views, with all 30 class members clicking on and seeing the post. Post A thus “reached” 10 students while Post B “reached” all 30 students. However, let’s say that—even though it reached only 10 students—Post A received three clicks from each student (each of the 10 students visited the post three times, thus “CL” would be 10 x 3 = 30) and three comments each (thus “C” would be 10 x 3 = 30). This would indicate that the 10 students were repeatedly engaging and interacting with Post A. Further, let’s say all 10 students “Liked” Post A (students can only “Like” a post once, so “L” becomes 10 x 1 = 10). Meanwhile, let’s say Post B—while reaching all 30 students—did not elicit any comments or “Likes” and was clicked on only once by all 30 class members (thus the maximum reach of 30). Both “CL” and “R” would be 30, but “C” and “L” would be 0. Data for both Posts A and B would look thus: Post A Post B CLICKS COMMENTS LIKES REACH 10 x 3 = 30 30 x 1 = 30 10 x 3 = 30 0 10 x 1 = 10 0 10 30 Feeding the data into the modified equation, the numerator and denominator for Post A would thus be: MPER (Post A) = (30+30+10) 10 and further: MPER (Post A) = 70 10 MPER (Post A) = 7 Post A’s engagement rate would be 7%. Using Facebook Metrics to Measure Student Engagement in Moodle 45 IJODeL, Vol. 2, No. 2, (December 2016) Meanwhile, the numerator and denominator for Post B would be: MPER (Post B)=(30+0+0) 30 and further: MPER (Post B) = 30 30 MPER(Post B) = 1 Post B’s engagement rate would only be 1%. Results and Discussions Facebook post engagement rates average between 1% and 3%, but higher figures are found in some industries and popular brand pages with more successful rates of engagement using other social media platforms such as Twitter or Instagram. 10 This shows that, with a higher engagement rate of 7% compared to Post B’s 1%, it is possible that Post A may have been considered by students as more interesting or engaging than Post B, despite the fact that only 10 students reached Post A (because those 10 students clicked and commented on Post A repeatedly, and all 10 “Liked” the post as well). This repeated interaction of clicking and commenting on the post may indicate genuine interest in the post. Compare this with Post B which reached all 30 students in the course, and yet none of them revisited the post (re-clicking), commented on it, nor “Liked” the post. This also illustrates why “Reach” alone or the number of students who saw the post is not a complete indicator of interest or engagement as no further interaction with the forum post occurs. Post A’s 7% engagement rate is high, and may indicate that—based on the number of times the post was clicked on (viewed), commented on, and “Liked”—Post A may be more “engaging” to the course members due to the interaction it received despite the fact that only a third of the class actually viewed the post. The use of social media metrics to measure phenomena outside social media have been explored before, like in the Twitter experiments of Hall and Asur and Huberman. Meanwhile, data mining has been used by the likes of Casey and Konstantinidis and Grafton to analyze user behavior in Moodle courses. Studies from Junco, Irwin, and Esteves have attempted to use Facebook as a separate platform to supplement the Moodle learning and communication experience. 10 Robinson, J. (2014). Measuring Facebook Engagement. We Are Social. Retrieved 6 July 2016 from http://wearesocial.com/uk/blog/2014/07/measuring-facebook-engagement 46 Leonardo Magno IJODeL, Vol. 2, No. 2, (December 2016) This proposal, however, seeks to integrate Facebook-like functionalities and to adopt Facebooklike metrics within Moodle itself, melding Facebook within Moodle and without treating it as a distinct and separate platform. Based on the dummy data fed into a calculation derived from a modified post engagement rate formula based on Facebook’s own engagement rate, the idea seems plausible. Conclusions and Recommendations Based on this study proposal’s objectives and after feeding dummy data into the modified Moodle post engagement rate formula, we conclude that it is possible to aggregate data using an existing social media formula and then apply that to an online course. That collected data can be used to gain insights and measure student interaction and engagement with Moodle posts. In turn, the types of posts that received high engagement rate numbers may be used to predict student behavior for future posts. It is recommended that the modified Facebook engagement rate formula be applied to an actual Moodle course for an entire term, for only then would the formula’s true utility in analyzing student interaction and engagement with specific posts be manifested. A possible next step could be content analysis of posts that receive high engagement rates among students to determine, codify, and even predict what types of posts would have a higher possibility of eliciting interaction and engagement among students in the future. While this proposal is meant specifically for Moodle, it is possible for the same concept to be applied to other learning management systems as well. References Asur, S., & Huberman, B. A. (2010). Predicting the future with social media. In Web intelligence and intelligent agent technology (WI-IAT), 2010 IEEE/WIC/ACM International Conference on (Vol. 1, pp. 492-499). IEEE. Casey, K., Gibson, P., & Paris, I. S. (2010). Mining moodle to understand student behaviour. International conference on engaging pedagogy. Retrieved 28 June 2016 from http://goo. gl/tLxOSQ Center for Rural Enterprise Management. (2016). Facebook analytics tips. Kansas State University. Retrieved 6 July 2016 from http://www.k-state.edu/cree/Facebookanalytics.html Edwards, C., Spence, P. R., Gentile, C. J., Edwards, A., & Edwards, A. (2013). How much Klout do you have: A test of system generated cues on source credibility. Computers in human behavior, 29 (5), A12-A16. Esteves, K. K. (2012). Exploring Facebook to enhance learning and student engagement: a case from the University of Philippines (UP) Open University. Malaysian journal of distance education, 14 (1), 1-15. Using Facebook Metrics to Measure Student Engagement in Moodle 47 IJODeL, Vol. 2, No. 2, (December 2016) Facebook. (2016). Help center: How is engagement rate defined? Retrieved from https://www. facebook.com/help/178043462360087 Gupta, S. (2015). Contify Musings: An effort to solve paradox of Facebook engagement rate. Retrieved from http://www.contify.com/blog/2015/10/01/effort-solve-paradox-facebookengagement-rate/ Hall, N. (2014). The Kardashian index: A measure of discrepant social media profile for scientists. Genome biology, 15 (7), 424. Huberman, B. et al. (2009). Social networks that matter: Twitter under the microscope. First monday, 14 (1), Jan 2009. Irwin, C., Ball, L., Desbrow, B., & Leveritt, M. (2012). Students’ perceptions of using Facebook as an interactive learning resource at university. Australasian journal of educational technology, 28 (7), 1221-1232. Junco, R. (2012). The relationship between frequency of Facebook use, participation in Facebook activities, and student engagement. Computers & education, 58 (1), 162-171. Klout. (2015). The klout score. Retrieved from https://klout.com/corp/score Konstantinidis, A., & Grafton, C. (2013). Using excel macros to analyse moodle logs. Retrieved from http://goo.gl/yfrjss Leskovec, et al. (2006). The dynamics of viral marketing. In Proceedings of the 7th ACM Conference on Electronic Commerce, 2006. Robinson, J. (2014). Measuring Facebook engagement. We are social. Retrieved from http:// wearesocial.com/uk/blog/2014/07/measuring-facebook-engagement Smitha, N. (2016). Simply measured. Facebook metrics defined: engagement rate. Retrieved from http://simplymeasured.com/blog/facebook-metrics-defined-engagement-rate/#sm.0001sr q731egmefaxeo1fnrz0fhib University of the Arts London. (2013) Moodle: Add a Facebook like etc. Retrieved from http:// elearningsupport.myblog.arts.ac.uk/files/2013/05/Create-a-Facebook-Like-box.pdf 48 Leonardo Magno