Journal Pre-proof

Can an online educational game contribute to developing information literate citizens?

Soeun Yang, Jae Woo Lee, Hyoung-Jee Kim, Minji Kang, EunRyung Chong, Eun-
mee Kim

PII: S0360-1315(20)30255-4
DOI: https://doi.org/10.1016/j.compedu.2020.104057
Reference: CAE 104057

To appear in: Computers & Education

Received Date: 5 March 2020

Revised Date: 31 July 2020
Accepted Date: 17 October 2020

Please cite this article as: Yang S., Lee J.W., Kim H.-J., Kang M., Chong E. & Kim E.-m., Can an online
educational game contribute to developing information literate citizens?, Computers & Education, https://
doi.org/10.1016/j.compedu.2020.104057.

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition
of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of
record. This version will undergo additional copyediting, typesetting and review before it is published
in its final form, but we are providing this version to give early visibility of the article. Please note that,
during the production process, errors may be discovered which could affect the content, and all legal
disclaimers that apply to the journal pertain.

© 2020 Published by Elsevier Ltd.

S. Yang: Conceptualization; Roles/Writing - original draft; Formal analysis; Investigation
J. W. Lee: Investigation; Roles/Writing - original draft; Writing - review & editing
H. Kim: Software; Visualization
M. Kang: Software; Visualization
E. Chong: Validation
E. Kim: Supervision; Conceptualization

of
ro
-p
re
lP
na
ur
Jo
 Title Page for Computer & Education

Can an online educational game contribute to developing information literate citizens?

Soeun Yang1, Jae Woo Lee1, Hyoung-Jee Kim2, Minji Kang2, EunRyung
Chong3, Eun-mee Kim1

of
ro
1
Department of Communication, Seoul National University, Seoul 08826, Korea;
-p
soeun022@snu.ac.kr (S.Y.); jwlee12@snu.ac.kr (J.W.L.); eunmee@snu.ac.kr (E.K.)
2
Institute of Communication Research, Seoul National University, Seoul 08826,
re
Korea; hjeekim.onto@gmail.com (H.K.); zemfira@snu.ac.kr (M.K.)
lP

3
SNUFactCheck Center, Institute of Communication Research, Seoul
National University, Seoul 08826, Korea; ryung0305@gmail.com (E.C.)
na
ur
Jo

Corresponding Author:
Eun-mee Kim, (Ph.D., Northwerstern University)
Professor
Department of Communication
Seoul National University
404 IBK Communication Center
1 Gwanak-ro, Gwanak-gu, Seoul, Korea
Tel: 82-2-880-5479
Fax: 82-2-885-8418
Email: eunmee@snu.ac.kr
 1. Introduction

Advances in Information and Communication Technology have drastically increased

not only access to information but also opportunities for people to create and share

information. With these changes came the proliferation of fake news, making the issues of

mis- and disinformation more prevalent and pressing than ever before. Against this backdrop,

concerns have emerged that decision making and social engagement based on false

of
information and misleading arguments will undermine democracy, and that unreliable

ro
information will pose a threat to both individuals and the society as a whole (Karlova &

-p
Fisher, 2013; Lewandowsky, Ecker, Seifert, Schwartz, & Cook, 2012). In relation to these
re
concerns, there is an especially heightened global concern over the younger generation’s
lP

limited competencies to combat mis- and disinformation. Dubbed “digital natives,” youth

today spend much time online and base many of their decisions on online communication and
na

information. However, their limited proficiency in judging and discerning online information
ur

has been repeatedly pointed out (Wineburg, McGrew, Breakstone & Ortega, 2016).
Jo

Various media regulations like real-time fact checking and algorithm-run rumor

detection are considered as possible solutions to the problems caused by misleading

arguments and false information. However, a more fundamental measure is to promote users’

abilities to navigate and discern online information, and for this reason, ways to help one

become an information literate citizen are getting greater attention today. This is in line with

the initiative to promote Media and Information Literacy (MIL, hereafter), a set of abilities

recognized by UNESCO as essential life skills for citizens’ participation and survival in

today’s media environment (Grizzle et al., 2013). Media education scholars and practitioners,

recognizing the need to address the issue of unreliable information, strive to implement in-

school programs against mis- and disinformation. These programs have resulted in
1
improvements in learners’ abilities to critically assess online information (Bonnet, Herakova,

& McAlexander, 2018; Kahne & Bowyer, 2017; Marten & Hobbs, 2015; McGrew, Smith,

Breakstone, Ortega, & Wineburg, 2019; Roozenbeek & van der Linden, 2019a). Overall

research suggests the need to include MIL education in school curriculum.

Despite research showing the benefits of classroom-based MIL education, the

limitations of offline-based education lead us to explore the effects of an online educational

game that can help people to become information literate citizens. That is, even though the

of
importance of offline-based education is well established, public school systems often face

ro
practical challenges in timely implementation of those education programs against mis- and

-p
disinformation (Hodgin & Kahne, 2018; Lee, 2018; Tully & Vraga, 2018). There's also an
re
increasing concern of marginalizing those outside the purview of the school system, like
lP

adults who have already graduated, people with limited access to learning opportunities, or

students whose school districts do not offer adequate MIL programs. For these learners, an
na

online intervention program open for public use is a more accessible alternative.
ur

Drawing from a body of game research, this study predicts that an online educational
Jo

game will effectively help the players to learn core competencies in MIL. Lately, games for

educational purposes have become subjects of much attention as instruments that could help

people advance as information literate citizens. Online games designed to combat

misinformation, such as the Bad News Game (Roozenbeek & van der Linden, 2019b) and

NewsFeed Defenders (FactCheck.org) have emerged as new kinds of educational resources

that generate interest and pleasure. Based on the recent scholarship and efforts to educate

information literate citizens, we developed an online educational game entitled Trustme! and

investigated whether and how playing the educational game can enhance attitudinal (i.e.

skepticism toward online information) and cognitive (i.e. information discernment)

2
competencies required for information literate citizens to thrive in the new media

environment.

2. Literature review

2.1. Information literate citizen in the MIL context

MIL is a composite construct that includes media literacy, information literacy,

digital literacy and other literacies1, and is widely accepted as “the essential competencies

of
that allow citizens to engage with media and other information providers effectively and

ro
develop critical thinking and lifelong learning skills for socializing and becoming active

-p
citizens” (Grizzle et al., 2013, p 191). MIL researchers deem media and information as
re
indispensable for democratic processes, and being able to process and judge information is
lP

considered especially important (Gretter & Yadav 2016; Wilson et al., 2013). Thus, MIL puts

special emphasis on competencies that empower citizens to critically evaluate and assess
na

media content, and to make informed decisions based on their understanding of the media
ur

and information ecology (Grizzle et al., 2013). Although MIL is a multifaceted concept that
Jo

1
MIL is a concept that includes elements of information literacy from the field of Library
Studies and Information Sciences and media literacy from the media, education and social
sciences. Information literacy focuses on individual’s ability to look for, evaluate and apply
information based on one’s informational need while media literacy underlines the ability to
analyze, understand and critically evaluate media messages (Cuervo Sánchez & Sandra
Liliana, 2019; Livingstone, van Couvering, & Thumin, 2008; Vaičiūnienė & Mažeikienė,
2016). Many of the core competencies are shared by information literacy and media literacy,
and the need to consider the two together in order to effectively cultivate the important
abilities for today’s media and information environment has been discussed by numerous
scholars (Cuervo Sánchez & Sandra Liliana, 2019; Lee & So, 2014; Livingstone et al., 2008;
Vaičiūnienė & Mažeikienė, 2016; Wilson, Grizzle, Tuazon, Akyempong, & Cheung, 2013).

3
embraces elements ranging from media access and use, language and critical comprehension,

production and programming processes to transforming one’s situation through

communication (Cuervo Sánchez & Sandra Liliana, 2019), this study puts at the forefront the

ability to critically evaluate the infinite amount of online information as a core MIL

competency.

Based on this concept of MIL, we focus on skepticism toward online information as a

learning outcome of this study, which refers to neither pessimism nor denial of the

of
information (Lewandowsky et al., 2012), but a balanced attitude that characterizes an

ro
information literate citizen. Due to what is known as the continued influence effect (see

-p
Lewandowsky et al., 2012), the impact of misinformation is often irreversible once an
re
individual is exposed to misinformation because efforts to correct or debunk earlier
lP

inaccuracies are often unsuccessful after the initial exposure to misinformation ingrains that

information on an individual. Such resilience of false information highlights the importance

na

of skepticism before exposure to unreliable information. Previous studies report that people
ur

with a higher level of skepticism are less susceptible to misleading arguments and false
Jo

information (DiFonzo, Beckstead, Stupak, & Walders, 2016; Lewandowsky et al., 2005,

2009). As Hurtt (1999, p. 5) defines, skepticism is “the act of withholding judgment while

searching for more information,” and refers to a healthy level of skepticism that helps one to

identify motivations behind information and access a more diverse information (Hodgin &

Kahne, 2018, Lewandowsky et al., 2012). Developing skeptical attitude about the information

one encounters online is one of the most discussed solutions for combating misleading and

false online information.

As the second learning outcome, we consider information discernment skills as a

cognitive learning outcome associated with information literate citizens. Walton and

Helpworth (2013, p.55) define information discernment as “the ability to use higher order
4
thinking skills in order to make sound and complex judgment regarding a range of text-based

materials,” and use the concept as the evaluating information component of information

literacy in the digital media context (Walton, Pickard & Dodd, 2018). Information

discernment skills is a useful concept that captures the ability to evaluate factuality and

motivation behind the myriad of information people encounter online, and help people to

actively fight the dissemination of unreliable information.

These specific discerning skills are also core capabilities included in the civic online

of
reasoning (Middaugh, 2019), a concept that has developed in the recent years and is gaining

ro
more attention from MIL researchers. As “the ability to effectively search for, evaluate and

-p
verify social and political information online” (McGrew et al., 2018, p.166), civic online
re
reasoning is an overarching concept that goes beyond the ability to simply detect fake news:
lP

It consists of multiple sub-skill sets, which include lateral reading skills or the ability to

cross-check multiple sources, as well as the ability to evaluate accuracy, infer who is behind
na

the message and identify evidence that supports one’s evaluation. This specific sub-skillset
ur

for critical evaluation and assessment is referred to as information discernment skills by

Jo

Walton and colleagues (Walton & Helpworth, 2013; Walton et al., 2018). Although various

competencies are included in civic online reasoning as illustrated, we hone in on online

information discernment as the core MIL competency in our study.

2.2 Digital game-based learning in MIL education

Breaking away from the traditional classroom learning model, educators and

researchers are increasingly embracing digital game-based learning as a viable method of

learning (Gee & Burke, 2001; Prensky, 2001). Digital game-based learning (DGBL, hearafter)

offers a highly interactive experience that enhances learner engagement and improves

learning outcomes (van Eck, 2007; Cheong, C., Cheong, F., & Filippou, 2013; Denny, 2013;
5
Erhel & Jamet, 2013; Hamari, Koivisto, & Sarsa, 2014). For example, Papastergiou’s (2009)

findings show the DGBL approach to be effective in both promoting content knowledge

acquisition and triggering student motivation. Proponents of digital game-based learning

advocate for games as effective educational instruments with various pedagogical benefits

(Gee & Burke, 2001; Prensky, 2001).

Also noteworthy is game’s potential contribution to promoting critical skills that are

in growing need today (Gerber & Scott, 2011). Online educational games have been reported

of
to foster the 21st century skills that include active critical thinking, communication, and

ro
collaboration (Amory, Naicker, Vincent, & Adams, 1999; Gerber & Scott, 2011; Romero,

-p
Usart, & Ott, 2015). As an innovative tool to help foster the players’ cognitive abilities and
re
encourage problem-solving (Felicia, 2009), online game is expected to be effective in
lP

promoting MIL competencies as well.

A recent study by Roozenbeek and van der Linden (2019b) reports positive effects of
na

an online game entitled Fake News Game on players’ abilities to evaluate the credibility of
ur

online information. Each of the players, after being assigned a role as a producer of fake news,
Jo

played the online game that teaches them the fake news techniques. The players earned

followers, credibility points and badges for strategically creating convincing misinformation.

The study reports that the online game helped participants to accurately identify misleading

arguments, suggesting that games can teach players the techniques used to create

misinformation and inoculate the players against the risk of misinformation. Similarly, Grace

and Hone (2019) and Katsaounidou, Vrysis, Kotsakis, Dimoulas & Veglis (2019) suggest the

effectiveness of DGBL in fostering abilities to evaluate online information.

However, to our knowledge, there has been no research that identifies an educational

game’s effects on actual evaluation and discernment skills for online information. Past

studies rely on self-reporting to measure game’s educational benefits or explore players’

6
game experience without examining the learning outcomes (e.g., Grace & Hone, 2019).

Several studies also argue for examining actual media literacies as opposed to relying on self-

reported media literacies, suggesting a gap between performance and perception (Bonnet et

al., 2018; Porat, Blau, & Barak, 2018; van Deursen & van Diepen, 2013). In order to address

the research gap, we developed a prototype of an educational computer game called Trustme!,

and examined whether the educational game indeed had any positive effect on information

discernment skills, by having participants complete an assessment to evaluate and judge

of
online information.

ro
In addition, this study examines the game effects on skepticism, an attitude essential

-p
for accurately discerning and correcting misinformation in the current media environment as
re
mentioned above. In Roozenbeek and van der Linden’s (2019b) study of the inoculation
lP

effect of The Bad News Game against misinformation, the results suggest that a game can

help cultivate skeptical attitude that raises a red flag for unreliable information rather than
na

leading the participants to unsystematically deny all information. Since this skepticism
ur

toward online information refers to an attitude toward a specific piece of information rather
Jo

than a dispositional skepticism (see Difonzo et al., 2016), frequent exposure to

misinformation in the game can be expected to cultivate skepticism. Therefore, we predicted

that playing Trustme! would heighten skepticism toward online information.

Despite such a growing interest in game as a tool to foster abilities to evaluate and

assess unreliable information, few studies systematically examine the effects of educational

computer games on learning outcomes in comparison to non-game treatments in the MIL

context. Gamification research, although distinct from game research, systematically

examines each game element and informs game design by identifying the link between

7
specific game attributes and learning outcomes (Landers, 2014).2 We took advantage of the

game attribute taxonomy from the gamification literature in designing our game (Trustme!),

so that the game addresses desired learning outcomes.

2.3 Intellectual civic skills as a moderator for learning outcomes

Since the game we developed is short in duration and is designed to be played only

once, we expected some individual differences in the learning outcomes of the game. We

of
considered the possibility of online game having limited effects on deeper knowledge-based

ro
skills and on attitude changes in some participants.

-p
A major individual characteristic that moderates the effects of online educational
re
game is preexistent competencies related to the learning outcomes. For example, based on the
lP

learning cycle model that posits that learners begin learning from familiar or concrete

experiences, Kebritchi, Hirumi and Bai (2010) explain how different individual abilities
na

2
Although there is much overlap between educational game research and gamification
ur

research, there are some important differences between the two. The effect of games on
Jo

learning is a subject of common interest of the two areas of research, but game research
literature focuses on the relationship between the game as a whole and desired learning
outcomes. Gamification literature, on the other hand, approaches the game attributes more
systematically to identify meaningful elements and the most effective combinations of them
for achieving specific learning outcomes (Landers, 2014). In short, games include almost all
game attribute categories at different degrees in game research, and the game’s overall effects
are of the primary interest, while “identification, extraction and application of individual
game elements (p.754)” in a non-game context is the focus of gamification research. This
study is a part of the game research in that it examines the overall effect of the game designed
based on the findings of gamification research. Past literature on specific game attributes and
their effects heavily informed our game design process as we selected and incorporated game
elements most commonly found to be effective for application, cognitive strategies and
attitudinal valuing as learning outcomes (Wilson et al., 2009).
8
result in different learning experiences, which in turn affects individual achievements.

Likewise, previous studies on educational games explore the role of prior competencies

related to learning outcomes: perceived level of preexisting competency in economics and

business administration in the study of an educational business game (Sørebø & Hæhre ,

2012) and second language proficiency in the study of digital game-based second language

vocabulary learning have been found to moderate the learning outcomes (Tsai & Tsai, 2018).

Based on these findings, we examined players’ perceived levels of intellectual civic skills as

of
a moderating variable, since our game aimed at promoting MIL competencies to help citizen

ro
effectively engage and participate in current media environment.

-p
Intellectual civic skills, also called cognitive civic skills, are defined as the citizen’s
re
abilities to analyze civic-related issues and make informed and reasoned judgements about
lP

civic and political situations (Patrick, 2002, Tolo, 1999; Educational Testing Service, 2017;

Branson & Quigley, 1998; Center for Information and Research on Civic Learning and
na

Engagement, 2010). Branson and Quigley (1998) refers that critical thinking abilities are
ur

significant intellectual civic skills to becoming an informed citizen as shown above. Thus,
Jo

individuals with these intellectual civic skills are, to some extent, already equipped with the

very competency that the MIL education aims to develop. The development of civic skills

begins in adolescence and continues into adulthood (Brady, Verba, & Schlozman, 1995). The

development of civic skills begins in adolescence and continues into adulthood (Brady et al.,

1995). We thus expected to find varying levels of intellectual civic skills in young adults, and

the game’s educational effects to differ depending on the participants’ perceived level of

intellectual civic skill levels.

Although there is not much research that considers individual differences as a factor

in learning outcomes of online games, several studies find that individuals competent in skills

related to the targeted learning objectives may benefit more from DGBL (Landers, Auer,
9
Collmus, & Armstrong, 2018; Sanchez, Langer, & Kaur, 2020; Sørebø & Hæhre, 2012). For

example, Sanchez et al. (2020) reports a greater effect of gamified quiz on student test scores

in higher achieving students and concludes that students with higher course grades tend to

have greater self-efficacy and show higher learning outcomes. For students with lower course

grades and less cognitive resource, however, gamified elements interfered with learning.

Such an argument has also been made by cognitive theory of multimedia learning. According

to this theory (Moreno & Mayer, 1999; Moreno, 2002), a wide range of stimuli and

of
interactivity in a multimedia program requires cognitive resources in a learner to effectively

ro
coordinate them, and multimedia program may have a detrimental effect on learning when

-p
students have limited cognitive capacity. It can be that learners equipped with prior
re
knowledge and relevant past experience may be less affected by the cognitive load in
lP

complex learning environment. Therefore, we predicted that the game may have a greater

effect on individuals with high levels of intellectual civic skills can result in greater
na

educational effects when treated with short-duration materials.

ur
Jo

2.4 Purpose of the study and research questions

In this study, an online educational game was developed to enhance students’ skepticism

toward online information and information discernment skills. Thus, we examined the effect

of the developed game on those learning outcomes and compared the effects of the quiz (non-

game intervention) to probe if DGBL is of any particular importance in this context.

Furthermore, we considered intellectual civic skills as the individual difference which

moderates the effects of the game. The experiment investigated the following research

questions:

1. Do participants who play the online game (Trustme!) score higher in the information

discernment skills measure as compared with those who complete a quiz or those in
10
 the control group?

2. Do participants who play the online game developed for the study (Trustme!) show

higher skepticism toward online information as compared with those who complete a

quiz or those in control group?

3. Are the learning outcomes of two interventions (quiz and game) moderated by

participants’ perceived levels of intellectual civic skills?

of
ro
3. Developing an online game to combat mis- and disinformation: Trustme!

-p
re
Trustme! is an online educational game devised specifically to examine the effects of
lP

DGBL in this study. It helps the players to become better information literate individuals by

teaching them to critically evaluate online information and accurately judge intention and
na

falsity behind online messages. The developed game sets the context by explaining to the
ur

player that the protagonist of the game is a famous influencer with the username, “Trustme,”
Jo

known for sharing reliable information on social media. The goal of the player is to take on

the role of the protagonist and successfully identify the reliability of each piece of

information presented on the screen.

Trustme! is unique in that its content and design were constructed with the

educational purpose in mind: feedback research and gamification research informed the game

design process.

3.1 The content of the game: the online information and feedback

Trustme! has two parts: examples of false and misleading online information for the

quiz, and the feedback to each of the player’s response (Fig. 1).
11
 First, the articles characterized by falsity and hidden intentions were composed to be

used in the game as stimuli. Falsity refers to inaccuracy regardless of the intention behind the

message, a concept associated with misinformation (i.e., inaccurate online information,

European Commission, 2018). Intention is related to motivations to deceive, and is thus

associated with disinformation (i.e., false information that is intentionally manipulated to

deceive, Ibid.): as two distinct concepts, falsity and intention can overlap at times. Articles

used in our game include erroneous information without intention to deceive (i.e. false report)

of
as well as information crafted and disseminated for political or commercial purposes (i.e.

ro
propaganda, native ads). A total of six items were created using the above criteria to construct

the quiz. -p
re
When developing the elaborated feedback messages, we turned to past feedback
lP

research to ensure that the feedback provided in the game has significant educational effect.

Given the importance of elaborated feedback that provides an explanation for higher order
na

learning outcomes (van der Kleij, Feskens, & Eggen, 2015), we developed immediate
ur

elaborated feedback that provides both task-level and process-level information for our game
Jo

(Narciss, 2005). Task-level feedback explains what the correct answer for the specific task is

and why it is so, while process-level feedback explains what strategies or procedures need to

be used to successfully complete the task. In Trustme!, after the player answers each quiz

item, the task-level feedback that explains why the article is unreliable is provided first, and

then the process-level feedback is provided in the form of a list of strategies to be used. This

way, the player can enhance skills to detect falsity and intention in online information. For

the list of strategies, we adapted the fake news checklists from the Poynter Institute (Krueger,

2016), the Seoul National University (SNU) FactCheck Center (SNU, Institute of

Communication Research, 2019), and the Stanford History Education Group’s standards for

civic online reasoning to fit the context.

12
 Fig. 1. Major contents presented in Trustme!

of
3.2 Game design elements

ro
In practice, games for DGBL are often built based on the common game components
-p
found in commercial games, of which scenario (narrative), feedback, reward, progress and
re
challenge are most frequently used (Hamari et al., 2014). These components can be matched
lP

to the game attributes defined and systematically categorized by Bedwell and colleagues

(Bedwell, Pavlas, Heyne, Lazzara & Salas, 2012), and rules and goals, assessment, progress,
na

representation, challenge, and safety are the major game attributes incorporated into our game.
ur

First, a scenario given at the beginning of the game (Fig. 2a) explains that the player is a
Jo

famous social media influencer who needs to identify as many pieces of reliable information

as possible while checking for the rationale behind her choice. The scenario explicitly states

the rules and goals of the game, but also adds to the reality of the experience by setting the

scene in a social media context for the player.

Feedback on the correctness of the player’s answers as well as the virtual achievements

and rewards given to the player (Fig. 2b) also provide the players with the means to assess

their performance and progress. The rewards (Fig, 2c) in the form of increase in the number

of virtual followers can be seen to add to the safety of the game experience, since there is no

risk of losing actual social media followers for choosing a wrong answer.

13
 Progress, which is made visible on the tab where the player can see the total number of

questions in the game and the number of questions answered, occurs only after the player

answers the given question. The questions on the reliability of given information corresponds

to the challenge in Bedwell’s taxonomy, and presents the player with a problem that needs to

be resolved. While the level of difficulty does not adapt to the player’s responses, the

challenge was carefully manipulated in the game design process so that it does not frustrate

the player with excessively difficult problems.

of
In terms of representation which is defined as a feature that “makes the game appear

ro
psychologically real” (Bedwell et al., 2012), the game has a fairly high level of representation
-p
in that it mimics reality with its social-media-like feedback (i.e. increase and decrease in the
re
number of followers) and appearance.
lP
na
ur
Jo

Fig. 2. Major game components incorporated in Trustme!

In order to find out if the game has any meaningful educational effect or if players’

exposure to online intervention is enough, we created a quiz with identical content and layout,

but without the game elements included in Trustme!: No scenario, points or game rules are

provided, and the quiz-only version’s interface relies strictly on text, unlike Trustme! (Fig. 3).

While the information provided and the content of the feedback messages are identical,

14
Trustme! offers a more interactive feedback and incorporates the game elements that are

commonly known to motivate user engagement. We thus compared the game effect in MIL

education against a regular quiz with elaborated feedback.

of
ro
-p
Fig. 3. A screenshot from the quiz without game elements
re
lP

4. Methods
na

4.1. Participants and procedure

We designed an online experiment to examine the effects of the game. Participants

ur

were recruited by Marketlink, a third-party survey company in Korea. A total of 210

Jo

participants between the ages 20 and 29 completed the experiment, with the average age of

26 (105 males and 105 females). All participants gave informed consent before completing

the assigned tasks and were compensated with $10, payed upon completion. The entire data

collection took about 25 minutes for each participant on a personal computer.

As Fig. 4 shows, participants were randomly assigned to one of three conditions: the

game (Trustme!), the quiz, and no intervention. The participants first completed a pre-survey

collecting their demographic information and measuring their intellectual civic skills. The

experiment, then post-testing, followed. During the experiment session, the game group

completed the Trustme! game, and the quiz group completed a quiz where identical tasks was

15
given but without game elements. The control group completed the post-test that assessed

their information discernment skills and skepticism toward online information without

completing a quiz or a game.

of
ro
-p
re
lP
na
ur

Fig. 4. Research procedure

Jo

4.2 Measuring tools

4.2.1 Pre-game instruments

Intellectual civic skills were measured as a moderating variable using self-reported

questionnaires based on the index of intellectual civic skills (Miller, 2004; Patrick, 2002).

Four items were measured (α =.80, M=3.51, SD=.68, e.g., “I critically engage in the political

and social issues of our society,” “I always pay close attention to political and civic issues or

events,” and “I have a good understanding of the political and social issues in our society

today.” All items were measured using a five-point Likert scale (ranging from 1 = completely

disagree to 5= completely agree).

16
 4.2.2 Post-game instruments: Skepticism and information discernment skills

In order to measure the learning outcomes of our study, the questionnaire on

skepticism toward online information was modified based on a scale of skepticism toward

electronic word-of-mouth (Zhang, Ko & Carpenter, 2016) and the dispositional skepticism,

developed by DiFonzo et al. (2016) based on professional skepticism scale (Hurtt, 1999). The

measurement consists of 10 items (α = .76, M =3.74, SD=0.46, e.g., “I usually accept things

of
that I see, read, or hear online at face value (R),” “online information is not generally truthful,”

ro
and “most online arguments are not intended to manipulate people (R)”) that are rated on a

-p
five-point Likert scale (ranging from 1 = completely disagree to 5= completely agree).
re
The second learning outcome, information discernment skills were measured by a
lP

performance-based assessment. Stanford History Education Group’s civic online reasoning

assessments (McGrew, Breakstone, Ortega, Smith, & Wineburg, 2018) were consulted for the
na

development of the assessment for information discernment skills (see Appendix). While the
ur

SHEG assessment puts a special emphasis on lateral comparison of different pieces of online
Jo

information, due to the restrictions of web-based experiment, our study limited the scope of

assessment to discerning only online information that is given to the players. The assessment

used in our study was modified from the four assessment items asking the participants to

evaluate who is behind the information and what the evidence is: News on Facebook,

Evaluating Evidence, Article Analysis, and Comparing Articles (sheg.stanford.edu). We

replaced the English articles used in the original assessments with the news articles and

information composed in Korean, and crafted multiple-choice questions asking the

participants to evaluate each piece of information and to choose the most appropriate reason

for their response from of the four possible answer choices.

17
 We assigned two points for each item when the respondent accurately discerned the

reliability of the given information and selected the correct reasoning behind their evaluation.

Respondents earned one point for each item that they accurately discerned the reliability of,

but failed to identify the correct explanation behind their choice. A score of zero was

assigned when the respondent failed to accurately evaluate the reliability of the information.

The possible score for each assessment item ranged from 0 to 2, and the sum of the points

earned was used for analysis, with a potential score ranging from 0 to 8 (M: 3.20, SD:1.61).

of
ro
4.3 Analysis

-p
Since this study aims to investigate the moderating effect of intellectual civic skills and
re
the main effect of educational interventions on learning outcomes, one-way between-group
lP

analyses of covariance (ANCOVA) were used as the statistical test. Specifically in our study,

one-way ANCOVA model allows us to compare dependent variables (i.e., skepticism and
na

information discernment skills) in three groups (i.e., control, quiz vs. game) while taking into
ur

account influences of a continuous variable a covariate (i.e., intellectual civic skills).

Jo

Moreover, we tested the interactions between intellectual civic skills and the experimental

conditions. Levene’s test for equality of variance was performed, and the homogeneous

assumption was supported.

5. Results

Table 1 shows descriptive differences among the experimental conditions in

skepticism toward online information and information discernment skills. The statistical

significance of the observed differences is reported in the following ANCOVA results.

18
Table 1
Mean and SD of Skepticism toward online information and Information discernment skills

Skepticism
Information discernment skills
toward online information
Experiment group Mean(SD) N Mean(SD) N

Control 3.69(0.41) 70 2.81(1.60) 70

Quiz 3.74(0.45) 70 3.14(1.49) 70

Game (Trustme!) 3.77(0.53) 70 3.63(1.66) 70

of
ro
Table 2 presents the results of an ANCOVA with the experimental conditions as independent

-p
variables, skepticism toward online information as the dependent variable, and intellectual

civic skills as a covariate. Since the interaction effect between the conditions and
re
lP

intellectual civic skills was not significant, F(2, 204) = 0.68, p = 0.510, we

reported the model without interaction terms. The main effects showed that the
na

experimental conditions do not yield a significant effect, F(2, 204) = 0.49, p = 0.611.
ur

Meanwhile, the results demonstrated that participants’ intellectual civic skills strongly predict
Jo

their levels of skepticism toward online information, F(1, 204) = 9.12, p = 0.003. That is,

people with high levels of intellectual civic skills were more likely to be skeptical about

online information.

Table 2
ANCOVA for skepticism toward online information

Source Df F η2 p
Condition 2 0.33 0.00 0.72

Intellectual civic skills 1 9.47 0.04 0.00

Error 206
R2 0.05*

19
Notes: *p<.05.

With regard to information discernment skills (Model 1 in Table 3), an ANCOVA

showed significant differences among the experimental conditions, F(2,

204) = 4.62, p = 0.011. Pairwise comparisons revealed that participants in the game group

(Madjusted=3.63) showed statistically significant achievement compared to those in the control

group ( Madjusted= 2.83, p = 0.003). On the other hand, quiz group (Madjusted=3.13) did not

show difference compared to the control group (p=.267), and showed marginally significant

of
difference to the game group (p=.064).

ro
Table 2
-p
ANCOVA for information discernment skills
re
source Df F η2 P
lP

Model 1
Condition 2 4.54 0.04 0.012*
na

Intellectual civic skills 1 1.15 0.00 0.285

Error 206
R2
ur

0.05*
Model 2
Jo

Condition 2 4.62 0.04 0.011*

Intellectual civic skills 1 1.17 0.00 0.280
Condition × Intellectual civic skills 2 2.72 0.03 0.068†
Error 204
2
R 0.07**
+
Notes: <. 01; *p<.05; **p<.01. Model 1 represents the main effect model without the
interaction term; Model 2 represents the model that includes the main and interaction effects.

More importantly, as model 2 shows, we found a marginally significant interaction

effect between the conditions and intellectual civic skills on information discernment

skills, F(2, 204) = 2.72, p = 0.068. Fig.1 shows this interaction effect. Pairwise comparisons

revealed that the levels of information discernment skills significantly differed between the

20
control group and the game group, regardless of whether the intellectual civic skills were low

(Mcontrol = 2.88, Mgame = 3.64, p = .031), moderate (Mcontrol = 2.80, Mgame = 3.63, p = 0.002) or

high (Mcontrol = 2.73, Mgame = 3.61, p = .021). That is, the game’s educational effect was

present regardless of the level of intellectual civic skills.

However, the effect of the quiz on information discernment skills was found to

depend on the participants’ perceived levels of intellectual civic skills. Mean comparison of

information discernment skills between the control group and the quiz group at three levels of

of
intellectual civic skills revealed that the effect of quiz intervention was apparent when

ro
intellectual civic skills was high (Mcontrol = 2.73, Mquiz =3.61, p = 0.024). There was no

-p
significant difference in achievement between the control group and the quiz group when
re
intellectual civic skills were low (Mcontrol =. 2.88, Mquiz =2.52, p = 0.328) and moderate
lP

(Mcontrol = 2.80, Mquiz =3.07, p = 0.376). At these two points, however, a mean comparison
na

yielded statistically significant differences between the game group and the quiz group (low:

Mcontrol = 2.52, Mgame = 3.64, p=.006, moderate: Mquiz = 3.07, Mgame = 3.63, p = 0.037). That is,
ur

the quiz and the game were equally effective for the participants with high levels of
Jo

intellectual civic skills, while the game, but not the quiz, was effective for the participants

with average or lower levels of intellectual civic skills. When we estimated the effect of

intellectual civic skills on information discernment skills under the quiz condition, we found

that people with higher levels of intellectual civic skills were more likely to score higher in

the information discernment skills measure in comparison to those with lower levels of those

skills (βintellectual civic skills = 0.80, p = 0.022).

21
 of
ro
Fig. 5. Relationships among the experimental conditions, participants’ intellectual civic skills,
-p
and information discernment skills
re
lP

6. Discussion
na

In this study, we developed an online educational game, examined its effects on the
ur

attitudinal and cognitive aspects, and investigated if DGBL has any significant effect on those

learning outcomes by comparing the effects of the game to those of the quiz. A few findings
Jo

can be highlighted.

First, as a prototype of an educational game to combat mis- and disinformation, the

game we developed improves information discernment skills that are one of the most

important competencies today. The game teaches the players to consider the source, evidence,

and the intentions of the information they encounter so that they can evaluate the information

and detect any falsity. When the participants of our study completed an assessment where

they had to discern online information, those who played the game performed significantly

better than those who did not. Even though the difference between the game group and the

control group is not sizeable, nor is the R-squared value from the ANCOVA results, the

22
current study reports findings that are in line with a few recent studies (Grace & Hone, 2019;

Roozenbeek & van der Linden, 2019b), and suggest that even a very brief and short computer

game can improve the ability to evaluate the online information. This result is meaningful

because (1) it examined actual performance on tasks rather than self-reports as a measure for

information discernment abilities, (2) and the tasks asked the participants not to simply

determine whether given information is fake news or not, but to identify the reason they are

unreliable.

of
Moreover, in order to better understand the effects of the game and compare the

ro
treatment effects, we prepared a quiz that contains identical content as the game but without

-p
the game elements. As a result, we found that the participants that completed the game
re
(Trustme!) scored higher on the information discernment skills measure than those in the
lP

control group. In the meantime, no significant difference was observed in information

discernment skills level between the quiz group and the control group. These findings point
na

to the importance of carefully designed games in not only traditional subject areas as past
ur

literature has found, but also in MIL education. According to the data from past research, in
Jo

general, individuals learning with interactive simulations or games saw significantly higher

cognitive gains in comparison to those treated with traditional teaching methods, which

aligns with our findings (Vogel et al., 2006). The current study invites future research to draw

from gamification research to identify the effects of different game elements on learning

outcomes for a more sophisticated understanding of the game effects.

From our examination of intellectual civic skills as a moderator for the educational

effects of the game, , it is notable that the quiz intervention show statistically significant

effect only when the participants have a high level of intellectual civic skills, even though the

interaction effect between intellectual civic skills and experimental condition is marginally

significant. In the meantime, we find that the game effects are statistically significant
23
regardless of the intellectual civic skills level. That is, for learners with limited competencies

or skills related to the targeted learning objectives, the quiz without any game element may

not be an effective intervention. The cognitive theory of multimedia learning warns that an

excessive cognitive load by a multimedia game or program can hurt individuals with low

cognitive capacity, highlighting the possibility of game elements offsetting the learning

outcomes of an educational treatment when the learners are not adequately equipped with the

cognitive and reasoning abilities (Moreno & Mayer, 1999; Moreno, 2002). However, our

of
research shows a significantly higher level of information discernment skills in the

ro
participants whose intellectual civic skills levels are higher in the quiz group, but not in the

-p
game group. It can be inferred from this observation that without additional motivational
re
affordances of game elements that enhance learning and engage learners computer-based
lP

education like our quiz will be beneficial to only those who are already better equipped in

intellectual civic skills. Researchers have pointed to the dearth of studies that consider
na

individual differences in gamification research (Landers et al., 2018; Sanchez et al., 2020).
ur

Our study has important implications in that it shows the importance of game elements in
Jo

better engaging the people who are more vulnerable to unreliable information.

On the other hand, our study did not find a statistically significant effect of the game

on skepticism toward online information. Skepticism toward online information is an

attitudinal variable that is effective in not only detecting mis- and disinformation, but also in

debunking them (DiFonzo et al., 2016; Lewandowsky et al., 2005, 2012). These findings can

be due to the limitations of an online intervention. While our study found that an online game

can teach the players how to evaluate and discern online information in the short-term, we did

not find any attitude changes that can function as a long-term motivation leading to routine

evaluation of information. Our findings are similar to the results of Zhao and Shute (2019)’s

study which examined the cognitive and attitudinal impacts of playing a video game targeting
24
computational thinking skills. Their findings show that the video game only influenced

computational thinking skills and did not foster changes in students’ attitude toward computer

science. One explanation offered by the authors is that the short duration of the game

intervention may be insufficient to bring about any attitudinal change. On a different note, the

game players with a higher level of intellectual civic skills were found to have a higher level

of skepticism, suggesting that skepticism also needs to be cultivated as a dimension in civic

education.

of
There are several practical implications of findings. First, when in-class, school-based

ro
MIL curriculum is inadequate to tackle the rampant problem of mis- and disinformation

-p
(Hodgin & Kahne, 2018; Lee, 2018), we can expect online educational resources like
re
Trustme! to help resolve such issues. The current study supports the importance of
lP

developing more interventions like online educational games and making them available to a

larger audience. Additionally, our research, through a comparison of different educational

na

treatments, shows that it is important to provide online game intervention for individuals with
ur

lower levels of civic skills. However, since Trustme! was developed as a prototype for the
Jo

experiment and only includes short content and the most basic game elements, future efforts

should be directed at developing a game that incorporates game elements with greater

motivational affordances and content that is richer and more diverse. This demands a closer

collaboration with game experts. Finally, that we found no desired effect related to

skepticism toward online information suggests that attitude changes require a more carefully

designed instruction, and that long-term, in-person education may be necessary.

7. Conclusion and limitations

25
 This study aims to investigate the effectiveness of DGBL in helping young adults to

grow into information literate citizens in a media environment riddled with unreliable

information. While games that combat mis- and dis- information are becoming more popular,

there is still very little research that systematically studies their effectiveness. To our

knowledge, this study is the first of its kind in that it demonstrates the effectiveness of an

online game in comparison to an identical intervention with no game elements and

specifically tests the effects of DGBL. Our results show that a short online educational game

of
can contribute to cultivating information literate citizens, especially by improving

ro
information discernment which constitutes a cognitive aspect of MIL. Moreover, unlike the

-p
quiz that was found to have statistically significant effect only in the participants with a high
re
level of intellectual civic skills, the game led to no individual difference in its learning
lP

outcome. At the same time, that

there was no desired effect related to skepticism toward online information suggests that a
na

short game is not enough to bring about changes in attitude. In sum, our study demonstrates
ur

the contributions and limitations of online games in helping individuals to grow into
Jo

information literate persons, and in doing so, extends the past research on DGBL to the field

of media education and invites future research to identify effective online intervention for

MIL competencies related to misinformation.

This study is not without limitations. First, the assessment for civic online reasoning

(McGrew et al., 2018, 2019; Wineburg et al., 2016), which was originally designed for in-

class use, was carefully modified to be administered in our experiment. However, because of

the restrictions of an online experiment, the items that ask the respondent to determine the

reliability of a given information and the rationale for the answer were modified into

multiple-choice questions. For future study, open-ended questions may allow for a more

thorough assessment of the participants’ information literacy.

26
 Second, while we find the game used in our study to have a significant educational

effect, it is yet unclear exactly how each of the game elements used in the experiment—rules

and goals, assessment, progress, challenge, representation and safety ,—affect learning. In

gamification research, confounding too many game elements in the experimental condition

interferes with precisely determining which game attribute or combination of attributes is

actually effective (Landers, 2014). Specific game elements were not tested in our study

because our goal concerned the overall effects of a game intervention, but future research on

of
game-element effects will also be valuable for the purpose of game design as well as for

ro
development of gamification research.

-p
Third, since our results explained small proportion of the dependent variable’s
re
variance, the R-squared value in the ANCOVA analysis in information discernment skills,
lP

future studies should consider more effective ways to enhance these skills as well as

predictors such as time spent reading the news.

na

Our final limitation regards the experiment design. First, our experiment design
ur

allowed for observation of only short-term effects. In order to better examine the educational
Jo

effects of the game, a follow-up study based on our beginning study is needed to find out if

the educational effects are well maintained in the long run. The effect size of the game when

it is played once versus multiple times may lead to interesting findings. Furthermore, because

our study used a short intervention, we took into consideration the possibility of testing threat

occurs in the pre-post designs: since pre-test can affect the scores on the post-test, we

examined the educational effect of the game based on a randomly assigned, posttest-only

design. It would be important to conduct future research that systematically uses pre-post test

design to examine within-subject achievement gains after a long-term or repeated

participation in the game.

27
References

Amory, A., Naicker, K., Vincent, J., & Adams, C. (1999). The use of computer games as an

educational tool: Identification of appropriate game types and game elements. British

Journal of Educational Technology, 30(4), 311-321. https://doi.org/10.1111/1467-

8535.00121

Bedwell, W. L., Pavlas, D., Heyne, K., Lazzara, E. H., & Salas, E. (2012). Toward a

of
taxonomy linking game attributes to learning: An empirical study. Simulation &

ro
Gaming, 43(6), 729-760.

-p
Bonnet, J., Herakova, L., & McAlexander, B. (2018). Play on? Comparing active learning
re
techniques for information literacy instruction in the public speaking course. Journal of
lP

Academic Librarianship, 44(4), 500-510. https://doi.org/10.1016/j.acalib.2018.04.014

Brady, H. E., Verba, S., & Schlozman, K. L. (1995). Beyond SES: A resource model of
na

political participation. American Political Science Review, 89(2), 271-294.

ur

https://doi.org/10.2307/2082425
Jo

Branson, M. S., & Quigley, C. N. (1998). The Role of Civic Education. Center for Civic

Education. Retrieved from https://www.civiced.org/papers/articles_role.html

Center for Information and Research on Civic Learning and Engagement. (2010). Civic Skills

and Federal Policy (Fact sheet). Medford, MA: CIRCLE.

Cheong, C., Cheong, F., & Filippou, J. (2013, June). Quick quiz: A gamified approach for

enhancing learning. In Proceedings of Pacific Asia Conference on Information Systems

(p. 206), Jeju, Korea.

Cuervo Sánchez, S. L., Foronda Rojo, A., Rodríguez Martínez, A., & Medrano Samaniego, C.

(2019). Media and information literacy: a measurement instrument for adolescents.

Educational Review, 1-16.https://doi.org/10.1080/00131911.2019.1646708

28
Denny, P. (2013, April). The effect of virtual achievements on student engagement. In

Proceedings of the SIGCHI conference on human factors in computing systems (pp.

763-772). Paris, France.

DiFonzo, N., Beckstead, J. W., Stupak, N., & Walders, K. (2016). Validity judgments of

rumors heard multiple times: The shape of the truth effect. Social Influence, 11(1), 22-

39. https://doi.org/10.1080/15534510.2015.1137224

Educational Testing Service (2017). HEIghten® Civic Competency and Engagement: Test-

of
at-a-glance. Retrieved from

ro
https://www.ets.org/s/heighten/pdf/heighten_cce_test_at_a_glance.pdf

-p
Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and
re
feedback on motivation and learning effectiveness. Computers & Education, 67, 156-
lP

167. https://doi.org/10.1016/j.compedu.2013.02.019

European Commission (2018). A Multi-dimensional Approach to Disinformation: Report of

na

the Independent High Level Group on Fake News and Online Disinformation.
ur

https://doi.org/10.2759/739290
Jo

Felicia, P. (2009). Digital Games in Schools: A Handbook for Teachers. Belgium: European

Schoolnet.

Gee, C., & Burke, M. E. (2001). Realising potential: The new motivation game. Management

Decision, 39(2), 131-137. https://doi.org/10.1108/00251740110384671

Gerber, S., & Scott, L. (2011). Gamers and gaming context: Relationships to critical thinking.

British Journal of Educational Technology, 42(5), 842-849.

https://doi.org/10.1111/j.1467-8535.2010.01106.x

Grace, L., & Hone, B. (2019, April). Factitious: Large Scale Computer Game to Fight Fake

News and Improve News Literacy. In Extended Abstracts of the 2019 CHI Conference

on Human Factors in Computing Systems (p. CS05). ACM.

29
Gretter, S., & Yadav, A. (2016). Computational thinking and media & information literacy:

An integrated approach to teaching twenty-first century skills. TechTrends, 60(5), 510-

516.https://doi.org/10.1007/s11528-016-0098-4

Grizzle, A., Moore, P., Dezuanni, M., Asthana, S., Wilson, C., Banda, F., & Onumah, C.

(2014). Media and information literacy: policy and strategy guidelines. Unesco.

Retrieved from: http://unesdoc.unesco.org/images/0022/002256/225606e.pdf.

Guo, Y. R., & Goh, D. H. L. (2016). Evaluation of affective embodied agents in an

of
information literacy game. Computers & Education, 103, 59-75.

ro
https://doi.org/10.1016/j.compedu.2016.09.013

-p
Hakulinen, L., Auvinen, T., & Korhonen, A. (2013, March 21-24). Empirical study on the
re
effect of achievement badges in TRAKLA2 online learning environment. In Learning
lP

and Teaching in Computing and Engineering (pp. 47-54). IEEE, Macau.

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - A literature review of
na

empirical studies on gamification. In Proceedings of the 47th Annual Hawaii

ur

International Conference on System Sciences (pp. 3025-3034), Hawaii, USA.

Jo

Hodgin, E., & Kahne, J. (2018). Misinformation in the information age: What teachers can do

to support students. Social Education, 82(4), 208-212.

Hurtt, R. K. (1999). Skeptical about Skepticism: Instrument Development and Experimental

Validation. (Doctoral dissertation, The University of Utah) Available from ProQuest

Dissertations and Theses database. (UMI No. 9935950)

Kahne, J., & Bowyer, B. (2017). Educating for democracy in a partisan age: Confronting the

challenges of motivated reasoning and misinformation. American Educational

Research Journal, 54(1), 3-34. https://doi.org/10.3102/0002831216679817

30
Karlova, N. A., & Fisher, K. E. (2013). “Plz RT”: A social diffusion model of misinformation

and disinformation for understanding human information behaviour. Information

Research, 18(1), 1-17.

Katsaounidou, A., Vrysis, L., Kotsakis, R., Dimoulas, C., & Veglis, A. (2019). MAthE the

game: A serious game for education and training in news verification. Education

Sciences, 9(2), 155. https://doi.org/10.3390/educsci9020155

Kebritchi, M., Hirumi, A., & Bai, H. (2010). The effects of modern mathematics computer

of
games on mathematics achievement and class motivation. Computers & Education,

ro
55(2), 427-443. https://doi.org/10.1016/j.compedu.2010.02.007

-p
Krueger, V. (2016, November 2) 9 questions to help you evaluate the credibility of news
re
sources. Poynter, Retrieved from https://www.poynter.org/educators-students/2016/9-
lP

questions-to-help-you-evaluate-the-credibility-of-news-sources/

Landers, R. N. (2014). Developing a theory of gamified learning: Linking serious games and
na

gamification of learning. Simulation & gaming, 45(6), 752-768.

ur

https://doi.org/10.1177/1046878114563660
Jo

Landers, R. N., Auer, E. M., Collmus, A. B., & Armstrong, M. B. (2018). Gamification

science, its history and future: Definitions and a research agenda. Simulation &

Gaming, 49(3), 315-337. https://doi.org/10.1177/1046878118774385

Lee, N. M. (2018). Fake news, phishing, and fraud: a call for research on digital media

literacy education beyond the classroom. Communication Education, 67(4), 460-466.

https://doi.org/10.1080/03634523.2018.1503313

Lee, A., & So, C. (2014). Media literacy and information literacy: Similarities and

differences. Comunicar. Media Education Research Journal, 22(1).137-146.

http://dx.doi.org/10.3916/C42-2014-13

31
Lewandowsky, S., Ecker, U. K., Seifert, C. M., Schwarz, N., & Cook, J. (2012).

Misinformation and its correction: Continued influence and successful debiasing.

Psychological Science in the Public Interest, 13(3), 106-131.

https://doi.org/10.1177/1529100612451018

Lewandowsky, S., Stritzke, W. G. K., Oberauer, K., & Morales, M. (2005). Memory for fact,

fiction, and misinformation: The Iraq War 2003. Psychological Science, 16(3), 190-

195. https://doi.org/10.1111/j.0956-7976.2005.00802.x

of
Lewandowsky, S., Stritzke, W. G. K., Oberauer, K., & Morales, M. (2009). Misinformation

ro
and the "war on terror": When memory turns fiction into fact. In W. G. K. Stritzke, S.

-p
Lewandowsky, D. Denemark, J. Clare, & F. Morgan (Eds.), Terrorism and Torture: An
re
Interdisciplinary Perspective (pp. 179-203). Cambridge University Press.
lP

Livingstone, S., Van Couvering, E., & Thumin, N. (2008). Converging traditions of research

on media and information literacies. In J. Coiro, M. Knobel, C. Lankshear, & D. Leu

na

(Eds.), Handbook of research on new literacies (pp. 103–132). New York: Lawrence
ur

Erlbaum Associates.
Jo

Martens, H., & Hobbs, R. (2015). How media literacy supports civic engagement in a digital

age. Atlantic Journal of Communication, 23(2), 120-137.

https://doi.org/10.1080/15456870.2014.961636

McGrew, S., Breakstone, J., Ortega, T., Smith, M., & Wineburg, S. (2018). Can students

evaluate online sources? Learning from assessments of civic online reasoning. Theory

& Research in Social Education, 46(2), 165-193.

https://doi.org/10.1080/00933104.2017.1416320

McGrew, S., Smith, M., Breakstone, J., Ortega, T., & Wineburg, S. (2019). Improving

university students' web savvy: An intervention study. The British Journal of

Educational Psychology, 89(3), 485-500. https://doi.org/10.1111/bjep.12279

32
Mekler, E. D., Brühlmann, F., Opwis, K., & Tuch, A. N. (2013, October). Do points, levels

and leaderboards harm intrinsic motivation? An empirical analysis of common

gamification elements. In Proceedings of the First International Conference on gameful

design, research, and applications (pp. 66-73).

Middaugh, E. (2019). More than just facts: promoting civic media literacy in the era of

outrage. Peabody Journal of Education, 94(1), 17-31.

https://doi.org/10.1080/0161956X.2019.1553582

of
Miller, J. J. (2004). Citizenship Education Policy at the School District Level. Denver, CO:

ro
Education Commission of the States (ED498663). Retrieved from

https://eric.ed.gov/?id=ED498663 -p
re
Moreno, R. (2002). Who Learns Best with Multiple Representations? Cognitive Theory
lP

Implications for Individual Differences in Multimedia Learning. In Proceedings of

World Conference on Educational Multimedia, Hypermedia & Telecommunications.

na

Denver, Colorado, USA.

ur

Moreno, R., & Mayer, R. E. (1999). Multimedia-supported metaphors for meaning making in
Jo

mathematics. Cognition and Instruction, 17(3), 215-248.

https://doi.org/10.1207/S1532690XCI1703_1

Narciss, S. (2008). Feedback strategies for interactive learning tasks. Handbook of research

on educational communications and technology In J. M. Spector, M. D. Merrill, J. van

Merrienboer, M. P. Driscoll (Eds.) Handbook of Research on Educational

Communications and Technology (3rd ed., pp. 125-144). Routledge.

Papastergiou, M. (2009). Exploring the potential of computer and video games for health and

physical education: A literature review. Computers & Education, 53(3), 603-622.

https://doi.org/10.1016/j.compedu.2009.04.001

33
Patrick, J. J. (2002, May 15). Defining, delivering, and defending a common education for

citizenship in a democracy. Paper presented at the Summit on Civic Learning in

Teacher Preparation, Boston, MA.

Porat, E., Blau, I., & Barak, A. (2018). Measuring digital literacies: Junior high-school

students' perceived competencies versus actual performance. Computers & Education,

126, 23-36. https://doi.org/10.1016/j.compedu.2018.06.030

Prensky, M. (2001). Fun, play and games: What makes games engaging. In M. Prensky (Ed.)

of
Digital Game-based Learning (Chapter 5). NY: McGraw-Hill.

ro
Romero, M., Usart, M., & Ott, M. (2015). Can serious games contribute to developing and

-p
sustaining 21st century skills? Games and Culture, 10(2), 148-177.
re
https://doi.org/10.1177/1555412014548919
lP

Roozenbeek, J., & van der Linden, S. (2019a). The fake news game: Actively inoculating

against the risk of misinformation. Journal of Risk Research, 22(5), 570-580.

na

https://doi.org/10.1080/13669877.2018.1443491
ur

Roozenbeek, J., & van der Linden, S. (2019b). Fake news game confers psychological
Jo

resistance against online misinformation. Palgrave Communications, 5, 65.

https://doi.org/10.1057/s41599-019-0279-9

Sanchez, D. R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining

the impact of gamified quizzes on student learning. Computers & Education, 144,

103666. https://doi.org/10.1016/j.compedu.2019.103666

Seaborn, K., & Fels, D. I. (2015). Gamification in theory and action: A survey. International

Journal of Human-Computer Studies, 74, 14-31.

https://doi.org/10.1016/j.ijhcs.2014.09.006

34
Seoul National University, Institute of Communication Research (2019). How to respond to

false information online (in Korean). Retrieved from

http://factcheck.snu.ac.kr/documents/193

Sørebø, Ø., & Hæhre, R. (2012). Investigating students' perceived discipline relevance

subsequent to playing educational computer games: A personal interest and self-

determination theory approach. Scandinavian Journal of Educational Research, 56(4),

345-362. https://doi.org/10.1080/00313831.2011.594609

of
Tolo, K. W. (1999). The Civic Education of American Youth: From State Policies to School

ro
District Practices. Calabasas, CA: Center for Civic Education.

-p
Tsai, Y. L., & Tsai, C. C. (2018). Digital game-based second-language vocabulary learning
re
and conditions of research designs: A meta-analysis study. Computers & Education,
lP

125, 345-357. https://doi.org/10.1016/j.compedu.2018.06.020

Tully, M., & Vraga, E. K. (2018). A mixed methods approach to examining the relationship
na

between news media literacy and political efficacy. International Journal of

ur

Communication, 12, 766-787.

Jo

Vaičiūnienė, V., & Mažeikienė, V. (2016) Media literacy and information literacy:

Conceptual convergence into a composite notion of MIL. Societal studies, 8(1), 78-94.

https://doi.org/10.13165/SMS-16-8-1-5

van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a

computer-based learning environment on students’ learning outcomes: A meta-analysis.

Review of Educational Research, 85(4), 475-511.

https://doi.org/10.3102/0034654314564881

van Deursen, A. J. A. M., & van Diepen, S. (2013). Information and strategic Internet skills

of secondary students: A performance test. Computers & Education, 63, 218-226.

https://doi.org/10.1016/j.compedu.2012.12.007
35
van Eck, R. (2007). Building artificially intelligent learning games. In D. Gibson, C. Aldrich,

& M. Prensky (Eds.) Games and Simulations in Online Learning: Research and

Development Frameworks (pp. 271-307). IGI Global.

Vogel, J. J., Vogel, D. S., Cannon-Bowers, J., Bowers, C. A., Muse, K., & Wright, M. (2006).

Computer gaming and interactive simulations for learning: A meta-analysis. Journal of

Educational Computing Research, 34(3), 229-243. https://doi.org/10.2190/FLHV-

K4WA-WPVQ-H0YM

of
Walton, G., & Hepworth, M. (2013). Using assignment data to analyse a blended information

ro
literacy intervention: A quantitative approach. Journal of Librarianship and

-p
Information Science, 45(1), 53-63. https://doi.org/10.1177/0961000611434999
re
Walton, G., Pickard, A. J., & Dodd, L. (2018). Information discernment, mis-information and
lP

pro-active scepticism. Journal of Librarianship and Information Science, 50(3), 296-

309. https://doi.org/10.1177/0961000618769980
na

Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., ... &
ur

Conkey, C. (2009). Relationships between game attributes and learning outcomes:

Jo

Review and research proposals. Simulation & gaming, 40(2), 217-266.

https://doi.org/10.1177/1046878108321866

Wilson, C., Grizzle, A., Tuazon, R., Akyempong, K., & Cheung, C. K. (2013). Media and

information literacy curriculum for teachers. UNESCO. Retrieved from:

http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CI/CI/pdf/media_and_infor

mation_literacy_curriculum_for_teachers_en.pdf.

Wineburg, S., McGrew, S., Breakstone, J., & Ortega, T. (2016). Evaluating Information: The

Cornerstone of Civic Online Reasoning. Stanford History Education Group. Available

at http://purl.stanford.edu/fv751yt5934

36
Zhang, X., Ko, M., & Carpenter, D. (2016). Development of a scale to measure skepticism

toward electronic word-of-mouth. Computers in Human Behavior, 56, 198-208.

https://doi.org/10.1016/j.chb.2015.11.042

Zhao, W., & Shute, V. J. (2019). Can playing a video game foster computational thinking

skills? Computers & Education, 141, 103633.

https://doi.org/10.1016/j.compedu.2019.103633

of
ro
-p
re
lP
na
ur
Jo

37
Acknowledgements

This work was supported by the Ministry of Education of the Republic of Korea and the

National Research Foundation of Korea (NRF-2018S1A5B8070398).

of
ro
-p
re
lP
na
ur
Jo
Highlight 1. Developed a game and evaluated its effects on MIL education
Highlight 2. Compared the learning outcomes of the game treatment and the non-game quiz
treatment
Highlight 3. The game improves information discernment regardless of intellectual civic
skills level
Highlight 4. The quiz fosters information discernment when intellectual civic skills level is
high
Highlight 5. Neither causes any attitudinal change in skepticism toward online information.

of
ro
-p
re
lP
na
ur
Jo