Preparing For The Digital Educationt PDF

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/284023691
Preparing for the digital university: a review of the history and

current state of distance, blended, and online learning
Technical Report · January 2015

DOI: 10.13140/RG.2.1.3515.8483
CITATIONS READS
145 1,646
3 authors, including:
Dragan Gasevic
The University of Edinburgh
529 PUBLICATIONS 9,392 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Physical Activity View project
WeLearn@Scale: Understanding and Supporting Peer Feedback at Massive Learning Scale View project
All content following this page was uploaded by Dragan Gasevic on 17 November 2015.
The user has requested enhancement of the downloaded file.

Preparing for the
digital university:
a review of the history and
current state of distance,
blended, and online learning
George Siemens, PhD

Dragan Gašević, PhD
Shane Dawson, PhD
Athabasca University,
University of Edinburgh,
University of Texas Arlington,
University of South Australia
This publication is based on research funded in part by the
Bill & Melinda Gates Foundation. The findings and conclu-
sions contained within are those of the authors and do not
necessarily reflect positions or policies of the Bill & Melinda
Gates Foundation.
Athabasca University served as the Principal Investigator

for this grant.
This work is licensed under a Creative Commons Attribution-ShareAlike

4.0 International License.
Table of contents
Introduction 6
The History and State of Distance Education 9
The History and State of Blended Learning 55
The History and State of Online Learning 93
The History and State of

Credentialing and Assessment 133
Where is Research on

Massive Open Online Courses Headed? 161
Future Technology
Infrastructures for Learning 199
Introduction
In the field of educational technology 2012 was touted as the year of the Massive Open
Online Course (MOOC). While the number of MOOC offerings have since rapidly increased,
the research in this space has been lagging. To help facilitate the development of research
and examine the potential of MOOCs in education the Bill and Melinda Gates Foundation
supported the Massive Open Online Course (MOOC) Research Initiative (MRI). Athabasca
University, long a pioneer in distance education, was selected as the principal investigator
for the grant.
The MOOC conversation was largely occurring in the popular media and was focused on
the technologies and the large numbers of learners enrolling. The sheer scale of numbers
of students led to bold proclamations of education disruption and a sector on the verge of
systemic change. However, from the perspective of 2015, these statements appear increas-
ingly erroneous as MOOCs have proven to be simply an additional learning opportunity
instead of a direct challenge to higher education itself. Many of the issues confronting
early MOOC development and offerings could have been reduced if greater consideration
was given to research literature in learning sciences and technology enabled learning. This
report is the final component of the MRI grant. Additional work in the MRI Grant includes
research reports1, conference2, and a special issue of the International Review of Research
in Open and Distributed Learning3.
The articles presented in this report provide an overview of research literature in:
• Distance education
• Blended learning
• Online learning
• Credentialing
• MOOC research
• Future learning technology infrastructures
1 http://www.moocresearch.com/reports
2 http://www.moocresearch.com/mooc-conference/program
3 http://www.irrodl.org/index.php/irrodl/issue/view/64
6
It is our intent that these reports will serve to introduce academics, administrators, and
students to the rich history of technology in education with a particular emphasis of the
importance of the human factors: social interaction, well-designed learning experiences,
participatory pedagogy, supportive teaching presence, and effective techniques for using
technology to support learning.
The world is digitizing and higher education is not immune to this transition. The trend is
well underway and seems to be accelerating as top universities create departments and
senior leadership positions to explore processes of innovation within the academy. It is our
somewhat axiomatic assessment that in order to understand how we should design and
develop learning for the future, we need to first take a look at what we already know. Any
scientific enterprise that runs forward on only new technology, ignoring the landscape of
existing knowledge, will be sub-optimal and likely fail. To build a strong future of digital
learning in the academy, we must first take stock of what we know and what has been well
researched.
During the process of completing this report, it became clear to us that a society or academic
organization is required to facilitate the advancement and adoption of digital learning re-
search. Important areas in need of exploration include faculty development, organizational
change, innovative practices and new institutional models, effectiveness of teaching and
learning activities, the student experience, increasing success for all students, and state
and provincial policies, strategies, and funding models. To address this need, we invite
interested academics, administrators, government and industry to contact us to discuss
the formation of an organization to advocate for a collaborative and research informed
approach to digital learning.
February 2015
George Siemens4
Dragan Gašević5
Shane Dawson6
4 gsiemens@uta.edu, Twitter: gsiemens

5 dragan.gasevic@ed.ac.uk, Twitter: dgasevic
6 shane.dawson@unisa.edu.au, Twitter: shaned07
7

The History and State
of Distance Education
Vitomir Kovanović
University of Edinburgh
Srećko Joksimović
Oleksandra Skrypnyk
Dragan Gašević
Shane Dawson
George Siemens
University of Texas Arlington
Athabasca University
9

Abstract
This report is one of a series of reports describing the histor-

ical developments and current state of distance education,
online learning, and blended learning. With the intent of
informing future research and practice in the emerging
discipline of digital learning, this tertiary study focuses on
the history and state of distance education, and the under-
standing of the large body of empirical research as captured
by secondary studies (i.e., meta-analyses and systematic
literature reviews). We conducted an automated search for
secondary studies in several online digital libraries, and a
manual search through Google Scholar and the ten most
relevant academic journals. Our search identified 339 sec-
ondary studies in the domains of distance education, online
learning, and blended learning. Of those, 37 secondary
studies on distance education research and practice met the
selection criteria for final inclusion in our study. Based on
the analysis of these secondary sources, three main themes
emerged: i) comparison of distance education and traditional
classroom instruction, ii) identification of important factors
of distance education delivery, and iii) factors of institutional
adoption of distance education. Our results indicate that
distance education, when properly planned, designed,
and supported by the appropriate mix of technology and
pedagogy, is equivalent to, or in certain scenarios more
effective than, traditional face-to-face classroom instruc-
tion. This highlights the importance of instructional design
and the active role of institutions play in providing support
structures for instructors and learners. The implications for
future research and practice are discussed.
Although the scope and definition of distance education
Introduction
are ever changing, the major premise remains the same:

students and teachers are separated by space, time, or both
for the majority or the complete duration of teaching and
learning (Moore & Kearsley, 2004). The origins of distance
education date back to the mid-19th century (Holmberg,
2005); however, the global shift towards knowledge-based
work — ongoing for the last fifty years — has made distance
education highly relevant today (Hanna, 2003). The rapid flow
of information, the fast decay of knowledge, and the pace
of modern societies have placed high demands on today’s
workers for continuous learning and the enhancement of
their own knowledge (Toffler, 1991).
The definition and scope of distance education also changed

as new forms of educational technology developed (Moore,
Dickson-Deane, & Galyen, 2011). Among many definitions
of distance education, one of the more popular and widely
used is given by Moore and Kearsley (2004) as “teaching and
planned learning in which teaching normally occurs in a dif-
ferent place from learning, requiring communication through
technologies as well as special institutional organization”
(p. 2). Given this inherent need to transcend physical dis-
tance between students and instructors, distance education
has always been highly dependent on the current state of
technological development (Anderson & Dron, 2010). The
12 Introduction
the history and state of distance education
biggest shift came with the development of digital computing technology, the Internet, and
the World Wide Web, which had an all-encompassing impact on the domain of education.
The introduction of various educational software systems dramatically changed the entire
process of educational delivery for both distance and on-campus modes of instruction. This
trend is likely to continue into the future. According to a recent report (Allen & Seaman,
2011) published by the Sloan Consortium, 6.1 million students took at least one online course
in the fall 2010 term, an increase of 10% from the fall 2009 term — far bigger than the 1%
increase in the same period for higher education overall. Finally, the recent development
of Massive Open Online Courses (MOOCs) has further stressed the need for lifelong, per-
sonalized, flexible education (Kovanović, Joksimović, Gašević, Siemens, & Hatala, 2014).
Novel educational software systems, such as Learning Management Systems (LMSs) have
not only influenced the practices of distance education. These software have also altered
the way traditional universities provide on-campus learning, as well as enabling a mix
between the two, which is typically known as blended learning (Lust, Juarez Collazo, Elen,
& Clarebout, 2012). The introduction of digital technology has also brought a plethora of
different terms and abbreviations, such as online learning, web-based learning, blended
learning, e-learning, learning management systems (LMS), computer-aided instruction (CAI),
computer-supported instruction (CSI), technology-enhanced learning (TEL), Internet-based
training (IBT), and virtual learning environments (VLE), which to a large extent all fall under a
broad definition of distance education (Moore & Kearsley, 2004). The most recent addition
to this group of terms is MOOC (Daniel, 2014; Siemens, 2012). While there is certainly a
need for more accurate descriptions of different features of new technology, many of these
terms were used without establishing an accepted and authoritative definition and often
described several completely different things (Moore et al., 2011).
Introduction 13
The ever-growing adoption of educational technology has also sparked debate on the relative
importance of instruction and pedagogy versus educational technology and media on the
quality of learning. The history of distance education teaches us that the general public will
readily assume that the technology alone can transform education (Blin & Munro, 2008).
Even today, this position can be seen in reports related to MOOCs and the “disruptive
change” of their technologically inspired approach to learning (Kovanović et al., 2014). More
than thirty years ago, Clark (1983) expressed his skepticism toward this belief. Clark (1983)
argued that different educational technologies and media are “mere vehicles that deliver
instruction but do not influence student achievement any more than the truck that delivers
our groceries causes changes in our nutrition. …the choice of vehicle might influence the
cost or extent of distributing instruction, but only the content of the vehicle can influence
achievement” (p. 445). Anderson and Dron (2010) express a similar view, recognizing the
importance of technology and pedagogy for the success of distance education. According
to Anderson and Dron (2010) “the technology sets the beat and creates the music, while
the pedagogy defines the moves” (p. 81). What is now needed is a synergetic effect of
pedagogies and novel technological approaches.
The purpose of this study

Given the large body of educational research conducted over the past forty years, there
is an imperative to collate this accumulated knowledge into a usable form for the devel-
opment of modern distance education. Distance education has come a long way and can
offer useful lessons for the development of the emerging field of digital education. The
connection with this previous long-lasting line of research is particularly relevant in light of
recent advances in MOOCs. While MOOCs may have been described as a “revolution” of
education (Friedman, 2012) they can be considered as a continuation in the long devel-
opment of distance education (Daniel, 2014). In addition, with many institutions trying to
innovate their learning and teaching practices (e.g., flipped classroom, active learning, and
clickers), it is important to provide a concise synthesis of the relevant knowledge that can
be used to guide these processes of innovation. As such, a first step is to examine what
is known from the large body of research in the domains of distance education, online
learning, and blended learning.
The study presented in this paper provides a systematic overview of relevant meta-analyses
and systematic literature reviews in the domain of distance education, while two additional
reports cover online and blended learning, respectively. We examined the important di-
mensions that can be leveraged from a large number of scientifically rigorous reports on
14 Introduction
the different forms of distance education. More specifically, in this report we focus on the
following research questions:
RQ1. What is the current state of distance education research, as reflected through
meta-analyses and systematic reviews?
RQ2. What are the key themes in the domain of distance education research?
RQ3. Based on available meta-analyses and systematic literature reviews, what is currently
known in the field of distance education?
As most social science studies are not strong enough to provide conclusive evidence, we
focused on meta-analyses, which are studies that statistically integrate empirical results
of multiple studies on the same topic and thus, provide better characterization of a given
phenomena (Glass, 1976; Grant & Booth, 2009; Hedges, 1982). Systematic literature reviews
are an important component of educational research (Andrews, 2005; Mulrow, 1994), and
include several related types of studies, such as scoping studies, literature mapping and
review studies (Rumrill, Fitzgerald, & Merchant, 2010), and rapid reviews and qualitative
evidence synthesis studies (Grant & Booth, 2009).
History of Distance Education:

From correspondence study to teleconferencing
The origins of modern distance education can be traced back to the early 18th century. In
1728, Bostonian Caleb Phillips, a teacher of a novel shorthand writing method, suggested to
the people living outside the city could participate in his learning program by having lessons
sent to them each week without diminishing the quality of their instruction (Holmberg, 2005,
p. 13). From that period, it is generally considered that distance education has gone through
either three (Anderson & Dron, 2010; Bates, 2005; Keegan, 1993) or five generations (An-
derson, 2008; Taylor, 2001), depending on whether the focus is on the adopted technology
or on the pedagogical approach.
The oldest form of distance education, universally recognized as the first generation, was
correspondence study (Anderson, 2008; Bates, 2005; Keegan, 1993; Taylor, 2001). In this form,
students received self-directed, paper-based study materials from instructors using the postal
service and then, as instructed, returned their written assignments for evaluation, grading,
and often written feedback (Holmberg, 2005). The primary drawback of this mode of delivery
is that it provided slow, one-to-one communication between instructors and students and
Introduction 15
did not provide opportunities for interaction among students (Anderson, 2003; Taylor, 2001).
Instead of relying solely on printed media, the next generation of distance education made
heavy use of a richer set of media and also enabled simpler and faster delivery of learning
materials through broadcasting (Keegan, 1993). Regarding the level of interaction, as with
correspondence study, opportunities were still quite limited (Bates, 2005; Garrison, 1985).
The broad availability of audio conferencing technology in the mid-1960s, however, enabled
distance education with limited student-student interaction (Anderson & Dron, 2010). Unlike
audio conferencing, videoconferencing technology was not widely used in practice, primarily
due to very high equipment costs (Garrison, 1985). In most cases where video technology
was used, only instructors would transmit video to students, while two-way communication
was supported only through audio conferencing technology (Garrison, 1985).
In terms of pedagogy, all forms of distance education were characterized by behaviorist and
later cognitivist models of learning in which the locus of control is heavily on the teacher
and instructional designer (Anderson & Dron, 2010). Students primarily learned individu-
ally as the interaction among learners was still limited and not incorporated into learning
activities. Those pedagogical models proved very successful and are still in wide use even
today, particularly for training purposes in which it is easy to define strict performance criteria
(Anderson & Dron, 2010). Given that they are primarily characterized by the purposeful
integration of different media and adopt similar pedagogical approaches, some authors
(Bates, 2005; Keegan, 1993) consider broadcasting and conferencing to be two forms of
the same, second generation of distance education.
Modern Distance Education:

Online Learning, Blended Learning, and Beyond
The development of digital computing technology has marked the most significant mile-
stone in the history of distance education (Anderson & Dron, 2010). The use of email, web-
based resources, learning management systems, and online discussion boards are some of
the primary technologies supporting interactive and flexible forms of distance education
(Harasim, 2000). We should emphasize that this is the first form of distance education that
truly enabled interaction between students and opened doors to new distance education
pedagogies. Novel pedagogical approaches, based on social-constructivist views of learning,
were developed (Jonassen, Davidson, Collins, Campbell, & Haag, 1995). Those ideas first
pioneered by the likes of Dewey (1897), Piaget (1959), and Vygotsky (1978), view learning
16 Introduction
as a process of knowledge construction by learners through their social interactions rather
than the plain acquisition of facts from instructors (Anderson & Dron, 2010).
Aside from distance education, in more recent times the terms online learning and web-
based learning have become more widely used. It is almost universally accepted that they
represent special forms of distance education (Clardy, 2009; Harasim, 2000; Mason, 2000;
Taylor, 2001) that are the most popular in the 21st century (Anderson, 2009). Another im-
portant form of learning that has gained significant research attention is blended learning.
This is generally defined as learning that encompasses both traditional classroom and dis-
tance delivery (Bonk, Graham, Cross, & Moore, 2005; Spector, Merrill, van Merrienboer, &
Driscoll, 2007). It should be noted that mixed-mode learning and hybrid learning are two
other terms typically used interchangeably with blended learning (Means, Toyama, Murphy,
& Bakia, 2013). Because of the mandatory integration with traditional classroom instruction,
blended learning cannot be considered just another form of distance education.
Although it is universally accepted that both types of instruction should be present for
learning to be considered blended, there is no clear consensus on their relative percentages.
Allen, Seaman, and Garrett (2007), for example, argued that even courses with 30–79% of
online instruction should be considered blended. Bernard, Borokhovski, Schmid, Tamim,
and Abrami (2014) debated that traditional instruction should represent at least 50% of the
course in order to be considered blended. An further view argues that all mixes of online
and traditional instruction should be considered blended, even traditional courses that
only use LMS for course communication (Bliuc, Goodyear, & Ellis, 2007). In any case, based
on this mandatory inclusion of traditional classroom instruction, in this paper we consider
blended learning as a special form of learning on its own, that draws from both distance
and traditional instruction and provides for their pedagogically sound integration.
Introduction 17
Methodology
Study search method and inclusion criteria

To provide a synthesis of scientifically rigorous knowledge in distance education, we con-
ducted a systematic literature review of published distance education research. Building
on the study search method of Means et al. (2013), we searched for published journal
articles, conference proceedings, doctoral dissertations, and institutional reports on ERIC1,
PsychINFO2, PubMed3, and ProQuest4 digital libraries. SCOPUS5 digital library was also in-
cluded in our search given its comprehensive coverage of many research domains. We also
conducted a manual search using Google Scholar6 for additional reports in our dataset. In
terms of the time-span, we did not put any constraint into the search query and considered
all studies regardless of their publication date. Similar to Means et al. (2013), we conducted
a manual search of all papers published in the following journals: American Journal of Dis-
tance Education, Journal of Distance Education, Distance Education, International Review
of Research in Distance and Open Education, Journal of Asynchronous Learning Networks,
Journal of Technology and Teacher Education, Career and Technical Education Research,
Internet and Higher Education, Journal of Computing in Higher Education, and Computers
and Education. We selected these particular journals because in our digital library search
these publications contained the highest number of results.
To consider the papers found in the search for inclusion in the study, we examined titles,
keywords, and abstracts for the required combination of important domain keywords (i.e.,
distance education, online learning, web-based learning, blended learning) and important
study-type keywords (i.e., meta-analysis, systematic review, tertiary study, scoping study).
Figure 1 shows the SCOPUS query used in our search. The actual query syntax depended
on the particular search platform, but they all followed the above-mentioned logical struc-
ture. We also included several different wordings of our keywords in order to provide more
flexible searching criteria.
1 http://eric.ed.gov
2 http://apa.org/pubs/databases/psycinfo/idex.aspx
3 http://www.ncbi.nim.nih.gov/pubmed
4 http://search.proquest.com
5 http://scopus.com
6 http://scholar.google.com
18 Methodology
After the search, three researchers coded the studies for distance education, online learning,
or blended learning based on the article title, abstract, and keywords. In cases where a code
could not be assigned based on the available information, the full text of the article was
consulted. In cases where several publications discussed the same data (e.g., dissertation
and journal article) we gave preference to the journal articles, based on the study by Bernard
et al. (2014) who found evidence of their superiority in terms of methodological quality
over other types of publications. In order to be included in this study, each report had to:
i. Be published in a peer-reviewed scientific journal, conference paper, doctoral

dissertation, or government report,
ii. Adopt systematic approach (i.e., systematic literature review or meta-analysis),
iii. Focus on distance education,
iv. Have a criteria for the inclusion of primary sources, and
v. Focus on higher or adult education; studies that focused on K–12 education were
included only if they also analyzed higher or adult education.
Figure 2 illustrates our complete literature search process. The initial search found 306 studies
from distance, online, and blended learning. An additional 19 studies were discovered using
Google Scholar and 14 more through the manual search of the selected academic journals.
In total, we identified 339 unique studies that fit our search criteria. Upon further screen-
ing, we were left with 102 studies on distance, online, and blended learning for analysis.
Given that this report focuses only on distance education, as a final dataset we included
37 papers that fit our criteria.
Methodology 19
( TITLE-ABS-KEY(“metaanalysis”) OR
TITLE-ABS-KEY(“meta-analysis”) OR
TITLE-ABS-KEY(“meta analysis”) OR
TITLE-ABS-KEY(“metasynthesis”) OR
TITLE-ABS-KEY(“meta synthesis”) OR
TITLE-ABS-KEY(“meta-synthesis”) OR
TITLE-ABS-KEY(“scoping study”) OR
TITLE-ABS-KEY(“systematic review”) OR
TITLE-ABS-KEY(“tertiary study”)
) AND (
TITLE-ABS-KEY(“distance learning”) OR
TITLE-ABS-KEY(“distance education”) OR
TITLE-ABS-KEY(“blended education”) OR
TITLE-ABS-KEY(“blended learning”) OR
TITLE-ABS-KEY(“hybrid education”) OR
TITLE-ABS-KEY(“hybrid learning”) OR
TITLE-ABS-KEY(“e-learning”) OR
TITLE-ABS-KEY(“online learning”) OR
TITLE-ABS-KEY(“online education”) OR
TITLE-ABS-KEY(“web-based learning”) OR
TITLE-ABS-KEY(“web-based education”) OR
TITLE-ABS-KEY(“web based learning”) OR
TITLE-ABS-KEY(“web based education”)
)
Figure 1 Example of SCOPUS search query
20 Methodology
Systematic search point
Digital Library Search Google Scholar Search Manual Journal Search
Identified 306 studies Identifed 19 studies Identifed 14 studies
Merging search results
Total of 339 unique

studies identifed
FOCUS OF THIS STUDY
Selection of distance Selection of online Selection of blended

education studies learning studies learning studies
Total of 37 distance Total of 32 online Total of 20 blended

Systematic search end
Figure 2 Systematic literature search process
Methodology 21
Overview of
distance education
studies
An overview of the sources included in the present study is outlined in Figure 3. A complete
list of all reports is presented in Table 1. Overall, 37 studies were included comprising 12
meta-analyses and 25 systematic literature reviews. Journal articles were the most repre-
sented, with 30 journal publications and only five conference papers and two dissertations
(Figure 3). In terms of the number of primary sources, most of the studies were of moderate
size, covering between 11 and 50 primary sources (Figure 4). The largest six studies, cov-
ering more than 300 sources, were exclusively literature reviews, which is expected given
the highly complex statistical calculations involved in conducting meta-analyses and high
expectations for the quality of the reported results in the studies (Hedges, 1982).
Regarding the coverage of different levels of education (Figure 5), most of the studies did
not focus on any particular level of education, but rather analyzed all the available studies.
Focus on higher education, or a combination of higher and adult education, was more
common for meta-analyses. What is interesting is that explicit focus on adult learning was
only seen in systematic literature reviews, as none of the meta-analyses focused explicitly
on this educational segment (Figure 5).
All 37 studies were published between 1998 and 2014, with the largest number of meta-stud-
ies (i.e., four meta-studies) published in 2004 (Figure 6). The highest number of systematic
literature reviews was published in 2010 (i.e., five studies). The number of published studies
seems to be decreasing from 2004 onwards, while the publication of systematic literature
reviews seems to be increasing. Recently, systematic literature reviews — probably due
to their more qualitative nature — seem to be the preferred way of interpreting existing
knowledge in the domain of distance education. Still, those numbers need to be put into
perspective of changing terminology, with a large number of studies published in online
and web-based learning, which are covered in two separate reports.
22 Overview of Distance Education Studies

As expected, the largest second-order studies, in terms of the number of included primary
sources, were almost exclusively described in journal publications (Figure 7). Likewise, the
largest second-order studies tend to cover all levels of education, or focus primarily on the
higher education domain (Figure 8). The studies focusing on adult learning tended to have
fewer than 50 primary sources, likely due to the much smaller number of studies related to
this particular form of learning.
12
30
number of studies
number of studies
10
20 8
10 4
2
0 0
Conf. Paper Journal Article Thesis 1-10 11-50 51-100 101-300 >300
type of report number of primary studies covered
meta-analysis meta-analysis
sys. review sys. review
Figure 3 Number of meta-analyses and systematic Figure 4 Number of meta-analyses and systematic
literature reviews published as different types of literature reviews covering a given number of primary
publications sources
Overview of Distance Education Studies 23

18 6
number of studies
number of studies
16
14
12 4
10
8
2
6
4
2 0
0
00
99
20 2
04
05
06
09
98
07
10
03
11
12
13
14
0
Adult HE HE & Adult HE, Adult
20
19
20
20
20
20
20
19
20
20
20
20
20
20
& K-12
publication year
covered level of education
meta-analysis meta-analysis
sys. review sys. review
Figure 5 Number of meta-analyses and systematic Figure 6 Number of meta-analyses and systematic
literature reviews covering different levels of education literature reviews published in different years
12 12
number of studies
number of studies
10 10
8 8
6 6
4 4
2 2
0 0
1-10 11-50 51-100 101-300 >300 1-10 11-50 51-100 101-300 >300
number of primary studies covered number of primary studies covered
conf. paper thesis adult he & adult

journal article he he, adult & k-12
Figure 7 Coverage of primary sources by different Figure 8 Coverage of primary sources about different
types of reports education levels

Table 1 Primary sources included in the study
Report Title Report Type Studies Years Level
Allen et al. (2002). Comparing student

satisfaction with distance education
to traditional classrooms in higher Meta-Analysis
1 24 1989–1999 HE
education: A meta-analysis. American Journal
Journal of Distance Education, 16(2),
83–97.
Allen et al. (2004). Evaluating the

effectiveness of distance learning: A Meta-Analysis
2 39 Up to 2003 All
comparison using meta-analysis. Journal Journal
of Communication, 54(3), 402–420.
Bernard et al. (2004). A methodological

morass? How we can improve
Meta-Analysis
3 quantitative research in distance 232 1985–2002 All
Journal
education. Distance Education, 25(2),
175–198.
Bernard et al. (2009) A meta-analysis of
three types of interaction treatments Meta-Analysis
4 74 1985–2006 All
in distance education. Review of Journal
Educational Research, 79(3),1243–1289.
Bernard et al. (2004). How does

distance education compare with
Meta-Analysis
5 classroom instruction? A meta-analysis 232 1985–2002 All
Journal
of the empirical literature. Review of
Educational Research, 74(3), 379–439.

Table 1 (Cont.) Primary sources included in the study
Bernard et al. (2004). The effects of

synchronous and asynchronous distance
education: A meta-analytical assessment
of Simonson’s “equivalency theory.”
Meta-Analysis
6 2004 Annual proceedings of selected 232 1985–2002 All
Conference
research and development papers
presented at the national convention
of the Association for Educational
Communications and Technology.
Borokhovski et al. (2012). Are contextual

and designed student-student interaction
Meta-Analysis
7 treatments equally effective in distance 32 1985–2006 HE
Journal
education? Distance Education, 33(3),
311–329.
Lou et al. (2006). Media and pedagogy

in undergraduate distance education: A
theory-based meta-analysis of empirical Meta-Analysis
8 103 1985–2002 HE
literature. Educational Technology Journal
Research and Development, 54(2),
141–176.
Machtmes & Asher (2000). A meta-

analysis of the effectiveness of
Meta-Analysis HE,
9 telecourses in distance education. 19 1943–1997
Journal Adult
American Journal of Distance Education,
14(1), 27–46.
Shachar & Neumann (2003). Differences

between traditional and distance
education academic performances: Meta-Analysis
10 86 1990–2002 All
A meta-analytic approach. The Journal
International Review of Research in Open
and Distance Learning, 4(2).

Storrings (2006). Attrition in distance

education: A meta-analysis. Annual
Proceedings of Selected Research and
Meta-Analysis HE,
11 Development Papers Presented at the 30 1984–2004
Conference Adult
National Convention of the Association
for Educational Communications
and Technology.
Williams (2006). The effectiveness of

distance education in allied health
Meta-Analysis HE,
12 science programs: A meta-analysis of 25 1990–2003
Journal Adult
outcomes. American Journal of
Distance Education, 20(3), 127–141.
Booth et al. (2009). Applying findings

from a systematic review of workplace-
based e-learning: Implications for Sys. Lit. Rev.
13 29 1992–2009 Adult
health information professionals. Health Journal
Information & Libraries Journal, 26(1),
4–21.
Borokhovski et al. (2011). An extended

systematic review of Canadian policy
Sys. Lit. Rev.
14 documents on e-learning: What we’re 138 2000–2010 All
Journal
doing and not doing. Canadian Journal
of Learning and Technology, 37(3).
Childs et al. (2005). Effective e-learning

for health professionals and students:
Barriers and their solutions. A systematic Sys. Lit. Rev.
15 57 1997–2004 Adult
review of the literature: Findings from Journal
the HeXL project. Health Information &
LibrariesJournal, 22, 20–32.

Chipps et al. (2012). Videoconference-

based education for psychiatry registrars
Sys. Lit. Rev.
16 at the University of KwaZulu-Natal, South 7 1998–2009 Adult
Journal
Africa. African Journal of Psychiatry,
15(4).
Chipps et al. (2012). A systematic review

of the effectiveness of videoconference-
Sys. Lit. Rev.
17 based tele-education for medical and 5 1990–2011 Adult
Journal
nursing education. Worldviews on
Evidence-Based Nursing, 9(2), 78–87.
De Freitas (2007). Post-16 e-learning
content production: A synthesis of the Sys. Lit. Rev. HE,
18 NR Up to 2006
literature. British Journal of Educational Journal Adult
Technology, 38(2), 349–364.
Hauser (2013). Qualitative research in
distance education: An analysis of journal Sys. Lit. Rev.
19 382 2005–2012 All
literature 2005–2012. American Journal Journal
of Distance Education,27(3), 155–164.
Hrastinski & Keller (2007). Computer-

mediated communication ineducation: Sys. Lit. Rev.
20 117 2000–2004 All
A review of recent research. Educational Journal
Media International,44(1), 61–77.
Lee & McElroy (2012). Telepractice is a

new method for providing services to
children with Autism Spectrum Disorder
(ASD): This scoping review summarizes Sys. Lit. Rev.
21 9 Up to 2011 Adult
existing research and identifies research Journal
gaps. Evidence-Based Communication
Assessment and Intervention, 6(4),
177–180.

Table 1: (Cont.) Primary sources included in the study
Lee et al. (2004). The past, present, and

future of research in distance education:
Sys. Lit. Rev.
22 Results of a content analysis. American 383 1997–2002 All
Journal
225–241.
Ludlow & Brannan (1999). Distance

education programs preparing personnel
for rural areas: Current practices, Sys. Lit. Rev.
23 32 1985–1999 Adult
emerging trends, and future directions. Journal
Rural Special Education Quarterly, 18(3),
4–15.
Mehlenbacher et al. (2010). Reviewing

the research on distance education and
Sys. Lit. Rev.
24 e-learning. Proceedings of the 28th ACM NR Up to 2009 All
Conference
International Conference on Design of
Communication, 237–242.
Neto & Santos (2010). Analysis of the

methods and research topics in a sample
of the Brazilian distance education Sys. Lit. Rev.
25 983 1987–2007 All
publications, 1992 to 2007. American Journal
119–134.
Ritzhaupt et al. (2010). An investigation

of distance education in North American
research literature using co-word Sys. Lit. Rev.
26 517 1987–2005 All
analysis. The International Review of Journal
Research in Open and Distance Learning,
11(1), 37–60.

Simpson (2003). Distance delivery of

pre-service teacher education: Lessons
Sys. Lit. Rev.
27 for good practice from twenty-one NR Up to 2003 HE
Thesis
international programs. Doctoral
Dissertation.
Singh & Hardaker (2014). Barriers and
enablers to adoption and diffusion of Sys. Lit. Rev.
28 340 2001–2013 HE
eLearning. Education + Training, 56(2/3), Journal
105–121.
Stall-Meadows (1998). Grounded

meta-analysis of qualitative case study Sys. Lit. Rev.
29 4 NR All
dissertations in distance education Thesis
pedagogy. Doctoral Dissertation.
Stewart (2010). What’s missing? The

next step towards universal distance
Sys. Lit. Rev.
30 education. Proceedings of 26th Annual 59 2004–2009 All
Conference
Conference on Distance Teaching &
Learning.
Tomlinson et al. (2013). How does tele-

learning compare with other forms of
education delivery? A systematic review Sys. Lit. Rev.
31 13 2000–2012 Adult
of tele-learning educational outcomes for Journal
health professionals. New South Wales
Public Health Bulletin, 24(2), 70–75.
Tuquero (2011). A meta-ethnographic

synthesis of support services in Sys. Lit. Rev.
32 5 2000–2008 Adult
distance learning programs. Journal of Journal
Information Technology Education, 10.

Uzuner (2009). Questions of culture in

distance learning: A research review. The Sys. Lit. Rev.
33 27 Up to 2008 All
International Review of Research in Open Journal
and Distance Learning, 10(3).
Waight et al. (2002). Recurrent themes

in e-learning: A meta-analysis of major Sys. Lit. Rev.
34 15 1999–2001 All
e-learning reports. Proceedings AHRD Conference
2002 Conference.
Wang & Lockee (2010). Virtual worlds in

distance education: A content analysis Sys. Lit. Rev.
35 4 2003–2009 All
study. Quarterly Review of Distance Journal
Education, 11(3), 183–186.
Wutoh et al. (2004). eLearning: A review

of Internet-based continuing medical
Sys. Lit. Rev.
36 education. Journal of Continuing 16 1966–2003 Adult
Journal
Education in the Health Professions,
24(1), 20–30.
Zawacki-Richter et al. (2009). Review

of distance education research (2000
to 2008): Analysis of research areas, Sys. Lit. Rev.
37 695 2000–2008 All
methods, and authorship patterns. The Journal
International Review of Research in Open
and Distance Learning, 10(6), 21–50.

State of distance
education literature
From the published second-order studies in the domain of distance education, we were able
to identify several themes that describe the focus of distance education research. The topics
of systematic literature reviews include the following: i) topic analysis of published literature,
ii) state of distance education research methods, iii) effectiveness of distance education,
and iv) success factors for distance education. Several studies covered more than one of
these topics or focused on a specific context in which they are investigated in more detail.
We identified four main topics captured by the meta-analyses included in our study: i)
comparison of distance education and traditional face-to-face education, ii) comparison of
different modes of distance education delivery, iii) success factors of distance education,
and iv) methodological quality of published distance education literature. In the following
sections, we report on the most important findings from our systematic review.
Topic analysis of published literature

A number of systematic reviews analyzed significant themes that emerged in the published
distance education literature. The most general analysis was done by Lee, Driscoll, and Nelson
(2004) who categorized each study using a predefined list of six important topic categories:
i) design-related, ii) development-related, iii) management-related, iv) evaluation-related,
v) institutional- and operational-related, and vi) theory- and research-related topics. What
Lee et al. (2004) found is that studies in distance education focus primarily on two of those
topic categories: i) design-related topics, such as course development, organization, and
instructional strategy (27% of all publications), and ii) theory and research related topics,
such as literature review papers, theory building studies, and research methods (30% of
all publications). Similar findings are given by Ritzhaupt, Stewart, Smith, and Barron (2010)
who found that the pre-web era was mostly focused on theory development to provide
a basis for further pragmatic research efforts. Also, a more recent analysis of almost 700
papers published between 2000 and 2008 by Zawacki-Richter, Baecker, and Vogt (2009)
identified a focus on the study of teaching and learning topics, such as learner character-
istics, instructional design, interaction, and communication.
32 State of Distance Education Literature

In the more concrete domain of workplace learning, Booth, Carroll, Papaioannou, Sutton,
and Wong (2009) analyzed 29 studies and identified five emerging topics: i) flexibility, ii)
peer communication, iii) support, iv) knowledge validation, and v) course presentation
and design. The need for flexibility — although a constant theme in distance education
— is increasingly manifested in many different forms, such as a flexible pace of learning,
flexible trajectories through course content, and adaptation for the particular context and
needs of individual learners (Booth et al., 2009). This need for a flexible pace of learning
is closely related to the need for asynchronous peer communication and collaboration,
a direct contradiction to the shared learning experience (Booth et al., 2009). Support —
either institutional support, peer support, or instructional support — is identified as one of
the prerequisites for a successful educational experience. Institutional support also plays
an important role in assuring knowledge validation through various means of assessment
focused on the particular needs of learners (Booth et al., 2009).
Given that distance education makes use of various technologies and media that enable
different modes and levels of interaction, it is important to examine which media and tech-
nologies were more investigated by the distance education research community. Looking at
117 studies between 2000 and 2004, Hrastinski and Keller (2007) found the primary focus
was on the analysis of asynchronous discussions, while mixed and fully synchronous modes
of interaction received much less attention. Also, with respect to three types of interaction
(i.e., student-student, student-content, and student-instructor) by Moore (1989), the primary
focus was on student-student interaction while other forms received less attention. This
particular period (2000–2004) was characterized by the wider adoption of learning manage-
ment systems and the Hrastinski and Keller (2007) study confirmed the focus on adoption
of learning management systems, rather than their development. This also resulted in an
extensive investigation of their use in traditional classroom settings and a higher focus on
blended rather than distance modes of delivery (Hrastinski & Keller, 2007). These findings
are also consistent with those of Ritzhaupt et al. (2010) who, through the analysis of 517
studies published between 1987 and 2005, identified decreasing interest in teleconferencing
and a move towards computer-based distance education. Over time, interaction became
the central point of investigation, with its scope shifting from instruction to collaboration
(Ritzhaupt et al., 2010).
In addition to the thematic analysis of the research literature, Waight, Willging, and Wentling
(2002) and Borokhovski et al. (2011) conducted interesting thematic analyses of published
government and business reports. In their study of Canadian provincial government reports,
Borokhovski et al. (2011) found that the most commonly discussed topic related to the
benefits of distance education technology, followed by the support for implementation,
State of Distance Education Literature 33

and the importance of coordinating efforts in implementing distance education in prac-
tice. Similar findings are given by Waight et al. (2002), who also identified the benefits of
technology as the main topic of published government, business, and association reports.
Effectiveness of distance education:

Comparison with traditional classroom instruction
With the broader acceptance of distance education, the educational research community
raised questions about its effectiveness and how it compares to traditional classroom in-
struction. A large number of empirical studies were conducted in order to provide evidence
on these important questions, together with a substantial number of systematic reviews and
meta-analyses that tried to aggregate this large body of empirical evidence.
Based on the results of the meta-analyses included in this review, it can be concluded
that distance education is more effective, or at least as effective as traditional classroom
instruction. The meta-analysis by Machtmes and Asher (2000) analyzed 19 true or quasi-ex-
perimental studies where a total of 1,426 students showed a non-significant difference in
terms of academic performance. Similarly, the meta-analysis by Bernard et al. (2004) looked
at 232 studies with more than 57,000 students and found no difference in effect size for
student academic performance. Looking at 103 studies of undergraduate distance edu-
cation, representing 25,320 students, Lou, Bernard, and Abrami (2006) also found similar
academic performance, with a non-significant difference in effect sizes. However, results from
several meta-analyses give a slight advantage to distance education. The study by Shachar
and Neumann (2003) looked at 86 studies with more than 15,000 students and found an
overall moderate effect size of Cohen’s d = 0.366 (Cohen, 1988). The meta-analysis of 39
primary sources with a total of 71,731 students conducted by Allen et al. (2004) also found
an average effect size (average r = 0.048) favoring distance education in terms of student
course grades. Finally, in the context of health education, the meta-analysis by Williams
(2006) looked at 25 experimental studies with a total of 2,702 students and found a small
overall effect size of Cohen’s d = 0.15, favoring distance education as a mode of delivery.
The potential of distance education has been acknowledged by Ludlow and Brannan (1999)
for training special education and services professionals in rural areas. Similarly, based on
the analysis of high quality studies of videoconferencing, tele-learning, and tele-practice
use, several studies (Chipps, Ramlall, & Mars, 2012; Chipps, Brysiewicz, & Mars, 2012;
Tomlinson et al., 2013) indicated comparable results for distance-education programs and

traditional face-to-face programs. The study by Chipps, Ramlall, and Mars (2012) looked
at the effectiveness of videoconferencing for doctors and nurses education, based on the
analysis of five randomized control trials. They concluded that there is enough evidence
to provide moderate evidence-based support for the use of videoconferencing in distance
education. Similar findings are reported by Tomlinson et al. (2013) who found — based
on the analysis of 13 studies — that tele-learning achieves comparable results to those of
traditional face-to-face instruction. In a similar manner, Chipps, Brysiewicz, and Mars (2012)
conducted a systematic review of seven published studies on the use of videoconferencing
for psychiatric education in Africa. The results of that review show support for videoconfer-
encing for psychiatric education, although they point out that for many educational programs
in the developing world, videoconferencing is still out of reach. The systematic review by
Wutoh, Boren, and Balas (2004) of 16 primary sources also reported comparable results
for Internet-based distance education programs and traditional face-to-face instructional
modes. Finally, the systematic review by Lee and McElroy (2012), based on the review of
six experimental and two non-experimental studies, concluded that teleconferencing is a
promising direction for teaching doctors who work with special-needs children.
While the above mentioned studies showed similar levels of effectiveness, it should be
noted that slightly higher satisfaction with traditional modes of instruction is observed.
This was reported in the systematic reviews by Chipps, Brysiewicz, and Mars (2012) and
Tomlinson et al. (2013) along with the meta-analysis by Allen, Bourhis, Burrell, and Mabry
(2002). Based on the meta-analysis of 24 studies with a total of 4,702 students, Allen et al.
(2002) found slightly higher student satisfaction with traditional mode of delivery over dis-
tance learning (average r = 0.031 after the deletion of outliers). Similar results are reported
in the meta-analysis by Bernard, Abrami, Wade, Borokhovski, and Lou (2004) who found
higher satisfaction with classroom instruction (Hedges’ g = −0.185 over 83 studies) over
synchronous mode of distance education delivery. The difference between asynchronous
mode of distance education delivery and traditional classroom instruction was non-significant
(Hedges’ g = −0.003 over 71 studies).
Institutional adoption of distance education

The fourth important set of systematic reviews and meta-analyses focused on institutional
adoption of distance education and the various aspects that contribute to or impede its
success. Several general, macro-level factors were found to be important for successful
adoption of distance education, together with several individual, micro-level factors (Singh
& Hardaker, 2014).

Among different macro-level factors, technology infrastructure is seen as an important pre-
requisite for the adoption of distance education (Childs, Blenkinsopp, Hall, & Walton, 2005;
Singh & Hardaker, 2014). The role of academic management is essential, as the introduction
of new technology requires institutional innovation and change (Childs et al., 2005; Simpson,
2003; Stewart, 2010) and the provision of sufficient resources for program implementation
(Childs et al., 2005; Singh & Hardaker, 2014). The coordination and collaboration of different
parties involved in the implementation of distance education programs is often necessary
(Ludlow & Brannan, 1999; Simpson, 2003). Finally, support for academic staff is recognized
as having important effects on the adoption of distance education (Childs et al., 2005; Singh
& Hardaker, 2014), especially related to the technological aspects of course implementation.
It is equally important to understand different personal and individual factors that affect
success in distance education. As Singh and Hardaker (2014) point out, current practices are
tailored to more technologically literate academic staff. This brings attention to important
issues related to staff development (Childs et al., 2005; Simpson, 2003; Singh & Hardaker,
2014; Stewart, 2010) and allocation of time dedicated to learning new technologies (Childs
et al., 2005). When academic staff have positive views of distance education, their primary
perceived benefit is related to serving students in remote locations (Stall-Meadows, 1998).
On the other hand, negative attitudes towards a new technology are a major limiting
factor for distance education, and they are found to be strongly related to the staff age
(Singh & Hardaker, 2014). There is also more time needed for the preparation of distance
education courses (Childs et al., 2005; Singh & Hardaker, 2014; Stall-Meadows, 1998),
and more challenge in keeping students motivated and engaged (Stall-Meadows, 1998).
Instructors often use the interactive exchange of information to overcome these limitations
(Stall-Meadows, 1998). Sometimes, instructors even visit students in their remote locations,
which is associated with improved relationships with students and more positive views of
distance education by course instructors (Stall-Meadows, 1998).
Although, in most cases, access to modern technology — such as the Internet — is seen as
a major prerequisite for successful distance education, many sources point to a need for
high quality materials in distance education. As indicated by Moore and Kearsley (2004),
“A far bigger problem [than Internet access] is the quality of media produced for distri-
bution via the technology” (p. 8). With this in mind, Simpson (2003) and De Freitas (2007)
analyzed the state of distance education in terms of content production. Simpson (2003)
showed that the production of quality learning materials, besides being highly valued by
the students, was also valued by academic staff, as it enabled a closer examination and
refinement of the course structure, and also for maintaining the quality of instruction. A
typical medium of content is print, although in many cases interactive forms of content are

also included (Simpson, 2003). In the context of workplace learning and adult learning,
practitioner-produced content is currently the most popular mode of production; the new
mode of learner-produced content is, however, also gaining popularity (De Freitas, 2007).
Learner production of content is particularly interesting, as it opens several new possibilities,
particularly in terms of adjusting to the needs of learners (De Freitas, 2007). Still, in order for
learner-produced content to achieve its full potential, further research — primarily related
to the challenges of adopting learner-produced content in formal educational settings — is
required (De Freitas, 2007).
Factors in distance education delivery

A large number of studies conducted empirical investigation into the differences between
various modes of distance education. These studies aimed at understanding what factors
affect the success of students in distance education courses. Machtmes and Asher (2000)
looked at second-order studies and found that the effectiveness of distance education
was increasing over time. The effect size (expressed as a standard deviation) of –0.09 in
favor of traditional instruction was observed during the 1960s, while during the 1990s, an
effect of 0.23 in favor of distance education was observed. The likely reason for this is the
overall maturation of the distance education field, together with the maturation of distance
education pedagogies, modes of instruction, and the introduction of new and more flexible
communication technologies.
Domain-related factors
Among the many factors affecting the success of distance education, particular characteristics
of a given domain are shown to be highly influential. Using a sample of 25 studies with a
total population of 2,702 students, a meta-analysis by Williams (2006) found that, despite
being overall more effective, distance education was far more effective for adult profes-
sional learners than for graduate and undergraduate students. With an overall effect size of
Cohen’s d = 0.74, Williams (2006) showed that adult professional learners had significantly
higher academic achievement than students in traditional classrooms. This aligns with the
findings of Machtmes and Asher (2000), who showed that the largest significant and positive
effects of distance education use were on workplace learning (i.e., 0.53 standard deviation).
In a similar manner, a study by Allen et al. (2004) showed that the effectiveness of distance
education is dependent on course content, although with slightly conflicting findings. For

military-related courses, distance education turned out to be less successful than traditional
models, although the sample of three studies with a total of 210 students was rather small
to draw any general conclusions. For natural sciences, Allen et al.’s (2004) study found no
significant differences between traditional and distance education modes (the aggregated
sample of three studies was 833 students), while for social sciences the effect was small
(average r = 0.075 over nine studies with 680 students). For education courses, the effect
was negative, but very close to zero (average r = −0.021, over 13 studies totaling 1,828
students). The largest effect in favor of distance education was achieved for foreign language
courses (average r = 0.218, over the three studies with 2,238 students). The primary reason
was that in the distance education mode, students were able to converse regularly with
native speakers, which enabled them to achieve better outcomes overall. However, a study
by Bernard, Abrami, Lou, et al. (2004) shows different results. According to that study, the
best results were achieved in business, military, or computing courses while math, science,
and engineering courses benefited more from face-to-face classroom instruction. The likely
reasons for the differences across all these second-order studies relate to generally small
effect sizes that are hard to estimate reliably over the small number of studies, particularly
given the impact of slightly different study contexts and operationalizations.
Instructional factors
One important, extensively analyzed aspect in second-order studies is the role of commu-
nication technology in the success of distance education courses. Machtmes and Asher’s
(2000) second-order study shows no significant differences arising from the use of different
communication technologies. This finding should be taken cautiously, as technology changed
dramatically during the analyzed period. Similar results are reported by Allen et al. (2004),
who, using a sample of 39 studies with 71,731 students, found no statistical differences
between written and audio communication. This aligns with the findings of Bernard, Abrami,
Wade, et al. (2004), who showed that pedagogy, rather than the adopted technology, played
a dominant role in the effectiveness of distance education.
It is equally important to understand what role different modes of course delivery (i.e.,
asynchronous, synchronous, or classroom) have on overall student performance. The results
are still not conclusive, although it seems likely that asynchronous delivery is superior to
traditional classroom delivery, which in turn is more effective than synchronous distance
education delivery. Based on a sample of 318 studies totaling 57,775 students, Bernard,
Abrami, Lou, et al. (2004) found significantly negative results for synchronous distance
education (Hedges’ g = −0.10 over 92 studies totaling 8,677 students), with significantly

positive results (Hedges g =.053 over 174 studies totaling 36,531 students) for asynchronous
distance education delivery. Similar results are reported by Lou et al. (2006), who analyzed
a sample of 58 studies of synchronous distance education in which the effect size slightly
favored traditional classroom instruction (Hedges’ g = −0.023), while a sample of 122 studies
of asynchronous delivery showed an effect size in favor of distance delivery (Hedges’ g =
−0.058). A study by Allen et al. (2004) found slight, but significantly larger effect sizes for both
asynchronous (average r = 0.074 over 10 studies totaling 1,319 students) and synchronous
modes of distance education (average r = 0.066 for 27 studies totaling 6,847 students), with
the asynchronous mode having a slightly larger effect size. In contrast, based on a sample
of 18 synchronous and 12 asynchronous studies, Williams (2006) showed that synchronous
distance education had a positive effect (Cohen’s d = 0.24), while asynchronous distance
education showed a negative effect (Cohen’s d = 0.06).
Based on the current evidence, it seems that both synchronous and asynchronous distance
education have the potential to be as effective as traditional classroom instruction (or
better). However, this might not be the case in the actual practice of distance education. To
improve this situation, Bernard, Abrami, Wade, et al. (2004) suggest including more person-
alized contact between students and instructors to make synchronous distance education
instruction similar to that commonly used in face-to-face education. On the other hand, for
the asynchronous mode of distance education, the use of problem-based learning shows
positive affects in both achievement and attitude outcomes (Bernard, Abrami, Wade, et al.,
2004). The results of the Lou et al. (2006) meta-analysis also indicate that for asynchronous
distance education, the use of media to enable collaborative learning among students was
the most effective (Hedges’ g = 0.11), while media that supports individual learning only
(e.g., student-content) was not significantly different from classroom instruction.
Alongside the study of different modes of distance education delivery is the investigation of
different types of interactions (i.e., student-student, student-content, and student-teacher)
and the role that different media play in supporting these interactions. Equivalency theory
states that different combinations of the three types of interactions can be equally effec-
tive for achieving learning outcomes, providing that instructors have the required space to
organize their pedagogical approaches according to the needs of a particular situation or
student (Miyazoe & Anderson, 2010). The Bernard et al. (2009) meta-analysis found that all
three forms of interaction produced positive effect sizes on academic performance, with
student-student and student-content interactions having higher effect sizes (Hedges’ g = 0.49
over 10 studies for student-student and Hedges’ g = 0.46 over 20 studies for student-content
interaction) than student-teacher interactions (Hedges’ g = 0.32 over 44 studies).

To foster quality interactions between students, an analysis of the role of instructional
design and instructional interventions planning is essential. In this regard, the study by
Williams (2006) showed that courses that incorporate three or more components of in-
teraction design (e.g., interaction, integration, innovation, introspection) were associated
with larger positive effects (Cohen’s d = 0.25 over 22 studies). In contrast, courses with
fewer than three components had negative effects (Cohen’s d = −0.09 over 12 studies),
which is indicative of the importance of proper instructional design on the effectiveness
of distance education courses. Similarly, a meta-analysis by Borokhovski, Tamim, Bernard,
Abrami, and Sokolovskaya (2012) of 32 studies showed that the planning of interactions
is equally important, with designed and planned interactions leading to higher academic
performance (Hedges’ g = 0.50 across 14 studies) than contextual interactions (Hedges’ g
= 0.22 across 22 studies).
Likewise, appropriate academic support is recognized to have an important effects on

student academic success; however, support is often limited to financial and technological
services only (Tuquero, 2011). In order to provide sufficient academic support, understanding
stakeholder needs is a main prerequisite alongside the understanding of student attrition
(Tuquero, 2011). A meta-analysis by Storrings (2006), conducted on 30 primary sources
with 9,769 students, found no significant characteristics predictive of attrition in distance
education. When compared to traditional classroom courses, Bernard, Abrami, Wade, et
al. (2004) showed that asynchronous distance courses had significantly higher attrition rates
(Hedges’ g = 0.093 over 53 studies), while the difference for synchronous courses was not
significant (Hedges’ g = 0.005 over 17 studies). Finally, the use of novel distance education
technologies, such as 3-D virtual worlds, has also been investigated and shows promising
results that require more empirical investigation (Wang, & Lockee, 2010).

Limitations
There are several limitations related to this study. First, there is always a possibility that
the selected studies are not representative of the whole body of empirical research in the
distance education domain. We performed a search across the main digital libraries in the
field, and we inspected the studies based on the inclusion criteria, as is commonly done in
systematic reviews and meta-analytic studies. However, there is always the possibility that
some important studies were not included, which would negatively reflect on the results
presented in this report. The second challenge is related to the operationalization of distance
education, e-learning, and online learning, as the borderline between them is not clear.
Likewise, second-order studies included in this report do not always provide sufficient detail
to classify each paper with the absolute certainty. We adopted definitions commonly used in
the research literature, but that does not fully eliminate the problem. Finally, given that any
tertiary study is limited by the quality of data reported in the secondary sources, this study is
dependent on the methodological qualities of those secondary sources. Overall, the quality
is very high for most, with only few studies having a noticeably lower quality of reporting.

Model of
Distance Education
Figure 9 shows the conceptual model that concisely synthesizes the findings of the present
study. At centre is distance education, which — when properly organized and supported
— is associated with reduced costs of education, and an increase in student retention and
effectiveness. Primary elements of distance education are learners, content, and instruc-
tors. Learning experience is primarily shaped by the interaction of learners with content,
other learners, and instructors. In order to successfully engage in interactions, learners are
required to possess high levels of digital literacy, to be self-efficient and properly motivated
to productively engage in learning activities. Likewise, it is instructors’ attitude towards
technology use and their levels of digital literacy play an important role in shaping overall
learning experience. Instructors should also pay a special attention to planning and designing
course interactions, given the evidence of its advantages over contextualized interactions.
The quality of learning content is also important, particularly in formal educational settings,
where standards of learning quality are of particular importance.
In addition to role of learners, instructors, and content, our findings indicate that other
factors — such as academic support, institutional adoption, and course design — play an
important moderating role on the final learning experience and achievement of learning
objectives. Important course design characteristics that shape learning experience are
flexibility, personalization, forms of assessment, use of small group learning and designed
interactions, and soundness of adopted mix of pedagogies, technologies, and media. Like-
wise, factors related to the level of institutional adoption of distance education include the
quality of technological infrastructure, support for academic staff, role of academic man-
agement, level of coordination between involved parties, and governmental support and
policy development. Finally, academic support for students — including technological and
financial support — is particularly important for students that do not possess required levels
of literacy and self-efficiency, and for understanding the reasons behind student attrition.
42 Model of Distance Education

oration
ab
ll
co
peer
tion interac
ac
Learner
r
tio
inte
• High Digital literacy
n
• Self-efficient
• Motivated
Content Instructors
• High quality learning materials • Positive attitudes towards
• Tailored to the students needs technology
• Maintains quality of teaching • High digital literacy
and learning • Plan course interactions
Distance
Education
Course Design • High Effectiveness
• Reduced costs
• Asynchronous vs Synchronous
• High Retention
• Flexible
• Adjusted to learners needs Academic Support
• Assessment
• Enable for collaborative • Technological
learning • Financial
• Small group interactions • Understanding attrition factors
• Designed interactions
• Media that supports
pedagogy
• Use of LMS Institutional Adoption
• Technological Infrastructure
• Support for academic staff
• Active role of academic management
• Coordinated actions
• Governmental support and policies
Figure 9 Conceptual model of distance education
Model of Distance Education 43

Summary
This report looked at published systematic literature reviews and meta-analyses in order
to provide an overview of the current state of distance education research. The primary
goal of this review is to inform the development and operationalization of digital learning
as a new construct to capture modern developments in the field of education. Our review
showed that:
1. Asynchronous forms of distance education received much more attention

than synchronous or mixed modes of education delivery.
2. The primary focus of research is exploratory, with all forms of research

design (i.e., quantitative, qualitative, and mixed) being widely used. In terms
of particular methodologies, case studies and survey-based research
represent some of those more commonly used.
3. There is strong evidence for the effectiveness of distance education, with

several studies pointing to the comparable, or better effectiveness of
distance education.
4. The early focus on comparing distance and traditional classroom instruction

that characterized early reviews made room for a more proactive analysis of
different factors of distance education.
5. Student satisfaction with distance education courses depends primarily on

the target student population, with professional and adult learners typically
being more satisfied with distance education, while undergraduate and
graduate students were slightly more satisfied with traditional face-to-face
learning.
6. Attrition between modes of instruction is on a comparable level, with

synchronous distance education having lower levels of attrition than
traditional face-to-face education, while asynchronous distance education
has slightly higher levels of attrition than traditional face-to-face education.
44 Summary
7. Institutional adoption of distance education depends on a multitude of
macro- and micro-level factors. Among different macro-level factors, the
most common include i) technological infrastructure, ii) the role of academic
management, and iii) academic and institutional support. Similarly, the most
important micro-level factors include i) views on distance education
technology use, ii) computing literacy of academic staff, iii) allocation
of time for technology-related academic staff training and distance
education course development, and iv) quality of learning materials.
8. The success of individual distance education courses is related to many

domain and instructional factors such as i) course subject area, ii) target
student group (e.g., professional, adult, graduate, undergraduate),
iii) conformity between adopted pedagogical approaches and the
technologies that enable them, iv) mode of course delivery (i.e.,
synchronous vs. asynchronous), v) type of interactions supported (i.e.,
student-student, student-instructor, and student-content), and
vi) instructional design and planning.
9. There is much room for improvement in terms of the methodological quality

of published distance education literature. There is a lot of missing
information in published reports, which makes the synthesis of research
findings much more complex and challenging.
With the rising diversity of forms of technology use in modern education, this tertiary
study overview presents a synthesis for informing the development of digital learning as
a new umbrella term for the 21st-century use of technology in education. While distance
education has been the most widely used term, the increasing diversity of educational
programs, learning personalization, and modes of assessment merits the development of
a more comprehensive and unified construct. The challenges directly targeted include the
development of a framework for the successful adoption of novel educational technology
and the development of novel educational programs, the identification of successful edu-
cational practices for different learning scenarios, the provisions for the further adoption
of technology in education, and the development of better connections between educa-
tional research and practice. With the rising diversity of forms of learning (e.g., MOOCs or
blended training programs), research on digital learning and its main characteristics can be
successfully used to advance the current state of education, which is a necessity for coping
with the challenges of the future.
Summary 45
References
Allen, I. E., & Seaman, J. (2011). Going the distance: Online education in the United
States, 2011 (Survey Report). Babson Survey Research Group. Retrieved from
http://www.onlinelearningsurvey.com/reports/goingthedistance.pdf
Allen, I. E., Seaman, J., & Garrett, R. (2007). Blending in: The extent and promise of
blended education in the United States. Sloan Consortium. Retrieved from
http://eric.ed.gov/?id=ED529930
Allen, M., Bourhis, J., Burrell, N., & Mabry, E. (2002). Comparing student satisfaction
with distance education to traditional classrooms in higher education: A meta-
analysis. American Journal of Distance Education, 16(2), 83–97. doi:10.1207/
S15389286AJDE1602_3
Allen, M., Mabry, E., Mattrey, M., Bourhis, J., Titsworth, S., & Burrell, N. (2004).
Evaluating the effectiveness of distance learning: A comparison
using meta-analysis. Journal of Communication, 54(3), 402–420.
doi:10.1111/j.1460-2466.2004.tb02636.x
Anderson, T. (2003). Modes of interaction in distance education: Recent developments

and research questions. In M. G. Moore & W. G. Anderson (Eds.), Handbook of
Distance Education (2nd revised edition, pp. 129–144). Mahwah, NJ: Routledge.
Anderson, T. (2008). The theory and practice of online learning. Athabasca, AB, Canada:
Athabasca University Press.
Anderson, T. (2009). A rose by any other name: Still distance education – A response to
D. R. Garrison implications of online and blended learning for the conceptual
development and practice of distance education. International Journal of
E-Learning & Distance Education, 23(3), 111–116. Retrieved from http://ijede.
ca/index.php/jde/article/view/653/981
46 References
Anderson, T., & Dron, J. (2010). Three generations of distance education pedagogy. The
International Review of Research in Open and Distance Learning, 12(3), 80–97.
Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/890/1663
Andrews, R. (2005). The place of systematic reviews in education research. British Journal
of Educational Studies, 53(4), 399–416. doi: 10.1111/j.1467-8527.2005.00303.x
Bates, A. W. (2005). Technology, e-learning and distance education (2nd edition).

London; New York: Routledge.
Bernard, R., Borokhovski, E., Schmid, R., Tamim, R., & Abrami, P. (2014). A meta-analysis
of blended learning and technology use in higher education: From the general
to the applied. Journal of Computing in Higher Education, 26(1), 87–122.
doi:10.1007/s12528-013-9077-3
Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A.,
& Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments
in distance education. Review of Educational Research, 79(3), 1243–1289.
Retrieved from http://www.jstor.org/stable/40469094
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovsk, E., Wade, A., Wozney, L., … Huang,
B. (2004). How does distance education compare with classroom instruction? A
meta-analysis of the empirical literature. Review of Educational Research, 74(3),
379–439. doi: 10.3102/00346543074003379
Bernard, R. M., Abrami, P. C., Wade, A., Borokhovski, E., & Lou, Y. (2004). The effects
of synchronous and asynchronous distance education: A meta-analytical
assessment of Simonson’s “equivalency theory.” Annual proceedings of selected
research and development papers presented at the national convention of
the Association for Educational Communications and Technology. Chicago, IL:
Association for Educational Communications and Technology. Retrieved from
Blin, F., & Munro, M. (2008). Why hasn’t technology disrupted academics’ teaching
practices? Understanding resistance to change through the lens of
activity theory. Computers & Education, 50(2), 475–490. doi:10.1016/j.
compedu.2007.09.017
References 47
Bliuc, A.-M., Goodyear, P., & Ellis, R. A. (2007). Research focus and methodological
choices in studies into students’ experiences of blended learning in higher
education. The Internet and Higher Education, 10(4), 231–244. doi:10.1016/j.
iheduc.2007.08.001
Bonk, C. J., Graham, C. R., Cross, J., & Moore, M. G. (2005). The handbook of blended
learning: Global perspectives, local designs (1st edition). San Francisco: Pfeiffer.
Booth, A., Carroll, C., Papaioannou, D., Sutton, A., & Wong, R. (2009). Applying findings
from a systematic review of workplace-based e-learning: Implications for health
information professionals. Health Information & Libraries Journal, 26(1), 4–21.
doi:10.1111/j.1471-1842.2008.00834.x
Borokhovski, E., Bernard, R., Mills, E., Abrami, P. C., Wade, C. A., Tamim, R., … Surkes,
M. A. (2011). An extended systematic review of Canadian policy documents
on e-learning: What we’re doing and not doing. Canadian Journal of Learning
and Technology, 37(3). Retrieved from http://www.cjlt.ca/index.php/cjlt/article/
view/589
Borokhovski, E., Tamim, R., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012).
Are contextual and designed student-student interaction treatments equally
effective in distance education? Distance Education, 33(3), 311–329. doi:10.108
0/01587919.2012.723162
Childs, S., Blenkinsopp, E., Hall, A., & Walton, G. (2005). Effective e-learning for health
professionals and students: Barriers and their solutions. A systematic review of
the literature: Findings from the HeXL project. Health Information & Libraries
Journal, 22, 20–32. doi:10.1111/j.1470-3327.2005.00614.x
Chipps, J., Brysiewicz, P., & Mars, M. (2012). A Systematic review of the effectiveness
of videoconference-based tele-education for medical and nursing education.
Worldviews on Evidence-Based Nursing, 9(2), 78–87. doi:10.1111/j.1741-
6787.2012.00241.x
Chipps, J., Ramlall, S., & Mars, M. (2012). Videoconference-based education for
psychiatry registrars at the University of KwaZulu-Natal, South Africa. African
Journal of Psychiatry, 15(4). doi:10.4314/ajpsy.v15i4.32
48 References
Clardy, A. (2009). Distant, on-line education: Effects, principles and practices. ERIC
Document ED506182. Retrieved from http://eric.ed.gov/?id=ED506182
Clark, R. E. (1983). Reconsidering research on learning from media. Review of

Educational Research, 53(4), 445–459. doi:10.3102/00346543053004445
Cohen, J. (1988). The analysis of variance. Statistical power analysis for the behavioral
sciences (pp. 273–406). Hillsdale, NJ: L. Erlbaum Associates.
Daniel, J. (2014). Foreword to the special section on massive open online courses
MOOCs: Evolution or revolution? Journal of Online Learning and Teaching,
10(1), i–iv. http://jolt.merlot.org/vol10no1/daniel_foreword_0314.pdf
De Freitas, S. (2007). Post-16 e-learning content production: A synthesis of the literature.

British Journal of Educational Technology, 38(2), 349–364. doi:10.1111/j.1467-
8535.2006.00632.x
Dewey, J. (1897). My pedagogical creed. School Journal, 54(3), 77–80. Retrieved from
http://dewey.pragmatism.org/creed.htm
Friedman, T. L. (2012, May 15). Come the revolution. The New York Times. Retrieved
from http://www.nytimes.com/2012/05/16/opinion/friedman-come-the-
revolution.html
Garrison, D. R. (1985). Three generations of technological innovations in distance

education. Distance Education, 6(2), 235–241. doi:10.1080/0158791850060208
Gašević, D., Kovanović, V., Joksimović, S., & Siemens, G. (2014). Where is research on
massive open online courses headed? A data analysis of the MOOC research
initiative. The International Review of Research in Open and Distance Learning,
15(5), 134–176. Retrieved from http://www.irrodl.org/index.php/irrodl/article/
view/1954
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational

Researcher, 5(10), 3–8. doi:10.2307/1174772
References 49
Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types
and associated methodologies. Health Information and Libraries Journal, 26(2),
91–108. doi:10.1111/j.1471-1842.2009.00848.x
Hanna, D. E. (2003). Organizational models in higher education, past and future. In M. G.

Moore & W. G. Anderson (Eds.), Handbook of Distance Education (2nd revised
edition). Mahwah, NJ: Routledge.
Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning.

The Internet and Higher Education, 3(1–2), 41–61. doi:10.1016/S1096-
7516(00)00032-4
Hedges, L. V. (1982). Statistical methodology in meta-analysis. Retrieved from http://eric.

ed.gov/?id=ED227133
Holmberg, B. (2005). The evolution, principles and practices of distance education.

Oldenburg: Bibliotheks und Informationssystem der Univ.
Hrastinski, S., & Keller, C. (2007). Computer-mediated communication in education:

A review of recent research. Educational Media International, 44(1), 61–77.
doi:10.1080/09523980600922746
Jonassen, D., Davidson, M., Collins, M., Campbell, J., & Haag, B. B. (1995).
Constructivism and computer-mediated communication in distance
education. American Journal of Distance Education, 9(2), 7–26.
doi:10.1080/08923649509526885
Keegan, D. (Ed.). (1993). Theoretical principles of distance education (1st edition).

London; New York: Routledge.
Kovanović, V., Joksimović, S., Gašević, D., Siemens, G., & Hatala, M. (2014). What public
media reveals about MOOCs: A systematic analysis of news reports. British
Journal of Educational Technology, in press.
Lee, S. A. S., & McElroy, S. (2012). Telepractice is a new method for providing services to
children with Autism Spectrum Disorder (ASD): This scoping review summarizes
existing research and identifies research gaps. Evidence-Based Communication
Assessment and Intervention, 6(4), 177–180. doi:10.1080/17489539.2013.780377
50 References
Lee, Y., Driscoll, M. P., & Nelson, D. W. (2004). The past, present, and future of research
in distance education: Results of a content analysis. American Journal of
Distance Education, 18(4), 225–241. doi:10.1207/s15389286ajde1804_4
Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and pedagogy in undergraduate
distance education: A theory-based meta-analysis of empirical literature.
Educational Technology Research and Development, 54(2), 141–176.
doi:10.1007/s11423-006-8252-x
Ludlow, B. L., & Brannan, S. A. (1999). Distance education programs preparing personnel
for rural areas: Current practices, emerging trends, and future directions. Rural
Special Education Quarterly, 18(3), 4–15.
Lust, G., Juarez Collazo, N. A., Elen, J., & Clarebout, G. (2012). Content management
systems: Enriched learning opportunities for all? Computers in Human Behavior,
28(3), 795–808. doi:10.1016/j.chb.2011.12.009
Machtmes, K., & Asher, J. W. (2000). A meta-analysis of the effectiveness of telecourses

in distance education. American Journal of Distance Education, 14(1), 27–46.
doi: 10.1080/08923640009527043
Mason, R. (2000). From distance education to online education. The Internet and Higher
Education, 3(1–2), 63–74. doi:http://dx.doi.org/10.1016/S1096-7516(00)00033-6
Means, B., Toyama, Y., Murphy, R. O. B. E. R. T., & Baki, M. (2013). The effectiveness
of online and blended learning: A meta-analysis of the empirical literature.
Teachers College Record, 115(3), 1-47. Retrieved from https://www.tcrecord.
org/library/Abstract.asp?ContentId=16882
Miyazoe, T., & Anderson, T. (2010). The interaction equivalency theorem. Journal of
Interactive Online Learning, 9(2), 94–104. Retrieved from http://www.ncolr.org/
issues/jiol/v9/n2/the-interaction-equivalency-theorem#.VPNUgnXcdhE
Moore, J. L., Dickson-Deane, C., & Galyen, K. (2011). e-Learning, online learning, and
distance learning environments: Are they the same? The Internet and Higher
Education, 14(2), 129–135. doi:10.1016/j.iheduc.2010.10.001
References 51
Moore, M. G. (1989). Editorial: Three types of interaction. American Journal of Distance
Education, 3(2), 1–7. doi:10.1080/08923648909526659
Moore, M. G., & Kearsley, G. (2004). Distance education: A systems view (2nd edition).
Belmont, CA: Wadsworth Publishing.
Mulrow, C. D. (1994). Systematic Reviews: Rationale for systematic reviews. British

Medical Journal, 309(6954), 597–599. doi: 10.1136/bmj.309.6954.597
Piaget, J. (1959). The language and thought of the child. (3rd edition). London:
Routledge and Kegan Paul.
Ritzhaupt, A. D., Stewart, M., Smith, P., & Barron, A. E. (2010). An investigation of
distance education in North American research literature using co-word analysis.
The International Review of Research in Open and Distance Learning, 11(1),
37–60. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/763
Rumrill, P. D., Fitzgerald, S. M., & Merchant, W. R. (2010). Using scoping literature
reviews as a means of understanding and interpreting existing literature.
Work: A Journal of Prevention, Assessment and Rehabilitation, 35(3), 399–404.
doi:10.3233/WOR-2010-0998
Shachar, M., & Neumann, Y. (2003). Differences between traditional and distance
education academic performances: A meta-analytic approach. The International
Review of Research in Open and Distance Learning, 4(2). Retrieved from ttp://
www.irrodl.org/index.php/irrodl/article/view/153
Siemens, G. (2012). MOOCs are really a platform. ELearnSpace Blog. Retrieved from
http://www.elearnspace.org/blog/2012/07/25/moocs-are-really-a-platform/
Simpson, M. G. (2003). Distance delivery of pre-service teacher education: Lessons

for good practice from twenty-one international programs (D.Ed.). The
Pennsylvania State University, United States.
Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of
eLearning. Education + Training, 56(2/3), 105–121. doi:10.1108/ET-11-2012-0123
52 References
Spector, J. M., Merrill, M. D., van Merrienboer, J., & Driscoll, M. P. (Eds.). (2007).
Handbook of research on educational communications and technology (3rd
edition). New York: Routledge.
Stall-Meadows, C. E. (1998). Grounded meta-analysis of qualitative case study

dissertations in distance education pedagogy (Ed.D.). Oklahoma State
University, United States.
Stewart, C. (2010). What’s missing? The next step towards universal distance education.
Proceedings of 26th Annual Conference on Distance Teaching & Learning.
Madison, WI: University of Wisconsin–Madison. Retrieved from http://www.
uwex.edu/disted/conference/resource_library/search_detail.cfm?presid=29845
Storrings, D. A. (2006). Attrition in distance education: A meta-analysis. Annual

Proceedings of Selected Research and Development Papers Presented at
the National Convention of the Association for Educational Communications
and Technology. Dallas, TX: Association for Educational Communications and
Technology. Retrieved from http://www.aect.org/pdf/proccedings/2006/06_35.pdf
Taylor, J. C. (2001). Fifth generation distance education. Instructional Science and

Technology, 4(1), 1–14. Retrieved from http://eprints.usq.edu.au/136/
Toffler, A., & Butz, B. (1990). Powershift: Knowledge, wealth, and violence at the edge of
the 21st century (pp. 341-3). New York: Bantam Books.
Tomlinson, J., Shaw, T., Munro, A., Johnson, R., Madden, D. L., Phillips, R., & McGregor,
D. (2013). How does tele-learning compare with other forms of education
delivery? A systematic review of tele-learning educational outcomes for
health professionals. New South Wales Public Health Bulletin, 24(2), 70–75.
doi:10.1071/NB12076
Tuquero, J. M. (2011). A meta-ethnographic synthesis of support services in distance

learning programs. Journal of Information Technology Education, 10, 157–
179. Retrieved from http://www.jite.org/documents/Vol10/JITEv10IIPp157-
179Tuquero974.pdf
Vygotsky, L. S. (1978). Mind in society: Development of higher psychological processes.

Cambridge, MA: Harvard University Press.
References 53
Waight, C. L., Willging, P. A., & Wentling, T. L. (2002). Recurrent themes in e-learning: A
meta-analysis of major e-learning reports. Proceedings AHRD 2002 Conference.
Honolulu, Hawaii, USA. Retrieved from http://eric.ed.gov/?id=EJ875539
Wang, F., & Lockee, B. B. (2010). Virtual worlds in distance education: A content analysis
study. Quarterly Review of Distance Education, 11(3), 183–186. Retrieved from
http://www.editlib.org/p/53214/
Williams, S. L. (2006). The effectiveness of distance education in allied health science

programs: A meta-analysis of outcomes. American Journal of Distance
Education, 20(3), 127–141. doi:10.1207/s15389286ajde2003_2
Wutoh, R., Boren, S. A., & Balas, E. A. (2004). eLearning: A review of Internet-based
continuing medical education. Journal of Continuing Education in the Health
Professions, 24(1), 20–30. doi:10.1002/chp.1340240105
Zawacki-Richter, O., Baecker, E. M., & Vogt, S. (2009). Review of distance education
research (2000 to 2008): Analysis of research areas, methods, and authorship
patterns. The International Review of Research in Open and Distance Learning,
10(6), 21–50. Retrieved from http://www.irrodl.org/index.php/irrodl/article/
view/741/1433
54 References
of Blended Learning
Oleksandra Skrypnyk
Srećko Joksimović
Vitomir Kovanović
Shane Dawson
Dragan Gašević
George Siemens
55

Abstract
This report forms one part in a series of articles offering

an overview of the state of distance, online, and blended
learning, and positioning them in relation to the emerging
domain of digital learning. This particular report focuses on
blended learning (BL), referring to the practices that combine
(or blend) traditional face-to-face (f2f) learning with online
learning (OL). As the concept of BL continues to gain trac-
tion in educational settings, researchers are attempting to
establish and verify the learning gains it brings. This report
seeks to outline the debate regarding BL definitions, ped-
agogical benefits, and deficiencies that arise in academic
studies, and reflect on the future direction for BL. Our critical
overview of the state and development of BL is structured
to reflect the dominant themes of twenty systematically
selected second-order academic studies of BL. This report
reviews main findings around such dominant themes as the
effectiveness of BL, recommended instructional practices in
BL delivery and design, as well as the state of research into
BL. The findings suggest that advances in technology have
fueled the development of BL from a grassroots practice
to an emerging research field. The implementation of BL
practices by including both online and f2f modes of delivery
positively influence student performance, making BL an at-
tractive educational provision. At present, the field of BL is
still dependent on the modes of delivery it is derived from,
drawing heavily on OL in both theory and in practice. The
field of BL is a dynamically changing area, and much of the
critique of the existing research noted here is likely to be
rapidly addressed in future work. That being said, a critical
overview of the field suggests that it can further mature by
adopting a digital learning perspective in its own activities.
In the traditional educational context, face-to-face (f2f)
introduction
classes refer to the cohorts of students able to commit to

on-campus instruction presented in conventional brick and
mortar classrooms. This model of education requires that
students be present for regular in-classroom instruction. As
such, the model can be seen as inequitable for those who
may have competing demands and priorities that make
regularly scheduled on-campus attendance difficult, if not
impossible.
In contrast, distance education (DE) represents a model that

affords students an opportunity to undertake programs of
study external to the institution. In essence, this model of
education places no requirements on students to be on-cam-
pus for course instruction, content, or study. This model
is not a recent introduction to the sector; DE has a long
history that evolved from specialist colleges and boutique
programs and courses (see Kovanović et al. (2015) in this
58 Introduction
the history and state of blended learning
series of papers). Despite a long-established record of offerings, DE has been stymied by
the perception that this mode of education delivery is less personal, and primarily caters
to working adults or marginal groups separated by both time and space (Moore & Kears-
ley, 2011). However, the recent growth and sophistication of Internet-based technologies
has seen distance education or online learning (OL) become a staple of post-secondary
and secondary instruction. This is well noted by Allen and Seamen (2013), who have been
tracking online course enrolment across the United States for over a decade. The authors
identified that the percentage growth in enrolment for online learning courses is now in-
creasing faster than for on-campus offerings.
Clearly, the development and adoption of asynchronous and synchronous communication

technologies into the learning space has provided an opportunity to leverage the affordances
of both education models (DE and f2f classes). At present, courses and programs can be
offered with flexibility for access and attendance while retaining opportunities for social
engagement and interaction. This report discusses the ongoing juxtaposition of traditional
modes of education with distance learning in what has been termed blended learning (BL),
or hybrid learning. We outline the debate regarding blended learning definitions, along with
the perceived pedagogical benefits and deficiencies that arise through the convergence of
technologies with f2f education as documented in the existing research literature.
Introduction 59
The rise of education
technologies
Over the past decade, technology has increasingly been used to enhance course and content
offerings both in the f2f and distance education settings. Technology for learning can be
divided into three broad categories: i) information technologies that support the delivery of
and access to information; ii) communication and interactive technologies that mediate user
interaction; and iii) social software technologies that support group-based activities such as
decision-making, planning, and higher order learning activities (C. Allen, 2004; Anderson,
2008; Hulsmann, 2004). While the initial information technologies adopted in education were
computer-based and for personal use, educational IT is now predominately located on the
web (or cloud) and is more socially oriented. This transition to the web or cloud provides for
greater, more diverse access to learning resources and effectively capitalizes on the growth
in web-based connectivity for end-users. Communication technologies, initially developed
as independent software for synchronous or asynchronous communication, also applied in
education, have recently been merged into common software platforms for learning, such
as learning management systems (LMS) or virtual learning environments (VLE). Web 2.0 tools
and social software technologies (C. Allen, 2004) have enabled two-way communication, as
well as sharing, extracting, and organizing knowledge, along with building social relationships
(Anderson, 2008). Due to their affordances for interactivity, social software technologies allow
distance learners to be exposed to group-based learning activities previously perceived as
exclusive to f2f teaching contexts. For example, students can communicate synchronously
through video conferencing, synchronous chat, or virtual classrooms without the need for
physical co-location (Helms, 2014).
The increased reliance on technology for education delivery and instruction has resulted in
changing perceptions of what now constitutes distance. Students can interact in real-time
in peer-groups or with instructors and even participate in lectures remotely. As such, the
reliance on online technology, alongside the diminishing concept of distance, has given
rise to the term online learning (OL). The adoption of smart devices, the wider use of the
Internet, and the gradual lowering cost of technology (Rainie, 2010) have all played a part
in redefining learning and teaching practice in the 21st century.
60 The Rise of Education Technologies

Defining
Blended Learning
A positive outcome stemming from the growth in OL has been the capacity for educators
to leverage these new technologies for on-campus teaching. This combination of online
technologies and f2f instruction has been described in the literature as blended learning,
mixed mode, hybrid, or online-supplemented. While blended learning is perhaps the most
commonly used phrase, essentially all these terms describe the merging of online technol-
ogies with f2f teaching.
This report focuses on blended learning (BL), specifically the practices that combine (or
blend) traditional f2f with OL. BL or hybrid practices represent a continual convergence
between traditional f2f and distributed, technology-mediated learning environments
(Bonk & Graham, 2006; Graham & Dziuban, 2008). Its variance as a changing practice can
be located on a continuum between fully f2f to fully online courses (Helms, 2014). Given
this broad spectrum, it is not surprising that BL has multiple and varied definitions. For
instance, BL can be defined as a version of OL, where 30% – 79% of the content is deliv-
ered in an online format (Allen & Seaman, 2003, 2004; Allen, Seaman, & Garrett, 2007;
Means, Toyama, Murphy, Bakia, & Jones, 2009; Means, Toyama, Murphy, & Bakia, 2013).
Some researchers limit BL practices to those where f2f instruction comprises at least half of
all class time (Bernard, Borokhovski, Schmid, Tamim, & Abrami, 2014). Others expand BL
practices to include web-facilitated classroom instruction where the LMS is used for syllabi
and course communication (Bliuc, Goodyear, & Ellis, 2007). In other words, any instructor
who employs technology in his/her teaching practice, whether in f2f or through web-based
distance education practices in online education (DE/OL), could legitimately refer to such
experiences as blended.
Defining Blended Learning 61

Although the discussion in the literature regarding the development of an authoritative
definition is still open, there are identifiable commonalities in the many attempts to define
BL. In all definitions, BL is considered a combination of traditional f2f modes of instruction
with online modes of learning (OL), drawing on technology-mediated instruction, where
all participants in the learning process are separated by distance some of the time. The
distinctions between mixed-mode, hybrid, and blended courses are not well articulated, and
the terms are often used interchangeably (Graham & Dziuban, 2008; Means et al., 2013).
Arguably, hybrid implies that one mode is unused while the other is used; while blended
suggests no perceptible difference between modes (McGee & Reis, 2012).
As the concept of BL continues to gain traction in educational settings, researchers are also
attempting to establish and verify the touted learning gains and benefits associated with
this model of education. Although technology positivists frequently espouse significant
learning gains when adopting educational technologies, the realities of such claims are
often difficult to measure. This report seeks to highlight the perceived benefits alongside
the reported deficiencies or gaps as currently reported in the research.
62 Defining Blended Learning

Methods
This report provides an overview of the state of evidence-based findings for BL approaches
by synthesizing the themes and findings from the meta-analyses and systematic literature
reviews on BL. Such a line of inquiry has been framed by the following research questions:
RQ1. What are the main themes of BL meta-analyses and systematic literature reviews?
RQ2. What is the state of BL as reflected through these meta-analyses and systematic
literature reviews?
To identify meta-analyses and systematic literature reviews on BL, a database search was
combined with a Google Scholar and journal search (Figure 1). First, a list of 306 studies
that included OL, DE, and BL was identified through a search of ERIC, Scopus, PsychIN-
FO, PubMed, and ProQuest databases1. An additional 19 studies were retrieved using a
Google Scholar search for similar combinations of terms. A journal search yielded a further
14 studies2. Hence, the compiled list included some 339 meta-analyses and systematic
reviews of DE, OL, and BL. Three researchers scanned titles, keywords, and abstracts to
verify the relevance of each extracted article and to assign one or more labels to the list of
studies; specifically distance education, online learning, or blended learning. If the coder
could not label the study based on the meta-information, the content of the article was
reviewed to provide further detail and clarification. During this stage, 67 studies potentially
relevant for BL were identified.
1 The database search was conducted using the following criteria: title, abstract, and/or keywords containing at least one of the following
terms: meta-analysis, meta-synthesis, scoping study, OR systematic review, AND title, abstract, and/or keywords containing at least one of the
following terms: distance learning, distance education, blended learning, blended education, hybrid education, hybrid learning, online learning,
online education, e-learning, web-based learning, OR web-based education.
2 The list of the relevant journals was obtained from the most influential meta-analyses in distance and online education, and included
American Journal of Distance Education, Journal of Distance Education, Distance Education, International Review of Research in Distance and
Open Education, Journal of Asynchronous Learning Networks, Journal of Technology and Teacher Education, Career and Technical Education
Research, Internet and Higher Education, Journal of Computing in Higher Education, and Computers and Education.
Methods 63
Total of 339 unique

studies identifed
FOCUS OF THIS STUDY


Figure 1 The process of systematic literature search
64 Methods
As previously discussed, BL has dual roots in traditional and online modes of learning. This
evolution of BL complicated the identification of readily available meta-analyses and sys-
tematic reviews for inclusion in this report. For instance, on the one hand, especially in the
early 2000s, BL was tied to technology integration into conventional classroom instruction
(See Figure 2). The academic literature in this case, connects BL to such keywords as virtual
learning environments, course management systems, and computer-aided instruction,
among others. On the other hand, BL largely overlaps with the literature on DE/OL. In this
context, the academic literature draws on such keywords as web-based learning, e-learning,
Internet-based learning, online and distance learning/education and distributed learning.
Furthermore, practices of BL are also referred to as hybrid learning, mixed mode learning,
and more recently the flipped classroom.
4000
Keyword
blended learning
for blended learning and related keywords
computer-aided instruction
3000
computer-assisted instruction
education technology
e-learning
flipped classroom
Number of articles
learning management system

online learning
virtual learning enviroment
2000
web-based training
1000
1980 1990 2000 2010
Figure 2 Distribution of studies indexed by Scopus as related to BL provisions (query: study’s title, abstract and keywords
Methods 65
To be included in the final list for analysis, a study had to meet the following criteria:
i. the study applied a systematic approach to literature analysis, e.g., meta-analysis,

systematic literature review, and meta-synthesis
ii. the study identified findings related to blended learning as part of the dataset
iii. the study was published in a peer-reviewed journal/conference proceedings or in a
dissertation, available in English
iv. the study was situated in higher education and professional development contexts,
excluding K–12 unless part of a larger dataset embracing higher and/or adult education
Additionally, OL meta-analyses with a sub-set of BL studies, such as Means et al. (2009) or

Bernard et al. (2009), were excluded if the same academic group had conducted a more
recent and BL-focused analysis using the dataset of primary studies that overlapped with
their prior work.
A list of second-order studies selected for this report comprised 20 meta-analyses and
systematic reviews of BL (Table 1, Figure 3)
Table 1 Summary of Systematic Analyses and Meta-Analyses of Blended Learning.
Primary
Study Title Type Studies
A meta-analysis of blended learning and

1 Bernard et al. (2014) technology use in higher education: From MA 96
general to the applied
What might online delivery teach us about

Arbaugh, J.
2 blended management education? Prior SLR 60
(2014)
perspectives and future directions
A thematic analysis of the most highly cited

3 Halverson et al. (2014) scholarship in the first decade of blended SLR, TA 85
learning research
The effectiveness of online and blended

Means et al.
4 learning: A meta-analysis of the empirical MA 45
(2013)
literature
66 Methods
Table 1 (Cont.) Summary of Systematic Analyses and Meta-Analyses of Blended Learning
Primary
Bishop & Verleger The flipped classroom: A survey of the

5 SLR 24
(2013) research
Literature review on hybrid/blended

6 Zhao & Breslow (2013) SLR 42
learning
An analysis of research trends in

7 Drysdale et al. (2013) dissertations and theses studying blended SLR 205
learning
Keengewe & Kang A review of empirical research on blended

8 SLR 23
(2013) learning in teacher education programs
A literature landscape of blended learning

Torrisi-Steel & Drew in higher education: The need for better
9 SLR, TA 827
(2013) understanding of academic blended
practice
Blended course design: A synthesis of best

10 McGee & Reis (2012) QMA 67
practices
The role of blended learning in the clinical
Rowe, Frantz, &
11 education of healthcare students: A MA 14
Bozalek (2012)
systematic review
An analysis of high impact scholarship and
12 Halverson et al. (2012) SLR 95
publication trends in blended learning
Gikandi, Morrow, & Online formative assessment in higher

13 SLR 8
Davis (2011) education: A review of the literature
What do we know about web-based

14 Cook et al. (2010) learning? A systematic review of the SLR 65
variability of interventions
Traditional, web-based and hybrid

15 Landers (2009) instruction: A comparison of training MA 126*
methods
Methods 67
Primary
Research focus and methodological choices

Bliuc, Goodyear, & Approx.
16 in studies into students’ experiences of SLR
Ellis (2007) 300
blended learning in higher education
The undergraduate experience of blended

17 Sharpe et al. (2006) e-learning: A review of UK literature and SLR & I 14
practice
The comparative effectiveness of web-

18 Sitzmann et al. (2006) based and classroom instruction: A meta- MA 96*
analysis
What makes the difference? A practical

19 Zhao et al (2005) analysis of research on the effectiveness of MA 52*
distance education
A meta-analytic review of factors that
influence the effectiveness of web-based
20 Paul (2001) MA 15
training within the context of distance
education
Note: SR: systematic review; MA: meta-analysis; SLR: systematic literature review; TA: thematic
analysis; QMA: qualitative meta-analysis; I: interviews. *The sample included both studies with BL
and OL instructional interventions
68 Methods
number of studies 6 6
number of studies
4 4
2 2
0 0
1-10 11-20 21-50 51-100 >100 1-10 11-20 21-50 51-100 >100
number of primary studies covered number of primary studies covered
Conf. Paper Report adult he & adult he & k-12

Journal article Thesis he he, adult & k12
Figure 3A Coverage of primary studies by different types Figure 3B Coverage of the domain in primary studies
of systematic reviews and meta-analyses
6 6
number of studies
number of studies
4 4
2 2
0 0
01
06
07
10
11
12
13
14
01
05
06
07
09
10
11
12
13
14
20
20
20
20
20
20
20
20
20
20
20
20
20
20
20
20
20
20
Conf. Paper Report 1-10 21-50 >100

Journal article Thesis 11-20 51-100
Figure 3C Number and type of second order studies Figure 3D Number of primary studies analysed in
published in different years systematic literature reviews and meta-analyses, published
in different years
Figure 3 Descriptive Analysis of the Selected Dataset
Methods 69
Synthesis of Selected
Scholarly Work
Analysis of the selected second-order studies revealed the following major themes:
i. Effectiveness of BL (Bernard et al., 2014; Landers, 2009; Paul, 2001; Rowe, Frantz, &
Bozalek, 2012; Zhao & Breslow, 2013; Zhao, Lei, Yan, & Tan, 2005)
ii. Review of instructional practices in BL delivery and design (Bishop & Verleger, 2013;
Cook et al., 2010; Gikandi, Morrow, & Davis, 2011; Keengwe & Kang, 2013; McGee &
Reis, 2012)
iii. Review of existing research (Arbaugh, 2014; Bliuc et al., 2007; Drysdale, Graham,
Spring, & Halverson, 2013; Halverson, Graham, Spring, & Drysdale, 2012; Halverson,
Graham, Spring, Drysdale, & Henrie, 2014; Torrisi-Steele & Drew, 2013)
A further theme, related to institutional adoption of BL, was included to address one sys-
tematic review and to draw attention to the lack of research that effectively systematizes
the experiences in the adoption of BL practices as reported by individual universities and
organizations.
One obvious limitation of the current synthesis is the challenge of coherence, since the
boundaries of what constitutes BL differed across reported studies. Furthermore, it has come
to our attention that the themes reflected in the second-order studies do not completely
reflect the dominant themes of primary research in the field (cf. Halverson et al., 2014).
Effectiveness of BL
The effectiveness of BL as compared to OL and f2f has been a prominent theme in scho-
larly work. This research has been of particular interest to policymakers and institutional
administrations seeking to understand the potential impact on investment or for further
insight into the allocation of future resources. As BL involves an alternate set of costs when
compared to other modes of instruction, there is an implicit expectation that this model
of learning has to be more cost-effective (Graham, 2013; Means et al., 2009; Twigg, 2003).
70 Synthesis of Selected Scholarly Work

From the corpus of literature for the study, six meta-analyses and two systematic literature
reviews addressing the effectiveness of BL were identified (Table 2). Following the tradition
of DE and educational technology research, these studies approached the effectiveness of
BL instruction as an observable increase in effect sizes. That is, a greater effect size indicates
an achievement gain for the treatment (BL) condition over the control condition within the
research design (Ross, Morrison, & Lowther, 2010, p. 19).
Essentially, all selected studies concluded that in situations where the students experienced
BL instruction, whether it was in an online course with some additional f2f time, or whether it
was mostly an f2f course with some online time, student academic achievement was higher
than that of students who experienced a fully f2f or fully online learning mode.
Table 2 Summary of Systematic Analyses and Meta-Analyses of Blended Learning
Primary Studies
Years Analyzed
BL Boundaries
Type of Study
Outcomes
Measured
Learning
Findings
Type of
Study
BL conditions exceed
Bernard
f2f = 50%; OL f2f conditions Any measure of academic 1990–
1 et al. 96 MA
< 50% (g+=0.334, k=117, p performance 2010
(2014) <.001)
Only objective and direct

measures of learning, such
as standardized scores,
scores on researcher-
Means OL = BL outperforms f2f
created assessment, 1996–
2 et al. 30%–80%; f2f conditions (g+=0.35, 45 MA
grades, or GPA (excluded 2008
(2013) = 70%–20% k=23, p=.001)
student/teacher
perceptions, satisfaction,
retention, attendance,
etc.)
Synthesis of Selected Scholarly Work 71

Primary Studies
Years Analyzed
BL Boundaries
Type of Study
Outcomes
Measured
Learning
Findings
Type of
Study
Quantitative indicator of
learning, in most cases
Mixed evidence
Zhao & OL = grades for homework,
regarding whether 1999–
3 Breslow 30%–80%; f2f quizzes, labs, exams, and 42 SLR
hybrid or BL is more 2013
(2013) = 70%–20% similar, in some cases
effective
combined with student
satisfaction scores
Rowe, BL shows some

Pre- and post-tests,
Frantz, measure of
Meaningful interviews, focus groups, 2000–
4 & improvement, but 14 SLR
integration surveys, reflective blog 2012
Bozalek claims are difficult to
posts, etc.
(2012) make
Knowledge, observable
BL appears superior to
skills, problem-solving
traditional courses but
skills, attitudes,
OL = examinations of the
Landers perceptions, e.g., 1991–
5 30%–80%; f2f effect of the degree 126 MA
(2009) = 70%–20%
multiple-choice tests, 2009
to which a course is
computer use tests,
online are difficult due
self-reported learning
to small cell sizes
outcomes
BL was more effective

than f2f. Effect on
declarative knowledge: Declarative knowledge,
Sitzmann Mostly f2f d=0.34, k=33; procedural knowledge
1996–
6 et al. with OL Effect on procedural assessed by either 96 MA
2005
(2006) enhancement knowledge: d= participating in activity or
0.52, k=6; Effect on taking a written test
reactions: d=–.15,
k=11

Primary Studies
Years Analyzed
BL Boundaries
Type of Study
Outcomes
Measured
Learning
Findings
Type of
Study
Grades, quizzes,
As DE with f2f independent/standardized
DE mixed with a
enhancement tests, students
certain amount of f2f
as moderating satisfaction, faculty
Zhao instruction seems to
variable, satisfaction, dropout 1966–
7 et al. be most effective. 52 MA
or how rate, student evaluation 2002
(2005) Media involvement of
frequently of learning, student
60–80%: d=0.49, k=18,
technology evaluation of the course,
p<.001
was used external evaluation, and
cost effectiveness
Perceptual skills,
f2f with online intellectual skills,
Mostly f2f
Paul enhancement was 11% motor skills, attitudes, 1980–
8 with OL 15 MA
(2001) enhancement
more effective than f2f interpersonal skills/ 2000
only; d=0.27 averaged across reactions
and learning criteria
Despite the near unanimous agreement about the effect of BL, some scholars conclude
that the existing evidence is mixed, and that the first-order studies used in meta-analyses
lack statistical control for confounding factors (Rowe et al., 2012; Zhao & Breslow, 2013). BL
tends to involve additional time, instructional resources, and course elements encouraging
interaction among learners, and either of these could serve as a confounding variable ex-
plaining why BL conditions have a significantly higher effect on achievement gains (Jaggars
& Bailey, 2010; Landers, 2009; Means et al., 2013). In other words, although the effects
of BL instruction can be seen, the assignment of causality is far more difficult to ascertain
(Rowe et al., 2012).
Generalizability of the findings in BL effectiveness research is problematic due to the lack

of consistency across the primary studies from which the datasets for meta-analyses are
drawn. First, the primary studies lack consistency in what comprises BL instructional con-

ditions. The commonality between contexts in BL research is the presence of two modes,
but such a broad definition embraces practices too diverse and varied in intensity to be
replicated without more detailed specifications. Second, the primary research is incongruent
in defining what constitutes academic achievement. Some meta-analyses and systematic
reviews included a diverse set of academic achievement measurements, e.g., standardized
test scores, researcher-made and teacher-made tests, attitude measures and inventories,
expressions of satisfaction, skill evaluations, evaluations of the course as a whole, as well
as retention outcomes (e.g., Zhao et al., 2005). Other researchers looking at the primary
studies identified whether student learning resulted in declarative or procedural knowledge
(Sitzmann, Kraiger, Stewart, & Wisher, 2006), or the learning of facts or problem solving skills
(Landers, 2009). Recent meta-analyses and systematic literature reviews tend to settle on
exclusively quantitative measures of academic achievement (Means et al., 2013), which do
not necessarily represent meaningful outcomes (Ross & Morrison, 2014). Since a comparison
of academic achievement between different instructional conditions is the foundation for
demonstrating the positive impact of BL, the diversity of meanings of academic achievement
further confounds the potential for generalizing the research findings.
To conclude, BL effectiveness studies support the premise that students who learn from
the combination of online and f2f modes develop better learning outcomes than their
peers exposed to either of the modes exclusively. Although the research speculates that
BL combines the “best of two worlds,” studies of effectiveness lack consistency in what
constitutes BL environments, and what outcomes are being compared. Thus, the research
offers limited evidence as to what aspects of BL pedagogy or technology influence learning
outcomes (Arbaugh, 2014; Torrisi-Steele & Drew, 2013).
Instructional Practices and Technology that Impact BL

Another prominent theme in the reviewed research focuses on instructional practices per-
ceived as favorable for BL modes. The discussion below is structured to reflect what the
research has reported in relation to the following:
i. Use of technology in BL
ii. Pedagogical considerations for instruction within online and f2f modes,
as well as across the two
v. Design of BL courses
vi. Gaps in research on instructional practices

There is a difference in how the researchers cited in this section have established evidence
for “best” practices. For example, some practices have been simply recommended by in-
structors with their experiences and recommendations summarized in systematic reviews.
Alternatively, other evidence has been statistically derived from meta-analyses that identify
the value of instructional interventions in relation to academic achievement.
The Use of Technology
It is to be expected that the themes of technology and pedagogy would have a strong
presence in the BL research. The tension between technology and pedagogy has a long-
standing history, ever since Clark (1983) challenged the notion that technology-as-is has an
effect on learning. Clark argued that the instructional practice, not the medium of delivery,
ultimately influences the learning process. Research demonstrates that using technology
in BL for communication, for presentation, for searching, and so on, has varying effects on
academic achievement. More specifically, technology used to provide cognitive support
has a much higher moderating effect on achievement in BL (g+=0.59) than technology
provided for content/presentational support (g+=0.24) or technology used to facilitate
communication among peers and with the instructor (g+=0.31) (Bernard et al., 2014;
Schmid et al., 2014). As noted by Bernard et al. (2014), although these findings seem to
challenge the neutrality of technology, such conclusions at this stage would be superficial.
These results do suggest, however, that technological tools have varying power. Therefore,
treating technological tools as the one and only intervention condition in research may
lead to over-generalizing the different affordances that have potential to influence student
learning. In conclusion, these findings support the standpoint of Clark’s opponents who
did not necessarily disagree as to the importance of instructional choices in enhancing the
learning outcomes in technology-facilitated interventions, but argued for a more inclusive
definition of such interventions that would reflect the importance of both the technology
choice and the instructional practice (Hannafin & Young, 2008).
In addition to the studies selected for the synthesis, a meta-analysis by Schmid et al. (2014)
investigated the impact of how much technology is used within the BL provision. They found
that students in f2f classrooms where technology is used at a low or medium intensity level
outperform students in a predominantly technology-based classroom (i.e., high intensity).
Similarly, Bernard et al. (2014) investigated whether spending a low-to-moderate amount
of time in the BL online mode (up to 30% with at least 50% f2f) has an impact on academic
achievement when compared to students spending longer periods in the online mode (ap-
proaching 50% with 50% f2f). Their results are not statistically significant, and indicate that

in courses where at least 50% of the class is f2f mode, students who spend more time in
the online mode outperform those whose exposure to the online mode is shorter. A similar
trend was noted by Means et al. (2013). That said, the meta-analysis undertaken by Schmid
et al. (2014) suggests that differences in the impact of the intensity of technology may be
influenced by the subject matter of the course itself. At this point, besides supporting the
standpoint that technology is a potent element in the educational setting, such findings
are difficult to parlay into well-defined practical implications that can inform the design of
BL courses.
BL research perhaps provides only initial insight into the potential impact of specific tech-
nological interventions, and solid conclusions are as yet difficult to make. As McGee and
Reis (2012) point out in their systematic review of best practices, there is no agreement
even when it comes to the question of whether having an LMS is a mandatory component
for BL let alone what components of technology or the balance of time allocated for tech-
nology-mediated learning will best facilitate student outcomes.
Instructional Practices
Meta-analyses of the effectiveness of BL also include various factors related to instructional

practices and their moderating effect on academic achievement. For instance, Means et
al. (2013) provide statistically derived evidence that the pedagogical approaches used
in BL can have a positive impact on student academic achievement. Their meta-analysis
integrated 12 instructional practices as moderating variables3. The researchers identified
the effects that instructional practices, i.e., instructor-directed (expository), independent
(active), and collaborative (interactive), have on academic achievement. From these analyses,
the authors demonstrate that effect sizes of collaborative interactive learning (g+=0.249)
3 Means et al. (2013) included the following pedagogical practices as moderator variables: pedagogy/learning experience contrasting in-
structor-directed (expository) with independent (active) and collaborative (interactive); computer-mediated communication with instructor
contrasting asynchronous only with a combination of asynchronous and synchronous; treatment duration contrasting less that 1 month
with more than 1 month; media features contrasting text-based only with the combination of text and other media; time on task contrasting
provisions where students spent longer time online than f2f, with courses where the f2f part of the course was greater than the online part; a
condition comparing the presence or absence of one-way video or audio; a condition comparing the presence or absence of computer-based
instruction elements; a condition comparing whether the students had an opportunity for f2f time with instructor during the instruction,
before or after the instruction, or none at all; a condition that compared whether the students had an opportunity to interact with peers during
the instruction, before or after the instruction, or not at all; a condition examining whether the students had an opportunity to practice, or
not; and the condition comparing whether the feedback was or was not provided.

and teacher-directed instruction (g+=0.386) on academic achievement are higher than the
effect of self-paced independent learning (g+=0.05) (Means et al., 2013).
Bernard et al. (2014) also integrated instructional-related factors as moderating variables. They
focused, however, on the types of interactions students had in BL provisions, and included
the three types of interactions — student-student, student-teacher and student-content, as
interaction treatments in the meta-analysis of BL effectiveness. Since it was not the focus
of the study, they did not establish the impact of the exact combinations of the three inter-
action types upon student learning outcomes. However, their findings do suggest that the
presence of two or three types of interaction yields a higher effect than only one (g+=0.44
for two types of interaction; g+=0.47 for three types; g+=0.26 for one type).
Prior research into OL further adds to our understanding of the combinations of interactions
that have more impact on academic achievement in the online modes of BL provision. OL
meta-analyses have demonstrated the impact of student-student and student-content
interactions, as well as student-teacher and student-content interactions (for further ref-
erence, see Joksimović et al. (2015) in this series of papers). Furthermore, the effects of
the combinations of interaction types in BL support the interaction equivalency theorem
(Anderson, 2003). Anderson’s hypothesizes that in OL high levels of more than one type
of interaction “will likely provide a more satisfying educational experience, though these
experiences may not be as cost or time effective as less interactive learning sequences” (p.
4). This research highlights the close relationship of BL to the dominant pedagogies in OL
and DE, where interactions are seen as the means of bridging the psychological distance
between the participants in the learning process (Moore, 1989, 1993).
Recommended instructional practices for BL mirror effective practices within f2f and online
modes respectively. For example, there has been strong evidence of the essential role that
feedback plays in learning from f2f education research (Hattie & Timperley, 2007), as well
as DE/OL research (Paul, 2001), and educational technology research (Azevedo, 1993). As
expected, prompt feedback is also a suggested practice in BL settings, reflecting OL/DE
experiences; recommended practices of BL also include active learning and varied inter-
activity (McGee & Reis, 2012). Or, in accord with both f2f and OL instructional principles,
general recommendations in BL include defining clear course objectives, which are the
foundation for the course activities, assignments, and assessments. The design of these
learning activities must account for the specificities of f2f and online modes. That being
said, literature on BL pedagogy has tied itself more strongly to OL, with far fewer insights
taken from f2f education research (Arbaugh, 2014).

Course Design
Clearly, OL and f2f offer a rich heritage for the delivery and design of specific elements in BL
provisions. However, there is more to BL design than just mirroring existing practices. The
discussion around BL is concerned with thoughtful and meaningful ways of combining the
two (Garrison & Kanuka, 2004). This part of BL course design is not yet based on sufficient
evidence. The challenge of how to mix intensifies when an existing course is re-designed
into a blend. For instance, in the study by Graham and Robison (2007), over one-third of
faculty reported having taught a BL course. However, in the majority of these instances
the teaching and learning practices did not change. Instead the instructors simply added
small technology-based enhancements for accessing course content or communication
with peers. As such, it is tempting to “translate” a practice that exists within a course into
a corresponding technology-based practice (Salomon, 2002), but research provides little
assistance as to the types of “translations” that would be meaningful in BL contexts.
Designing activities for BL involves understanding the differences between OL and f2f
modes. Research indicates that different modes of learning are best suited for achieving
particular learning outcomes. For example, effectively designed OL facilitates the develop-
ment of factual and declarative knowledge (Landers, 2009; Rowe et al., 2012; Sitzmann et
al., 2006), while problem-based f2f learning has a positive effect on skills, and a negative
effect on knowledge (Dochy, Segers, Van den Bossche, & Gijbels, 2003; Gijbels, Dochy, Van
den Bossche, & Segers, 2005). The premise that different instructional modes are suited for
different tasks is further confirmed by studies that show that the same amount of time spent
on a task will impact the learning outcome in the OL mode, but will not have a comparable
impact in the f2f setting (Means et al., 2013).
Since computer-mediated communication with the instructor and among students asyn-
chronously tends to have higher effect when used in combination with synchronous mode
(Bernard et al., 2004), research and practice recommend that students and the instructor
carry on their conversations across both the online and f2f modes (Stacey & Gerbic, 2009).
Such continuous conversations allow for leveraging the convenience and deeper levels of
asynchronous discussion (Bernard et al., 2004; Bonk & Graham, 2006; Hrastinski & Keller,
2007) with the potential for a stronger sense of community in both web-based and f2f
synchronous communication (Rovai & Jordan, 2004).
In contrast with recommendations on how to mix student-student and student-instructor

interactions, the literature reviewed lacks advice regarding the blending of student-content
interactions (Helms, 2014). Here the concept of the flipped classroom (FC) becomes rele-

vant. The FC regained popularity as institutions began developing scaled online courses
for external non-fee-paying students (e.g., MOOCs). In essence, institutions sought a
better way to recoup the costs of such courses through greater integration of the new
online resources into the established fee-based course offerings (Bruff, Fisher, McEwen, &
Smith, 2013). FC is a specific course design that combines f2f interactive group learning
activities with web-based components taught at distance, such as lectures, close-ended
quizzes, and practice exercises. A systematic review of FC initiatives by Bishop & Verleger
(2013) attempts to evaluate this practice. Their results indicate that most primary studies
use single-group study design, providing no point of comparison for learning outcomes.
According to Bishop and Verleger (2013), the evidence that FC outperforms the traditional
classroom is anecdotal, and little work using controlled experimental or quasi-experimental
designs actually investigates what they call objective learning outcomes.
McGee and Reis (2012) offer an exhaustive synthesis of best practices reported by BL prac-
titioners in primary studies, including strategies and techniques related to course design,
pedagogy, implementation, and assessment. They recommend that BL provisions be de-
signed, rather than redesigned from f2f models, and that course components be aligned,
especially regarding assessment practices. They also highlight that although student-stu-
dent and student-teacher interactions are reported to enhance learning outcomes, in BL
provisions, that is not always the expectation, and sometimes instructors are not willing or
are incapable of continuous interaction. As a result, the alignment of expectations between
instructors and students in BL courses is of utmost importance.
Learner Support
Although student perceptions and perspectives on BL are among the most prominent
themes addressed in BL primary research, we were unable to identify a systematized review
of this topic. The available meta-analyses suggest differences as to the characteristics of
the students who prefer and/or benefit from BL provisions. For instance, Landers (2009)
found that older students seem to prefer the OL mode. Students in undergraduate-level BL
courses also tend to outperform those at the graduate level (Bernard et al., 2014; Means
et al., 2009, 2013). Furthermore, learners oriented towards information and ideas appear
to be more satisfied with the online components of BL, while those more people and feel-
ings oriented showed higher satisfaction with f2f components (Akkoyunlu & Yilmaz-Soylu,
2008). Despite these trends being noted in two recent BL meta-analyses, these findings
are statistically insignificant.

A common thread across BL studies is that, regardless of the type of learner, extra atten-
tion should be paid to developing additional student support to reduce the dependence
on the teacher and to assist with self-regulated learning skills (Bernard et al., 2014; Bonk
& Graham, 2006; McGee & Reis, 2012; Schmid et al., 2014; Torrisi-Steele & Drew, 2013).
Assessment in BL
Only 2.36% of all articles on BL in higher education, indexed in the Web of Science, ad-
dressed the topic of assessment (Torrisi-Steele & Drew, 2013). Consequently, recommenda-
tions related to assessment in BL practices are limited. In their synthesis of best practices,
McGee and Reis (2012) report that practitioners prefer BL assessment to be conducted
online, along with traditional assessments such as quizzes, exams, and essays. They also
acknowledge the interest in evaluating projects, threaded discussions, and presentations, as
well as assessing groups in collaborative contributions rather than individuals. McGee and
Reis (2012) note that if BL assessment is conducted in the f2f format, then it also tends to
be traditional, e.g., final exams, term papers, and so on. In conclusion, the authors confirm
that the minimal presence of assessment in BL research is puzzling. They also point out
that such conservative focus of BL assessment seems to mismatch the diversity of learning
activities promoted by BL practitioners.
Instructor’s Role in BL
Besides assessment, a further under-represented theme in the research relates to the role
of the instructor (or teacher). Two meta-analyses reported the rather obvious fact that the
teacher’s role has a significant influence on student learning outcomes. For example, in the
results of their DE meta-analysis, Zhao et al. (2005) reported that when instructor involvement
is low, the outcomes of DE are not as positive as those of fully f2f classes. However, when
instructor involvement in DE is at its highest, learning outcomes show a significantly better
effect than those of f2f. Similarly, Means et al. (2013) showed the significantly high effect that
instructor-directed (expository) instructional practices can have on academic achievement
(g+=0.386). Although, as pointed out by Graham (2013), neither of these studies actually
identified the aspects of instructor involvement that facilitated student learning.
Although the meta-analyses and systematic reviews lack detail on the types of instructor
involvement that improve student outcomes, further insight can be gained from primary
research studies. For instance, Shea and Bidjerano (2013) demonstrate that students in BL

courses report much higher levels of instructional design, facilitation of productive discourse,
and direct instruction — all elements of teaching presence (Garrison, Anderson, & Archer,
2001). The authors argue that this student perception may explain the higher effect sizes
related to academic performance in BL as compared to those of OL or f2f. Prior OL research
has demonstrated that teaching presence can predict levels of social presence and that it
influences cognitive presence.
The State of BL Research

A further prominent theme relates to the state of BL research itself. That is, undertaking
specific research on how the field is evolving through for example the use of bibliometrics
and content analyses. For instance, Halverson et al. (2012) analyzed high-impact scholar-
ship in the field, reviewing the most cited articles, books, authors, and influential journals.
Drysdale et al. (2013) also conducted a thematic analysis of doctoral dissertations, as well
as a thematic analysis of the high-impact scholarship (Halverson et al., 2014). Torrisi-Steele
and Drew (2013) undertook a similar task, reviewing and thematically coding all BL research
on higher education indexed by the Web of Science. In addition, Bliuc, Goodyear, and Ellis
(2007) conducted a review of methodological choices and research focus in student experi-
ences of BL, while Arbaugh (2014) reviewed academic work on BL in management education.
Several of these systematic reviews report that the majority of primary research on BL can
be classified as how-to papers related to instructional design or best practices reported
through single-case experiences at the course, program, or faculty level of implementing
a blended course (Arbaugh, 2014; Halverson et al., 2014; Torrisi-Steele & Drew, 2013). This
is consistent with the reported dominant methodologies, i.e., the majority of research has
been derived from “teachers as researchers analyzing their own students’ experiences of
BL” (Bliuc et al., 2007, p. 235). Such a trend illustrates that BL practice has predominately
been implemented “bottom-up” by individual teachers in their own classrooms in a desire
to improve student learning (Drysdale et al., 2013).
The authors of systematic reviews of the BL research are consistent in their observations
regarding the state of that research. There is agreement that the field of BL has matured
(Arbaugh, 2014; Drysdale et al., 2013; Halverson et al., 2012) but still has a disconnect
between BL practice and theory (Drysdale et al., 2013). To date, BL relies heavily on OL
theories (Arbaugh, 2014), as there has been little scholarly work on the development of
new theories or modifications to established theory (Halverson et al., 2014). To address this
disconnection, there is a need for theories that can better explain the influence of blending

practices on student learning (Graham, Henrie, & Gibbons, 2013).
Researchers also highlight significant gaps in the current research. First, the role of the
instructor is not sufficiently addressed, nor are the staff or institutional perspectives due to
the lack of information on professional development and BL institutional policy and adoption
(Drysdale et al., 2013; Halverson et al., 2014; Torrisi-Steele & Drew, 2013). Second, both
state-of-the-field reviews and meta-analyses that address the convergence of technology
with f2f contexts suggest that future directions for research include investigating blended
instructional practices, specifically their relationship to student motivation and engagement,
with particular focus on the student characteristics that benefit most from set designs.
Finally, suggestions for further research include investigating the role of learner choice and
self-regulated learning (Bernard et al., 2014; Drysdale et al., 2013; Graham, 2006; Means
et al., 2013).
Institutional Adoption
Only one systematic review on the issue of BL adoption, by Sharpe, Benfield, Roberts,
& Francis (2006), was found in our search of the literature. This review does not fully fall
under the broad definition of BL adopted for this synthesis, as it deals with the adoption
of LMS/VLE-supported f2f classrooms (excluding those combining online and f2f modes).
However, the Sharpe et al. study is highly relevant as it highlights the approaches and
processes that have promoted BL adoption. More specifically, Sharpe et al. observed that
a major advantage of adopting BL has been its ambiguous definition, “which allows staff
to negotiate their own meaning” (p. 4). In line with that, Picciano, Dziuban, & Graham
(2013) observe that there are no reliable estimates of the number of students enrolled in
BL. Essentially, faculty are not fully cognizant of when they are, or are not, teaching in BL
format. Furthermore, colleges and universities do not readily keep records of faculty who
teach blended courses (Picciano et al., 2013).
Besides the general lack of studies reviewing adoption experiences, no studies were iden-
tified that analysed the various reports and models related to the cost-effectiveness of BL.
Although, Graham (2013) has recently noted that cost-effectiveness is an obvious rationale
driving institutions to adopt BL. He reviewed positive experiences of return on investment
reported throughout both corporate (e.g., IBM, Intel) and higher educational contexts
(e.g., University of Central Florida). Corporations list such factors as reduction in wait times
for training, reduction in training hours and associated salary expenses, and reduction in
training costs to be relevant for cost-effectiveness (Graham, 2013).

Conclusions
and Implications
for Research
and Practice
This report presented a synthesis of the themes and findings from some 20 meta-analyses
and systematic studies to offer an evidence-based perspective on the practices known as
Blended Learning — a combination of f2f and web-based, technology-mediated educational
contexts. Studies investigating BL have evolved from grassroots practice into a maturing
research field. This is well evidenced in the changing definitions, growing number of doctoral
dissertations, and expanding conversations presently taking place in the primary research.
This transformation of the BL field has been fueled by rapid advances in technology, facil-
itating OL instruction that mirrors the properties of f2f contexts, thus enabling the rapid
convergence of these instructional modalities.
Despite the development of BL, its current dependence on its “parent” modes of delivery is
omnipresent in every theme synthesized in this report. First, findings from the effectiveness
studies conclude that combining f2f and online modes of delivery has a higher effect on
student academic achievement than either one of the modes independently. However, so
far there is limited evidence as to what particular methods of blending impacts academic
achievement. Second, recommended instructional practices mirror existing best practices
developed within OL and f2f modes, with heavy reliance on OL/DE. Course (re-)design
maintains its focus on approaches that help capitalize on the perceived benefits from these
separate modes of delivery, e.g., the enhanced social presence and relationship building
through f2f modes (Rovai & Jordan, 2004; Shea & Bidjerano, 2013), and the learner control
and the flexibility of access through online modes (Graham, 2013). Third, the research field
relies heavily on concepts developed in OL/DE while lacking its own theories to address
blending itself. Consequently, despite the abundance of individual accounts of blended
experiences, there is a lack of empirical research that would feed back to refine the blen-
ding-specific theoretical lenses.
Conclusions and Implications for Research and Practice 83

Such dependence on the two delivery modes (f2f/OL) from which BL emerged, results in
little evidence about the actual blending — a diverse set of practices with the potential
to overgrow its status of a combined delivery mode to become an effective pedagogical
method. While the choices behind pedagogy are highly related to how the process of
learning unfolds, there is actually little evidence about learning in BL practices. Despite
their complex and nuanced research design, recent BL effectiveness studies fall under
the category of “surface learning research” (Ross & Morrison, 2014), as they fail to show
the effect of various types of learning while maintaining focus on academic achievement
and performance. Furthermore, instructional practices barely address student-content
interactions, and the role of teacher has been mostly neglected. Finally, research does
not undertake theories that synergistically unite the learning that takes place across the
physical and the virtual.
BL research has provided some evidence that certain types of technology are more con-
ducive to producing higher measured learning outcomes, which brings the medium back
into the conversation about learning and pedagogy (Clark, 1983; Kozma, 1991, 1994).
Moreover, the development of technological affordances and technological ubiquity in some
parts of the world suggest that technology can help extend informal learning processes,
both socially and cognitively. In other words, besides bridging the psychological distance
between the separated participants in the learning process, despite their perceived prox-
imity (Thompson, 2007), pedagogical activities mediated by technology need to fit what
that technology can afford.
In sum, based on the evidence synthesized in this report, we argue that deeper insights
and focus on digital learning — i.e., learning mediated by various technological methods of
transcending physical and virtual space — would allow practitioners of BL to make better
pedagogical choices. In addition, a more detailed reporting of BL practices, both by ad-
ministrators and researchers, will aid our understanding of the nuances of BL beyond that
of a combined delivery mode. Last, the focus on the interplay between learning-processes
and technological affordances would allow researchers of BL to reframe their inquiries in a
way that leads to further maturation of the field.
84 Conclusions and Implications for Research and Practice

References
Akkoyunlu, B., & Yilmaz-Soylu, M. (2008). A study of student’s perceptions in a blended

learning environment based on different learning styles. Educational Technology
& Society, 11(1), 183–193.
Allen, C. (2004). Life with alacrity: Tracing the evolution of social software. Retrieved from
http://www.citeulike.org/group/1218/article/1613220
Allen, I. E., & Seaman, J. (2003). Seizing the opportunity: The quality and extent of online
education in the United States, 2002 and 2003. Sloan Consortium. Retrieved from
Allen, I. E., & Seaman, J. (2004). Entering the mainstream: The quality and extent of online
education in the United States, 2003 and 2004. ERIC. Retrieved from http://eric.
ed.gov/?id=ED530061
Allen, I. E., & Seaman, J. (2013). Changing course: Ten Years of Tracking Online Education
in the United States. Sloan Consortium. PO Box 1238, Newburyport, MA 09150.
Allen, I. E., Seaman, J., & Garrett, R. (2007). Blending in: The extent and promise of
blended education in the United States. ERIC. Retrieved from http://eric.
ed.gov/?id=ED529930
Anderson, T. (2003). Getting the mix right again: An updated and theoretical rationale for
interaction. The International Review of Research in Open and Distance Learning,
4(2). Retrieved from http://www.irrodl.org/index.php/irrodl/article/viewArticle/149
Anderson, T. (2008). Social software technologies in distance education: Maximizing learning

freedoms. In International Handbook of Distance Education, (pp. 167–184). Emerald
Group Publishing.
Arbaugh, J. B. (2014). What might online delivery teach us about blended management
education? Prior perspectives and future directions. Journal of Management
Education. doi: 10.1177/1052562914534244
References 85
Azevedo, R. (1993). A meta-analysis on the effects of computer-presented feedback on learning
from computer-based instruction. The Department of Education, Concordia University
& Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in
distance education. Review of Educational Research, 79(3), 1243–1289.
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., … Huang,
379–439. doi:10:3102/0034654309333844
Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-
analysis of blended learning and technology use in higher education: From the
general to the applied. Journal of Computing in Higher Education, 26(1), 87–122.
doi:10.1007/s12528-013-9077-3
Bishop, J. L., & Verleger, M. A. (2013). The flipped classroom: A survey of the research.
ASEE National Conference Proceedings. Atlanta, GA. Retrieved from http://www.
studiesuccesho.nl/wp-content/uploads/2014/04/flipped-classroom-artikel.pdf
Bliuc, A.-M., Goodyear, P., & Ellis, R. A. (2007). Research focus and methodological choices
in studies into students’ experiences of blended learning in higher education. The
Internet and Higher Education, 10(4), 231–244.
Bonk, C.J., & Graham, C.R. (Eds.)(2006). Handbook of blended learning: Global perspectives,
local Designs. San Francisco, CA:Pfeiffer Publishing.
Bruff, D. O., Fisher, D. H., McEwen, K. E., & Smith, B. E. (2013). Wrapping a MOOC:
Student perceptions of an experiment in blended learning. MERLOT Journal of
Online Learning and Teaching, 9(2), 187–199. Retrieved from http://jolt.merlot.
org/vol9no2/bruff_0613.pdf
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational

Research, 53(4), 445–459.
86 References
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2010).
Instructional design variations in Internet-based learning for health professions
education: A systematic review and meta-analysis. Academic Medicine, 85(5),
909–922.
Dochy, F., Segers, M., Van den Bossche, P., & Gijbels, D. (2003). Effects of problem-based
learning: A meta-analysis. Learning and Instruction, 13(5), 533–568.
Drysdale, J. S., Graham, C., Spring, K. J., & Halverson, L. R. (2013). An analysis of research
trends in dissertations and theses studying blended learning. The Internet and
Higher Education, 17, 90–100. doi:10.1016/j.iheduc.2012.11.003
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence,
and computer conferencing in distance education. American Journal of Distance
Education, 15(1), 7–23. doi:10.1080/08923640109527071
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative
potential in higher education. The Internet and Higher Education, 7(2), 95–105.
doi:10.1016/j.ihedue.2004.02.001
Gijbels, D., Dochy, F., Van den Bossche, P., & Segers, M. (2005). Effects of problem-based
learning: A meta-analysis from the angle of assessment. Review of Educational
Research, 75(1), 27–61. doi: 10.3102/00346543075001027
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher
education: A review of the literature. Computers & Education, 57(4), 2333–2351.
doi:10.1016/j.compedu.2011.06.004
Graham, C. (2013). Emerging practice and research in blended learning. Handbook of

Distance Education, 333–350.
Graham, C. (2006). Blended learning systems. CJ Bonk & CR Graham, The handbook of
blended learning: Global perspectives, local designs. Pfeiffer.
Graham, C., & Dziuban, C. (2008). Blended learning environments. In J. M. Spector, M.

D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational
communications and technology. Springer. Retrieved from http://link.springer.
com/content/pdf/10.1007/978-1-4614-3185-5.pdf
References 87
Graham, C., Henrie, C. R., & Gibbons, A. S. (2013). Developing models and theory for blended
learning research. In Blended Learning: Research Perspectives (Vol.2). Routledge.
Graham, C., & Robison, R. (2007). Realizing the transformational potential of blended
learning: Comparing cases of transforming blends and enhancing blends in higher
education. Blended Learning: Research Perspectives, 83–110.
Halverson, L. R., Graham, C., Spring, K. J., & Drysdale, J. S. (2012). An analysis of high impact
scholarship and publication trends in blended learning. Distance Education, 33(3),
381–413. doi:10.1080/01587919.2012.723166
Halverson, L. R., Graham, C., Spring, K. J., Drysdale, J. S., & Henrie, C. R. (2014). A
thematic analysis of the most highly cited scholarship in the first decade of blended
learning research. The Internet and Higher Education, 20, 20–34. doi:10.1016/j.
iheduc.2013.09.004
Hannafin, R. D., & Young, M. (2008). Research on educational technologies. In M. Spector,

M. D. Merrill, J. V. Merrienboer, & M. Driscoll (Eds.), Handbook of Educational
Communications and Technology (3rd ed., pp. 731–739). New York: Routledge.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research,
77(1), 81–112. doi: 10.3102/003465430298487
Helms, S. A. (2014). Blended/hybrid courses: a review of the literature and recommendations

for instructional designers and educators. Interactive Learning Environments, 22
(6), 80-810. doi:10.1080/10494820.2012.745420
Hrastinski, S., & Keller, C. (2007). Computer-mediated communication in education: A

review of recent research. Educational Media International, 44(1), 61–77. doi:
10.1080/09523980600922746
Hulsmann, T. (2004). The two-pronged attack on learner support: Costs and centrifugal forces
of convergence. In Supporting the Learner in Distance Education and E-Learning:
Proceedings of the Third EDEN Research Workshop, Oldenburg: Bibliotheks-und
Information system der Universitat Oldenburg, pp. 498–504.
88 References
Jaggars, S., & Bailey, T. R. (2010). Effectiveness of fully online courses for college students:
Response to a Department of Education meta-analysis. Retrieved from http://
academiccommons.columbia.edu/item/ac:172120
Keengwe, J., & Kang, J.-J. (2013). A review of empirical research on blended learning in
teacher education programs. Education and Information Technologies, 18(3),
479–493. doi: 10.1007/s10639-011-9182-8
Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61(2), 179–211.
doi: 10.3102/00346543061002179
Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational
Technology Research and Development, 42(2), 7–19. doi: 10.1007/BF0229908
Landers, R. N. (2009). Traditional, web-based, and hybrid instruction: A comparison of

training methods. University of Minnesota. Retrieved from http://conservancy.
umn.edu/handle/11299/52260
McGee, P., & Reis, A. (2012). Blended course design: A synthesis of best practices. Journal
of Asynchronous Learning Networks, 16(4), 7–22.
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based
practices in online learning: A meta-analysis and review of online learning studies.
US Department of Education. Retrieved from http://eric.ed.gov/?id=ED505824
Means, B., Toyama, Y., Murphy, R., & Bakia, M. (2013). The effectiveness of online and
blended learning: A meta-analysis of the empirical literature. Teachers College
Record, 115(3), 1–47.
Moore, M. G. (1989). Editorial: Three types of interaction. Retrieved from http://www.

tandfonline.com/doi/pdf/10.1080/08923648909526659
Moore, M. G. (1993). Theory of transactional distance. In D. Keegan (Ed.), Theoretical

principles of distance education (p.22-38). London: Routledge.
Moore, M. G., & Kearsley, G. (2011). Distance education: A systems view of online
learning. Cengage Learning. Retrieved from https://books.google.com.au/
books?id=dU8KAAAAQBAJ
References 89
Paul, D. S. (2001). A meta-analytic review of factors that influence the effectiveness of
Web-based training within the context of distance learning. Texas A&M University.
Picciano, A. G., Dziuban, C. D., & Graham, C. (2013). Research Perspectives. In Blended
Learning: Research Perspectives (Vol.2). Routledge.
Rainie, L. (2010). Internet, broadband, and cell phone statistics. Pew Internet & American
Life Project, 5. Retrieved from http://www.distributedworkplace.com/DW/Research/
Internet%20broadband%20and%20cell%20phone%20statistics%20-%20Pew%20
Internet%20Report%20Jan%202010.pdf
Ross, S. M., Morrison, G. R., & Lowther, D. L. (2010). Educational technology research past
and present: Balancing rigor and relevance to impact school learning. Contemporary
Educational Technology, 1(1), 17–35. Retrieved from http://aulavirtual.eaie.cvudes.
edu.co/publico/lems/L.000.002.MG/Documentos/Anexos/Cap3/1.pdf
Ross, S. M., & Morrison, J. R. (2014). Measuring meaningful outcomes in consequential

contexts: Searching for a happy medium in educational technology research
(Phase II). Journal of Computing in Higher Education, 26(1), 4–21. doi: 10.1007/
s12528-013-9074-6
Rovai, A. P., & Jordan, H. (2004). Blended learning and sense of community: A comparative
analysis with traditional and fully online graduate courses. The International Review
of Research in Open and Distance Learning, 5(2). Retrieved from http://www.irrodl.
org/index.php/irrodl/article/viewArticle/192
Rowe, M., Frantz, J., & Bozalek, V. (2012). The role of blended learning in the clinical
education of healthcare students: A systematic review. Medical Teacher, 34(4),
e216–e221. doi: 10.3109/0142159X.2012.642831
Salomon, G. (2002). Technology and pedagogy: Why don’t we see the promised revolution?
Educational Technology, 42(2), 71–75.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M. A.,
… Woods, J. (2014). The effects of technology use in postsecondary education:
A meta-analysis of classroom applications. Computers & Education, 72, 271–291.
doi: 10.1016/j.compedu.2013.11.002
90 References
Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience
of blended e-learning: A review of UK literature and practice. Higher Education
Academy London. Retrieved from http://www.islamicstudiesnetwork.ac.uk/assets/
was%20York%20-%20delete%20this%20soon/documents/ourwork/archive/
blended_elearning_full_review.pdf
Shea, P., & Bidjerano, T. (2013). Understanding distinctions in learning in hybrid, and online
environments: An empirical investigation of the community of inquiry framework.
Interactive Learning Environments, 21(4), 355–370. doi:10.1080/10494820.2011
.584320
Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness
of web-based and classroom instruction: A meta-analysis. Personnel Psychology,
59(3), 623–664. doi: 10.1111/j.1744-6570.2006.00049.x
Stacey, E., & Gerbic, P. (2009). Introduction to blended learning practices. In Stacey, E.
(Ed.) Effective Blended Learning Practices: Evidence-Based Perspectives in ICT-
Facilitated Education. IGI Global.
Thompson, M.M. (2007). From distance education to e-learning. In R. Andrews & C.

Haythornthwaite (Eds.). SAGE handbook of e-learning research (pp.159-178).
London: Sage Publications Ltd.
Torrisi-Steele, G., & Drew, S. (2013). The literature landscape of blended learning in
higher education: The need for better understanding of academic blended
practice. International Journal for Academic Development, 18(4), 371–383. doi:
10.1080/1360144X.2013.786720
Twigg, C. A. (2003). New models for online learning: Improving learning and reducing costs
EDUCAUSE review, 38 (5), 28–38. Retrieved from: http://www.educause.edu/ir/
library/pdf/erm0352.pdf
Zhao, Y., & Breslow, L. (2013). Literature review on hybrid/blended learning. Retrieved
from http://tll.mit.edu/sites/default/files/library/Blended_Learning_Lit_Reveiw.pdf
Zhao, Y., Lei, J., Yan, B., & Tan, S. (2005). What makes the difference? A practical analysis
of research on the effectiveness of distance education. Teacher’s College Record,
107, 1836–1884.
References 91
of Online Learning
Srećko Joksimović
Vitomir Kovanović
Oleksandra Skrypnyk
Dragan Gašević
Shane Dawson
George Siemens
93

Abstract
This report analyzes findings from research into online learn-

ing in order to provide guidelines for further research and
practice. Within this tertiary study, we performed a systematic
review of thirty-two second-order studies that address issues
of teaching and learning in online settings. From the exam-
ination of the studies included in the review, four prominent
topics emerged: i) comparison of online learning with the
traditional classroom, ii) comparison of various instructional
practices within two or more online courses, iii) perspectives
of students and instructors regarding learning and teaching
in online settings, and iv) adoption of online learning in in-
stitutions of higher and adult education. Except for showing
no significant difference in effectiveness of online learning
compared to traditional face-to-face settings, the studies
within the first theme also provided directions for further
research, necessary to better understand what practices
work best in online settings. Our findings further indicate
that contemporary research into online learning almost
univocally agrees that structured online discussions with
clear guidelines and expectations, well-designed courses
with interactive content and flexible deadlines, and con-
tinuous instructor involvement that includes the provision
of individualized, timely, and formative feedback are the
most promising approaches to fostering learning in online
environments. However, this also implies a more complex
role for the instructor in online settings, and a need for
research on instructional strategies that would allow for the
development of student self-regulatory skills. Implications
for future research and practice, as well as the position of
online learning within the broader aspect of digital learning
are further discussed.
From the first offering of a fully online course in 1981
introduction
(Harasim, 2000) it was clear that this new model of educa-
tion had much potential to impact the design and delivery
of education at all levels. Initial attempts to offer education
online tended to replicate existing distance education prac-
tice. As such, online education was text heavy, and mirrored
the previous postal packages of handbooks and required
readings (Garrison, 2011; Harasim, 2000). However, these
early offerings did provide rapid and valuable lessons into
what constitutes effective learning in this new mode of
education delivery. For instance, long textual lectures were
clearly not suitable for the online environment and students
did not readily engage in discussion activity (Harasim, 2000).
These early insights guided the development of online
learning (technical and pedagogical) including the intro-
duction of more collaborative learning activities, such as
course discussions.
Online forum discussions have remained an essential compo-

nent of online education (Harasim, 2000) and even now con-
tinue to be central in massive open online courses (MOOCs).
The current proliferation of technological affordances and
pedagogical developments influence the evolution of online
learning and the transformation of teaching and learning in
higher and adult education (Clardy, 2009; Garrison, 2011),
allowing for broad adoption of distance and online educa-
tion and incorporation of more interactive approaches to
learning (Anderson, 2009).
96 Introduction
the history and state of online learning
Although online learning presents a form (i.e., the fifth generation1), of distance education
(Taylor, 2001), it has its own provenance (Ally, 2004; Garrison, 2011). Online and traditional
distance education approaches do share common attributes, including the emphasis on
“any time — any place” learning, the assumption that students are at a distance from the
instructor (Moore, 1993), and the use of some form of technology to access course materials
(Ally, 2004; Garrison, 2011; Harasim, 2000). However, in part due to the interactive nature
of online learning, it “is very different from traditional distance education with its [DE]
historical focus on content delivery and independent learning [... and] has evolved from a
different field of theory and practice” (Garrison, 2011, p. 3). Given that online learning draws
from constructivist approaches to learning, it presents a significant shift in comparison to
traditional distance education, which is based on “the ideal of autonomy and the industrial
production of prepackaged study materials” (Garrison, 2011, p. 3).
Online learning transforms education from instructor-centered (traditional classroom)

to student-centered, where students have more responsibility for their learning (Koch,
2014; Peterson, 2008). Given that students are now able to choose what to learn, when
to learn, and who to learn with, a certain level of self-directedness is necessary in order to
succeed in an online course. On the other hand, the lack of unity of time and place leads
to greater interdependence between students and instructors and significantly changes the
instructor’s role within this “new environment” (Harasim, 2000; Koch, 2014). For example,
online asynchronous communication results in new engagement and learning patterns.
Peer interaction is high and, unlike more traditional approaches, “there is a multiplicity of
voices or perspectives, and students are exposed to a variety of possible interpretations
or solutions, rather than just the ‘right’ or the ‘textbook’ answer” (Harasim, 2000, p. 16).
Instructor participation, on the other hand, is still significant and highly valuable; however,
1 Taylor (2001) identified the following five generations of distance education development: 1) the correspondence model, 2) the
multi-media model, 3) the tele-learning model, 4) the flexible learning model, and 5) the intelligent flexible learning model. The fifth gen-
eration includes online interactive multimedia, Internet-based access to resources, computer-mediated communication using automated
response systems, and campus portal access to institutional processes and resources.
Introduction 97
the online environment assumes a “more passive and nondirective role” in teaching and
learning in these settings (Koch, 2014, p. 1385). In general terms we would agree with Koch’s
(2014) view of the instructor’s role. However, we would prefer a more inclusive definition
that reflects Marks, et al., (2005) earlier perspective. That is, the instructor is not simply
passive in this “new” learning environment. The context calls for a more active “supportive
and guiding” role (Marks, Sibley, & Arbaugh, 2005, p. 549).
The advances in educational technology and increased interest in asynchronous discussion

groups, gave rise to the term e-learning in the mid-1990s, which aimed to describe learn-
ing delivered fully online and learning that combines online with face-to-face components
(blended or hybrid learning) (Garrison, 2011). With the continued growth and interest in
online learning, many researchers aimed to investigate the equivalency of these forms of
instruction against traditional on-campus courses. Researchers began to examine: i) whether
new technological affordances were more effective than traditional modes of learning (Cook,
Levinson, & Garside, 2010; Means, Toyama, Murphy, Bakia, & Jones, 2009; Tallent-Runnels
et al., 2006), ii) what best instructional practices foster learning in online settings (Bernard
et al., 2009; Borokhovski, Tamim, Bernard, Abrami, & Sokolovskaya, 2012; Darabi, Liang,
Suryavanshi, & Yurekli, 2013), and iii) how the roles of instructors and learners evolved
within these new educational models (Carroll, Booth, Papaioannou, Sutton, & Wong, 2009;
Peterson, 2008; Styer, 2007). Nevertheless, the challenge in obtaining a clear understanding
of best practices in online learning settings originates in the “multiplicity of terms used to
describe a phenomenon” (Rudestam & Schoenholtz-Read, 2010).
The wide variety of terms used to describe the same or similar learning approaches and the
high heterogeneity among the results presented were the main motivations for conducting
this study. Thus, the purpose of this report is to summarize findings from contemporary
research into online learning in order to understand its current state and to identify potential
guidelines for further research and practice.
98 Introduction
Operational Definitions:
Changing the Landscape of the Language
One of the challenges for research into online learning is the lack of an authoritative defi-
nition of what constitutes this mode of education delivery. As well noted by Clardy (2009),
Garrison (2011), and Rudestam and Schoenholtz-Read (2010) (among others), there is a
wide diversity of terms used to describe online learning and what activities neatly fit under
the umbrella of purely online learning. The most commonly used terms to describe online
learning include web-based learning, e-learning, Internet-based learning, online learning,
distance learning, distance education, distributed learning, computer-mediated learning, and
computer-assisted learning (Ally, 2004; Means et al., 2009; Rudestam & Schoenholtz-Read,
2010; Tallent-Runnels et al., 2006). Figure 1 shows the distribution of the most commonly
used keywords over years. Online learning is on the other hand, considering that online
learning is considered as the “fifth generation” of distance education (Taylor, 2001), as well
as the current state of the available technological affordances, it is rather challenging to
identify what is considered pure online learning. For example, most studies analyzed for this
report (Means et al., 2009; Styer, 2007; Tallent-Runnels et al., 2006), defined online learning
as a subset of distance education; that is, courses delivered completely online, excluding
“print-based correspondence education, broadcast television or radio, videoconferencing,
videocassettes, and stand-alone educational software programs” (Means et al., 2009, p. xii).
However, it is questionable whether videoconferencing in the forms available today (such as
Google Hangouts or Skype) should be a part of distance education only, or perhaps both
distance and online education. Therefore, for the purpose of our study, we build further
on definitions proposed by Clardy (2009), Gikandi, Morrow, and Davis (2011), Means et al.
(2009), Schlosser and Simonson (2006), Styer (2007), and Tallent-Runnels et al. (2006), and
we define distance and and online learning as:
Distance education is teaching and planned learning where the teaching

occurs in a different place from learning, requiring communication through
technologies and special institutional organization2.
2 For further discussion on distance learning, please refer to the report entitled “The History and State of Distance Education.”
Operational Definitions 99
Online learning is a form of distance education where technology mediates
the learning process, teaching is delivered completely using the Internet, and
students and instructors are not required to be available at the same time
and place. It does not include more traditional distance education instruction
methods, such as print-based correspondence education, broadcast television
or radio, videoconferencing in its traditional form, videocassettes/DVDs and
stand-alone educational software programs.
Keyword
3000 online learning
for online learning and related keywords
e-learning
web-based learning
internet-based learning
distance education
distance learning
Number of articles
distributed learning
2000 computer aided learning
computer assisted learning
computer-mediated learning
1000
1990 1995 2000 2005 2010 2015
Figure 1 Distribution of studies on online learning with the most commonly used keywords used to describe
learning delivered online, as indexed by Scopus.
100 Operational Definitions

It is interesting to note that terms such as computer-based instruction, web-based instruction,
or problem-based learning, have become synonymous with distance (online) and blended
learning. As such, a further challenge to synthesizing the findings of research on online
learning involves the morphing or aggregation of these terms with online learning. Many
studies analyzed learning and teaching in online and blended (hybrid) settings, without a
clear distinction between the two approaches. In order to provide a more comprehensive
description of the current state of the research and practice of online learning, we will
refer to blended learning where appropriate. Therefore, in our study, blended learning is
defined as follows:
Blended learning refers to the practices that combine (or blend) traditional
face-to-face instruction with online learning3.
3 For further discussion on blended learning, please refer to the report entitled “The History and State of Blended Learning.”
Operational Definitions 101

Method
Research questions
The aim of this tertiary study is to identify themes that have framed research into online
learning and to summarize the current state of research and practice, as well as to reveal
prospective directions for further research and practice. Therefore, we defined the following
questions to guide our research:
RQ1. What are the main topics emerging from the contemporary literature on online
learning?
RQ2. What is the state of research and practice in online learning, as reflected through
meta-studies and systematic literature reviews?
Literature Search and Inclusion Criteria

We performed a computer-based literature search (with no time limit) through three phases
(Figure 2). The first phase included searching the ERIC, Scopus, PsychINFO, PubMed, and
ProQuest databases. In order to be included in our analysis, the following search criteria
had to be met:
Title, abstract, and/or keywords must contain at least one of the following
terms: meta-analysis, meta-synthesis, scoping study, OR systematic review AND
Title, abstract, and/or keywords must contain at least one of the following
terms: distance learning, distance education, blended learning, blended
education, hybrid education, hybrid learning, online learning, online education,
e-learning, web-based learning, OR web-based education.
The initial search resulted in a list of 306 studies, including those on online, blended, and
distance learning. Further, we searched Google Scholar for different combinations of the
above-mentioned terms. Specifically, we included one of the terms from the first group of
102 Method
concepts (meta-analysis, meta-synthesis, scoping study, and systematic review), and one of
the terms from the second group (distance learning, distance education, blended learning,
blended education, hybrid education, hybrid learning, online learning, online education,
e-learning, web-based learning, and web-based education). The second phase resulted in
an additional 19 studies that satisfied the search criteria. Finally, we manually searched the
following journals: American Journal of Distance Education, Journal of Distance Education,
Distance Education, International Review of Research in Distance and Open Education, Journal
of Asynchronous Learning Networks, Journal of Technology and Teacher Education, Career
and Technical Education Research, Internet and Higher Education, Journal of Computing
in Higher Education, and Computers and Education to identify relevant studies. The list
of relevant journals was obtained from the most influential meta-analyses in distance and
online education (Bernard et al., 2009; Means, Toyama, Murphy, & Bakia, 2013). The third
search phase resulted in an additional 14 studies, providing a final list of some 339 papers.
After completing the search, three researchers coded the identified studies as distance,
online, and/or blended learning. The coding process comprised reading the title, keywords,
and abstract for each study and assigning one or more labels, indicating that study predom-
inantly analyzed distance, online, and/or blended learning. In cases where a code could not
be assigned based on the available information, the coders examined the article in detail
(e.g., reading the Methods section) to identify the most appropriate categorization. The
coding process and search yielded a total of 102 second-order studies that satisfied the
criteria for inclusion in this review:
i. the study applied a systematic approach (e.g., meta-analysis, systematic literature

review, meta-synthesis) in analyzing studies that addressed any aspect of online learning
ii. the study analyzed online along with distance and/or blended learning
iii. the study was published in a peer-reviewed journal/conference proceedings or in a
dissertation, available in English
iv. the study participants in primary studies were non-disabled undergraduate students,
graduate students, and/or employees (e.g., teachers and nurses)
v. the study analyzed K–12 education along with higher and/or adult education
After the final screening of the 102 studies, we identified 32 studies that met the above-de-
fined criteria for inclusion. These studies all defined online learning equivalently to the defini-
tion provided in this report. We also identified 37 studies related to distance education and
20 second-order studies that analyzed teaching and learning practices in blended learning.
Method 103
Total of 339 unique

studies identifed
FOCUS OF
THIS STUDY

edutcation studies learning studies learning studies

Figure 2 The systematic literature search process
104 Method
Analysis and Data Set
To address the research questions, we performed a synthesis of the systematically se-
lected literature. Given the focus of the study, we aimed to identify the most prominent
themes within the second-order studies that satisfied the specified inclusion criteria and
to summarize the findings on the state of research and practice in online learning at the
time when the studies were conducted or for a set period. Thus, each individual study was
coded for the following attributes: author(s), publication year, publication venue (the name
of the journal/conference), keywords (keywords assigned by author), type of publication
venue (journal, conference, or dissertation), important definitions (distance, online, and/or
blended learning), domain (distance, online, and/or blended), level of education, type of
method applied (e.g., meta-analysis, scoping study, systematic literature review), number
of primary studies analyzed, years analyzed, research questions, major topics, effect sizes
reported (yes/no), and main findings, methodology applied in the primary studies. Coding
was performed by the first author only, and further discussed with the co-authors until a
consensus was reached.
Table 1 shows the author(s), title, publication year, study type, number of primary studies
analyzed, and the number of participants4 for all the studies included in this report. The
vast majority of second-order studies (i.e., 24) were published in journals (Figure 4). We
also included five theses, one conference paper, and one report. Figure 3 also shows that
literature reviews (including systematic reviews) were the most commonly utilized approach,
followed by meta-analyses and a single scoping study.
4 Several studies did not report precise information about the number of participants included, thus we noted “more than” a certain
number of participants. For example, Tallent-Runnels et al. (2006) in one case reported “30 undergraduate students,” while in the other
case it was “Those who successfully completed any of 2,300 college courses over 2-year period (2000–2002)” (p. 44, Appendix).
Method 105
25
number of studies
20
15
10
0
Conf. Paper Journal Article Report Thesis
Lit. Reveiw Meta-analysis Scoping study Sys. Review
Figure 3 Number of studies per type of the publication venue (i.e.,

conference, journal, report, and thesis), with bars showing the respective
number of papers using the method applied (i.e., literature review, meta-
analysis, scoping study, and systematic review)
Only four studies included here were published before 2009 (Figure 4). Our search also
revealed meta-analyses and literature reviews published before 2006; however, these were
primarily focused on distance education rather than specifically on online learning (e.g.,
Bernard et al., 2004). Given that most of the studies were published during and after 2009,
it is no surprise that the majority of those second-order studies analyzed the period between
2000 and 2008 (Figure 5). Finally, the number of primary studies analyzed within the sec-
ond-order studies varied (Figure 5), whereas the primary focus in meta-analysis, systematic
and literature reviews was on the Higher and Adult education (Figure 5).
Thirteen (out of 32) studies reported effect sizes, and those studies focused primarily on
the effectiveness of online learning instruction compared to face-to-face learning or to
another online course. Three studies (i.e., Jurewitsch, 2012; Beinkowski, Feng, & Means,
2012; Sitzmann, Kraiger, Stewart, & Wisher, 2006) focused solely on experimental and
quasi-experimental studies, while only five analyses (Cook, Levinson, Garside, et al., 2010;
Cook, Garside, Levinson, Dupras, & Montori, 2010; Cook, Levinson, & Garside, 2010; Du
et al., 2013; Wong, Greenhalgh, & Pawson, 2010) included primary studies that used ran-
domized trials (all studied online learning in medical education).
106 Method
6
number of studies
4
06
07
08
09
10
11
12
13
14
20
20
20
20
20
20
20
20
20
Conf. Paper Report
Journal article Thesis
Figure 4 The count of second-order studies published

with the respective year, with bars showing counts for
the publication venue type (i.e., conference, journal,
report, or thesis)
8
number of studies
0
1-10 11-20 21-50 51-100 >100
adult he & adult he & k-12

he he, adult & k12
Figure 5 Count of primary studies analyzed within

the second-order study, with bars showing the counts
of second-order studies that addressed a particular
level of education
Method 107
Limitations and Challenges of the Synthesis
The diversity in terms used to describe distance, online, and blended learning provided a
substantial challenge for undertaking this systematic review. Researchers frequently defined
the three learning approaches in multiple ways. Therefore, the majority of initially obtained
second-order studies required a detailed investigation of methods applied and the descrip-
tion of the primary studies included in those reviews. Although we followed the definitions
provided in the previous section, inconsistency in the terminology used left a certain level
of subjectivity in applying the specified inclusion criteria, leading to potential challenges in
internal validity. Moreover, the emergence of new technological affordances and the dearth
of second-order studies related to certain themes (e.g., assessment and teaching practices
in online education) may limit the generalizability of these findings.
Table 1 Second-order studies included in the analysis, with the information about the author(s), title, type of
the study, number of primary studies included, number of participants, and publication year
Studies
Study
Num.
Num.
Type
Part.
Title
Chumley-Jones, Web-based learning: Sound educational

1 Dobbie, & method or hype? A review of the LR 76 >5,471
Alford (2002) evaluation literature
Tallent-Runnels Teaching courses online: A review of the

2 LR 76 >10,000
et al. (2006) research
The comparative effectiveness of web-

Sitzmann et al.
3 based and classroom instruction: A meta- MA 96 19,331
(2006)
analysis
A grounded meta-analysis of adult learner

4 Styer (2007) motivation in online learning from the MA 14 >4,000
perspective of the learner
A meta-analysis of three types of

Bernard et al.
5 interaction treatments in distance MA 74 NR
(2009)
education
108 Method
Table 1 (Cont.) Second-order studies included in the analysis, with the information about the author(s), title,
type of the study, number of primary studies included, number of participants, and publication year
Studies
Study
Num.
Num.
Type
Part.
Title
UK health-care professionals’ experience
Carroll et al.
6 of on-line learning techniques: A SR 19 >2,290
(2009)
systematic review of qualitative data
Evaluation of evidence-based practices in

Means et al.
7 online learning: A meta-analysis and review MA 45 >1,635
(2009)
of online learning studies
Arbaugh & Research in online and blended learning in

8 Benbunan-Fich the business disciplines: Key findings and LR 182 NR
(2007) possible future directions
Cook, Levinson, Time and learning efficiency in Internet-

9 & Garside based learning: A systematic review and SR 20 1,814
(2010) meta-analysis
What do we mean by web-based learning?

Cook, Garside,
10 A systematic review of the variability of SR 266 NR
et al. (2010)
interventions
Instructional design variations in internet-
Cook, Levinson,
based learning for health professions
11 Garside, et al. SR 51 8,416
education: A systematic review and meta-
(2010)
analysis
Traditional, web-based, and hybrid
12 Landers (2009) instruction: A comparison of training MA 126 NR
methods
Internet-based medical education: a realist

Wong et al.
13 review of what works, for whom and in SR 249 NR
(2010)
what circumstances
Gikandi et al. Online formative assessment in higher

14 SR 18 NR
(2011) education: A review of the literature
Method 109
Studies
Study
Num.
Num.
Type
Part.
Title
Student satisfaction with online courses
15 Macon (2011) MA 13 2,071
versus traditional courses: A meta-analysis
A meta-analysis of e-learning technology
Šumak, Heričko,
16 acceptance: The role of user types and MA 42 12,986
& Pušnik (2011)
e-learning technology types
Cohen,
The design, implementation, and
Carbone, &
17 evaluation of online credit nutrition SR 9 1,017
Beffa-Negrini
courses: A systematic review
(2011)
Best instructional practices for distance

18 Roberts (2011) MA 59 5,779
education: A meta-analysis
Are contextual and designed student-

Borokhovski et
19 student interaction treatments equally SR 32 3,634
al. (2012)
effective in distance education?
A meta-analytic and qualitative review of

Jurewitsch
20 online versus face-to-face problem-based MA 5 291
(2012)
learning
A meta-analytic study of adult self-directed

21 Peterson (2008) learning and online nursing education: A MA 9 NR
review of research from 1995 to 2007
Wolbrink & Internet-based learning and applications

22 LR 6 NR
Burns (2012) for critical care medicine
Ravenna, Foster, Increasing student interaction online: A

23 LR 19 >2,196
& Bishop (2012) review of the literature
Cook & Steinert Online learning for faculty development: A

24 LR 20 1,458
(2013) review of the literature
110 Method
Studies
Study
Num.
Num.
Type
Part.
Title
Darabi et al., Effectiveness of online discussion
25 MA 8 NR
2013 strategies: A meta-analysis
Web-based distance learning for nurse

26 Du et al. (2013) SR 9 1,125
education: A systematic review
Stepanyan,
Littlejohn, & Sustainable e-Learning: Toward a coherent
27 SR 46 NR
Margaryan body of knowledge
(2013)
Research trends in meaningful learning

Chia-Wen, research on e-learning and online
28 Pei-Di, & education environments: A review of LR 38 NR
Yi-Chun (2013) studies published in SSCI-indexed journals
from 2003 to 2012
Exploring the use of asynchronous online

29 Thomas (2013) discussion in health care education: A LR 14 1,179
literature review
Jumaat & Tasir Instructional scaffolding in online learning
30 LR 10 NR
(2014) environment: A meta-analysis
The nursing educator’s role in e-learning: A

31 Koch (2014) LR 40 NR
literature review
Barriers and enablers to adoption and

Singh & diffusion of eLearning : A systematic
32 SR 340 NR
Hardaker (2014) review of the literature - A need for an
integrative approach
Note: SR – systematic review, MA – meta-analysis, SC – scoping study, LR – literature review; NR –

not reported.
Method 111
Synthesis of Selected
Scholarly Work
From the examination of the 32 second-order studies, four prominent themes were identified:
vi. comparison of online learning with the traditional classroom

vii. comparison of instructional treatments within two or more online courses
viii. learning and teaching in online settings from the perspective of students and instructors
ix. adoption of online learning in institutions of higher and adult education
Comparison of online learning with

the traditional classroom
Soon after online learning emerged as a learning approach, certain advantages became
apparent — such as flexibility, alleviation of overcrowded classrooms, increased enrollment,
reduced cost, and increased profit (Clardy, 2009; Grandzol & Grandzol, 2006). On the other
hand, wide adoption of online learning also revealed various disadvantages of teaching
and learning in the online environment — such such as the cost of training instructors,
feelings of isolation, and technology gaps. Therefore, recognizing a great opportunity and
numerous potential threats with the introduction of a new educational medium, instructors,
policymakers, and other relevant stakeholders raised questions about whether instructional
technology affects learning and contributes to student achievement (Grandzol & Grandzol,
2006; Means et al., 2009; Ross, Morrison, & Lowther, 2010; Schmid et al., 2014). This resulted
in the now well-known “great media debate” (Clark, 1983, 1994; Kozma, 1994) in which
many researchers aimed to provide evidence about whether media influence learning or
whether pedagogy is “all that matters” (Means et al., 2009; Sitzmann et al., 2006). Thus,
the theme attracting the most attention in the existing research on online learning is the
effectiveness of learning in online learning practices. Initially, researchers compared online
learning with the traditional classroom in order to confirm whether the new learning mode
worked. Specifically, those studies compared the two delivery media in terms of effectiveness
for improving learning outcome (Cook, Garside, et al., 2010; Cook, Levinson, & Garside,

2010; Landers, 2009; Means et al., 2009; Sitzmann et al., 2006), student satisfaction with
online courses (Chumley-Jones, Dobbie, & Alford, 2002; Du et al., 2013; Macon, 2011;
Sitzmann et al., 2006), time and learning efficiency (Cook, Levinson, & Garside, 2010), and
the effectiveness of problem-based learning (Jurewitsch, 2012). Researchers also analyzed
whether contextual or methodological factors moderated the comparison between the two
and, if so, which had greater impact. Most studies have demonstrated that online learning
is (at least) as effective as face-to-face learning. Hence, existing research (e.g., Bernard, et
al., 2009; Cardy 2009) has suggested using an alternate approach to researching online
learning and that studies comparing online with classroom practice do not provide the
necessary insights into best practices that can effectively inform learning and teaching more
broadly. Most second-order studies conducted on the topic of online learning found support
for Clark’s (1983) view of technology and pedagogy — i.e., pedagogy and instructional
method affect learning while “media are mere vehicles that deliver instruction but do not
necessarily influence student achievement” (Clark, 1983, p. 445).
Comparison of instructional treatments within

two or more online courses
Besides knowing that distance and online education “can be much better and also much
worse” (Bernard et al., 2009, p. 5) than traditional learning, the studies that compared
two did not reveal much about “what works best” in online learning (Bernard et al., 2009;
Clardy, 2009). Roberts (2011) further stated, “media comparison studies have served their
purpose in pointing the way to the next generation of studies” (p. 61). Therefore, numerous
studies comparing various instructional treatments in distance education in general, and
online learning in particular, have emerged. The importance of various interaction treat-
ments (e.g., student-student, student-teacher, student-content interactions) for student
achievement (Bernard et al., 2009; Borokhovski et al., 2012), fostering collaboration among
students, increasing engagement with content, and supporting interaction with instructors
(Darabi et al., 2013; Ravenna, Foster & Bishop, 2012; Thomas, 2013), were some of the
most prominent themes in this line of research. The studies tended to indicate that asyn-
chronous, purposefully structured discussions, with clear guidelines and timely, summative,
and individualized feedback from the instructor or peer students are the best instructional
strategies to support learning in an online environment (Borokhovski et al., 2012; Darabi
et al., 2013). Means et al. (2009) also showed that incorporating mechanisms to prompt
reflection and self-assessment tend to be more successful than learning in groups. There-

fore, different practices to support asynchronous online discussions (Thomas, 2013) and
various instructional scaffolds — e.g., metacognitive scaffolding, strategic scaffolding, and
question prompts (Jumaat & Tasir, 2014) — were examined in order to reveal what works
best within the online learning paradigm. Furthermore, summarizing research on online
teaching and learning, Tallent-Runnels et al. (2006), highlighted the importance of creating
learning communities, instructor-active participation through scaffolding and timely (forma-
tive) feedback, and the promotion of student-instructor and student-student interactions
as significant aspects of an online course. Thus, these second-order studies of instructional
practices in online learning tend to agree on several aspects:
i. online courses should provide good support for student-student and student-content
interactions
ii. those interactions should include co-operative and collaborative learning
iii. the most common approach to fostering interactions within the online learning
environment is through structured online discussions
iv. the instructor’s moderating role in guided discussions is of great importance
v. instructors should be able to provide timely, formative feedback on learning progress
for every student
vi. instructional scaffolds should be wisely considered and applied according to
student needs
vii. content provided should be visually engaging and interactive
Considering these general guidelines for organizing online courses, the expected amount
of participation in course design and facilitation seems hardly sustainable for instructors
(Moallem, 2003). Cook, Levinson, and Garside (2010) showed that instructional strategies
that enhance feedback and interactivity tend to prolong learning time in the online learn-
ing environment; however, it seems that even more effort is needed from instructors to
support learning in online settings than in face-to-face settings. Although Bernard et al.
(2009), Borokhovski et al. (2012), and Ravenna et al. (2012) (among others) noticed that
the instructor is “not alone” in that process, but rather a member of a team, and that some
responsibilities can be delegated to students, the obvious line of further research might be
on how to support instructors to teach more effectively.

Perspectives of students and instructors
regarding learning and teaching in online settings
Another prominent line of research that emerged examined online learning from the student’s
perspective. For instance, the factors that motivate students to enroll in an online course
(Styer, 2007), aspects that influence student satisfaction with the course and the instructor
(Carroll et al., 2009; Tallent-Runnels et al., 2006), and the importance of self-directedness in
online students (Peterson, 2008) were the most notable topics aimed at revealing/defining
characteristics of the successful student/learner in online environments. Research revealed
that students tend to value well-designed, frequently updated courses that incorporate
extrinsic motivating factors, with tasks/examples immediately relevant for their practice, a
reasonable level of control and flexibility (primarily in terms of deadlines), support to col-
laborate with their peers, and a high level of instructor involvement in providing summative
and timely feedback (Carroll et al., 2009; Styer, 2007).
Despite the fact that most of the research on online learning stressed the importance of the
instructor, and clearly the instructor’s role within the online learning environment differs from
the traditional classroom, there have been very few studies that investigated how teaching
has evolved with the new learning paradigm (Koch, 2014). Given that the focus in course
design has shifted from an instructor-centered to a student-centered role, where instructors
need to take a more passive, non-directive position in leaving students to pace their own
learning activities (Koch, 2014), it became clear that instructors need to redistribute their
responsibilities among a team of instructors, or by assigning more obligations to students,
in order to make their role sustainable.
Assessment is considered an essential component of formal higher education and effective

learning (Gikandi et al., 2011). However, there is an evident lack of research on assessment
in online education. Based on the 18 studies published between 2000 and 2010, Gikandi et
al. (2011) identified validity, reliability, and dishonesty as the key components of assessment.
Formative and immediate feedback, engagement with critical processes, and promotion
of equitable education were recognized as the main opportunities facilitated by online
formative assessment. The main findings of the Gikandi et al. (2011) study suggest that
an “effective online formative assessment can foster a learner and assessment centered
focus through formative feedback and enhanced learner engagement with valuable learn-
ing experiences” (Gikandi et al., 2011, p. 1). Furthermore, online formative assessment
requires constant monitoring of student activity, using discussion tools, quizzes, and trace
data from the learning management system and in turn fosters student engagement and

self-regulation. Assessment still depends on the instructor’s “belief” about what should
be valued as learning. Therefore, further empirical research is necessary in order to inform
effective practice.
Adoption of online learning in institutions

of higher and adult education
Current research of institutional and administrative aspects of adoption of online learning
showed that individual attitudes towards technology and digital literacy are the main
factors influencing online learning adoption (Singh & Hardaker, 2014; Šumak, Heričko, &
Pušnik, 2011; Tallent-Runnels et al., 2006). Although most contemporary institutions of
higher education have developed policies for online courses, the main challenge is that
some institutions still have to develop policies for support, course development, and as-
sessment (Tallent-Runnels et al., 2006). Perhaps, the most comprehensive set of guidelines
for adoption of online learning has been provided in Singh and Hardaker’s (2014) study.
Their findings suggest that:
In deciding how online learning will be incorporated into current practices, all
the interested parties should be included in the decision-making process
A clear, strategic vision should be developed and communicated to all faculties

and departments
Such a significant decision for institutional development should not originate

from an individual or a small group of top management; rather, policymakers
must be aware of the “cultural configuration” within their organization in order
for the project to succeed
“Academics’ fear” of innovations and loss of control over teaching must be

considered by top management; the most prominent approach will consider
motivational factors for each individual “and avoid imposing institutional
constraints through standardized frameworks that assume a single solution for
all eventualities”(p. 13)
Institutional management needs to act as a role model in creating a culture that

promotes the adoption of online learning

Cook and Steinert (2013) concluded that online courses were more likely to succeed when
the course topic addressed a relevant need; supported collaboration, co-operative work,
and social interaction; and provided reasonably flexible timelines to complete all the ac-
tivities. On the other hand, Wolbrink and Burns (2012) argue that there are still challenges
with the implementation of innovative and interactive online resources that would allow
students actively to engage and attain hands-on skills (e.g., in critical care medicine).
Singh and Hardaker (2014) further concluded that both institutional and individual factors
should be considered in conjunction with modeling the adoption and diffusion of online
learning. However, most lessons learned are based on anecdotal, descriptive studies that
tend to offer insubstantial evidence to permit strong recommendations for online faculty
development (Cook & Steinert, 2013), and more critical evidence around the sustainability
of online learning is needed for policymakers and vital institutional decisions (Stepanyan,
Littlejohn, and Margaryan, 2013).

Conclusion and Implications
for Research and Practice:
Towards a Model of Digital Learning
One of the most significant requirements for further adoption of online learning is the
development of well-designed courses with interactive and engaging content, structured
collaboration between peers, flexible deadlines to allow students to pace their learning,
continuous monitoring of student progress, and the provision of formative feedback when
needed (Figure 6). Certainly, every aspect of such a design can be interpreted in different
ways. For example, a well-designed course with interactive and engaging content could
have many interpretations, and it is probable that instructors in different settings will have
different understandings and expectations as to what constitutes well-designed and engag-
ing. It is important to note that early second-order studies, such as that by Tallent-Runnels
et al. (2006), indicated that instructors requested support in online course development.
However, even when such resources were provided, instructors seldom made use of such
support services (Tallent-Runnels et al., 2006). Nevertheless, a set of general guidelines,
related to particular learning contexts, needs to exist as a commencement point for sup-
porting instructors. Here we stress the notion of general, since it is highly unlikely that there
is a single best course design for any particular context for all instructors.
Research shows that structured asynchronous online discussions are the most prominent
approach for supporting collaboration between students and to support learning (Darabi
et al., 2013; Macfadyen & Dawson, 2010; Rovai, 2007; Thomas, 2013). Darabi et al. (2013)
posit that the greatest impact on student performance is gained through “pedagogically
rich strategies” that include instructor participation, interaction with students, and facilita-
tion of student collaboration as well as continuous monitoring and moderating discussions.
In order to sustain the instructor’s role and provide effective support for the pedagogical
features that will foster learning, some of the instructor’s roles could be (or need to be)
delegated to students (Koch, 2014; Ravenna et al., 2012). A promising approach to de-
veloping self-regulatory skills using externally facilitated scaffolds is presented in Gašević,
Adescope, Joksimović, & Kovanović’s (2014) study. Gašević et al. (2014) argues that mean-
ingful student-student interaction that results in deep learning could be organized without
the instructor’s direct involvement in discussions. Specifically, the study showed a significant
118 Conclusion and Implications for Research and Practice

effect of instructional design that provides students with qualitative guidelines on how to
discuss rather than setting quantitative expectations only (e.g., number of messages posted)
(Gašević et al., 2014; Rovai, 2007).
Provision of formative, timely, and individualized feedback has also been identified as an
important challenge in the online learning environment (Barker, 2011; Gikandi et al., 2011;
Whitelock, 2010). Azevedo (1993) claimed that “[t]he use of the computer as the deliverer
of truly effective feedback will never be attained until it can be programmed to identify the
cause of user’s mistakes rather than merely verify (e.g., correct or wrong) and explain the
correct method” (p. 116). Likewise, more recent studies also highlighted the importance
of timely, formative, effective, and individualized feedback in order to efficiently support
learning (Gikandi et al., 2011; Tallent-Runnels et al., 2006). To our knowledge, there is no
fully automated process for the provision of formative feedback that would allow for more
scalable applications of this instructional method in teaching and learning. Such an interven-
tion also requires continuous monitoring of the learning process for every student. Given
the current development of learning management systems (Dabbagh, 2007; Macfadyen &
Dawson, 2010), collecting data is not an issue nowadays. Nevertheless, this brings numerous
ethical and privacy issues, such as who the data belongs to, who is allowed to use the data
and for what purposes, and what happens if a student does not allow his or her data to be
used to perform analytics. Further research and practice needs to provide clear answers to
those and many similar questions.
Conclusion and Implications for Research and Practice 119

oration
ab
ll
co
peer
ac
tion Learner interac
r
• Intrinsic motivation
inte
tio
• Self-efficacy and self-directness
n
• Responsible for their learning
• High digital literacy
Instructors
content
• Positive attitude towards
technology • Interactive and engaging
• Facilitation of learning Online Learning • Learning context with world
process examples
• Learner-centered
• Continuous monitoring of • Tasks/examples relevant
• Reduced costs
learner progress for practice
• High retention
• Guide students and provide • Flexibility
scaffolds • Increased enrollment
• Shared responsibilities • High interconnectedness
• among team members
Media
between learners and
• with students instructors • Rich set of affordances
• High digital literacy • Synergy of media and • Easy to navigate VLE/LMS
pedagogy
Instructional
Course Design
Strategies
• Guided and structured
• Rich pedagogical strategies discussions with clear
• Instructor’s involvement
Institutional Adoption expectations
• Instructional scaffold • small groups
• metacognitive • Institutional polices • extrinsic motivation
• strategic • Academic support (e.g., grades)
• prompts • technical • Flexible deadlines
• Support collaboration, • financial • Knowledge validation
cooperative work and social • infrastructure aligned with outcome
interaction • Clear strategic vision
• Provision of formative • Understanding of
timely, and individualized “cultural configuration”
feedback • Management as a role model
Figure 6 A conceptual diagram of the most significant factors that frame educational experience in online
learning settings

With the emergence of technological affordances, the landscape of online learning tends to
evolve as well. When we defined online learning at the beginning of this paper, we outlined
how certain technological advances might change our understanding of what fits under
the online, distance, or blended learning umbrella. One of the most obvious examples is
the use of video-conferencing, which was originally considered a technology belonging
to distance education (Bernard et al., 2009). However, now that videoconferencing is
easily available through the Internet, it could be considered part of online learning. With
the further development of online education, it seems that both learning paradigms are
evolving into a single learning approach — digital learning. Building on existing research
in distance, online, and blended learning and relying on findings from research in learning
sciences, digital learning emerges as a new approach to learning using technology (Siemens,
2014b). Digital learning might be structured as formal/informal, self-regulated, structured/
unstructured, and “lifelong”; however, the main goal of the new learning approach will be
promoting “research as practice and practice as research mindsets in college and university
systems engaged in researching digital learning and teaching” (Siemens, 2014b).
This report has revealed that initial studies of the effectiveness of online learning support
Clark’s (2000) view of the role of technology in the great media debate (e.g., Means et al.,
2009). However, we tend to agree with Ross and Morrison (2014) and Schmi d et al. (2014)
that the “synergy” of media and pedagogy is what actually matters. As Schmid et al. (2014)
noticed, Clark’s (1983) original argument dates back to the era when technology was barely
used for presentation purposes, thus not contributing much to the learning process. When
technological affordances are used to support meaningful interaction and engage students
in collaboration with their peers and instructors, technology plays an important role in the
learning process and even in improving pedagogy (Bernard et al., 2009; Schmid et al., 2014).
Therefore, pedagogy defines collaborative activities but media enables such activities to
occur (Ross et al., 2010; Ross & Morrison, 2014; Schmid et al., 2014).
Digital technologies and their applications in distance, online, and blended learning has
had significant influence on academic research and practice. However, “these technologies
have not revolutionized teaching and access to higher education as thoroughly as was
predicted by some” (OECD, 2007, p. 21). Within a general tendency to expand access to
learning for everyone, an open educational resources (OER) movement emerged with the
aim of accelerating the development of formal and informal learning. As a most promising
trend in that direction, Massive Open Online Courses (MOOCs) were developed within
the OER movement5 as a new form of online learning that aims at unlimited participation
5 It should be noted that this association of MOOCs with OERs should be considered in the context of the pioneering work on MOOCs
(Fini, 2009; Mackness, Mak, & Williams, 2010). The majority of the MOOCs offered since the hype in 2012 however do not build on the
principles of OERs.
Conclusion and Implications for Research and Practice 121

and open access to learning (and teaching resources) using the Internet (Pappano, 2012;
Siemens, 2005). Although recognized as a new trend in online and distance education
these large-scale courses required changes in instructional design, given that the majority
of widespread, effective instructional practices used in the online learning environment
could not scale well to the new massive settings (De Laat, 2006; Fournier, Kop, & Durand,
2014). Therefore, future research should also examine how online learning practices might
scale to MOOCs. A recent study of emerging topics in public discourse related to MOOCs,
Kovanović, Joksimović, Gašević, Siemens, and Hatala (2014), showed that within the last year
a general opinion has developed that MOOCs have failed to fulfill their purpose and their
promise. Specifically, Kovanović et al. (2014) showed that public media tend to criticize the
“overall MOOC experience” rather than a particular topic and suggest thorough research
on every aspect being criticized in order to understand major concerns in MOOC research
and adoption. One possible reason for such a trend in the current opinion on MOOCs in the
mainstream media might be the lack of the focus on instructional design, as indicated by
Margaryan, Bianco, and Littlejohn (2015). The Margaryan et al. study, based on 76 randomly
selected MOOCs, showed that course content was well designed and delivered while most
courses analyzed showed low-quality instructional design. Further research is required to
investigate the possibilities for scaling up instructional methods that have proven successful
in online learning in order to address the requirements of learning with MOOCs.
Current research into online learning and MOOCs has provided new evidence for teaching and
learning in the digital environment and has raised many questions as well (Siemens, 2014b).
As Siemens (2014b) argues, there is great opportunity for further research to examine how
(and whether) institutions are redesigning distance and online courses based on the lessons
learned from MOOCs. Moreover, another potential line of research might be investigating how
universities position online and blended learning with respect to on-campus learning (Siemens,
2014b). Finally, current research and practice also shows that higher education has been pri-
marily focused on content design and curriculum development (Siemens, 2014b). However, in
order to move forward and “develop personalized and adaptive learning,” the development of
personal knowledge graphs and profiles is crucial (Siemens, 2014b). Personalized knowledge
graphs present a promising approach for collecting and mapping an individual’s knowledge
from learning in various settings (e.g., formal, informal, and workplace), and using the accu-
mulated knowledge to bridge the knowledge gaps and provide focused learning materials
(Siemens, 2014a, 2014b). On the other hand, some of the main challenges in distance, online,
and blended learning relate to developing personalised and adaptive learning pathways and
the provision of timely, formative and individualised feedback. Given that personalised and
adaptive learning are increasingly being incorporated within the research into a new learning
approach — digital learning, it is likely that the future of distance, online and blended learning
models will be ultimately subsumed by this new learning approach.

References
Ally, M. (2004). Foundations of educational theory for online learning. Theory and
Practice of Online Learning, 2, 15–44. Retrieved from http://cde.athabascau.ca/
online_book/ch1.html
Anderson, T. (2009). A Rose by Any Other Name: Still Distance Education - A Response
to D.R. Garrison Implications of Online and Blended Learning for the
Conceptual Development and Practice of Distance Education. International
Journal of E-Learning & Distance Education, 23(3), 111–116. Retrieved from
http://www.editlib.org/p/105545/
Arbaugh, J. B., & Benbunan-Fich, R. (2007). The importance of participant interaction in

online environments. Decision Support Systems, 43(3), 853 – 865. doi:http://
dx.doi.org/10.1016/j.dss.2006.12.013
Azevedo, R. (1993). A Meta-analysis on the Effects of Computer-presented Feedback

on Learning from Computer-based Instruction. Concordia University. Retrieved
from https://archive.org/details/ERIC_ED385235
Barker, T. (2011). An Automated Individual Feedback and Marking System: An Empirical

Study. Electronic Journal of E-Learning, 9(1), 1–14. Retrieved from www.ejel.
org/issue/download.html?idArticle=163
Beinkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching and Learning
Through Educational Data Mining and Learning Analytics: An Issue Brief (No.
ED-04-CO-0040) (pp. 1–57). US Department of Education, Office of Educational
Technology. Retrieved from http://www.cra.org/ccc/files/docs/learning-analytics-
ed.pdf
& Bethel, E. C. (2009). A Meta-Analysis of Three Types of Interaction Treatments
in Distance Education. Review of Educational Research, 79(3), 1243–1289.
doi:10.3102/0034654309333844
References 123
Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., … Huang,
379–439. doi: 10.3102/00346543074003379
Borokhovski, E., Tamim, R., Bernard, R. M., Abrami, P. C., & Sokolovskaya, A. (2012).
Are contextual and designed student–student interaction treatments equally
effective in distance education? Distance Education, 33(3), 311–329. doi:10.108
0/01587919.2012.723162
Carroll, C., Booth, A., Papaioannou, D., Sutton, A., & Wong, R. (2009). UK health-care
professionals’ experience of on-line learning techniques: A systematic review
of qualitative data. Journal of Continuing Education in the Health Professions,
29(4), 235–241. doi:10.1002/chp.20041
Chia-Wen, T., Pei-Di, S., & Yi-Chun, C. (2013). Research trends in meaningful learning
research on e-learning and online education environments: A review of studies
published in SSCI-indexed journals from 2003 to 2012. British Journal of
Educational Technology, 44(6), E179 – E184. doi: 10.1111/bjet.12035
Chumley-Jones, H. S., Dobbie, A., & Alford, C. L. (2002). Web-based learning: Sound
educational method or hype? A review of the evaluation literature. Academic
Medicine, 77(Suppl10), S86–S93. doi:10.1097/00001888-200210001-00028
Clardy, A. (2009). Distant, On-line Education: Effects, Principles and Practices. Online
Submission, Retreived from ERIC Database. Retrieved from http://files.eric.
ed.gov/fulltext/ED506182.pdf
Clark, R. E. (1983). Reconsidering research on learning from media. Review of

Educational Research, 53(4), 445–459. doi: 10.3102/00346543053004445
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research
and Development, 42(2), 21–29. doi:10.1007/BF02299088
Clark, R. E. (2000). Evaluating distance education: Strategies and cautions. Quarterly

Review of Distance Education, 1(1), 3–16. Retrieved from http://www.editlib.
org/p/91945
124 References
Cohen, N. L., Carbone, E. T., & Beffa-Negrini, P. A. (2011). The Design, Implementation,
and Evaluation of Online Credit Nutrition Courses: A Systematic Review.
Journal of Nutrition Education and Behavior, 43(2), 76 – 86. doi:http://dx.doi.
org/10.1016/j.jneb.2010.04.001
Cook, D. A., Garside, S., Levinson, A. J., Dupras, D. M., & Montori, V. M. (2010). What
do we mean by web-based learning? A systematic review of the variability
of interventions. Medical Education, 44(8), 765–774. doi:10.1111/j.1365-
2923.2010.03723.x
Cook, D. A., Levinson, A. J., & Garside, S. (2010). Time and learning efficiency in
Internet-based learning: a systematic review and meta-analysis. Advances in
Health Sciences Education, 15(5), 755–770. doi:10.1007/s10459-010-9231-x
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M.
(2010). Instructional Design Variations in Internet-Based Learning for Health
Professions Education: A Systematic Review and Meta-Analysis. Academic
Medicine, 85(5). Retrieved from http://journals.lww.com/academicmedicine/
Fulltext/2010/05000/Instructional_Design_Variations_in_Internet_Based.42.aspx
Cook, D. A., & Steinert, Y. (2013). Online learning for faculty development:
A review of the literature. Medical Teacher, 35(11), 930 – 937. doi:
10.3109/0142159X.2013.827328
Dabbagh, N. (2007). The online learner: Characteristics and pedagogical implications.

Contemporary Issues in Technology and Teacher Education [Online Serial], 7(3),
217–226. Retrieved from http://www.citejournal.org/vol7/iss3/general/article1.cfm
Darabi, A., Liang, X., Suryavanshi, R., & Yurekli, H. (2013). Effectiveness of Online
Discussion Strategies: A Meta-Analysis. American Journal of Distance
Education, 27(4), 228–241. doi:10.1080/08923647.2013.837651
De Laat, M. (2006). Networked learning. Police Academy of the Netherlands: Apeldoorn.

Retrieved from http://www.e-learning.nl/files/dissertatie%20maarten.pdf
Du, S., Liu, Z., Liu, S., Yin, H., Xu, G., Zhang, H., & Wang, A. (2013). Web-based distance
learning for nurse education: a systematic review. International Nursing Review,
60(2), 167–177. doi:10.1111/inr.12015
References 125
Fini, A. (2009). The technological dimension of a massive open online course: The case
of the CCK08 course tools. The International Review of Research in Open and
Distance Learning, 10(5). Retrieved from http://www.irrodl.org/index.php/irrodl/
article/view/643
Fournier, H., Kop, R., & Durand, G. (2014). Challenges to Research in MOOCs. Journal of
Online Learning & Teaching, 10(1), 1–15. Retrieved from http://jolt.merlot.org/
vol10no1/fournier_0314.pdf
Garrison, D. R. (2011). E-learning in the 21st century: A framework for research and
practice. Taylor & Francis. ISBN 9780203166093
Gašević, D., Adescope, O., Joksimović, S., & Kovanović, V. (2015). Externally facilitated
Regulation Scaffolding and Role Assignment to develop Cognitive Presence
in Asynchronous Online Discussions. The Internet and Higher Education, 24,
53–65. doi:http://dx.doi.org/10.1016/j.iheduc.2014.09.006
Gikandi, J. W., Morrow, D., & Davis, N. E. (2011). Online formative assessment in higher
education: A review of the literature. Computers & Education, 57(4), 2333 –
2351. doi:http://dx.doi.org/10.1016/j.compedu.2011.06.004
Grandzol, C. J., & Grandzol, J. R. (2006). Best Practices for Online Business Education.
The International Review of Research in Open and Distance Learning, 7(1).
Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/246
Harasim, L. (2000). Shift happens: online education as a new paradigm in learning. The
Internet and Higher Education, 3(1–2), 41 – 61. doi:http://dx.doi.org/10.1016/
S1096-7516(00)00032-4
Jumaat, N. F., & Tasir, Z. (2014). Instructional Scaffolding in Online Learning Environment:
A Meta-analysis. In 2014 International Conference on Teaching and Learning
in Computing and Engineering (LaTiCE 2014) (pp. 74–77). Johor, Malasyia.
doi:10.1109/LaTiCE.2014.22
Jurewitsch, B. (2012). A Meta-Analytic and Qualitative Review of Online versus Face-

to-Face Problem-Based Learning. Journal of Distance Education, 26(2), n2.
Retrieved from http://www.ijede.ca/index.php/jde/article/view/787/1399
126 References
Koch, L. F. (2014). The nursing educator’s role in e-learning: A literature review. Nurse
Education Today, 34(11), 1382 – 1387. doi:http://dx.doi.org/10.1016/j.
nedt.2014.04.002
media reveals about MOOCs? British Journal of Educational Technology, in-press.
Technology Research and Development, 42(2), 7–19. doi: 10.1007/BF02299087
Landers, R. N. (2009). Traditional, web-based, and hybrid instruction: A comparison

of training methods. ProQuest Information & Learning. Retrieved from
http://proxy.lib.sfu.ca/login?url=http://search.ebscohost.com/login.
aspx?direct=true&db=psyh&AN=2009-99211-008&site=ehost-live
Macfadyen, L. P., & Dawson, S. (2010). Mining {LMS} data to develop an “early warning
system” for educators: A proof of concept. Computers & Education, 54(2), 588 –
Mackness, J., Mak, S., & Williams, R. (2010). The ideals and reality of participating in
a MOOC. In Proceedings of the 7th International Conference on Networked
Learning 2010 (pp. 266–275), Aalborg, Denmark. Retrieved from http://www.
lancaster.ac.uk/fss/organisations/netlc/past/nlc2010/abstracts/PDFs/Mackness.pdf
Macon, D. K. (2011). Student Satisfaction with Online Courses versus Traditional Courses:
A Meta-Analysis. Retrieved from http://search.proquest.com.proxy.lib.sfu.ca/
docview/858611481?accountid=13800
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional quality of Massive Open
Online Courses (MOOCs). Computers & Education, 80(0), 77 – 83. doi:http://
dx.doi.org/10.1016/j.compedu.2014.08.005
Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A Structural Equation Model of
Predictors for Effective Online Learning. Journal of Management Education,
29(4), 531–563. doi: 10.1177/1052562904271199
References 127
Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of Evidence-
Based Practices in Online Learning: A Meta-Analysis and Review of Online
Learning Studies. US Department of Education. Retrieved from http://eric.
ed.gov/?id=ED505824
Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and
blended learning: A meta-analysis of the empirical literature. Teachers College
Record, 115(3), 1–47. Retrieved from http://www.sri.com/sites/default/files/
publications/effectiveness_of_online_and_blended_learning.pdf
Moallem, M. (2003). An interactive online course: A collaborative design model.

doi:10.1007/BF02504545
Moore, M. G. (1993). Theory of transactional distance. In In D. Keegan (Ed.), Theoretical

principles of distance education (pp. 22–38). New York: Routledge.
Organization for Economic Co-operation and Development. (2007). Giving knowledge

for free: The emergence of open educational resources. Paris, France: OECD
Publishing. Retrieved from http://www.worldcat.org/title/giving-knowledge-for-
free-the-emergence-of-open-educational-resources/
Pappano, L. (2012, November 2). The Year of the MOOC. The New York Times.
Retrieved from http://goo.gl/6QUBEK.
Peterson, D. S. (2008). A meta-analytic study of adult self-directed learning and online

nursing education: A review of research from 1995 to 2007. Retrieved from
http://search.proquest.com.proxy.lib.sfu.ca/
Ravenna, G., Foster, C., & Bishop, C. (2012). Increasing Student Interaction Online:
A Review of the Literature in Teacher Education Programs. Journal of
Technology and Teacher Education, 20(2), 177–203. Retrieved from http://eric.
ed.gov/?id=EJ975827
Roberts, R. M. (2011). Best instructional practices for distance education: A meta-

analysis. Retrieved from http://search.proquest.com.proxy.lib.sfu.ca/
128 References
Ross, S. M., Morrison, G. R., & Lowther, D. L. (2010). Educational technology research
past and present: Balancing rigor and relevance to impact school learning.
Contemporary Educational Technology, 1(1), 17–35. Retrieved from http://
aulavirtual.eaie.cvudes.edu.co/publico/lems/L.000.002.MG/Documentos/
Anexos/Cap3/1.pdf
Ross, S. M., & Morrison, J. R. (2014). Measuring meaningful outcomes in consequential

contexts: searching for a happy medium in educational technology research
(Phase II). Journal of Computing in Higher Education, 26(1), 4–21. doi:10.1007/
s12528-013-9074-6
Rovai, A. P. (2007). Facilitating online discussions effectively. Internet and Higher

Education, 10(1), 77–88. doi: 10.1016/j.iheduc.2006.10.001
Rudestam, K. E., & Schoenholtz-Read, J. (2010). Handbook of Online Learning. SAGE

Publications. Retrieved from http://books.google.ca/books?id=gkq4lX7-V-sC
Schlosser, L. A., & Simonson, M. R. (2006). Distance Education: Definition and Glossary
of Terms. IAP, Information Age Pub. Retrieved from http://books.google.ca/
books?id=Vf4ayMDBnFEC
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M. A.,
… Woods, J. (2014). The effects of technology use in postsecondary education:
A meta-analysis of classroom applications. Computers & Education, 72(0), 271 –
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International
Journal of Instructional Technology and Distance Learning, 2(1), 3–10. Retrieved
from http://www.itdl.org/journal/jan_05/article01.htm
Siemens, G. (2014a, July 5). elearnspace › Activating Latent Knowledge Capacity.

Retrieved from http://www.elearnspace.org/blog/2014/07/05/activating-latent-
knowledge-capacity/
Siemens, G. (2014b, November 18). elearnspace › Digital Learning Research Network

(dLRN). Retrieved from http://www.elearnspace.org/blog/2014/11/18/digital-
learning-research-network-dlrn/
References 129
Singh, G., & Hardaker, G. (2014). Barriers and enablers to adoption and diffusion of
eLearning. Education + Training, 56(2/3), 105–121. doi:10.1108/ET-11-2012-
0123
Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The Comparative Effectiveness
of Web-based and Classroom Instruction: A Meta-analysis. Personnel
Psychology, 59(3), 623–664. doi:10.1111/j.1744-6570.2006.00049.x
Stepanyan, K., Littlejohn, A., & Margaryan, A. (2013). Sustainable e-Learning: Toward
a Coherent Body of Knowledge. Journal of Educational Technology &
Society, 16(2). Retrieved from http://search.proquest.com.proxy.lib.sfu.ca/
Styer, A. J. (2007). A grounded meta-analysis of adult learner motivation in online

learning from the perspective of the learner. Retrieved from http://search.
proquest.com.proxy.lib.sfu.ca/docview/304723729?accountid=13800
Šumak, B., Heričko, M., & Pušnik, M. (2011). A meta-analysis of e-learning technology
acceptance: The role of user types and e-learning technology types. Computers
in Human Behavior, 27(6), 2067 – 2077. doi:http://dx.doi.org/10.1016/j.
chb.2011.08.005
Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., &
Liu, X. (2006). Teaching Courses Online: A Review of the Research. Review of
Educational Research, 76(1), 93–135. doi:10.2307/3700584
Taylor, J. C. (2001). Fifth generation distance education. Presented at the Keynote

address delivered at the ICDE 20th World Conference, Dusseldorf, Germany.
Retrieved from http://www.usq.edu.au/users/taylorj/conferences.htm
Thomas, J. (2013). Exploring the use of asynchronous online discussion in health care
education: A literature review. Computers & Education, 69(0), 199 – 215.
doi:http://dx.doi.org/10.1016/j.compedu.2013.07.005
Whitelock, D. (2010). Activating Assessment for Learning: are we on the way with Web
2.0. In In: Lee, Mark J. W. and McLoughlin, Catherine eds. Web 2.0-Based-E-
Learning: Applying Social Informatics for Tertiary Teaching (Vol. 2, pp. 319–342).
IGI Global. Retrieved from http://oro.open.ac.uk/id/eprint/19706
130 References
Wolbrink, T. A., & Burns, J. P. (2012). Internet-Based Learning and Applications for
Critical Care Medicine. Journal of Intensive Care Medicine, 27(5), 322–332.
doi:10.1177/0885066611429539
Wong, G., Greenhalgh, T., & Pawson, R. (2010). Internet-based medical education: a
realist review of what works, for whom and in what circumstances. BMC Medical
Education, 10(1), 12. doi: 10.1186/1472-6920-10-12
References 131
The History and State of
Credentialing and Assessment
Charles Lang
Harvard University
George Siemens
Dragan GaševiĆ
Shane Dawson
133

Abstract
The complexity of the credentialing landscape has dramat-

ically increased over the last twenty years. For most of the
20th century, the value of a university degree and the path
to its attainment was broadly understood and the terms of
the agreement between the state, educational institutions,
and students were clear. State bodies would accredit higher
education institutions with the power to grant degrees and
those institutions would develop programs so that students
could earn degrees that held meaning for employers. Over
the last two decades however, each step in this system has
come under question: accreditation, program development,
degree earning, and signaling. This new landscape of inquiry
and design reveals a tension between the desire to stan-
dardize how credentials are measured and an explosion in
the possible ways that this standardization could occur. This
tension is being played out along two axes with respect to
how credentials are defined: 1) whether by a fixed or flexible
amount of time and 2) whether based on knowledge or
competency. A review of novel credentialing forms follows,
classified in these terms.
Despite the long dispute over the role of universities within
Credentialing
Shifts Over Time
society, the meaning of their credentials is often considered
standard. At least since Kant’s Conflict of the Faculties (1798)
and Cardinal Newman’s The Idea of the University (1854), a
core tension of the university has been between its role as a
utilitarian trainer of workers on the one hand and bastion of
justice, ideas, and change on the other. In the present day,
this tension plays out in the demands of industry to produce
more computer science majors while humanities departments
and liberal arts colleges seek to preserve their funding and
a grander view of education as a central component of de-
mocracy and social progress. To some extent, this contest is
intensified because the meaning of the credential is seen as
standard and monolithic. Whichever becomes the dominant
meaning will dictate the future priorities of institutions and
the role of institutions within society.
The standard model of the university credential is to some

extent a matter of branding. Employers have tended to
differentiate between applicants based on the status of their
institution rather than the particular assessments that led
to the acquisition of the credential (Bordón & Braga, 2013).
In this way, credentials act as a value statement about the
trustworthiness of institutions generally (universities as an
institution are trusted) and in particular (some universities
are more trusted than others). But how the credential has
been achieved is often overlooked. Policy makers and aca-
136 Credentialing Shifts Over Time

the history and state of credentialing and assessment
demics have argued over how best to assess educational outcomes, but the expectations
set by governments for assessment are loose and largely have not changed over time
(Laitinen, 2012). Generally, universities and colleges were seen to be the best arbiters of
how to assess their students’ progress, with governments setting broad goals, such as
how much time a degree should take and employers paying little attention to the details
of assessment (Eaton, 2001). We can look to recent political statements about the urgent
need for more graduates, such as the Bradley Report in Australia or the Obama Admin-
istration’s “American Graduation Initiative.” The presumption behind both these plans is
that university credentials mean something and that their assessments are sound. Yet, the
longstanding global tradition of professional assessment bodies, from the National Board
of Medical Examiners in the US to the Honourable Society of King’s Inns who administer
legal examinations in Ireland, seem to validate the idea that assessment accountability is
not part of the role of a university.
Historically, time has been a key component of how governments and colleges have chosen
to assess their students. This focus originated with a drive to bring business-style accounting
practices to higher education in the late 19th and early 20th centuries. The credit hour —
the time requirement for academic credit — emerged and became the major component of
degree attainment (Shedd, 2003, p. 5). An easily understood, standardized metric provides
a veneer of accountability to the process of teaching and learning. Its growth as a marker
of quality completion can be seen in its global spread; agreements between governments
and universities generally specify time assignment as the benchmark that students must
meet to acquire credentials across countries. For example, a specified number of credit
hours in the United States and Canada, 10 hours of work per credit in the UK, or course
completion composed of a certain number of hours of on-campus time in Australia, New
Zealand, and Hong Kong.
Credentialing Shifts Over Time 137

Both the meaning of a degree and its correlated measure of assessment (credit hours) have
come under increasing pressure over the last two decades due to several shifting contexts.
A major contributor to this pressure revolves around the economic value of a credential.
Rising tuition justifiably provokes governments and students to ask what the return on their
investment is going to be and this has generated considerable discussion around how to
measure that value both in terms of economic returns and in terms of learning (Barrett,
2014). As such, the utility of the credit hour, a metric never intended to measure learning
(Learned & Wood, 1938), has come under increased scrutiny. Furthermore, the need for
universities to justify the value of their degrees is compounded by globalization through
greater study-related migration. A global marketplace for students has forced universities
to compete by demonstrating globally recognized credentials, thus feeding into a sense
of urgency about how this can be achieved.
In parallel to this economic pressure, there is uncertainty about the role of the university
within society. Universities have moved from isolated research and education institutions
toward being an essential piece of national economic planning (Duderstadt, 2000). The rise
of the “Triple Helix” of innovation, where universities are more integrated with industry and
government, requires credentials that serve these new goals to satisfy these stakeholders
(Leydesdorff, 1995; Leydesdorff & Etzkowitz, 1996). Besides the demands that industry
makes about the content that students should be learning, industry leaders are also ques-
tioning the effectiveness of that learning. One industry report claims that managers believe
that less than 50% of applicants can “communicate ideas or explain information clearly”
(Dua, 2013) — an essential skills for all industries. That industry has an opinion, not only
about what content is taught but how effectively it is being taught, is important. It places
further pressure on colleges to demonstrate student learning and therefore, the value of
the credentials they award. However, institutions have a choice about how they frame their
response. They may meet this challenge in the same terms as the problem is posed by
industry, by attempting to prove that they produce valuable graduates in economic terms.
Conversely, they may broaden the debate by insisting that education is about more than
work-readiness and earnings-potential. There are many “bottom lines,” including produc-
ing citizens who can be morally productive, positioning learning and assessment outside
the purview of industry. In either case, the imperative remains: establishing the means for
recording and validating learning outcomes.
Finally, the Internet has altered the educational landscape considerably. It has provided
new pathways for delivering education to larger numbers of people and measuring both
what and whether or not they have learned. Prior to the developments facilitated by In-
ternet technology, there were few alternative models to the lecture-examination cycle of

the traditional university. There are a myriad of ways to both engage students in learning
and to measure whether that learning has occurred. For the first time in 200 years, the
tools may exist to reformulate the educational process and what constitutes a credential.
This has generated both a sense of possibility and uncertainty within higher education as
innovations in degree programs continue to develop and, with them, new actors who may
challenge the traditional university model.
The shifting educational landscape has brought with it new questions about the traditional
credentials that have been assigned meaning and how those credentials should be reas-
sessed to understand that meaning. On the one hand, this inquiry has generated uncer-
tainty for administrators and policy makers, creating urgency around decision-making and
committing to re-standardizing the credentialing process. On the other hand, the growth
of online computing coupled with society’s desire to prepare students for the workforce
has meant parallel growth in the ways that credentials could be assessed, increasing the
possibility that standardization might not in fact be either possible or desirable. The re-
sulting dilemma essentially pits the desire to validate credentials through standardization
against an explosion in the number of possible ways that this standardization could occur.
Standardization
The major initiative to standardize credentialing to date has been the Bologna Process, which
began in 1999. Signatories to the Bologna Accords agreed to “harmonize” the structure
of European education to create equivalence between degrees across 29 countries. This
standardization approach was designed to address some of the tensions described above.
Free movement of intellectual capital and the consolidation of research and education
within Europe aimed to make the Eurozone more globally competitive within the knowl-
edge economy. This move was intended not only to attract bright students from countries
outside the Bologna states but also, by making credential requirements and labels more
similar to those in the US, to allow students and employers less friction across the Atlantic.
The success of the Bologna Process and the speed with which standardization was im-
plemented, despite being a voluntary set of recommendations, came as somewhat of a
surprise to policy analysts, especially in the United States (Gaston, 2012). The cornerstone
of the Process is the European Credit Transfer and Accumulation System (ECTS) — a way
for students to transfer credits between the institutions of signatory countries.
In contrast to this growth in standardization in Europe, credentialing in the United States

remains fractured between states and among schools. For example, state accreditation
bodies have differing requirements for professional accreditation, forcing professionals to
retrain if they wish to practice in a different state (Darling-Hammond, 2000, pp. 19–22).
Likewise, within a given degree (even in the same state), credits attained at one college are
rarely transferable to another (Quaye & Harper, 2014, p. 277). There are clear economic
costs to this situation, labor shortages in one state cannot easily be met by people moving
from another and degree seekers do not receive recognition for their knowledge and skills
and must pay to repeat courses if they change institutions.
The cumulative impact of credentialing issues seems to make the US a particularly fertile
place for the development of alternatives. Dissatisfaction with a fractured system, high
Internet connectivity and accessibility, and a mismatch between the needs of employers
and the skills of graduates have created a flood of alternate pathways to credentials not yet
being replicated globally. This growth further complicates the notion that the US will ever
achieve standardization in credentialing, either in the manner that Europe has, or even in
the manner the US historically had. Further complicating the path to standardization are
emerging questions around assessment of student knowledge and learning.
Assessment
The functionality and connectivity afforded by the Internet has enabled alternative forms of
credentialing by facilitating new forms of assessment. These alternatives can be categorized
into two basic approaches: proficiency testing (also called knowledge or objective testing)
and competency-based testing (also called performance assessment). The difference between
the two is the focus on “what people know” versus “what people do” (Davey et al., 2000).
Proficiency testing has its roots both in Imperial China (Miyazaki, 1976) and in the rapid
growth of psychological methodology at the end of the 19th century. This history has lead to
the development of a complex system of methods with strict rules of validity and reliability
and to the field of psychometrics. The fundamental idea behind proficiency assessments is
that they can attest to what a person knows; the implied utility is that this information can
reliably predict future performance. Although more commonly associated with entrance into
degree programs, these tests are also used for such professional credentialing as nursing
certification (DeVon et al., 2007).

Conversely, competency-based assessment seeks to understand precisely the relevant tasks
that people can perform. Competency-based assessments may involve traditional tests
but also embrace a wider variety of assessment methods such as portfolios, interviews,
presentations, and skill demonstrations. As such, there is no single understanding of what
validity looks like within competency-based assessment and no formal manner in which
reliability is established (Thomson, Saunders, & Foyster, 2001). Rather, validity and reliability
are established by aligning the competence demonstration with the task to be performed;
the more similar, the better the assessment (Ten Cate & Scheele, 2007). Ultimately, the
Internet and mobile computing enable a wider range of competency-based assessment
through the quantification of more contexts and skills. This leads to assessment innovation
through the application of data analytics and machine learning methodologies to student
data streams (Donkers, Govaerts, Driessen, & Verhoeven, 2008).
Beyond competency, proficiency, and which measures will be applied, questions have also
emerged in the domain of assessment regarding how time is treated: is it fixed or flexible?
Currently, the vast majority of credentials are dictated by fixed time: the test lasts an hour,
a school year is 40 weeks, the degree takes four years. Fixed time standardizes assessment
in a convenient way, though it is rigid and its correspondence to learning, knowledge, or
skill is neither assured nor obvious (Laitinen, 2012). In contrast, flexible time asks how long
it took to complete the task or degree. It provides information about individual capacity
and the dynamics of learning but also allows for degree requirements to adjust to students’
lives. The ability to work through a degree either faster or slower allows students to accom-
modate jobs and families, thus increasing accessibility to educational opportunities for a
wider range of the population. Frameworks for considering time components are bound
to be critical in navigating the new complexities in credentialing.

Taxonomy of
Credentials
We can use the distinction between competency/proficiency and flexible/fixed time (Table
1) to understand new forms of credentialing, categorize and discuss historical forms of
credentialing, and hypothesize the future of the credentialing debate. In the top left are
traditional credentials, such as college and associates degrees. These rely on fixed amounts
of time, usually years, and proficiency assessments such as exams and essays to test knowl-
edge. The three other boxes represent combinations of time and assessments suggested
and implemented as replacements to the traditional fixed/proficiency model.
Table 1 Categories of credentials based on whether they use proficiency or competency assessment and whether
they must be acquired over a fixed amount of time or done at a pace determined by the student
Assessment
Proficiency Competency
Demonstrates knowledge Demonstrates competency

Fixed
Uses tests or assignments Uses a range of assessments
Occurs within a set timeframe Occurs within a set timeframe

time
Demonstrates knowledge Demonstrates competency

Flexible
Uses tests or assignments Uses a range of assessments
Student paced Student paced
142 Taxonomy of Credentials

Flexible Time, Proficiency-Based Credentials
Flexible time, proficiency-based credentials rose in the late 1990s and early 2000s with the
emergence of for-profit online degrees and distance education programs such as those
pioneered by the Open University. Distance education largely followed the lead of tradi-
tional universities, using proficiency as the basis for assessment (Holmberg, 2005, p. 37).
Similarly, online for-profit assessment originally mirrored that of the traditional university,
but began to shift over the last four years.
Flexibility and convenience have been major attractions of both online, for-profit creden-
tials and distance education. Online for-profits have flexible course schedules and credit
arrangements that do not assign deadlines with respect to degree completion. Online
components also allow greater flexibility, allowing students to study at their own pace. In
order to increase convenience, the University of Phoenix, for example, also geographically
locates their brick and mortar institutions close to the student workplaces, such as within
malls and shopping areas. However, this flexibility has been a double-edged sword, as
tiered tuition links cost to the speed of student progress. For example, DeVry University
has a discounted rate when students take seven or more classes at one time, and all of the
for-profit online schools have recurring fees that can further inflate costs for each semester
a student enrolls.
Given emerging options, the flexible time, proficiency assessment model may fall by the
wayside as innovations in competency-based assessment grow and as flexibility becomes
an expectation rather than a novel feature. In fact, the online for-profit sector has been in
decline for several years, with enrollment down at the University of Phoenix since 2010 and
DeVry since 2013. As such, these schools, along with other online institutions, are currently
in a painful reorientation process that includes demonstrating that they are able to provide
the types of graduates that industry wants. This includes being able to show that their
graduates have particular skills — skills that are best demonstrated and assessed using
competency-based assessments. The University of Phoenix, for example, takes content
suggestions from various Chambers of Commerce and, having articulation agreements
with many corporations that define desirable skills, has now started a competency-based
program by partnering with the National Association of Manufacturers. It appears, there-
fore, that the flexible time, proficiency-based model of credentialing may be on the way
out. Providers will either remain traditional fixed time, proficiency models or innovate into
remain competitive. However, the flexible time, proficiency-based model may continue
within the world of no-work learning.
Taxonomy of Credentials 143

Learning about the preservation of democracy or the struggle for equity are hopefully
life-long pursuits for which a flexible timeline is likely more appropriate. In addition, these
are not aspects of society that students should be able to fail, so the ability to continue
to work on a skill over a long period, and get credit for that perseverance, is important.
Social perspective may take years to learn in any permanent way, and it is difficult to see
how it could be credited without the dual structure of flexibility and the demonstration of
proficiency (Gehlbach, Young, & Roan, 2012, p. 298).
Fixed Time, Competency-Based Credentials

The fixed time, competency-based credential may seem a strange category, as many
people, including the competency-based assessment theorist Malan (2000), define com-
petency-based credentials as those with flexible scheduling. However, it is informative to
split these apart, as it is possible to assess student competency while still adhering to a
fixed time period within which degree requirements must be met. This category includes
Massive Open Online Course (MOOC) providers who offer certificates for single courses —
students complete a MOOC course within a specified time to receive a certificate. Often
the competency piece of these certificates is a function of the MOOC content: technical
and computer science courses ask students to produce artifacts that lend themselves to
competency-based assessment. These artifacts require the demonstration of skills in coding
and analysis that can readily be converted into a competency measure.
One step more complex than a single certification is the nanodegree. Nanodegrees, launched
by Udacity in 2013, attempt to capitalize on the substantial success of certificates, the fastest
growing form of credentialing (Carnevale, Rose, & Hanson, 2012, p. 4). In less than a year,
these online qualifications are designed to provide specific technical skills for a particular
industry. A nanodegree is more than a single MOOC course, but less than a traditional
professional qualification and comes with a stamp of legitimacy from a partner company; in
Udacity’s case, AT&T. Udacity nanodegrees are currently limited to the technology sector,
with offerings in web development, app building, and data analytics. Rather than compete
with traditional degrees, the nanodegree is explicitly aimed at students unable to attend
traditional universities or technical schools. It is mostly marketed at current jobholders, and
designed to be pursued after work or on weekends. These degrees are competency based
by design and explicitly marketed as bridging the gap between what traditional degree
holders know and what skills companies need.
One area of innovation in the fixed time, competency-based model is the recently popular
144 Taxonomy of Credentials

“boot camp” phenomena. Boot camps, such as App Academy, Byte Academy, or Code
Core, are short (3 month), intensive, often selective programs that aim to train students in a
particular skill set, such as web development, app design, or a particular coding language.
Part of their mission is to supply workers for the growing IT sector, particularly development
jobs. As such, they tend to have close relationships with particular corporations to whom
they intend to send their students for employment. In economic terms, this system seems
to be working well, with demand from students high and graduate placements averaging
90% (McGuire, 2014). Students see an immediate return on their investment, allowing boot
camps to charge substantial sums, in the region of $1,000 USD a week. The price tag and
the residence requirement mean that the pool this model services is small; however, the
success of the boot camp model is attracting other sectors to emulate it. A marketing boot
camp, Market Campus, opened its doors in Provo, Utah, in 2014. If the boot camp model
can be applied to a broader range of industries then it may become a substantial source of
credentials in the future. It remains to be seen though how resilient this model is to broader
economic forces — will boot camps still exist when the need for developers has subsided?
Flexible Time, Competency-Based Credentials

Over the last decade, there has been rapid growth in the number of credentials that rely
on skill demonstrations and flexible time lines. However, the Western Governor’s University
(WGU) substantially developed this model of credential in the late 1990s. WGU set out to
find a model of college education that could support the growing number of applicants, at
a reasonable cost, with a high employability of graduates. Over the last 20 years, they have
developed a competency-based model that is time flexible, skill based, and aligned with
what employers are looking for in graduates. A key component of the time flexibility in the
WGU model is generous crediting of prior experience. If a student can demonstrate their
ability because they have worked in the industry related to the degree, they are credited
with that competency. This is in contrast to a traditional university course or a fixed time,
competency model in which prior experience may be able to substitute for a small part of
a degree but is limited to a few credits per year through placement exams. More generous
placement credits are available through the University of Wisconsin’s Flex Program but
currently, only two universities in the US allow students to “place out” of an entire degree,
Excelsior College and Thomas Edison State College. Although students rarely take this
option, it signals an institutional belief that people enter formal learning environments with
valuable pre-existing knowledge. This philosophy does seem to be gaining traction at other
schools. For example, Southern New Hampshire University’s College for America program
is marketed as a degree that builds on current skills to align students with employers.
Taxonomy of Credentials 145

The University of Maryland University College similarly advertises its belief that “learning
acquired outside of the classroom is valuable” and SUNY advertises a philosophy “that
people deserve credit for college-level learning no matter how it was acquired.” Yet even
with this philosophy, the organizational structures to value or use that credit remain limited.
Measuring
Competencies
In terms of measuring competencies, the original competency-based programs still lead

the way since competency-based assessment development is both difficult and resource
intensive (Kinser, 2007). Spearheading these efforts are universities with strong distance
education traditions such as Western Governors University; Southern New Hampshire
University; Excelsior College; Chapman University’s subsidiary, Brandam University; and
the University of Wisconsin.
Western Governor’s University has had more than two decades to develop its competency
assessments within a flexible time model. Particular innovations include developing difficulty
metrics so that students can be awarded credit based on the difficulty of a competency
rather than making all competencies equivalent, tracking the use of learning resources and
student performance so that resources can be evaluated, and putting all assessments into
a single location that can be accessed securely online.
A major hurdle for competency assessment has not been demonstrating competency, but
rather how should a failure to demonstrate competency be interpreted. After all, there may be
consequences to awarding a degree to a nurse who has demonstrated only eight out of nine
competencies. To address this issue, Rush University has developed a three-tiered system for its
nursing competencies, consisting of awareness, knowledge, and proficiency (Swider et al., 2006).
These competency levels are available to future instructors within the program and appear on
transcripts. However, the time and cost of generating these assessments is currently a barrier to
expanding them to other courses within the university. Further, competency models require the
146 Measuring Competencies

clear delineation of learning outcomes (Shapiro, 2014), a task that may require considerable time
and effort. These difficulties may drive institutions to seek out other flexible time, competen-
cy-based options, such as utilizing online functionality or better prior-knowledge assessments.
They may be beaten to this goal, of course, by some non-institutional actors — those who see
profitability in data such as the College Board’s College Level Examination Program (CLEP) or
the credit-by-examination collaboration between Excelsior College and Pearson, UExcel. In this
partnership Pearson provides testing technology to Excelsior students to allow them to gain
college credit for learning from other institutions or on-the-job training.
Peer-Based Credentials
Peer-based credentials represent one possible flexible time, competency-based alternative.
The essential ingredient is that an assessment is evaluated by a group of peers. Based on
a set of loose criteria, peers judge whether or not they believe an individual is competent
at a task. They may also judge whether that individual has been active enough within a
given period to deserve recognition. In this innovation, peers take over both regulating
time and assessment. As such, unlike the tightly defined competencies within a traditional
professional accreditation program (such as those in nursing), competencies within a peer-
based credentialing environment are socially negotiated and may vary substantially from
one community to another.
Both informal and formal peer-based credentials have developed specifically within the
software development and online technical communities. We might consider the prototypical
examples of each to be the project management platform Github (informal) and the Q&A
website Stack Overflow (formal). Github users interact in various ways to understand the work
of other users, seek answers to technical questions, and develop possible collaborations.
This requires users to become adept at estimating each other’s value. Marlow et al. (2013)
describe the main metrics used in this appraisal as “history of activity across projects” and
“successful collaborations with key, high status projects.” This informal peer assessment
process informs the value of individuals within the platform — those who can successfully
produce work that is deemed high quality and important — advancing their careers both
within the platform and in the real world.
The Q&A site Stack Overflow (http://stackoverflow.com/) has demonstrated the power of
a more formal system of peer-based reputation credits. Stack Overflow is a question and
answer website for programmers, facilitating code and technical problem solving through
community-sourced answers. The efficiency of this system is remarkable, with 92% of ques-
Measuring Competencies 147

tions being answered within 11 minutes (median) (Mamykina, Manoim, Mittal, Hripcsak, &
Hartmann, 2011). Incentives for both questioners and answer givers seem very well aligned.
Questioners get their queries answered while answer givers are rewarded with reputation
credits. Actions that elevate reputation credits include getting upvotes on a given answer,
getting upvotes for asking a useful question, or having your answer officially accepted by
the questioner. Reputation can also be reduced by having your question or answer down
voted or when you down vote an answer. These rules are clearly designed not just to
provide credentialing information but also to regulate interactions within the community.
This illuminates an important aspect of peer-based credentialing — it is a social metric. For
peer-based assessment to operate effectively, it must be on some level, an assessment of
someone’s ability to operate within a given social environment. The ability to play by the
social rules is the assessment and is considered an integral part of the knowledge. However,
in contrast to traditional proficiency assessment, which is concerned with removing the bias
of the assessor, peer-based assessment risks bias and the tyranny of the majority, which still
needs to be addressed. One possible remedy may be that the peer metrics not be fully
trusted until algorithmic processing can reveal something useful.
The peer-based assessment model can be seen in the endorsement system employed by
LinkedIn. Endorsements within LinkedIn have a very low bar, anyone can endorse anyone
else for any particular skill and stories abound of people acquiring skills outside their fields.
As such, LinkedIn endorsements have been ridiculed as “meaningless,” “pointless,” and
a “waste of time” (Wasserman, 2013). In fact, it has been suggested that the only utility
endorsements have is to increase click rates and therefore advertising revenue for the
company (Naughton, 2012). Certainly, within a traditional assessment and credentialing
framework this would be true, but LinkedIn is not such a tool. Although endorsements
clearly provide a very noisy signal, if the signal can be parsed from the noise it may provide
a useful metric. Indeed, in October 2013 LinkedIn applied for a patent that utilized en-
dorsements to determine the level of expertise of their members (Work, Blue, & Hoffman,
2013). In this case, validity of the measure has been sacrificed to acquire it. Before the
Internet, this would not have happened, as the logistical cost of data collection would have
been too high, involving paper surveys and human coders. The cost in this case may be the
irritation of its user base, but the implementation of such data collection is almost trivial.
This represents a shift methodologically, with a move away from trying to collect perfect
data sets toward preferencing data collection itself. This “Big Data” approach represents
not just a technical, but also a philosophical shift in credentialing. It puts far less weight on
any particular measure and presumes impermanence. It predicts a world in which skills are
fluid and everyone must re-skill constantly. In this world, sacrificing time for accuracy no
longer makes sense; an accurate but no longer relevant measure is worth less than a less

accurate but immediate measure. In future, we may see more credentials developed from
noisy data, as relationships between measures are discovered.
Two research fields in particular are poised to take advantage of the growth in educational
data: Learning Analytics and Knowledge (LAK) and Educational Data Mining (EDM). The
expansion of experimentation in new ways and across large populations would seem to be a
great opportunity, though it is unclear what form the pipeline from research to implementa-
tion will eventually take. Both fields struggle with common problems around access to data
(Siemens, 2012, pp. 5–7), infrastructure limits (Duval, 2011, p. 12), and cultural differences
between university researchers and institutional technology departments (Lonn, Aguilar, &
Teasley, 2013, p. 238). The next step — from reliable analytic method to scaled implemen-
tation for trusted credentials — is even less clear. Even though these fields are young, they
have already developed a dizzying number of metrics that test anything from detecting
the propensity of students to game a task (Baker, Corbett, Roll, & Koedinger, 2008) to how
the type of interaction between students might impact their learning (Schreurs, Teplovs,
Ferguson, de Laat, & Buckingham Shum, 2013). In a world where the number of things
that can be measured and the number of ways those things can be measured increases,
the question about how to organize all this new information becomes more pressing. We
are passing from a time when measures were limited and easily controlled to a time when
there are many measures and almost anyone can build their own. There are many possible
futures: the state may assert control over educational measures, making some measures
official and therefore possibly more trustworthy; institutions or corporations may convince
the public that particular measures are the most appropriate; or perhaps a technological
solution will arise, as Google did, to organize all the measures in a useful way. That said,
the role of the human assessor might well change to take on this aspect of the assessment
equation. Rather than being a “grader,” or being involved in the mechanics of assessment,
it will be the role of the teacher — or assessment specialist — to choose the appropriate
assessment and data for a particular educational goal or student.
Digital Badges
Another innovation within the flexible time, competency assessment category is the digital
badge or micro-credential. Badges draw inspiration from both the Boy/Girl Scout badge
system and the way that online games keep track of achievements. They use a digital image
to represent skill-related experiences verified through a rich set of associated metadata.
Badges seek to address the lack of transferable skill recognition across different educa-
tional experiences, allowing people to have a transparent, standardized record across

educational courses, work experience, professional development, and self-directed learn-
ing. For example, attendance at a professional development workshop may come with a
digital badge to designate the skills learned, the provider, the place, and other relevant
information to anyone who is interested in the bearer’s expertise. In this way, digital badges
incorporate features of competency-based models, but within a technological framework
that standardizes assessment.
Proponents of digital badges suggest that they have the flexibility required of a 21st-cen-
tury credential. Unlike a traditional diploma covering a lot of information over a standard
period, micro-credentials can be smaller in scope, with no particular time period deter-
mined. They also classify knowledge in a very loose way (Olneck, 2012), allowing them to
draw from many sources concurrently and adapting much faster in response to changes in
the job market (Young, 2012). The consequences of a widespread badge system, however,
have been contested. The ability of badges to gain critical mass, the technical solutions to
badge-fraud, and the impact of badges on the learning process, particularly for younger
students, has been questioned (Rughinis, 2013).
Efforts to standardize badges were given a substantial boost in 2011 with the creation
of the Mozilla Open Badge Infrastructure or MOBI (openbadges.org). MOBI is the most
ambitious badging program to date, but rather than grant badges themselves, Mozilla
provides open source code and a technical standard that allows educational vendors to
design their own badges. Mozilla also provides the software implementation for individuals
to maintain their acquired badges: the digital backpack. The initiative is driven by the same
“open web” sentiment that Mozilla brings to its other products. The ultimate goal of the
open badge framework is to ensure that all people can “level up” educationally, not just
those with access to resources. However, studies of the MOBI framework suggest that the
move toward badges will be mediated by how credible the format can appear, which will
depend on who is willing to create badges (Gibson, Ostashewski, Flintoff, Grant, & Knight,
2013). This may well be dependent on traditional higher education institutions and credible
institutions, such as the Smithsonian, who can lend their brands to the endeavour.

data driven
Credentials
Proficiency Competency
Online Education
degrees (EG -
U of Phoenix)
Traditional Distance education MOOC

certificates
University
Nanodegrees
degrees
WGU Model Boot Camps
Peer Assessment
credentials
Digital badges
FIXED FLEXIBLE
Figure 1 Examples of credentials (connections) based on whether they use proficiency or competency
assessment, and whether they must be acquired over a fixed amount of time or done at a pace determined by the
student (boxes).

The Future
Badges and endorsement analysis hint at a future where algorithmic processing of online
behavior plays a greater role in credentialing. In this way, standardization may occur through
technology, bringing proficiency testing and competency-based assessment together. The
proficiency goal of creating a general prediction about performance based on quantitative
methodology, and the competency goal of providing a prediction about discreet tasks and
skills, may merge in the form of a complex data store for each individual.
We can imagine a world where the data associated with a badge is so detailed that it pro-
vides the means to make predictions about a person’s performance on a particular project.
A complex variable store that can be queried to match an individual to a job or educational
program, perhaps even to predict the amount of time it will take the person to acquire the
skills to tackle a job. So in the future, perhaps time will be neither fixed nor flexible but will
itself become an outcome.
The closest example may be the Minerva Project (https://minerva.kgi.edu/), a for-profit

residential college/start up in San Francisco. One of Minerva’s main points of differentia-
tion from a traditional college is that it revolves around an online platform technologically
designed with pedagogical and psychological research in mind. Although located in the
same room, students interact with each other and the instructor largely through the online
platform, the Live Interactive Seminar. This allows easy implementation of various pedagog-
ical techniques, such as flexible grouping and short quizzes. It also tracks student answers,
which are then stored as profile data that can be used in a competency-based model or
as a traditional milestone. Instructors and program directors receive detailed information
and analysis of their students in real time that helps guide instruction, provide feedback
for students, and alter course design. It is not mentioned in Minerva publicity materials,
but presumably all their student data means that students could graduate with a detailed
profile of themselves that could inform their future job searches, demonstrate their skills,
or pursue future study.
There are, of course, many barriers to such a future, some technical but many adaptive in
nature. Open infrastructure, in particular open competencies, remains elusive but essential if
only for reasons of practicality. Progress will be slow if every institution, state, and company
has to reinvent their own competencies and tests for all content (Caswell, Henson, Jensen,
152 The Future

& Wiley, 2008). Open frameworks that can be remixed, mashed-up, and built upon would
considerably accelerate the process of developing credentials that take advantage of new
assessment methods and data sources. Openness has taken large strides with respect
to content in the last few years as MOOC providers compete to attract students. Some
frameworks, such as the Australian Qualifications Framework (http://www.aqf.edu.au/), seek
to establish hierarchically and in detail the competencies required across several domains
and job areas. Assessment remains the provision of training organizations, so the connec-
tion between assessment and competency is not complete. In fact, assessment remains a
highly proprietary area, raising serious questions about whether the current credentialing
system can adapt. If anyone can see the inner workings of an assessment, do we need to
re-conceptualize education in a more fundamental way? Can a system built on maintaining
secrecy about what percentage of course material is on the examination really transform to
allow openness of all aspects of a course, including what and how students are assessed?
Conclusion
The complexity of the credentialing landscape has dramatically increased over the last
decade. The rise of online education and the accompanying changes in the goals of the
higher education sector have produced a confusing mix of uncertainty and possibility. In-
novation in credentialing is occurring in several areas, both as part of traditional university
programming and from outside, for-profit entities as varied as boot camps and MOOCs.
This new landscape of inquiry and design reveals a tension between a desire to standardize
how credentials are measured and an explosion in the possible ways that this standardiza-
tion could occur. It has been argued here that this tension is being played out along two
axes with respect to how credentials are defined: 1) whether the credential is defined by
a fixed or flexible amount of time and 2) whether assessment is based on knowledge or
competency. Examination of novel credentialing forms can be classified in these terms to
try to come to grips with the possible future of credentialing.
Conclusion 153
References
Baker, R. S. J. d, Corbett, A. T., Roll, I., & Koedinger, K. R. (2008). Developing a

generalizable detector of when students game the system. User Modeling and
User-Adapted Interaction, 18(3), 287–314. doi:10.1007/s11257-007-9045-6
Barrett, L. O. (2014). Exploring the Return on Investment of a Liberal Arts Degree:

Perceived Connections Between Education and Work. Retrieved from http://
kuscholarworks.ku.edu/handle/1808/14554
Bordón, P., & Braga, B. (2013). Employer Learning, Statistical Discrimination and
University Prestige. Retrieved from http://www-personal.umich.edu/~bgbraga/
Bordon_Braga_August2013.pdf
Carnevale, A., Rose, S., & Hanson, A. (2012). Certificates: Gateway to gainful
employment and college degrees. Georgetown University Center for Education
and the Workforce. Retrieved from https://georgetown.app.box.com/s/
w6bzsdoxvqcywwoog6yl
Caswell, T., Henson, S., Jensen, M., & Wiley, D. (2008). Open content and open
educational resources: Enabling universal education. The International Review
of Research in Open and Distributed Learning, 9(1). Retrieved from http://www.
irrodl.org/index.php/irrodl/article/view/469
Darling-Hammond, L. (2000). Solving the Dilemmas of Teacher Supply, Demand, and

Standards: How We Can Ensure a Competent, Caring, and Qualified Teacher for
Every Child. National Commission on Teaching & America’s Future, Kutztown
Distribution Center, P.O. Box 326, Kutztown, PA 19530-0326 ($8). Tel: 888-492-
1241 (Toll Free). Retrieved from http://eric.ed.gov/?id=ED463337
Davey, D. D., McGoogan, E., Somrak, T. M., Allen, K. A., Beccati, D., Cramer, S. F., …
Suprun, H. Z. (2000). Competency assessment and proficiency testing. Acta
Cytologica, 44(6), 939–943.
154 References
DeVon, H. A., Block, M. E., Moyle-Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J.,
… Kostas-Polston, E. (2007). A psychometric toolbox for testing validity and
reliability. Journal of Nursing Scholarship, 39(2), 155–164. doi:10.1111/j.1547-
5069.2007.00161.x
Donkers, J., Govaerts, M., Driessen, E., & Verhoeven, B. (2008). An E-Portfolio for Post-
Graduate Competency-Based Assessment. Proceedings of Student Mobility
and ICT: Can E-LEARNING Overcome Barriers of Life-Long Learning? Maastricht
University, 81.
Dua, A. (2013). Voice of the Graduate. McKinsey & Company. Retrieved from http://
www.chegg.com/pulse
Duderstadt, J. J. (2000). New roles for the 21st-century university. Issues in Science and
Technology, 16(2), 37–44.
Duval, E. (2011). Attention please!: Learning analytics for visualization and

recommendation. In Proceedings of the 1st International Conference on
Learning Analytics and Knowledge (pp. 9–17). New York, NY, USA: ACM.
doi:10.1145/2090116.2090118
Eaton, J. S. (2001). Distance learning: Academic and political challenges for higher
education accreditation. Council for Higher Education Accreditation
Washington, DC.
Gaston, P. L. (2012). The challenge of Bologna: What United States higher education has
to learn from Europe, and why it matters that we learn it. Sterling, VA: Stylus
Publishing, LLC.
Gehlbach, H., Young, L. V., & Roan, L. K. (2012). Teaching social perspective taking: how
educators might learn from the Army. Educational Psychology, 32(3), 295–309.
doi:10.1080/01443410.2011.652807
Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., & Knight, E. (2013). Digital badges in
education. Education and Information Technologies, 1–8. doi:10.1007/s10639-
013-9291-7
References 155
Holmberg, B. (2005). Theory and Practice of Distance Education. Routledge.
Kant, I. (1798). Der Streit der Fakultäten (The conflict of the faculties). (M. J. Gregor,
Trans.). Lincoln: University of Nebraska Press, 1992.
Kinser, K. (2007). Innovation in higher education: A case study of the Western Governors
University. New Directions for Higher Education, 2007(137), 15–25. doi:10.1002/
he.243
Laitinen, A. (2012). Cracking the Credit Hour. New America Foundation. Retrieved from
Learned, W. S., & Wood, B. D. (1938). The student and his knowledge: A report to the
Carnegie Foundation on the results of the high school and college examinations
of 1928, 1930, and 1932. The Carnegie Foundation for the Advancement of
Teaching.
Leydesdorff, L. (1995). The Triple Helix–University-industry-government relations: A

laboratory for knowledge based economic development. Easst Review, 14(1),
14–9.
Leydesdorff, L., & Etzkowitz, H. (1996). Emergence of a Triple Helix of university—

industry—government relations. Science and Public Policy, 23(5), 279–286.
doi:10.1093/spp/23.5.279
Lonn, S., Aguilar, S., & Teasley, S. D. (2013). Issues, challenges, and lessons learned
when scaling up a learning analytics intervention. In Proceedings of the Third
International Conference on Learning Analytics and Knowledge (pp. 235–239).
New York, NY, USA: ACM. doi:10.1145/2460296.2460343
Malan, S. (2000). The’new paradigm’of outcomes-based education in perspective.

Journal of Family Ecology and Consumer Sciences/Tydskrif Vir Gesinsekologie
En Verbruikerswetenskappe, 28(1).
Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011). Design
Lessons from the Fastest Q&A Site in the West. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 2857–2866). New
York, NY, USA: ACM. doi:10.1145/1978942.1979366
156 References
Marlow, J., Dabbish, L., & Herbsleb, J. (2013). Impression formation in online peer
production: Activity Traces and Personal Profiles in Github. In Proceedings of
the 2013 Conference on Computer Supported Cooperative Work (pp. 117–128).
New York, NY, USA: ACM. doi:10.1145/2441776.2441792
McGuire, R. (2014). Hacking the hacker school: How the bootcamp is being taken to
scale outside the coding world. Retrieved December 20, 2014, from http://
venturebeat.com/2014/06/26/hacking-the-hacker-school-how-the-bootcamp-is-
being-taken-to-scale-outside-the-coding-world/
Miyazaki, I. (1976). China’s examination hell: The civil service examinations of Imperial
China. Yale University Press.
Naughton, J. (2012, December 29). LinkedIn endorsements turn you into the
product. Retrieved January 16, 2015, from http://www.theguardian.com/
technology/2012/dec/30/linkedin-endorsements-turn-you-into-the-product
Newman, J. H. (1854). The idea of a university. Yale University Press. New Haven, CT:
Yale University Press.
Olneck, M. (2012). Insurgent credentials: A challenge to established institutions of

higher education?. Presented at the Education in a New Society: The Growing
Interpenetration of Education in Modern Life, Cambridge, MA. Retrieved from
http://www.hastac.org/documents/insurgent-credentials-challenge-established-
institutions-higher-education
Quaye, S. J., & Harper, S. R. (2014). Student engagement in higher education:

Theoretical perspectives and practical approaches for diverse populations.
Oxford: Routledge.
Rughinis, R. (2013). Talkative Objects in Need of Interpretation. Re-thinking Digital

Badges in Education. In CHI ’13 Extended Abstracts on Human Factors
in Computing Systems (pp. 2099–2108). New York, NY, USA: ACM.
doi:10.1145/2468356.2468729
References 157
Schreurs, B., Teplovs, C., Ferguson, R., de Laat, M., & Buckingham Shum, S. (2013).
Visualizing social learning ties by type and topic: Rationale and concept
demonstrator. In Proceedings of the Third International Conference on
Learning Analytics and Knowledge (pp. 33–37). New York, NY, USA: ACM.
doi:10.1145/2460296.2460305
Shapiro, J. (2014, February 17). Competency-based degrees: Coming soon to a campus

near you. The Chronicle of Higher Education. Retrieved from http://chronicle.
com/article/Competency-Based-Degrees-/144769/ Shedd, J. M. (2003). The
History of the Student Credit Hour. New Directions for Higher Education,
2003(122), 5–12. doi:10.1002/he.106
Shedd, J. M. (2003). The history of the student credit hour. New Directions for Higher
Education, 2003(122), 5–12. doi:10.1002/he.106
Siemens, G. (2012). Learning Analytics: Envisioning a Research Discipline and a

Domain of Practice. In Proceedings of the 2Nd International Conference
on Learning Analytics and Knowledge (pp. 4–8). New York, NY, USA: ACM.
doi:10.1145/2330601.2330605
Swider, S., Levin, P., Ailey, S., Breakwell, S., Cowell, J., McNaughton, D., & O’Rourke, M.
(2006). Matching a graduate curriculum in public/community health nursing to
practice competencies: The Rush University experience. Public Health Nursing,
23(2), 190–195. doi:10.1111/j.1525-1446.2006.230211.x
Ten Cate, O., & Scheele, F. (2007). Viewpoint: Competency-based postgraduate training:
Can we bridge the gap between theory and clinical practice? Academic
Medicine, 82(6), 542–547.
Thomson, P., Saunders, J., & Foyster, J. (2001). Improving the validity of competency-
based assessment. National Centre for Vocational Education Research.
Retrieved from http://eric.ed.gov/?id=ED457376
Wasserman, T. (2013, January 3). LinkedIn’s Endorsements Have Become Meaningless.

Retrieved December 20, 2014, from http://mashable.com/2013/01/03/linkedins-
endorsements-meaningless/
158 References
Work, J. D., Blue, A., & Hoffman, R. (2013, October 31). Determining reputations of
users based on endorsements. Retrieved from http://www.google.com/patents/
US20130290420
Young, J. R. (2012, January 8). “Badges” earned online pose challenge to traditional
college diplomas. Chronicle of Higher Education. Retrieved from https://
chronicle.com/article/Badges-Earned-Online-Pose/130241/#disqus_thread
References 159
Where is Research on Massive
Open Online Courses Headed?
A data analysis of the MOOC Research Initiative
Dragan Gašević
Vitomir Kovanović
Srećko Joksimović
George Siemens
161

Abstract
This paper reports on the results of an analysis of the research

proposals submitted to the MOOC Research Initiative (MRI)
funded by the Gates Foundation. The goal of MRI was to
mobilize researchers to engage into critical interrogation of
MOOCs. The submissions — 266 in Phase 1, out of which
78 were recommended for resubmission in the extended
form in Phase 2, and finally, 28 funded — were analyzed
by applying conventional and automated content analy-
sis methods as well as citation network analysis methods.
The results revealed the main research themes that could
form a framework of the future MOOC research: i) student
engagement and learning success, ii) MOOC design and
curriculum, iii) self-regulated learning and social learning,
iv) social network analysis and networked learning, and v)
motivation, attitude and success criteria. The theme of social
learning received the greatest interest and had the highest
success in attracting funding. The submissions that planned
on using learning analytics methods were more successful.
The use of mixed methods was by far the most popular.
Design-based research methods were also suggested com-
monly, but the questions about their applicability arose
regarding the feasibility to perform multiple iterations in the
MOOC context and rather a limited focus on technological
support for interventions. The submissions were dominated
by the researchers from the field of education (75% of the
accepted proposals). Not only was this a possible cause of
a complete lack of success of the educational technology
innovation theme, but it could be a worrying sign of the
fragmentation in the research community and the need
to increased efforts towards enhancing interdisciplinarity.
Massive Open Online Courses (MOOCs) have captured the
introduction
interest and attention of academics and the public since fall

of 2011 (Pappano, 2012). The narrative driving interest in
MOOCs, and more broadly calls for change in higher edu-
cation, is focused on the promise of large systemic change.
The narrative of change is some variant of:
Higher education today faces a range of challenges, in-

cluding reduced public support in many regions, questions
about its role in society, fragmentation of the functions of
the university, and concerns about long term costs and
system sustainability.
In countries like the UK and Australia, broad reforms have

been enacted that will alter post-secondary education dra-
matically (Cribb & Gewirtz, 2013; Maslen, 2014). In the USA,
interest from venture capital raises the prospect of greater
privatization of universities (GSV Advisors, 2012). In addition
to economic questions around the sustainability of higher
education, broader socio-demographic factors also influence
the future of higher education and the changing diversity of
the student population (OECD Publishing, 2013).
164 Introduction
where is research on massive open online courses headed?
Distance education and online learning have been clearly demonstrated to be an effective
option to traditional classroom learning1. To date, online learning has largely been the
domain of open universities, separate state and provincial university departments, and
for-profit universities. Since the first offering of MOOCs by elite universities in the US and the
subsequent development of providers edX and Coursera, online learning has now become
a topical discussion across many campuses2. For change advocates, online learning in the
current form of MOOCs has been hailed as transformative, disruptive, and a game changer
(Leckart, 2012). This paper is an exploration of MOOCs; what they are, how they are reflected
in literature, who is doing research, the types of research being undertaken, and finally, why
the hype of MOOCs has not yet been reflected in a meaningful way on campuses around
the world. With a clear foundation of the type of research actually happening in MOOCs,
based on submissions to the MOOC Research Initiative3, we are confident that the conver-
sation about how MOOCs and online learning will impact existing higher education can be
moved from a hype and hope argument to one that is more empirical and research focused.
1 For details please refer to the reports on Distance and Online Learning
2 In this paper, we consider MOOCs to belong to the broader field of online education and learning and that their research should be built
on and expand the existing body of research knowledge of online education and learning.
3 http://www.moocresearch.com
Introduction 165
Massive Open Online Courses (MOOCs)
Massive Open Online Courses (MOOCs) have gained media attention globally since the
Stanford MOOC first launched in fall of 2011. The public conversation following this MOOC
was unusual for the education field where innovations in teaching and learning are often
presented in university press releases or academic journals. MOOCs were prominent in
the NY Times, NPR, Time, ABC News, and numerous public media sources. Proclamations
abounded as to the dramatic and significant impact that MOOCs would have on the future
of higher education. In early 2015, the narrative has become more nuanced and researchers
and university leaders have begun to explore how digital learning influences on campus
learning (Kovanović, Joksimović, Gašević, Siemens, & Hatala, 2015; Selwyn & Bulfin, 2014).
While interest in MOOCs appears to be waning from public discourse, interest in online
learning continues to increase (Allen & Seaman, 2013). Research communities have also
formed around learning at scale4 suggesting that while the public conversation around
MOOCs may be fading, the research community continues to apply lessons learned from
MOOCs to educational settings.
MOOCs, in contrast to existing online education which has remained the domain of open
universities, for-profit providers, and separate departments of state universities, have been
broadly adopted by established academics at top tier universities. As such, there are poten-
tial insights to be gained into the trajectory of online learning in general by assessing the
citation networks, academic disciplines, and focal points of research into existing MOOCs.
Our research addresses how universities are approaching MOOCs (departments, research
methods, and goals of offering MOOCs). This results that we share in this article provide
insight into how the gap between existing distance and online learning research, dating
back several decades, and MOOCs and learning at scale research, can be addressed as
large numbers of faculty start experimenting in online environments.
Much of the early research into MOOCs has been in the form of institutional reports by
early MOOC projects, which offered many useful insights, but did not have the rigor —
methodological and/or theoretical expected for peer-reviewed publication in online learning
and education (Belanger & Thornton, 2013; McAuley, Stewart, Siemens, & Cormier, 2010).
Recently, some peer reviewed articles have explored the experience of learners (Breslow et
al., 2013; Kizilcec, Piech, & Schneider, 2013; Liyanagunawardena, Adams, & Williams, 2013).
In order to gain an indication of the direction of MOOC research and representativeness
of higher education as a whole, we explored a range of articles and sources. We settled
on using the MOOC Research Initiative as our dataset.
4 http://learningatscale.acm.org
166 Introduction
MOOC Research Initiative
The MOOC Research Initiative was an $835,000 grant funded by the Bill & Melinda Gates
Foundation and administered by Athabasca University. The primary goal of the initiative
was to increase the availability and rigor of research around MOOCs. Specific topic areas
that the MRI initiative targeted included: i) student experiences and outcomes; ii) cost,
performance metrics and learner analytics; iii) MOOCs: policy and systemic impact; and
iv) alternative MOOC formats. Grants in the range of $10,000 to $25,000 were offered.
An open call was announced in June 2013. The call for submissions ran in two phases: 1.
Short overviews of two pages of proposed research including significant citations; 2. Full
research submissions, eight pages with influential citations, invited from the first phase. All
submissions were peer reviewed and managed in Easy Chair. The timeline for the grants,
once awarded, was intentionally short in order to quickly share MOOC research. MRI was
not structured to provide a full research cycle as this process runs multiple years. Instead,
researchers were selected who had an existing dataset that required resources for proper
analysis.
Phase one resulted in 266 submissions. Phase two resulted in 78 submissions. A total of 28
grants were funded. The content of the proposals and the citations included in each of the
phases were the data source for the research activities detailed below.
Research Objectives
In this paper, we report the findings of an exploratory study in which we investigated (a)
the themes in the MOOC research emerging in the MRI proposals; (b) research methods
commonly proposed for use in the proposals submitted to the MRI initiative, (c) demo-
graphics (educational background and geographic location) characteristics of the authors
who participated in the MRI initiative; (d) most-influential authors and references cited in
the proposals submitted in the MRI initiative; and (e) the factors that were associated with
the success of proposals that were accepted for funding in the MRI initiative.
Introduction 167
Methods
In order to address the research objectives defined in the previous section, we adopted
the Content Analysis and Citation Network Analysis research methods. In the remainder
of this section we describe both of these methods.
Content Analysis
To address research objectives (a) and (b), we performed content analysis methods. Spe-
cifically, we performed both automated a) and manual b) content analyses. The choice of
content analysis was due to the fact that it provides a scientifically sound method for con-
ducting an objective and systematic literature review, thus enabling for the generalizability
of the conclusions (Holsti, 1969). Both variations of the method have been used for analysis
of large amounts of textual content (e.g., literature) in educational research.
Automated content analysis of research themes and trends
Given that content analysis is a very costly and labor intensive endeavor, the automation of
content analysis has been suggested by many authors and this is primarily achieved through
the use of scientometric methods (Brent, 1984; Cheng et al., 2014; Hoonlor, Szymanski, &
Zaki, 2013; Kinshuk, Huang, Sampson, & Chen, 2013; Li, 2010; Sari, Suharjito, & Widodo,
2012). Automated content analysis assumes the application of the computational methods
— grounded in natural language processing and text mining — to identify key topics and
themes in a specific textual corpus (e.g., set of documents, research papers, or proposals)
of relevance for the study.
For extraction of key concepts from each submission, we selected Alchemy API, a platform
for semantic analyses of text that allows for extraction of the informative and relevant set
of concepts of importance for addressing research objective (c) as outlined in Table 1. In
addition to the list of relevant concepts for each submission, Alchemy API produced the
associated relevance coefficient indicating the importance of each concept for a given
submission. This allowed us to rank the concepts and select the top 50 ranked concepts
for consideration in the study. After the concept extraction, we used the agglomerative
168 Methods
hierarchical clustering in order to define N groups of similar submissions that represent the
N important research themes and trends in MOOC research, as aimed in research objective
(c). Finally, we were able to discover nine clusters in the first phase of the MRI granting
process, whereas in the second phase we discovered five clusters.
Table 1 Concept Categories for Describing Clusters
Category Description Example
Intelligent tutoring systems;

The most frequent keywords that identify
Topics Educational technology;
topics mentioned in the specific cluster.
Networked contexts
Keywords that identify specific theory

Competence-based education;
Theory/Approach recognized within documents in each
Social constructivist method
cluster.
MOOC platform identified within the

Environment Coursera; edX; MiriadaX
cluster.
STEM disciplines;
Keywords that represent a specific
Domain Red Cross;
domain of a MOOC course.
Health Sciences
Engagement data;
Keywords representing data used for
Data sources Qualitative data;
studies within the cluster.
Study logs
Measures and Keywords representing measures used Student outcome measures;

variables for studies within the cluster. Early motivation measures;
Parallel multi-method analysis;

Keywords representing various analysis
Analysis techniques Nonparametric statistical
used for studies within the cluster.
analysis;
Keywords representing various In-depth interviews;

Research instruments instruments used to collect data for Focus group interview;
studies within the cluster. Questionnaire
Identifies whether Control groups are

Use of control group used in at least one study within the Control group
cluster.
Methods 169
To assess the produced clusters and select the key concepts in each cluster, we created a
concept-graph consisting of the important concepts from each cluster. The nodes in a graph
were concepts discovered in a particular cluster, while the links between them were made
based on the co-occurrence of the concepts within the same document. More precisely,
the undirected link between two concepts was created in case that both of them were ex-
tracted from the same document. To evaluate the relative importance of each concept we
used the betweenness centrality measure, as the key concepts are likely the ones with the
highest betweenness centrality. Besides the ranking of the concepts in each cluster based
on their betweenness centrality, we manually classified all important concepts into one of
the several categories that are shown in Table 1. Provided categories represent important
dimensions of analysis and we describe each of the clusters based on the provided cat-
egories of key concepts. Thus, when we describe a particular cluster, we cover all of the
important dimensions to provide the holistic view of the particular research trend that is
captured in that cluster.
Content analysis of important characteristics of authors and submissions
A manual content analysis of the research proposals was performed in order to address
research objective (b). Specifically, each submission was categorized into one of the four
categories in relation to research objective (a):
1. Qualitative method, which meant that the proposal used a qualitative

research method such as grounded theory.
2. Quantitative method, which meant that a proposal followed some of the
quantitative research methods on data collected through (Likert-scale
based) surveys or digital traces recorded by learning platforms in order to
explore different phenomena or test hypotheses.
3. Mixed-methods, which reflected a research proposals that applied some
combination of qualitative and quantitative research methods.
4. Other, which comprised of the research proposals that did not explicitly
follow any of these methods, or it was not possible to determine from their
content which of the three methods they planned to use.
170 Methods
For all the authors5 of submitted proposals to the MRI initiative, we collected the infor-
mation related to their home discipline and the geographic location associated with their
affiliation identified in their proposal submissions in order to address research objective (c).
Insight into researchers’ home discipline was obtained from the information provided with
a submission (e.g., if a researcher indicated to be affiliated with a school of education, we
assigned education as the home discipline for this research). In cases when such informa-
tion was not available directly with the proposal submission, we performed a web search,
explored institutional websites, and consulted social networking sites such as LinkedIn or
Google Scholar.
Citation Analysis and Success Factors

The citation analysis was performed to address research objective (d). It entailed the investi-
gation of the research impact of the authors and papers cited in the proposals submitted to
the MRI initiative (Waltman, van Eck, & Wouters, 2013). In doing so, the counts of citations
of each reference and author, cited in the MRI proposals, are used as the measures of the
impact in the citation analysis. This method was suitable, as it allowed for assessing the
influential authors and publications in the space of MOOC research.
Citation network analysis was performed in order to assess the success factor of individual
proposals to be accepted for funding in the MRI initiative, as set in research objective (e).
This way of gauging the success was a proxy measure of the quality and importance of the
proposals. As such, it was appropriate to be used as an indicator of specific topics based on
the assessment of the international board of experts who reviewed the submitted proposals.
5 Information about the geographic location as extracted from the application forms submitted by the authors to EasyChair, a software
system used for the submission and review process.
Methods 171
Submission 1
author a1, Author a2
s cite
ite
s
Reference 1 Reference 2
author Ra1, Author Ra2 Produces
author Ra3, Author Ra4
a1 a2
ra1 ra2 ra3 ra4
Figure 1 The citation networks — connecting the authors of a research proposal (A1 and A2) with the authors
of two cited references (RA1, RA2, RA2 and RA4).
Social network analysis was used to address research objective (e). In this study, social
networks were created through the links established based on the citation and co-author-
ing relationships. The use of social network analysis has been shown as an effective way
to analyze professional performance, innovation, and creativity (Burt, Kilduff, & Tasselli,
2013; Dawson, Tan, & McWilliam, 2011). Moreover, Centola (2010) showed that the spread
of behavior was more effective in networks with higher clustering and larger diameters.
Therefore, for research objective (e), we expected to see the association between the larger
network diameter and the success in receiving funding.
In this study, we followed a method for citation network analysis suggested by Dawson et
al. (2014) in their citation network analysis of the field of learning analytics. Nodes in the
172 Methods
network represent the authors of both submissions and cited references, while links are
created based on the co-authorship and citing relations. Figure 1 illustrates the rules for
creating the citation networks in the simple case when a submission written by the two
authors references two sources, each of them with two authors as well.
We created a citation networks for each cluster separately and analyzed them by following
three measures commonly used in social network analysis (Bastian, Heymann, & Jacomy,
2009; Freeman, 1978; Wasserman, 1994):
1. Degree: the number of edges a node has in a network,

2. Diameter: the maximum eccentricity of any node in a network, and
3. Path: the average graph-distance between all pairs of nodes in a network.
All social networking measures were computed using the Gephi open source software
for social network analysis (Bastian et al., 2009). The social networking measures of each
cluster were then correlated (Spearman’s ρ) with the acceptance ratio — computed as a
ratio of the number of accepted proposals and the number of submitted proposals — for
both phases of the MRI initiative.
Results
Phase 1 Results
In order to evaluate the direction of the MOOC related research, we looked at the most
important research themes in the submitted proposals. In total, there were nine research
themes with similar number of submissions, from 19 (i.e., “MOOC Platforms” research
theme) to 40 (i.e., “Communities” and “Social Networks” research themes). Likewise,
submissions from all themes had on average slightly more than 2 authors and from 7 to 9
citations. However, in terms of their acceptance rates, more than a half of the papers from
the “Social Networks” research theme moved to the second phase and finally 25% of them
were accepted for funding, while none of the submissions from the “Education Technology
Results 173
Improvements” theme was accepted for funding. Other themes within the first phase include
“Processes” (7.7% accepted for funding), “Higher Educational Institutions and MOOCs”
(4.0%), “Motivational and Behavioral Patterns” (13.8%), “Mobile and Adaptive Learning”
(11.4%), “MOOC Platforms” (5.3%), and “Learner Performance” (8.3%). Further, results show
that mixed research was the most common research methodology, while the purely qualita-
tive research was the least frequent. Researchers from the field of education (around 53%)
were represented by far the biggest group, followed by the researchers from the industry
and computer science (both around 20%). Finally, we observed a strong presence of the
authors of the proposals from North America (N=305), followed by authors from Europe
(N=137) and Asia (N=87). For more details on the extracted themes, most frequent keywords
used, and the citation analysis, see Gašević, Kovanović, Joksimović, and Siemens (2014).
We looked at the correlations between the centrality measures of citation networks (for
details see Gašević et al., 2014) and the second phase acceptance rates. Spearman’s rho
revealed that there was a statistically significant correlation between the citation network
diameter and number of submissions accepted into the second round (ρs= .77, n=9, p<.05),
a statistically significant correlation between citation network diameter and second round
acceptance rate (ρs= .70, n=9, p<.05), and a statistically significant correlation between
citation network path and number of submissions accepted into the second round (ρs=
.76, n=9, p<.05). In addition, a marginally significant correlation between citation network
path length and second phase acceptance rate was also found (ρs= .68, n=9, p=0.05032).
Phase 2 Results
Phase 2 Research Themes
Following the analysis of popular research themes, we applied the same automated content
analysis method to the submissions that were accepted into the second phase (78 submis-
sions). We found five research themes (Table 2) that were the focus of an approximately
similar number of submissions. In order to give a better insight in the discovered research
themes, in the following paragraphs, we provide a description of each of the research themes.
Research theme 1: engagement and learning success

The main topics in this cluster are related to learners’ participation, engagement, and be-
havioral patterns in MOOCs. Submissions in this cluster aimed to reveal the most suitable
methods and approaches to understanding and increasing retention, often relying on peer
174 Results
learning and peer assessment. Studies encompassed a wide variety of courses (e.g., biology,
mathematics, writing, EEG-enabled courses, art, engineering, mechanical, and engineering)
on diverse platforms. However, most of the courses, used in the studies from this cluster,
were offered on the Coursera platform.
Research theme 2: MOOC design and curriculum

Research proposals in this cluster were mostly concerned with improving learning process
and learning quality and with studying students’ personal needs and goals. Assessing
educational quality, content delivery methods, MOOC design and learning conditions,
these studies aimed to discover procedures that would lead to better MOOC design and
curriculum, and thus improving learning processes. Moreover, many visualization techniques
were suggested for investigation in order to improve learning quality. Courses suggested
for the use in the proposed studies from this cluster were usually delivered by using the
edX platform and the courses were in the fields of mathematics, physics, electronics and
statistics. The cluster was also characterized by a diversity of data types planned for col-
lection — from surveys, demographic data, and grades to engagement patterns and data
about brain activity.
Research theme 3: Self-regulated learning and social learning

Self-regulated learning, social learning, and social identity were the main topics discussed
in the third cluster. Analyzing cognitive (e.g. memory capacity and previous knowledge),
learning strategies and motivational factors, the proposals from this cluster aimed to identify
potential trajectories that could reveal students at risk. Moreover, this cluster addressed issues
of intellectual property and digital literacy. There was no prevalent platform in this cluster,
while courses were usually in the fields such as English language, mathematics and physics.
Research theme 4: SNA and networked learning

A wide diversity in analysis methods and data sources is one of the defining characteristic of
this cluster. Applying networked learning and social network analysis tools and techniques,
the proposals aimed to address various topics, such as, identifying central hubs in a course, or
improving possibilities for students to gain employment skills. Moreover, learners’ interaction
profiles were analyzed in order to reveal different patterns of interactions between learners
and instructors, among learners, and learners with content and/or underlying technology.
Neither specific domain, nor platform was identified as dominant within the fourth cluster.
Research theme 5: Motivation, attitude and success criteria

The proposals within the fifth cluster aimed to analyze diverse motivational aspects and
correlation between those motivational facets and course completion. Further, research-
Results 175
ers analyzed various MOOC pedagogies (xMOOC, cMOOCs) and systems for supporting
MOOCs (e.g. automated essay scoring), as well as attitudes of higher education institutions
toward MOOCs. Another stream of research within this cluster was related to principles and
best practices of transformation of traditional courses to MOOCs, as well as exploration of
reasons for high dropout rates. The Coursera platform was most commonly referred to as
a source for course delivery and data collection.
Table 2 Phase 2 Research Themes
Quantitative
Qualitative
Accepted
Avg. (SD)
Avg. (SD)
Citations
Funding
Authors
Cluster
Theme
Mixed
Fields
Major
Size
Engagement Education (14)

6 2.2 15.0
1 and Learning 14 Computer Science (4) 1 3 10
(42.9 %) (1.3) (9.8)
Success Engineering (3)
Education (19)
MOOC Design 2 2.9 20.2
2 14 Computer Science (7) 3 5 6
and Curriculum (14.3 %) (2.1) (13.7)
Engineering (4)
Self-Regulated
6 2.3 21.7 Education (25)
3 Learning and 15 8 6 1
(40.0 %) (0.9) (9.2) Computer Science (3)
Social Learning
SNA and
9 2.1 20.7 Education (23)
4 Networked 19 2 12 5
(47.4 %) (0.8) (15.6) Computer Science (5)
Learning
Motivation, Education (25)

5 2.8 23.1
5 Attitude and 16 Engineering (5) 5 7 4
(31.2 %) (1.1) (9.2)
Success Criteria Social Sciences (4)
28
Total 78
(35.8 %)
176 Results
Phase 2 Research Methods
Table 3 indicates that mixed methods was the most common methodological approach
followed by purely quantitative research, which was used just slightly more than qualitative
research. This suggests that there was no clear “winner” in terms of the adopted meth-
odological approaches, and that all three types are used with a similar frequency. Also,
the average number of authors and citations shows that the submissions mixed methods
tended to have slightly more authors than quantitative or qualitative submissions, and that
quantitative submissions had a significantly lower number of citations than submissions
adopting both mixed and qualitative methods.
Table 2 shows that the submissions centered around engagement and peer assessment
(i.e., cluster 1) used mainly quantitative research methods, while submissions dealing with
self-regulated learning and social learning (i.e., cluster 3) exclusively used qualitative and
mixed research methods. Finally, submissions centered around social network analysis (i.e.,
cluster 4) mostly used mixed methods, while submissions dealing with MOOC design and
curriculum (i.e., cluster 2), and ones dealing with motivation, attitude and success criteria
(i.e., cluster 5) had an equal adoption of all the three research methods.
Table 3 Phase 2 Distribution of Research Methodologies
Methodology Submissions Authors Avg. (SD) Citations Avg. (SD)
Mixed 33 (42.3%) 2.7 (1.5) 21.8 (13.2)
Qualitative 19 (24.4%) 2.1 (0.9) 22.8 (12.10
Quantitative 26 (33.3%) 2.4 (1.2) 16.7 (10.3)
Total 78(100%) 2.5 (1.3) 20.3 (12.3)
Results 177
Table 4 Phase 2 Top 5 Research Fields Table 5 Phase 2 Geographic Distribution of the Authors
Authored Accepted
Field Authors Continent Authors
Proposals Proposals
Education 106 Asia 17 4.64 0.14
Computer Science 21 Australia/NZ 11 4.25 1
Engineering 13 Europe 40 15.66 4
Industry 8 North America 137 52.44 22.85
Social Sciences 6 South America 3 1 0
Phase 2 Demographic Characteristics of the Authors
With respect to the primary research areas of the submission authors, Table 4 shows that
Education was the primary research field of the large majority of the authors and that Com-
puter Science was the distant second. In terms of the average number of authors, we can see
in Table 2 that submissions related to MOOC design and curriculum (i.e., research theme 2)
and motivation, attitude and success criteria (i.e., research theme 5) had on average a slightly
higher number of authors than the other three research themes. In terms of their number of
citations, submissions dealing with the engagement and peer assessment had on average
15 citations, while the submissions about other research themes had a bit higher number
of citations ranging from 20 to 23. Similar to Phase 1, in all research themes, the field of
education was found to be the main research background of submission authors. This was
followed by the submissions authored by computer science and engineering researchers,
and in the case of submissions about motivation, attitude and success criteria, by social
scientists. Finally, similarly to the Phase 1, we see the strong presence of researchers from
North America, followed by the much smaller number of researchers from other parts of
the world (Table 5).
178 Results
Table 6 Phase 2 Most Cited Papers
Citation
Paper Name
Count
Kizilcec, R. F., Piech, C. and Schneider, E. (2013). Deconstructing disengagement:

15
analyzing learner subpopulations in massive open online courses.
Liyanagunawardena, T. R., Adams, A. A. and Williams, S. (2013). MOOCs: a Systematic

13
Study of the Published Literature 2008-2012.
McAuley, A., Stewart, B., Siemens, G. and Cormier, D. (2010). The MOOC model for
13
digital practice.
Breslow, L. B., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D. and Seaton, D. T.
13
(2013). Studying learning in the worldwide classroom: Research into edX's first MOOC.
Siemens, G. (2005). Connectivism: A Learning Theory for the Digital Age. 12
Pappano, L. (2012). The Year of the MOOC. 10
Yuan L. and Powell S. (2013). MOOCs and Open Education: Implications for Higher
9
Education.
Jordan, K. (2013). MOOC Completion Rates : The Data. 7
Belanger, Y. and Thornton, J. (2013). Bioelectricity: A Quantitative Approach. Duke

7
University First MOOC.
Long, P. and Siemens, G. (2012). Penetrating the fog: analytics in learning and
6
education.
Kop, R. (2011). The Challenges to Connectivist Learning on Open Online Networks:

6
Learning Experiences during a Massive Open Online Course.
Daniel, J. (2012). Making Sense of MOOCs: Musings in a Maze of Myth, Paradox and
6
Possibility.
Mackness, J., Mak, S. F. J. and Williams, R. (2010). The Ideals and Reality of
5
Participating in a MOOC.
Means, B., Toyama, Y., Murphy, R., Bakia, M. and Jones, K. (2010). Evaluation of
Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online 5
Learning Studies.
Results 179
Belanger Y. Qualitative
Ajzen I. Mixed
Ho A. D. Quantitative
Other
McAuley A.
Mackness J.
Stewart B.
Kizilcec R. F.
Stump G. S.
Garrison D. D.
Williams R.
Cormier D.
Schneider E.
DeBoer J.
Anderson T.
Williams S.
Kop R.
Seaton D. T.
Piech C.
Pritchard D. E.
Siemens G.
0 10 20 30 40 50
number of citations
Figure 3 Phase 2 most cited authors
180 Results
Rodriguez O.
Keppell M.
Mackintosh W.
Cormier D.
Dumitrica D. Tomkin J. Kelly N.
Bossu C.
Charlesvoix D. McAuley A. Padgett L.
Daniel J. Mak S. Willams Amanda Siemens G.
Stewart B.
Williams R. Brown J. S.
Mackness J.
Anderson T.
Powell S. Breden J. Toven-Lindsey B.
Raynauld J. Talbot J. Littlejohn A. Rhoads R.
Yuan L. Pappano L. Sujitparapitaya S.
Millgan C.
Poellhuber B. Kizlcec R. F. Ghadiri K.
Cheal C.
Lou H.
Piech C. Junn E.
MOOC Schneider E.
Kop R.
Pritchand D.E.
Seaton D.T.
Eynon R. Sump G.S.
DeBoer J. Breslow L.
Ho A.D.
Harrison L.
Figure 4 Phase 2 citation network.
Results 181
Table 7 Phase 2 Citation Network Metrics
Shortest Path
Degree (SD)
Diameter
Average
Average
Density
Cluster
Theme
(SD)
1 Engagement and Peer Assessment 4.6 (8.4) 8 4.5 (1.6) 0.014
2 MOOC Design and Curriculum 5.3 (10.9) 9 4.3 (1.8) 0.017
3 Learning Characteristics and Social Learning 5.4 (8.7) 7 4.1 (1.3) 0.023
4 SNA and Networked Learning 4.9 (9.6) 8 3.9 (1.4) 0.015
5 Motivation, Attitude and Success Criteria 6.9 (9.0) 8 3.7 (1.5) 0.033
Total 5.1 (7.3) 11 4.0 (1.3) 0.012
Phase 2 Citation Analysis and Success Factors
We calculated a total number of citations (Table 6) for each publication, and extracted a list
of the most cited authors (Figure 3). We can observe that the most cited authors were not
necessarily the ones with the highest betweenness centrality, but the ones whose research
focus was most relevant from the perspective of the MRI initiative and researchers from
different fields and with different research objectives.
We also extracted the citation network graph which is shown on Figure 4. At the centre of
the network is L. Pappano, the author of very popular New York Times article “The Year of
the MOOC”, as the author with the highest betweenness centrality value. The reason for
this is that his article was frequently cited by a large number of researchers from variety
of academic disciplines, and thus making him essentially a bridge between them, which is
clearly visible on the graph.
182 Results
We also analyzed citation networks for each research theme independently and extracted
common network properties such as diameter, average degree, path and density (Table
7). However, further investigation of Phase 2 success factors did not reveal any significant
correlation between the citation network centrality measures and the final acceptance rates
(Table 7) — i.e., Spearman’s rho correlation was not statistically significant correlation at
the α=0.05 significance level.
Discussion
Emerging Themes in MOOC Research

The results of the analysis indicated a significant attention of the researchers to the issues
related to MOOCs that have received much public (media) attention. Specifically, the issue
of low course completion and high degree of student attrition was often pronounced as the
key challenge of MOOCs (Jordan, 2013; Koller, Ng, Do, & Chen, 2013). Not only was the
topic of engagement and learning success (Cluster 1 in Phase 2) identified as a key theme
in the MRI submissions, but it was also identified as a theme that was clearly cross-cutting
all other research themes identified in Phase 2, including motivation, attitudes and success
criteria in Cluster 5, course design in Cluster 2, and learning strategies, social interaction,
and interaction with learning resources in Cluster 3. With the aim to understand the factors
affecting student engagement and success in MOOCs, the proposals had suggested a
rich set of data collection methods — e.g., surveys, physiological brain activity, knowledge
tests, and demographic variables. The theory of planned behavior (TBP) (Ajzen, 1991) was
found as the main theoretical foundation for research of student engagement and learning
success. While TBP is a well-known framework for studying behavioral change — in this
case changing students intention to complete a MOOC and thus, increase their likelihood
of course completion — it remains to be seen to what extent a student’s intention can be
changed if the student did not have an intention to complete a MOOC in the first place.
What would be a reason that could motivate a student to change their intention in cases
when she/he only enrolled into a MOOC to access information provided without intentions
Discussion 183
to take any formal assessments? In that sense, in seems necessary first to understand stu-
dents’ intentions for taking a MOOC, before trying to study the effects of interventions
(e.g., motivational messages) on the students with different initial intentions.
The results also confirmed that social aspects of learning in MOOCs were the most success-
ful theme in the MRI initiative (see Table 2). Total of 15 out of the 28 accepted proposals
(Clusters 3 and 4) were related to different factors of social learning in MOOCs. Not only has
it become evident recently that students require socialization in MOOCs through different
forms (of self-organization) such as local meet-ups (Coughlan, 2014)6 and that social factors
contribute to attribution in MOOCs (Rosé et al., 2014), but educational research is also very
clear about numerous educational benefits of socialization. The Vygotskian approach to
learning posits that higher levels of internalization can be achieved through social interac-
tion most effectively (Vygotsky, 1980). These benefits have been shown to lead to deeper
approaches to learning and consequently to higher learning outcomes (Akyol & Garrison,
2011). Moreover, students’ positions in social networks have been found in the existing
literature to have a significant positive effect on many important learning outcomes such
as creative potential (Dawson et al., 2011), sense of belonging (Dawson et al., 2011), and
academic achievement (Gašević, Zouaq, & Janzen, 2013). Yet, the lack of social interaction
can easily lead to the sense of social isolation which is well documented as one of the main
barriers in distance and online education (Muilenburg & Berge, 2001; Rovai, 2002). Finally,
Tinto’s (1997) influential theory recognizes social and academic integration as the most
important factors of student retention in higher education.
Research Methods in MOOC Research

The high use of mixed methods is a good indicator of sound research plans that recognized
the magnitude of complexity of the issues related to MOOCs (Greene, Caracelli, & Graham,
1989). The common use of design-based research is likely a reflection of MOOC research
goals aiming to address practical problems, and at the same time, attempting to build and/
or inform theory (Design-Based Research Collective, 2003; Reeves, Herrington, & Oliver,
2005). This assumes that research is performed in purely naturalistic settings of MOOC of-
fering (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003), always involves some intervention
(Brown, 1992), and typically has several iterations (Anderson & Shattuck, 2012). According
to Anderson and Shattuck (2012), there are two types of interventions — instructional and
6 It is important to acknowledge that the importance of a “face-to-face contact with other students” was found in the Lou et al. meta-anal-
ysis (2006) of the literature — published in the period from 1985 to 2002 — about the effects of different aspects of distance and open
education on academic success.
184 Discussion
technological — commonly applied in online education research. Our results revealed that
the focus of the proposals submitted to the MRI initiative were primarily focused on the
instructional interventions. However, it is reasonable to demand from MOOC research to
study the extent to which different technological affordances, instructional scaffolds and
the combinations of the two can affect various aspects of online learning in MOOCs. This
objective was set long ago in online learning research, led to the Great Media debate (Clark,
1994; Kozma, 1994), and the empirical evidence that support either position (affordances
vs. instruction) of the debate (Bernard et al., 2009; Lou, Bernard, & Abrami, 2006). Given
the scale of MOOCs, a wide spectrum of learners’ goals, differences in roles of learners,
instructors and other stakeholders, and a broad scope of learning outcomes, research
of the effects of affordances vs. instruction requires much research attention and should
produce numerous important practical and theoretical implications. For example, an im-
portant question is related to the effectiveness of the use of centralized learning platforms
(commonly used in xMOOCs) to facilitate social interactions among students and formation
of learning networks that promote effective flow of information (Thoms & Eryilmaz, 2014).
Our analysis revealed that the issue of the number of iterations in design-based research
was not spelled out in the proposals of the MRI initiative (Anderson & Shattuck, 2012). It
was probably unrealistic to expect to see proposals with more than one edition of a course
offering given the timeline of the MRI initiative. This meant that the MRI proposals, which
aimed to follow design-based research, were focused on the next iteration of existing courses.
However, given the nature of MOOCs, which are not necessarily offered many times and
in regular cycles, what is reasonable to expect from conventional design-based methods
that require several iterations? Given the scale of the courses, can the same MOOC afford
for testing out several interventions that can be offered to different subpopulations of the
enrolled students in order to compensate for the lack of opportunity of several iterations?
If so, what are the learning, organizational, and ethical consequences of such an approach
and how and whether at all they can be mitigated effectively?
The data collection methods were another important feature of the proposal submissions to
the MRI initiative. Our results revealed that most of the proposals planned to use conven-
tional data sources and data collection methods such as grades, surveys on assessments,
and interviews. Of course, it was commendable to see many of those proposals being
based on the well-established theories and methods. However, it was surprising to see a
low number of proposals that had planned to make use of the techniques and methods of
learning analytic and educational data mining (LA/EDM) (Baker & Yacef, 2009; Siemens &
Gašević, 2012). With the use LA/EDM approaches, the authors of the MRI proposals would
be able to analyze trace data about learning activities, which are today commonly collected
Discussion 185
by MOOC platforms. The use of LA/EDM methods could offer some direct research benefits
such as absence and/or reduction of self-selection and being some less unobtrusive, more
dynamic, and more reflective of actual learning actives than conventional methods (e.g.,
surveys) can measure (Winne, 2006; Zhou & Winne, 2012).
Interestingly, the most successful themes (Clusters 3-4 in Phase 2) in the MRI initiative had
a higher tendency to use the LA/EDM methods than other themes. Our results indicate
that the MRI review panel expressed a strong preference towards the use of the LA/EDM
methods. The data types and analysis methods in Clusters 3-4 were also mixed by combin-
ing the use of trace data with conventional data sources and collection methods (surveys,
interviews, and focus groups). This result provided a strong indicator of the direction in
which research methods in the MOOC arena should be going. It will be important however
to see the extent to which the use of LA/EDM can be employed to advance understanding
of learning and learning environments. For example, it is not clear whether an extensive
activity in a MOOC platform is indicative of high motivation, struggling and confusion with
the problem under study, or the use of poor study strategies (Clarebout, Elen, Collazo,
Lust, & Jiang, 2013; Lust, Juarez Collazo, Elen, & Clarebout, 2012; Zhou & Winne, 2012).
Therefore, we recommend a strong alignment of the LA/EDM methods with educational
theory in order to obtain meaningful interpretation of the results that can be analyzed
across different context and that can be translated to practice of learning and teaching.
Importance of
Interdisciplinarity in MOOC Research
The analysis of the research background of the authors who submitted their proposals to
the MRI initiative revealed an overwhelmingly low balance between different disciplines.
Contrary to the common conceptions of the MOOC phenomena to be driven by computer
scientists, our results showed that about 53% in Phase 1, 67% in Phase 2, about 67% of the
finally accepted proposals were the authors from the discipline of education. It is not clear
the reason for this domination of the authors from the education discipline. Could this be
a sign of the networks to which the leaders of the MRI initiative were able to reach out? Or,
is this is a sign of fragmentation in the community? Although not conclusive, some signs of
fragmentation could be traced. Preliminary and somewhat anecdotal results of the new ACM
international conference on learning at scale indicate that the conference was dominated
by computer scientists. It is not possible to have a definite answer if the fragmentation is
actually happening or not based on only these two events. However, the observed trend
186 Discussion
is worrying. A fragmentation would be unfortunate for advancing understanding of a
phenomenon such as MOOCs in particular and education and learning, in general, which
require strong interdisciplinary teams (Dawson et al., 2014).
The positive association observed between the success of individual themes of the MRI
submissions and citation network structure (i.e., diameter and average network path)
warrants research attention. The significance of this positive correlation indicates that the
themes of the submitted proposals, which managed to reach out to a broader and more
diverse citation networks, were more likely to be selected for funding in the MRI initiative.
Being able to access information in different social networks is already shown to be posi-
tively associated with achievement, creativity, and innovation (Burt et al., 2013). Moreover,
the increased length of network diameter — as shown in this study — was found to boost
spread of behavior (Centola, 2010). In the context of the results of this study, this could mean
that the increased diameters of citation networks in successful MRI themes were assessed
by the MRI review panel as more likely to spread educational technology innovation in
MOOCs. If that is the case, it would be a sound indicator of quality assurance followed by
the MRI peer-review process. On the other hand, for the authors of research proposals, this
would mean that trying to cite broader networks of authors would increase their chances of
success to receive research funding. However, future research in other different situations
and domains is needed in order to be able to validate these claims.
Discussion 187
Conclusions and
Recommendations
Research needs to create with theoretical underpinnings that will explain factors related to
social aspects in MOOCs that have a completely new context and offer practical guidance
of course design and instruction (e.g., Clusters 2, 4, and 5 in Phase 2). The scale of MOOCs
does limit the extent to which existing frameworks for social learning proven in (online)
education can be applied. For example, the Community of Inquiry (CoI) framework posits
that social presence needs to be established and sustained in order for students to build
trust that will allowed them to comfortably engage into deeper levels of social knowledge
construction and group-based problem solving (Garrison, Anderson, & Archer, 1999; Garri-
son, 2011). The scale of and (often) shorter duration of MOOCs than in traditional courses
limits opportunities for establishing sense of trust between learners, which likely leads to
much more utilitarian relationships. Furthermore, teaching presence — established through
different scaffolding strategies either embedded into course design, direct instruction, or
course facilitation — has been confirmed as an essential antecedent of effective cognitive
processing in both communities of inquiry and computer-supported collaborative learning
(CSCL) (Fischer, Kollar, Stegmann, & Wecker, 2013; Garrison, Cleveland-Innes, & Fung,
2010; Gašević, Adesope, Joksimović, & Kovanović, 2015). However, some of the pedagog-
ical strategies proven in CoI and CSCL research — such as role assignment — may not fit
to the MOOC context due to common assumptions that the collaboration and/or group
inquiry will happen in small groups (6-10 students) or smaller class communities (30-40
students) (Anderson & Dron, 2011; De Wever, Keer, Schellens, & Valcke, 2010). When this
is combined with different goals with which students enroll into MOOCs compared to
those in conventional (online) courses, it becomes clear that novel theoretical and practical
frameworks of understanding and organizing social learning in MOOCs are necessary. This
research direction has been reflected in the topics identified in Cluster 4 of Phase 2 such
as network formulation and peer-to-peer, online, learners and asynchronous interaction.
However, novel theoretical goals have not been so clearly voiced in the results of the anal-
yses performed in this study.
The connection with learning theory has also been recognized as another important feature
of the research proposals submitted to MRI (e.g., Clusters 3-5 in Phase 2). Likely responding
to the criticism often attributed to the MOOC wave throughout 2012 not to be driven by
188 Conclusions and Recommendations

rigorous research and theoretical underpinnings, the researchers submitting to the MRI
initiative used frameworks well-established in educational research and the learning sci-
ences. Of special interest were topics related to self-regulated learning (Winne & Hadwin,
1998; Zimmerman & Schunk, 2011; Zimmerman, 2000). Consideration of self-regulated
learning in design of online education has been already recognized. To study effectively
in online learning environments, learners need to be additionally motivated and have an
enhanced level of metacognitive awareness, knowledge and skills (Abrami, Bernard, Bures,
Borokhovski, & Tamim, 2011). Such learning conditions may not have the same level of
structure and support as students have typically experienced in traditional learning envi-
ronments. Therefore, understanding of student motivation, metacognitive skills, learning
strategies, and attitudes is of paramount importance for research and practice of learning
and teaching in MOOCs.
The new educational context of MOOCs triggered research for novel course and curriculum
design principles as reflected in Cluster 2 of Phase 2. Through the increased attention to
social learning, it becomes clear that MOOC design should incorporate factors of knowledge
construction (especially in group activities), authentic learning, and personalized learning
experience that is much closer to the connectivist principles underlying cMOOCs (Siemens,
2005), rather than knowledge transmission as commonly associated with xMOOCs (Smith &
Eng, 2013). By triggering the growing recognition of online learning in world-wide, MOOCs
are also interrogated from the perspective of their place in higher education and how they
can influence blended learning strategies of institutions in the post-secondary education
sector (Porter, Graham, Spring, & Welch, 2014). Although the notion of flipped classrooms
is being adopted by many in the higher education sector (Martin, 2012; Tucker, 2012), the
role of MOOCs begs many questions such as those related to effective pedagogical and
design principles, copyright, and quality assurance.
Finally, it is important to note that the majority of the authors of the proposals submitted to
the MRI were from North America, followed by the authors from Europe, Asia, and Australia.
This clearly indicates a strong population bias. However, this was expected given the time
when the MRI initiative happened — proposals submitted in mid-2013. At that time, MOOCs
were predominately offered by the North-American institutions through the major MOOC
provides to a much lesser extent in the rest of the world. Although the MOOC has become
a global phenomenon and attracted much mainstream media attention — especially in some
regions such as Australia, China and India as reported by Kovanovic et al. (2015) — it seems
the first wave of research activities is dominated by researchers from North America. In the
future studies, it would be important to investigate whether this trend still holds and to what
extent other continents, cultures, and economies are represented in the MOOC research.
Conclusions and Recommendations 189

References
Abrami, P. C., Bernard, R. M., Bures, E. M., Borokhovski, E., & Tamim, R. M. (2011).
Interaction in distance education and online learning: using evidence and
theory to improve practice. Journal of Computing in Higher Education, 23(2-3),
82–103. doi:10.1007/s12528-011-9043-x
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human
Decision Processes, 50(2), 179–211. doi:10.1016/0749-5978(91)90020-T
Akyol, Z., & Garrison, D. R. (2011). Understanding cognitive presence in an online and
blended community of inquiry: Assessing outcomes and processes for deep
approaches to learning. British Journal of Educational Technology, 42(2),
233–250. doi:10.1111/j.1467-8535.2009.01029.x
Allen, I. E., & Seaman, J. (2013). Changing Course: Ten Years of Tracking Online
Education in the United States. Newburyport, MA: Sloan Consortium. Retrieved
from http://www.onlinelearningsurvey.com/reports/changingcourse.pdf
Anderson, T., & Dron, J. (2011). Three generations of distance education pedagogy. The
International Review of Research in Open and Distance Learning, 12(3), 80–97.
Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/890/1663
Anderson, T., & Shattuck, J. (2012). Design-Based Research A Decade of

Progress in Education Research? Educational Researcher, 41(1), 16–25.
doi:10.3102/0013189X11428813
Baker, R. S. J. d, & Yacef, K. (2009). The State of Educational Data Mining in 2009: A
Review and Future Visions. Journal of Educational Data Mining, 1(1), 3–17.
Retrieved from http://www.educationaldatamining.org/JEDM/index.php/JEDM/
article/view/8
190 References
Bastian, M., Heymann, S., & Jacomy, M. (2009). Gephi: An Open Source Software for
Exploring and Manipulating Networks. In Proceedings of the Third International
AAAI Conference on Weblogs and Social Media (pp. 361–362). San Jose: AAAI
Press. Retrieved from http://www.aaai.org/ocs/index.php/ICWSM/09/paper/
view/154
Belanger, Y., & Thornton, J. (2013). Bioelectricity: A Quantitative Approach.

Duke University. Retrieved from http://dukespace.lib.duke.edu/dspace/
handle/10161/6216
& Bethel, E. C. (2009). A Meta-Analysis of Three Types of Interaction Treatments
in Distance Education. Review of Educational Research, 79(3), 1243–1289.
doi:10.3102/0034654309333844
Brent, E. E. (1984). The Computer-Assisted Literature Review. Social Science Computer

Review, 2(3), 137–151. doi: 10.1177/089443938600200303
Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D., & Seaton, D. T. (2013).
Studying Learning in the Worldwide Classroom: Research into edX’s First
MOOC. Journal of Research & Practice in Assessment, 8, 13–25. Retrieved
from http://www.rpajournal.com/studying-learning-in-the-worldwide-classroom-
research-into-edxs-first-mooc/
Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in

Creating Complex Interventions in Classroom Settings. Journal of the Learning
Sciences, 2(2), 141–178. doi:10.1207/s15327809jls0202_2
Burt, R. S., Kilduff, M., & Tasselli, S. (2013). Social Network Analysis: Foundations
and Frontiers on Advantage. Annual Review of Psychology, 64(1), 527–547.
doi:10.1146/annurev-psych-113011-143828
Centola, D. (2010). The Spread of Behavior in an Online Social Network Experiment.

Science, 329(5996), 1194–1197. doi:10.1126/science.1185231
Cheng, B., Wang, M., Mørch, A. I., Chen, N.-S., Kinshuk, & Spector, J. M. (2014). Research
on e-learning in the workplace 2000–2012: A bibliometric analysis of the literature.
Educational Research Review, 11, 56–72. doi:10.1016/j.edurev.2014.01.001
References 191
Clarebout, G., Elen, J., Collazo, N. A. J., Lust, G., & Jiang, L. (2013). Metacognition and
the Use of Tools. In R. Azevedo & V. Aleven (Eds.), International Handbook of
Metacognition and Learning Technologies (pp. 187–195). Springer New York.
Retrieved from http://link.springer.com/chapter/10.1007/978-1-4419-5546-3_13
Clark, R. E. (1994). Media will never influence learning. Educational Technology Research
and Development, 42(2), 21–29. doi:10.1007/BF02299088
Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design
Experiments in Educational Research. Educational Researcher, 32(1), 9–13.
doi:10.3102/0013189X032001009
Coughlan, S. (2014, April 8). The irresistible urge for students to talk. Retrieved April 18,
2014, from http://www.bbc.com/news/business-26925463
Cribb, A., & Gewirtz, S. (2013). The hollowed-out university? A critical analysis of
changing institutional and academic norms in UK higher education. Discourse:
Studies in the Cultural Politics of Education, 34(3), 338–350. doi:10.1080/01596
306.2012.717188
Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current State and
Future Trends: A Citation Network Analysis of the Learning Analytics
Field. In Proceedings of the Fourth International Conference on Learning
Analytics And Knowledge (pp. 231–240). New York, NY, USA: ACM.
doi:10.1145/2567574.2567585
Dawson, S., Tan, J. P. L., & McWilliam, E. (2011). Measuring creative potential: Using
social network analysis to monitor a learners’ creative capacity. Australasian
Journal of Educational Technology, 27(6), 924–942. Retrieved from http://
ascilite.org.au/ajet/submission/index.php/AJET/article/view/921
De Wever, B., Keer, H. V., Schellens, T., & Valcke, M. (2010). Roles as a structuring tool
in online discussion groups: The differential impact of different roles on social
knowledge construction. Computers in Human Behavior, 26(4), 516–523.
doi:10.1016/j.chb.2009.08.008
192 References
Design-Based Research Collective. (2003). Design-Based Research: An Emerging
Paradigm for Educational Inquiry. Educational Researcher, 32(1), 5–8.
doi:10.3102/0013189X032001005
Fischer, F., Kollar, I., Stegmann, K., & Wecker, C. (2013). Toward a Script Theory of
Guidance in Computer-Supported Collaborative Learning. Educational
Psychologist, 48(1), 56–66. doi:10.1080/00461520.2012.748005
Freeman, L. C. (1978). Centrality in social networks conceptual clarification. Social

Networks, 1(3), 215–239. doi:10.1016/0378-8733(78)90021-7
Garrison, D. R. (2011). E-learning in the 21st century a framework for research and
practice. New York: Routledge. ISBN 9780203166093
Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical Inquiry in a Text-Based
Environment: Computer Conferencing in Higher Education. The Internet and
Higher Education, 2(2–3), 87–105. doi:10.1016/S1096-7516(00)00016-6
Garrison, D. R., Cleveland-Innes, M., & Fung, T. S. (2010). Exploring causal relationships
among teaching, cognitive and social presence: Student perceptions of the
community of inquiry framework. The Internet and Higher Education, 13(1–2),
31–36. doi:10.1016/j.iheduc.2009.10.002
Gašević, D., Adesope, O., Joksimović, S., & Kovanović, V. (2015). Externally-facilitated
Regulation Scaffolding and Role Assignment to develop Cognitive Presence
inAsynchronous Online Discussions. The Internet and Higher Education, 24, 53-
65. doi:10.1016/j.iheduc.2014.09.006
Gašević, D., Kovanović, V., Joksimović, S., & Siemens, G. (2014). Where is Research
on Massive Open Online Courses Headed? - A data analysis of the MOOC
Research Initiative. The International Review of Research in Open and Distance
Learning, 15(5), 134 – 176. Retrieved from http://www.irrodl.org/index.php/
irrodl/article/view/1954
Gašević, D., Zouaq, A., & Janzen, R. (2013). “Choose Your Classmates, Your GPA
Is at Stake!” The Association of Cross-Class Social Ties and Academic
Performance. American Behavioral Scientist, 57(10), 1460–1479.
doi:10.1177/0002764213479362
References 193
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a Conceptual Framework
for Mixed-Method Evaluation Designs. Educational Evaluation and Policy
Analysis, 11(3), 255–274. doi:10.3102/01623737011003255
GSV Advisors. (2012). Fall of the wall: Capital flows to education innovation. Chicago, IL:
Global Silicon Valley (GSV) Advisors. Retrieved from http://goo.gl/NMtKZ4
Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Boston, MA:
Addison-Wesley Pub. Co.
Hoonlor, A., Szymanski, B. K., & Zaki, M. J. (2013). Trends in Computer Science Research.
Commun. ACM, 56(10), 74–83. doi:10.1145/2500892
Jordan, K. (2013). MOOC Completion Rates: The Data. Retrieved from http://www.
katyjordan.com/MOOCproject.html
Kinshuk, Huang, H.-W., Sampson, D. G., & Chen, N.-S. (2013). Trends in Educational
Technology through the Lens of the Highly Cited Articles Published in the
Journal of Educational Technology and Society. Educational Technology &
Society, 16(2) 3–20. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/
summary?doi=10.1.1.299.5508
Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing Disengagement:

Analyzing Learner Subpopulations in Massive Open Online Courses.
In Proceedings of the Third International Conference on Learning
Analytics and Knowledge (pp. 170–179). New York, NY, USA: ACM.
doi:10.1145/2460296.2460330
Koller, D., Ng, A., Do, C., & Chen, Z. (2013). Retention and intention in massive open
online courses: In depth. Educause Review, 48(3), 62–63. Retrieved from http://
www.educause.edu/ero/article/retention-and-intention-massive-open-online-
courses-depth-0
media reveals about MOOCs: A systematic analysis of news reports. British
Journal of Educational Technology, in press. doi: 10.1111/bjet.12277
194 References
Technology Research and Development, 42(2), 7–19. doi:10.1007/BF02299087
Leckart, S. (2012, March 20). The Stanford Education Experiment Could Change Higher
Learning Forever. Wired. Retrieved from http://www.wired.com/2012/03/
ff_aiclass/
Li, Feng, Textual Analysis of Corporate Disclosures: A Survey of the Literature (February
7, 2011). Journal of Accounting Literature. Available at SSRN: http://ssrn.com/
abstract=1756926. Retrieved from http://webuser.bus.umich.edu/feng/LI%20
JAL%202011%20(Textual%20Analysis%20of%20Corp%20Disclosures).pdf
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic

study of the published literature 2008-2012. The International Review of
Research in Open and Distance Learning, 14(3), 202–227. Retrieved from http://
www.irrodl.org/index.php/irrodl/article/view/1455/2531
Lou, Y., Bernard, R. M., & Abrami, P. C. (2006). Media and Pedagogy in Undergraduate
Distance Education: A Theory-Based Meta-Analysis of Empirical Literature.
doi:10.1007/s11423-006-8252-x
Lust, G., Juarez Collazo, N. A., Elen, J., & Clarebout, G. (2012). Content Management
Systems: Enriched learning opportunities for all? Computers in Human Behavior,
28(3), 795–808. doi:10.1016/j.chb.2011.12.009
Martin, F. G. (2012). Will Massive Open Online Courses Change How We Teach?
Commun. ACM, 55(8), 26–28. doi:10.1145/2240236.2240246
Maslen, G. (2014, May 20). Australia Debates Major Changes to Its Higher-Education
System. The Chronicle of Higher Education. Retrieved from http://chronicle.
com/article/Australia-Debates-Major/146693/
McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for
digital practice. elearnspace.org. Retrieved from http://www.elearnspace.org/
Articles/MOOC_Final.pdf
References 195
Muilenburg, L., & Berge, Z. L. (2001). Barriers to distance education: A factor-
analytic study. American Journal of Distance Education, 15(2), 7–22.
doi:10.1080/08923640109527081
OECD Publishing. (2013). Education at a Glance 2013: OECD Indicators. Retrieved from
http://dx.doi.org/10.1787/eag-2013-en
Pappano, L. (2012, November 2). The Year of the MOOC. The New York Times.
Retrieved from http://goo.gl/6QUBEK
Porter, W. W., Graham, C. R., Spring, K. A., & Welch, K. R. (2014). Blended learning in
higher education: Institutional adoption and implementation. Computers &
Education, 75, 185–195. doi:10.1016/j.compedu.2014.02.011
Reeves, T. C., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible
approach to instructional technology research in higher education. Journal of
Computing in Higher Education, 16(2), 96–115. doi:10.1007/BF02961476
Rosé, C. P., Carlson, R., Yang, D., Wen, M., Resnick, L., Goldman, P., & Sherer, J. (2014).
Social Factors That Contribute to Attrition in MOOCs. In Proceedings of the
First ACM Conference on Learning @ Scale Conference (pp. 197–198). New
York, NY, USA: ACM. doi:10.1145/2556325.2567879
Rovai, A. P. (2002). Building Sense of Community at a Distance. The International Review

of Research in Open and Distance Learning, 3(1). Retrieved from http://www.
irrodl.org/index.php/irrodl/article/view/79
Sari, N., Suharjito, & Widodo, A. (2012). Trend Prediction for Computer Science Research
Topics Using Extreme Learning Machine. International Conference on Advances
Science and Contemporary Engineering 2012, 50(0), 871–881. doi:10.1016/j.
proeng.2012.10.095
Selwyn, N., & Bulfin, S. (2014). The discursive construction of MOOCs as educational
opportunity and educational threat. MOOC Research Initiative (MRI) - Final
Report. Retrieved from http://goo.gl/ok43eT
196 References
Siemens, G. (2005). Connectivism: A learning theory for the digital age. International
Journal of Instructional Technology and Distance Learning, 2(1), 3–10. Retrieved
from http://www.itdl.org/journal/jan_05/article01.htm
Siemens, G., & Gašević, D. (2012). Special Issue on Learning and Knowledge Analytics.
Educational Technology & Society, 15(3), 1–163. Retrieved from http://www.
learninganalytics.net/?p=174
Smith, B., & Eng, M. (2013). MOOCs: A Learning Journey. In S. K. S. Cheung, J. Fong, W.
Fong, F. L. Wang, & L. F. Kwok (Eds.), Hybrid Learning and Continuing Education
(pp. 244–255). Springer Berlin Heidelberg. Retrieved from http://link.springer.
com/chapter/10.1007/978-3-642-39750-9_23
Thoms, B., & Eryilmaz, E. (2014). How media choice affects learner interactions in
distance learning classes. Computers & Education, 75, 112–126. doi:10.1016/j.
compedu.2014.02.002
Tinto, V. (1997). Classrooms as Communities: Exploring the Educational Character

of Student Persistence. The Journal of Higher Education, 68(6), 599–623.
doi:10.2307/2959965
Tucker, B. (2012). The Flipped Classroom. Education Next, 12(1), 82–83. Retrieved from
http://educationnext.org/the-flipped-classroom/
Vygotsky, L. S. (1980). Mind in Society: Development of Higher Psychological Processes.

(M. Cole, V. John-Steiner, S. Scribner, & E. Souberman, Eds.) (1 edition.).
Cambridge (Mass.): Harvard University Press.
Waltman, L., van Eck, N. J., & Wouters, P. (2013). Counting publications and citations:
Is more always better? Journal of Informetrics, 7(3), 635–641. doi:10.1016/j.
joi.2013.04.001
Wasserman, S. (1994). Social Network Analysis: Methods and Applications. Cambridge

University Press.
Winne, P. H. (2006). How Software Technologies Can Improve Research on Learning and
Bolster School Reform. Educational Psychologist, 41(1), 5–17. doi:10.1207/
s15326985ep4101_3
References 197
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker,
J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory
and practice (pp. 277–304). Mahwah, NJ, US: Lawrence Erlbaum Associates
Publishers. Retrieved from http://psycnet.apa.org/psycinfo/1998-07283-011
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus
traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.
learninstruc.2012.03.004
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M.

Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp.
13–39). San Diego, CA, US: Academic Press. doi: http://dx.doi.org/10.1016/
B978-012109890-2/50031-7
Zimmerman, B. J., & Schunk, D. H. (2011). Handbook of self-regulation of learning and

performance. New York: Routledge.
Future Technology
Infrastructures for Learning
George Siemens
Dragan Gašević
Shane Dawson
199

Abstract
Technology infrastructure shapes learning and learning

opportunities. First generation infrastructure emulated
classroom models where the instructor or institution con-
trolled social interaction and content access. With continued
development of the internet, mobile devices and apps, as
well as the growth of social media and the participative
web, explorations of future technology infrastructures are
required in order to help higher education prepare for next
generation learning opportunities. This paper explores four
factors that influence future technologies: who has control,
how well are the technologies integrated with other tool-
sets and the experiences of learners, who has ownership
of the data and the technology, and what is the nature of
the learning structure in terms of centralization and decen-
tralization. These four factors are then used to explore a
range of emerging technologies that provide an indication
of the types of learning infrastructures that academics and
institutional leaders need to consider in their resource and
pedagogical planning.
Though they are far from only the determinants of behavior,
introduction
enterprise-level technology platforms drive workflow and

organizational processes, thus enabling, supporting, and
encouraging certain activities and limiting or discouraging
others. Nowhere is this truer than with educational technol-
ogy. Learning has traditionally been a process largely under
the control of in-classroom teachers and faculty, albeit acting
within the constraints of regulations, norms, standards,
policies, schedules, imposed curricula, and the physical and
social constraints of classrooms and their facilities. Through
the development of online learning and related technologies,
such as social media and learning management systems
(LMS), administrators and researchers have accessible data
trails to evaluate what happened during the learning process.
Educational technology, however, also defines the scope
and opportunities available to both teachers and learners.
In some instances, the trade-off of greater consistency of
both the learner and the teacher experience, at the expense
of learner options, can be helpful when managing learner
enrolment and grading. In other instances, the trade-off
impacts both teaching practices and learner activities, lim-
iting options for engagement and creativity.
202 Introduction
future technology infrastructures for learning
Technology has received much attention within the education sector over the past several
decades as a means for providing more flexible and scalable access alongside its perceived
capacity for enhancing student learning outcomes. As noted in earlier articles in this series,
the early adoption of distance learning has laid the foundation for the development of
computer-based learning, and then online and blended learning. This progression of ed-
ucational technologies is in part driven by the growing demand and interest in perpetual
learning in all aspects of work, society, and life. The so-called knowledge revolution is
rapidly transitioning into a learning revolution1. In essence, knowledge has become an easily
accessible commodity, resulting in greater emphasis on learning opportunities. Workers
are required to transition from being knowledgeable to more self-managed learners with
the capacities and abilities to recognize their personal learning requirements in order to
address organizational problems and challenges. In addition to increasingly sophisticated
technologies, the proliferation of data and analytics increase the potential for personalized,
adaptive learning. As such, provisioning individuals with the opportunities for relevant and
timely education to meet their learning needs is now a critical challenge for companies,
universities, and governments alike.
One area that has received limited attention in research and in practice is the role of varying
technology infrastructures to address organizational and individual learning needs. While
much has been written about learning management systems, enterprise systems, and social
media, a nuanced evaluation of how educational technology infrastructures are changing is
1 www.forbes.com/sites/jacobmorgan/2014/12/30/the-death-of-knowledge-work-and-the-rise-of-learning-workers/
Introduction 203
required. This paper details the scope of educational technology models and provides some
guidance for future transitions and how higher education can prepare for the adoption of
the next generation of educational software.
Educational technology has gone through three distinct generations of development and
now a fourth is emerging:
Generation 1 — Basic technology use: Computer-based Training (CBT) and websites

Generation 2 — Enterprise systems: learning management systems (LMS) and content
management systems (CMS)
Generation 3 — Fragmentation and diversification: social media, e-portfolio software
and MOOC providers, integrated vendor/publishers
Generation 4 — Distributed and digitally shaped technologies: adaptive learning,
distributed infrastructures, and competency models
Each stage represents a complex relationship of factors, including the following2:
1. Control between learner and faculty/institution, including structured and

unstructured learning activities
2. Ownership of data and content — the learner or the institution
3. Institutional integration — loosely coupled with data exchange happening
through APIs and related industry standards or tightly connected with
enterprise level systems
4. Structure — centralized and decentralized teaching and learning approaches
These four points — control, integration, ownership, and structure — form the basis of
analysis of different technology toolsets and ways in which these toolsets are utilized in
higher education.
2 Existing research in learning sciences on distance, online, and blended learning suggests that the learning design approaches primarily,
and technology secondarily, determines effective learning. As such, the framework provided here should be considered as a way of un-
derstanding how different technologies are experienced by the learner. Important factors, such as scaling a technology for broad adoption
or ensuring standard look and feel across courses, or even integrating technologies into system/state-level enterprise systems, require
considerations beyond the interest of learners and administrators. Context, resource availability, learning design, regulatory requirements,
and other considerations will influence the selection of optimal toolsets in specific courses and institutions.
204 Introduction
CBT and basic websites were the norm in higher education institutions during much of the
1990s. These generally consisted of faculty webpages that included contact information,
course readings, and (in some cases) discussion forums or links to Usenet News. These
faculty webpages were rarely mandated by the university and were generally used to
provide profile information of the faculty member, their courses, and their areas of research.
In the late 1990s, Virtual Learning Environments and Learning Management Systems (LMSs)
at first in the form of homegrown solutions and later through off-the-shelf toolsets, such
as WebCT and Blackboard, emerged as an option to provide more integrated learning
options with greater institutional control. An LMS enabled storage of content, tracking of
learner activity, integration with institutional systems (such as Student Information Systems),
standardized look and feel of courses, and (typically) relatively limited interaction (mainly
threaded discussions in early offerings, though many provided blogs, wikis, and other more
sophisticated tools). Starting in the early 2000s, the LMS marketplace was dominated by
Blackboard as it actively acquired its main competitors, including WebCT and Angel. Open-
source LMSs, such as Moodle and Sakai, gained significant market share during much of the
2000s. Desire2Learn, generally a strong competitor in the marketplace, adopted a “build
slow and integrated” approach in contrast with Blackboard’s acquisition approach. More
recently, Instructure — a cloud-based LMS — has overtaken the number two LMS position
in terms of learner enrolment, largely due to its “web native” usability.
From the late 1990s into the early 2000s, the growth of social media generated interest
within the academy as it offered a learner-controlled approach instead of one that was
institutionally controlled. As the concept of Web 2.0 developed in 2004, faculty increas-
ingly adopted blogs, wikis, syndication through RSS, and social bookmarking as teaching
tools. While this approach provided learners with opportunities to experience collaborative
and knowledge generative learning, the weakly connected toolset sometimes resulted in
frustration for institutions as learner data were spread across a range of toolsets, apart
from when the tools were integrated into the LMS, which led to limiting compromises, in
particular inasmuch as few enabled interaction beyond the course. The learning process
enabled by these tools has resulted in a thinning of classroom walls where learners are
now able to use a range of technologies and interactions with learners and content from
around the world (see Figure 1).
Introduction 205
Peripheral External
Learners experts
Faculty
Co-created Co-created
Content Core Content
content
Learners
Open
Educational
Resources
Figure 1 The impact of networks on learning
Additionally, the learning experience of students varied significantly based on the technical
and pedagogical proficiency of the faculty member and the comfort and skills of learners to
use a range of software products instead of a single integrated suite. While RSS aggrega-
tion tools and simple approaches like the use of HTML frames allowed some integration, it
tended to involve manual effort and skills that, in mainstream teachers, were thinly spread.
The uneven experience for students and the lack of institutional control of large parts of
the learning process meant that adoption of Web 2.0, and later social media, was limited
to technically proficient faculty and learning designers. Institutional adoption of these
technologies was limited, even rare. Several notable projects, including the first MOOC in
2008, utilized a collection of social media technologies and placed greater emphasis on
the process of “stitching” together distributed interactions (see Figure 2). This approach
enabled learners to control the toolsets they personally preferred for learning while still
meeting the needs of educators to make sense of distributed interactions. The software,
gRSShopper, developed by Stephen Downes, produced a daily email that captured inter-
actions occurring on blogs, Twitter, and even the course LMS if RSS feeds were enabled3.
3 See here for examples: http://connect.downes.ca/cgi-bin/archive.cgi?page=thedaily.htm
206 Introduction
Distributed content & converstations
blog
Comment Ecosystem
Learning
Management
Aggregate
Systems Daily email
• Blogs
Filter • Social Media
Social Media • Learner-owned
Bookmarking • ProSolo
Twitter &
Social MedIA
Figure 2 Distributed Content
LMS providers, including open source ones, increasingly included more of the functionality
of social media in their platforms. Blogs and wikis were made available in closed (“walled
garden”) environments. Unfortunately, this presented challenges for pushing truly innova-
tive teaching and learning experiences as courses within these contained systems limited
interaction outside of a class or cohort. While using blogs and wikis within an LMS reflected
participatory pedagogical models, the experience for learners failed to capture interaction
with international students and boundary-less classrooms.
More recently, with the promotion of MOOCs by top research and state universities, ad-
ditional software platforms, notably edX/Open edX and Coursera, have gained interest
as delivery platforms. Both edX and Coursera systems are still developing and lack many
features of established LMS providers. The major LMS companies now offer some version
Introduction 207
of open or MOOC courses4, though the registration in those courses involves significantly
fewer learners than those in edX and Coursera (which can include tens of thousands or
even hundreds of thousands of learners).
The growing influence of vendors in the educational marketplace is often described in terms
of “unbundling.” The process of unbundling results in changes to existing power structures
of education. Instead of an integrated, end-to-end system of education, the future looks
more like a content and technology marketplace, with many vendors offering to take over
core university functionality, such as recruitment of students, testing, and even teaching.
These changes are providing lucrative opportunities to vendors5, but also backlash from
other stakeholders in education.
Another aspect of third generation educational technology, though it has received less
attention than social media and MOOCs, is the growing interest in e-portfolios. Portfolios
in the learning process have long been a staple in art or performance fields. Portfolios are
particularly useful when combined with some form of Prior Learning Assessment and Rec-
ognition (PLAR). This approach, which provides the foundation for current developments
in competency-based education, requires learners to produce evidence regarding what
they know and are able to do and then to compare that evidence with existing knowledge
and skill requirements in a particular subject area. Nursing and education have both seen
reasonable adoption of e-portfolios, especially with a growing range of commercial (such
as PebblePad or D2L’s portfolio offering in their LMS) and open-source products (such as
Mahara).
The first three generations of educational technology innovation have resulted in a complex
and multifaceted landscape. Product offerings range from integrated suites (such as Bright-
Space, Canvas, and Blackboard), to single functionality tools such as Mahara, to social media
products like Facebook and Elgg. The technology offerings within LMSs have, to date,
largely emulated existing classroom pedagogical models. This is starting to change. More
recently, a number of research projects and software products have become available that
address competency-based learning, adaptive and personalized learning, and self-regulated
(student driven) learning. These software products include Knewton, Smart Sparrow, OLI
(now at Stanford and CMU), and LoudCloud.
4 See https://www.canvas.net/, https://opencourses.desire2learn.com/cat/, and https://openeducation.blackboard.com/mooc-catalog/cata-

log?tab_tab_group_id=_10_1
5 The stakes and the sources of conflict are reflected in the recent Fortune publication detailing how Pearson is facing mounting criticism
for its activity in the education sector: http://fortune.com/2015/01/21/everybody-hates-pearson/
208 Introduction
Numerous innovative research projects, often overlooked in favour of commercial vendors,
are important to review as they provide insight into how educators and researchers are
thinking about restructuring higher education and the learning process. The section that
follows introduces a range of projects and software products that we feel provide important
insights into future technology infrastructure.
Systemic-level
innovations
Several university systems and multi-system collaborations have been initiated in order to
take advantage of scale and elimination of duplicated efforts across multiple institutions.
Three examples provide some direction about future systems-level infrastructure: University
of Texas System (single institution but multiple universities), Unizin (multiple institutions), and
the Apereo Foundation (multiple institutions, multiple international partners). Systems-level
innovation attempts to improve end-user experience while reducing operating expenses
and frustrations for individual universities.
University of Texas System: TEx6

In partnership with numerous educational technology providers, the Institute for Transfor-
mational Learning (ITL) is developing TEx, a set of platforms, applications, and services for
use across all ITL industry-oriented, competency-based educational initiatives. The acronym
TEx derives from the words “Total Education Experience” and will deliver personalized,
adaptive educational programming and support services at scale across the system.
6 Thanks to Steven Mintz, Executive Director of the University of Texas System’s new Institute for Transformational Learning, for his
contributions to this section on TEx
Systemic-level Innovations 209

Built around a personal profile, TEx is designed to assist students from the moment they
express interest in UT System programs through graduation and entry into their chosen
career. It will aggregate information about the student’s learning experiences, making it
possible for instructors to deliver customized content and learning pathways tailored to
individual students’ unique needs and interests. TEx will also encourage networking and
collaborative experiences among faculty, students, and others.
In addition, TEx will generate data about the student learning experience to support
continuous improvement of program and curriculum design, pedagogy, and student ser-
vices operations. TEx will unify data across traditional SIS, CRM, and LMS systems to drive
timely and personalized recommendations and support services tailored to each student’s
unique set of capabilities, experiences, personal circumstances, and aspirations. Analytics
will improve advising, drive continuous improvements in pedagogy, and allow institutions
to better assess the effectiveness of their student support programs.
CONTROL
Learner Institution
OWNERSHIP
Learner Institution
TEx
INTEGRATION
Loosely Enterprise level

coupled
Structure
Decentralized Centralized
Figure 3 TEx
210 Systemic-level Innovations

Unizin7
In June 2014, four universities announced the formation of Unizin with the stated goal of
becoming “the Internet2 of Digital Education” as a member-owned consortium. The meta-
phorical reference to Internet2 draws on the history of the university-operated infrastructure
for advanced digital networks. Internet2 ensures that decisions regarding the use of the
network, its privacy policies, ownership, etc. are vested in the hands of its 250+ college and
university members. Ownership affords decision rights and influence, and Unizin can ensure
that content, privacy, and analytics policies and costs are steered by Unizin’s members8.
CONTROL
Learner Institution
OWNERSHIP
Learner Institution
Unizin
INTEGRATION

coupled
Structure
Figure 4 Unizin
7 Thanks to Brad Wheeler, Indiana University Vice President for IT & CIO, Dean, and Professor, for his contributions to this section on
Unizin.
8 See http://unizin.org/2014/06/why-unizin

Apereo9
The Apereo Foundation is a not-for-profit entity supporting the creation and sustenance
of open-source software serving the academic mission — learning, teaching, and research.
Apereo is a global membership organization with around a hundred higher education and
commercial partners on six continents.
Apereo works in close partnership with other organizations with similar or adjacent areas
of interest. It has a close partnership with the seventy-institution-strong ESUP consortium
in France10 — creating a network some one hundred and seventy strong. Apereo also has
a developing relationship with the Open Source Initiative, Society for Learning Analytics
Research, and InCommon.
Since Apereo was formed from the merger of two pioneers of open source in higher
education — Jasig and Sakai — in 2012, it has given particular emphasis to creating a
framework for developing software project sustainability. This incubation process — a
scaffolded process assisting projects on the path from innovation to sustainability — has
proved a major success. Apereo currently acts as a legal, licensing, and service umbrella
for a dozen projects.
The Sakai environment has served higher education for almost a decade as a collaboration
and learning environment. While mainly deployed as a Learning Management System,
Sakai has also found a number of adoptions supporting research collaboration. Sakai is
designed for campus and above-campus deployment. In addition to considerable visual
and tool flexibility, Sakai has played a considerable role in promoting the development and
adoption of key educational technology standards, including IMS GLobal Learning Tools
Interoperability, and standards supporting learner analytics. Emerging consensus in the Sakai
community is driving the environment to provide further, and more effective, integration
points for external or alternative tools. This “hollowed out LMS” or “Enterprise Learning
Bus” approach will provide the flexibility to fit Sakai more closely to the needs of a greater
range of institutions and disciplines. It is highly likely, therefore, that Sakai will continue to
play a significant role in education going forward.
9 Thanks to Josh Baron, Assistant Vice President, Information Technology for Digital Education Marist College, and Ian Dolphin, Execu-
tive Director, Apereo Foundation, for their contributions to this section on Apereo.
10 https://www.esup-portail.org/
212 Systemic-level Innovations

The Apereo Open Academic Environment (OAE) is a next-generation platform for academic
collaboration. While OAE can be (and is) used to support formal education effectively, it
intentionally breaks with the LMS organizational models of courses and classes, instead
resting on more flexible concepts of individuals and groups collaborating around content.
In developing OAE, particular emphasis has been placed on user-experience designers
working with academic practitioners to provide not only the required functionality, but to
provide that functionality in a way that is easy for end-users. OAE is designed from the
ground up as a multi-tenant, hosted environment, with optional permeability between
tenants. This latter feature enables controlled collaboration between institutions for faculty
and students. Apereo OAE currently has around forty adopters, with significant rollouts in
the near future in France and Australia.
CONTROL
Learner Institution
OWNERSHIP
Apereo
Learner Institution
INTEGRATION

coupled
Structure
Figure 5 Apereo

Research
Projects
The Landing11
The Landing12 is a social site exclusively for members of Athabasca University and a few
invited guests. Athabasca University is a publicly funded distance education and research
university, with almost all of its undergraduates taking self-paced courses. Numerous
master’s and doctoral programs are also offered. Like its students, the faculty are also
distributed (primarily in Alberta). Although it offers a great deal of freedom and flexibility,
the distributed distance model results in a range of challenges caused by isolation for both
staff and students. Among other things, this model led to difficulties in communication, an
over-emphasis on structured courses and processes, difficulties spreading knowledge across
the institution, and problems sustaining motivation. The Landing was created in 201013 in
response to these unusual conditions using the Elgg social framework. It incorporates blogs,
wikis, microblogs, file sharing, social bookmarking, podcasting, photo sharing, and many
other read-write tools, all with discretionary access control, from fully private to completely
public. It supports social networking as a feature rather than a destination (see Chris An-
derson http://www.thelongtail.com/the_long_tail/2007/09/social-networki.html) as well as
having facilities for creating more or less open groups that act as containers for focused
group working. The site was conceived as a space between the hierarchical, structured
space of an LMS and the unstructured, informal, personal free-for-all of email, VOIP, social
media, and chat. In many ways, it might be seen as playing the role of similar in-between
spaces in traditional physical universities, such as common rooms, hallways, quads, and
library working areas, including not just the people but also the artefacts that they create
and share with one another. It is a space where other people and their activities are visible
and reified. With its large and diverse toolset, it can be shaped to fit a wide range of social
and learning needs.
11 Thanks to Jon Dron, Professor, School of Computing and Information Systems, Athabasca University, for his contributions to this
section on the Landing
12 http://landing.athabascau.ca/
13 Terry Anderson and Jon Dron are the principal investigators and leads on the Landing. Disclaimer: report author George Siemens was
project lead from 2010–2013.
214 Research Projects

The Landing was deliberately designed and nurtured as a diverse learning commons.
Though a lot of use has been made of the system to support formal courses, it has at least
equal value as a networking space, a place for ad hoc working groups and committees,
a space to share ideas, work in progress, observations, and — sometimes — complaints.
With a nod to Jane Jacobs’ observations on what makes a city area thrive, it was built so
that there should be many reasons one might need or wish to be there, the intent being to
keep it lively and filled with visible activity. This has many benefits, not least of which are the
opportunities to learn from what others are doing, to make serendipitous encounters, to
feel a part of a large and active learning community, and to spread learning within courses
beyond the silos that are an inevitable result of the hierarchical role-oriented structure of
a learning management system.
CONTROL
Learner Institution
OWNERSHIP
The landing
Learner Institution
INTEGRATION

coupled
Structure
Figure 6 The Landing
Research Projects 215

Federated Wiki14
Federated wiki is a new technology developed largely by the inventor of wiki, Ward
Cunningham. Like wiki, it encourages revision, reuse, and extension of community ideas.
However, it does this not through a centralized site, but through a federation of individually
owned wikis that fork text15, data, formulas, and media from one another. It has been called
“Github for Wiki,” but is perhaps better explained as a unique mix of blogging and wiki.
With Ward Cunningham’s assistance, Mike Caulfield (Washington State University Vancouver)
has been investigating the application of the technology to education. Historically, blogging
has provided a reflection and communication space for distributed courses. It is hoped that
federated wiki could provide a similar space in such courses for loosely coupled collaboration
and cooperation around text and data16. The “Fedwiki Happening” run in December 2014
explored the use of federated wiki in a distributed learning environment. This experiment
was a follow-up to the successful use of federated wiki in a traditional college class.
The results were intriguing. Happening participants were told to explore their academic or
professional interests on their own wiki, and to fork and edit elements of other participant
wikis if they found them useful to their own learning goals. In the Happening, rather than
have them reflect in social space, students were asked to engage in the “mining” of various
things they read for ideas, examples, and data that might be applied to other problems (in
this way, the wiki borrows from design patterns methodology in software). By abstracting
ideas and examples from texts, participants increased their understanding of the texts, and
by presenting the results in a modular way, they provided materials through which other
students could advance their own investigations. Even with the small number of participants,
a surprising number of serendipitous connections occurred.
14 Thanks to Mike Caulfield, Director, blended and networked learning at Washington State University, Vancouver, for his contributions
to this section on the Landing
15 Forking refers to the process of taking one resource and essentially copying it to create an additional resource where changes made
to the forked version do not impact the first. This allows for personalization, customization, and development opportunities that might
advance beyond what the original creators had intended.
16 The process of information flow in a federated system is described here: https://www.youtube.com/watch?v=2Gi9SRsRrE4

Cunningham and other volunteer programmers are currently redesigning the software
based on feedback and analysis from the Fedwiki Happening and plan a retooling based
on the new educational focus.
CONTROL
Learner Institution
Federated Wiki
OWNERSHIP
Learner Institution
INTEGRATION

coupled
Structure
Figure 7 Federated Wiki

gRSShopper17
Created to facilitate production of the OLDaily newsletter, gRSShopper18 has evolved into
a content management tool that aggregates contents in a variety of formats and facilitates
the publication of newsletters and RSS feeds. gRSShopper was used to support the first
MOOC, Connectivism and Connective Knowledge, in 2008. Course participants contribut-
ed their blog or content site URL and used these, rather than a course-specific discussion
board, to contribute comments and resources to the course. gRSShopper aggregated the
submissions and prepared a daily summary of contents. gRSShopper also manages event
listings, file uploads, and a chat room. gRSShopper is free and open source and available
for download, but is a research environment, not production-grade code.
CONTROL
Learner Institution
OWNERSHIP
gRSShopper
Learner Institution
INTEGRATION

coupled
Structure
Figure 8 gRSShopper
17 Thanks to Stephen Downes, LPSS Program Lead, National Research Council Canada, for his contributions to this section on
gRSShopper
18 http://grsshopper.downes.ca/

Learning and Performance Support Systems19
LPSS20 is a personal learning environment being developed by the National Research Council.
The project was announced in 2013 and launched as a $19 million, five-year project. Cur-
rently offered only as a hosted service and still in pre-Alpha, LPSS will enable learners to
manage their learning activities from a single environment while connecting to resources
and services offered by multiple vendors. LPSS supports competence-based learning and
generates resource recommendations from among a set of personally selected learning
providers. The system also supports personal cloud technology for document storage and
synchronization, a personal learning record (including badges and an e-portfolio), and a
personal learning assistant intended to enable LPSS access through desktop and mobile
devices as well as in third-party products and software.
CONTROL
Learner Institution
OWNERSHIP
Learner Institution
LPSS
INTEGRATION

coupled
Structure
Figure 9 LPSS
19 Thanks to Stephen Downes, LPSS Program Lead, National Research Council Canada, for his contributions to this section on Learning
and Performance Support Systems
20 http://lpss.me

ProSolo
ProSolo21 is a platform designed to facilitate self-directed and competency-based learning
through social interaction. ProSolo has been developed as a response to the on-going
challenges of traditional educational models primarily focused on classroom education
and training typically associated with the notion of credit hours as the (only) route towards
formal credentials. In contrast, the demands for education are far more diverse, with many
individuals now undertaking greater forms of informal learning or professional development
and education opportunities within the workplace. The “non-traditional learner” is now the
dominant student characteristic in the majority of institutions in higher education22. This
suggests a need for establishing a greater diversity of learning opportunities. At the same
time, institutions recognize a need to innovate educational practices through initiatives such
as active learning and flipped classrooms in order for learners to be actively engaged in
learning and developing core skills — such as information seeking, critical thinking, creativ-
ity, and teamwork — necessary lifelong skills and capacities for productive participation in
contemporary society. Therefore, the shift in higher education to more novel learning and
instructional models enabled through adaptive and personal technologies are essential.
ProSolo provides users with the ability to unbundle education programs, courses, and units
into discrete yet inter-related competencies, allowing learners to construct their education
pathway in a manner that better reflects their interests and future career motivations and
requirements. ProSolo is developed with the intention of providing learners with oppor-
tunities to customize, modify, and personalize their self-directed learning journey. In this
way, ProSolo’s design and conceptual model captures the essence of personalized learning
whereby learners take control over, drive, and make decisions about their learning. ProSolo’s
mapping of personal goals and experiences directly to the achievement of competencies
and granting of credentials provides learners with greater flexibility in their study options, as
well as the recognition of alternate learning pathways and prior experience. For education
providers, ProSolo maps curriculum activities directly to learner competencies and outcomes,
allowing for easy unbundling and reassembling of degree programs and courses. ProSolo
unpacks the rigidity of existing programs to cater to alternate educational pathways, pro-
viding students with new opportunities to gain recognition for prior learning and achieved
credentials and relevant life and work experiences. ProSolo doesn’t just break the concept
of the credit hour — it totally removes it.
21 Report authors Shane Dawson, Dragan Gašević, and George Siemens are all involved in the research and development of ProSolo.
22 http://www.census.gov/prod/2013pubs/acsbr11-14.pdf

ProSolo supports the development of skills for self-directed learning by allowing learners to
control the planning, learning, and presentation of outcomes associated with their learning
(see Figure 2). Learning in ProSolo occurs within a socially rich environment that aggregates
learners’ information created and shared in their existing online spaces. ProSolo encourag-
es learners to continue using those existing online spaces — such as blogs, YouTube, and
SlideShare — and assists them in the aggregation of their personal information to establish
their personal and publically shared profiles. The aggregated information can then be used
as evidence for competency achievement in the credentialing process. With the possibility
of creating and sharing personal profiles, learners also have opportunities to establish a
strong social presence that goes beyond a single course offering. This method promotes
the creation of communities and peer collaborations. Learners have opportunities to find
or be matched with others based on numerous factors, such as shared interests, similarity
of their profiles, and proximity of their geographic locations. To validate competencies
acquired even through informal and authentic contexts, ProSolo has a robust credentialing
pipeline to assess and validate the evidence learners provide in order to demonstrate the
achievement of the competencies claimed. The pipeline has its own functionality of support-
ing instructor and peer assessment, and has integration — through the IMS Learning Tools
Interoperability (LTI) standard23 — with other technologies commonly used in assessment,
such as open badging platforms, quizzing engines, or automated essay-scoring technologies.
23 http://www.imsglobal.org/toolsinteroperability2.cfm

o rk place Learn
W ing

Op
en
Socialize
ed
uc

at
io
N
LA
n Re

sources For
Lea
LEARNER
rsera
rn
ou
es
PR
en
C
ma
ed
X
l

Credential
Figure 10 The conceptual model for self-directed and competency-based learning through social interaction in
ProSolo.

The design of ProSolo recognizes that learners may not be prepared for self-directed
learning and may require a different form of scaffolding. To support learners with different
levels of prior knowledge, study skills, and cultural backgrounds, ProSolo offers features
for supporting self-directed learning through three types of scaffolds:
Instructional — enabling promotion and incorporation of the best principles

established for instructional design in online and blended learning; this form
of scaffold provides instructors with direct involvement and control over the
instructional processes. Rather than replacing instructors, ProSolo empowers
educators with a technology that creates the possibility for interaction with and
instruction of their learners in diverse and authentic contexts.
Social — building on established empirical evidence that asynchronous online

discussions among students are one of the most potent instructional strategies.
At the same time, the use of social media — embedded in and aggregated by
ProSolo — offers seamless communication and information sharing.
Technological — harnessing the power of learning analytics and machine

learning, as well as learner modeling, ProSolo provides automatically generated
recommendations to assist learners in their learning decisions about plans for
future learning and feedback on the ongoing learning.
ProSolo has been successfully used to support an innovative dual-layer instructional design
used in the Data, Learning, and Analytics MOOC24 offered in partnership with edX in late
2014. Presently, ProSolo is piloted on sites of several institutions in the USA and Australia
in supporting their transition to competency-based programs and credentialing for con-
tinuous career development.
24 http://www.edx.org/course/utarlingtonx/utarlingtonx-link5-10x-data-analytics-2186 and http://dalmooc.prosolo.ca

CONTROL
Learner Institution
OWNERSHIP
Learner Institution
ProSolo
INTEGRATION

coupled
Structure
Figure 10 ProSolo
Domain of One’s Own/Reclaim25

Started at the University of Mary Washington in the Fall of 2013, Domain of One’s Own
(DoOO) provides any interested faculty, students, and staff with their own web hosting
and domain. This personal cyber-infrastructure not only enables individuals to manage and
control the work they do over the course of their academic career, but also experiment
with a wide range of open-source applications from WordPress to Omeka to Moodle to
Mahara to Known. So rather than being a single tool provided by central IT, DoOO provides
a toolkit for innovation to an entire community.
These personal environments are hosted and supported by the university, providing a
platform to start encouraging and scaffolding a deeper understanding of how the web
works, and how 21st century learners can fully embrace its possibilities. Most importantly,
unlike most proprietary “all-in-one solution systems,” the infrastructure behind DoOO is
highly affordable, eminently portable, and easily transferable. The community is working
25 Thanks to Jim Groom, Executive Director of the Division of Teaching and Learning Technologies and adjunct professor at the Universi-
ty of Mary Washington, for his contributions to this section on Domain of One’s Own

on the industry standard, CPanel, which fuels most major hosting sites across the web, i.e.,
BlueHost, HostGator, etc. So, at the end of their time at university, those interested can
seamlessly transition their domain and/or hosting to the domain registrar and web host of
their choosing and continue to build and define their digital presence well beyond their
time at UMW. This approach to empowering faculty, students, and staff has caught on at
numerous universities. Emory University, University of Oklahoma, CSU Channel Islands,
Davidson College, and several others have adopted this approach on an institutional level,
not to mention the scores of initiatives at the departmental and course level.
Alongside Domain of One’s Own, there has been increased focus on reclaiming the work
we do online from larger, siloed sites such as Facebook, Twitter, Flickr, Instagram, etc. While
not advocating for secession from the social web, this segment of users is concerned with
taking greater control over the updates we post, images we upload, bookmarks we save,
videos we share, etc. More often than not, we are sharing our work through social media
conglomerates that often have unilateral control over the content posted, not to mention
limited interest in any long-term preservation (see Blip.tv, posterous, Geocities, del.icio.us,
and many, many more). So, a move to personalize infrastructure provides a shift away from
social media sites as the sole source of online work. This approach to publishing in one’s own
space while still sharing widely is one of the tenets of the IndieWeb26 movement, namely
POSSE: publish on your own site and syndicate everywhere. Your domain becomes the hub
from which you manage the many forms of work that you do across the web, a concept
with implications in every facet of our digital life well beyond education, from managing our
digital medical records to property insurance, taxes, etc. The reclaim movement provides
an alternative path based on an affordable infrastructure that empowers our communities
to take back control of the web.
Reclaim Hosting is a service born out of this vision that currently provides hosting for thou-
sands of students, faculty, and institutions. With so many IT departments either unable or
unwilling to support the changing nature of infrastructure to provide such personalized
spaces as part of the institutional ecosystem, Reclaim Hosting has become an option for
many to explore what this user innovation toolkit might mean on an individual, course, de-
partment, or even institutional level. One way to think about Reclaim Hosting is affordable,
distributed ed-tech infrastructure for any interested party.
26 indiewebcamp.org

CONTROL
Domain of One’s Own Learner Institution
OWNERSHIP
Learner Institution
INTEGRATION

coupled
Structure
Figure 11 DooO
Open Learning Initiative27

Carnegie Mellon University’s Open Learning Initiative (OLI) leverages learning science and
emerging technologies to design, deliver, and improve web-based learning experiences.
OLI’s methodology is supported by an online platform for developing, delivering, and con-
tinuously refining learning materials while driving ongoing research in learning science; in
this way, OLI’s platform integrates research and practice, providing learners and educators
with an experience informed by the best science while advancing our understanding of
how humans learn.
The OLI platform is a collection of tools for creating and delivering online instruction that
embeds core learning science principles in the system’s design, capabilities, and navigation.
Content in the system combines traditional expository materials with extensive opportuni-
ties for practice, targeted feedback, and robust hints for a complete, supported learning
27 Thanks to Candace Thille, Senior Research Fellow for the Office of the Vice Provost for Online Learning, Stanford University, and
Norm Brier, Director of the Open Learning Initiative (OLI) and Core Collaborations at Carnegie Mellon University, for their contributions
to this section on Open Learning Initiative

experience. The design of the system has been enhanced by Universal Design for Learning
principles to increase flexibility, address learner variability, and allow learners multiple ways
to recognize, act on, and engage with knowledge. The system exhaustively captures data
on learner interactions; this data is then used to provide meaningful feedback to learners,
instructors, and course developers, as well as to support researchers. The system provides
a core set of functionality for content, including traditional expository materials (text,
examples, images, videos, etc.). The system also provides mechanisms for incorporating
other non-core technologies through APIs. Such non-core technologies include standard
elements used frequently in courses, including certain types of labs, simulations, and cogni-
tive tutors. These technologies can also include less standard, more experimental elements
— as technologies and associated pedagogical approaches become less experimental and
better tested, their use becomes more standardized, eventually moving towards integration
with the core system.
The system exhaustively captures data on student learning and behaviors, tracking student
actions through the course, including questions, responses, feedback, hinting, login, page
views, video watching, etc. This data, in association with an underlying cognitive model,
drives an analytics dashboard that can provide a real-time view of student learning and
performance. The data is used to create various summary reports for improvement and
evaluation; it can also be exported to the Pittsburgh Science of Learning Center’s DataShop.
DataShop provides analytic methods for understanding learning interaction data and pro-
vides the infrastructure for sharing these datasets with other researchers. This data capture
and use supports continuous course improvement, a scientific approach to course design,
and a virtuous cycle that can improve our understandings of human learning.
The OLI system supports semantic tagging of content, associating metadata such as skills
and learning objectives with specific pieces of content at various levels of granularity (ranging
from pages to individual question parts). The system also strictly enforces a semantic rep-
resentation of all content, thus supporting design and research. These semantic represen-
tations distinguish OLI’s approach to learner data capture from other clickstream-based
systems, allowing insights into underlying pedagogical intent and learner misconceptions.
The OLI platform is currently provided as a hosted environment; the core platform consists
of a Java enterprise application running on Linux using the Apache web server, JBoss ap-
plication server, and MySQL database software — an open-source development stack. The
systems architecture supports open-source application servers that can be run locally, using
Red Hat Enterprise Linux or EC2 using Amazon Web Services. Additional technologies are

used in the non-core tools. OLI provides support for Basic LTI, integrating with LMS systems
as an LTI tool. The OLI system also provides some minimum set of LMS functions — spe-
cifically scheduling and a grade book — that are not well integrated into LTI consumers.
CONTROL
Learner Institution
OWNERSHIP
Learner Institution
OLI
INTEGRATION

coupled
Structure
Figure 12 OLI
Known28
Known is an open-source social publishing platform created by Ben Werdmuller (who
previously co-founded Elgg) and Erin Richey. Students publish content to a site that they
control and then optionally syndicate it to systems they have less control over, including
central course spaces, learning management systems, and social networks like Facebook,
Twitter, Flickr, and LinkedIn. Known supports status updates, blog posts, photos, audio,
and more. New content types, themes, and syndication integrations can be added easily.
Like Elgg, Known contains granular access permissions and extensive social functionality.
However, Known has been developed with today’s web in mind; for example, it is fully
28 Thanks to Ben Werdmuller, CEO and co-founder at Known, for his contributions to this section Known

responsive and works well on any mobile device, reflecting the reality that students use
these devices to browse the web over 50% of the time.
Unlike Elgg, each student’s profile is its own stand-alone site. This means that profiles
can easily be moved; for example, when students graduate, they may take their site and
content with them. It also allows students and faculty to customize their profiles to their
liking, including by radically changing the visual style. Due to this structure, Known users
can reply to each other and interact with each other’s content in a peer-to-peer manner,
wherever their site is hosted.
Known is structured as a start-up, based in San Francisco, California. The company provides
a hosted service, as well as consultancy, customization, and development services. A course
aggregator and hub software for easier deployment are also available29. Jim Groom at the
University of Mary Washington, among others, is piloting the software.
CONTROL
Learner Institution
OWNERSHIP
Known
Learner Institution
INTEGRATION

coupled
Structure
Figure 13 Known
29 Further details and open-source software available at https://withknown.com/

Conclusion
Higher education is changing. Central to this change is the transition from a physically
based learning model to one that makes greater use of digital technologies. A brave, new
landscape of toolsets is now emerging, each with various elements of control, integration,
ownership, and structure. As leaders, educators, and students begin selecting tools for en-
terprise deployment, questions of control and ownership become as important as questions
of integration and structure. More importantly, the technologies selected will determine
the quality of learning, the scope of teaching practices, and ultimately, how well learners
are equipped for both employment and engagement in democratic and equitable models
of modern global society.
230 Conclusion
View publication stats

Preparing For The Digital Educationt PDF

Uploaded by

Copyright:

Available Formats

Preparing For The Digital Educationt PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Preparing For The Digital Educationt PDF

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

Preparing for the digital university: a review of the history and

Technical Report · January 2015

Physical Activity View project

The user has requested enhancement of the downloaded file.

George Siemens, PhD

Athabasca University served as the Principal Investigator

This work is licensed under a Creative Commons Attribution-ShareAlike

The History and State of Distance Education 9

The History and State of Blended Learning 55

The History and State of Online Learning 93

The History and State of

Where is Research on

4 gsiemens@uta.edu, Twitter: gsiemens

This report is one of a series of reports describing the histor-

are ever changing, the major premise remains the same:

The definition and scope of distance education also changed

The purpose of this study

History of Distance Education:

Modern Distance Education:

Study search method and inclusion criteria

i. Be published in a peer-reviewed scientific journal, conference paper, doctoral

Figure 1 Example of SCOPUS search query

Digital Library Search Google Scholar Search Manual Journal Search

Identified 306 studies Identifed 19 studies Identifed 14 studies

Merging search results

Total of 339 unique

Selection of distance Selection of online Selection of blended

Total of 37 distance Total of 32 online Total of 20 blended

Systematic search end

Figure 2 Systematic literature search process

22 Overview of Distance Education Studies

Overview of Distance Education Studies 23

conf. paper thesis adult he & adult

24 Overview of Distance Education Studies

Report Title Report Type Studies Years Level

Allen et al. (2002). Comparing student

Allen et al. (2004). Evaluating the

Bernard et al. (2004). A methodological

Bernard et al. (2004). How does

Overview of Distance Education Studies 25

Report Title Report Type Studies Years Level

Bernard et al. (2004). The effects of

Borokhovski et al. (2012). Are contextual

Lou et al. (2006). Media and pedagogy

Machtmes & Asher (2000). A meta-

Shachar & Neumann (2003). Differences

26 Overview of Distance Education Studies

Report Title Report Type Studies Years Level

Storrings (2006). Attrition in distance

Williams (2006). The effectiveness of

Booth et al. (2009). Applying findings

Borokhovski et al. (2011). An extended

Childs et al. (2005). Effective e-learning

Overview of Distance Education Studies 27

Report Title Report Type Studies Years Level

Chipps et al. (2012). Videoconference-

Chipps et al. (2012). A systematic review

Hrastinski & Keller (2007). Computer-