See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/254088681

Design-Based Research

Article  in  Educational Researcher · February 2012

DOI: 10.3102/0013189X11428813

CITATIONS READS

570 10,746

2 authors:

Terry Anderson Julie Shattuck

Athabasca University Frederick Community College
139 PUBLICATIONS   13,442 CITATIONS    8 PUBLICATIONS   678 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Identifying community among self-paced online and distance learners View project

Crowd Science View project

All content following this page was uploaded by Julie Shattuck on 17 May 2014.

The user has requested enhancement of the downloaded file.

 Educational Researcher http://er.aera.net

Design-Based Research: A Decade of Progress in Education Research?

Terry Anderson and Julie Shattuck
EDUCATIONAL RESEARCHER 2012 41: 16
DOI: 10.3102/0013189X11428813

The online version of this article can be found at:

http://edr.sagepub.com/content/41/1/16

Published on behalf of

American Educational Research Association

and

http://www.sagepublications.com

Additional services and information for Educational Researcher can be found at:

Email Alerts: http://er.aera.net/alerts

Subscriptions: http://er.aera.net/subscriptions

Reprints: http://www.aera.net/reprints

Permissions: http://www.aera.net/permissions

>> Version of Record - Feb 3, 2012

What is This?

Downloaded from http://er.aera.net by guest on October 22, 2013

 Feature Articles
Design-Based Research: A Decade of Progress
in Education Research?
Terry Anderson1 and Julie Shattuck2

Design-based research (DBR) evolved near the beginning of the 21st study. For a more in-depth overview, the interested reader is
century and was heralded as a practical research methodology that invited to read the articles in the three special issues of education
could effectively bridge the chasm between research and practice in journals published in 2003 and 2004: Journal of the Learning
Sciences, vol. 13, no. 1; Educational Researcher, vol. 32, no. 1; and
formal education. In this article, the authors review the characteristics
Educational Psychologist, vol. 39, no. 4.
of DBR and analyze the five most cited DBR articles from each year
Our own analysis of the original and emerging definitions of
of this past decade.They illustrate the context, publications, and most methodology suggest that a quality DBR study is defined by the
popular interventions utilized. They conclude that interest in DBR is following:
increasing and that results provide limited evidence for guarded opti-
Being Situated in a Real Educational Context
mism that the methodology is meeting its promised benefits.
Being situated in a real educational context provides a sense of
validity to the research and ensures that the results can be effec-
Keywords: classroom research; design-based research; mixed tively used to assess, inform, and improve practice in at least this
methods; research utilization; teacher research one (and likely other) contexts.
Focusing on the Design and Testing of a Significant
Intervention

T
he first decade of this century has seen the emergence of Ann Brown (1992), the American researcher credited with first
a new research methodology for education research— developing DBR, noted that “an effective intervention should be
design-based research (DBR). A number of respected able to migrate from our experimental classroom to average class-
education researchers and special issues of well-known journals rooms operated by and for average students and teachers, sup-
have celebrated the potential of DBR to make a significant differ- ported by realistic technological and personal support” (p. 143).
ence in the quality and utilization of education research. But has The selection and creation of the intervention is a collaborative
DBR lived up to these expectations? task of both researchers and practitioners. The creation begins
DBR is a methodology designed by and for educators that with an accurate assessment of the local context; is informed by
seeks to increase the impact, transfer, and translation of educa- relevant literature, theory, and practice from other contexts; and
tion research into improved practice. In addition, it stresses the is designed specifically to overcome some problem or create an
need for theory building and the development of design princi- improvement in local practice. The intervention may be a learn-
ples that guide, inform, and improve both practice and research ing activity, a type of assessment, the introduction of an admin-
in educational contexts. istrative activity (such as a change in holidays), or a technological
In this article, we review the basic features of DBR, describe intervention—to mention just a few of the common types of
the trends toward increasing its use, and highlight and analyze interventions.
the most cited articles that focus on DBR in education. We note The design of these interventions is a key feature of the quality
the challenges that researchers deploying the methodology face and results of the research project. Mingfong, Yam San, and Ek
and speculate on its application in the near future. Ming (2010) identified four design characteristics that they sug-
gest must be aligned to create effective interventions. These are
What Is DBR? “frameworks for learning, the affordances of the chosen instruc-
Descriptions of DBR abound and have even found a home in a tional tools, domain knowledge presentation, and contextual
short article on Wikipedia, so we will do no more than give a limitations” (p. 470). The researcher is careful to document the
cursory overview of the methodology. We should note that the time, commitment, and contingencies that are involved in the
terms “design-research” (Oha & Reeves, 2010) and “develop-
ment research” (Conceicao, Sherry, & Gibson, 2004; Oha & 1Athabasca University, Centre for Distance Education, Edmonton, Alberta,

Reeves, 2010) also have been used to describe this methodology, Canada
but we use the more popular term design-based research in this 2Frederick Community College, Centre for Distributed Learning, Frederick, MD

Educational Researcher,Vol. 41, No. 1, pp. 16–25

DOI: 10.3102/0013189X11428813
© 2012 AERA. http://er.aera.net
16 educational Researcher
creation and implementation of the intervention. These are doc- with equal effect in all contexts. Rather, design principles reflect
umented so that readers of the research can judge for themselves the conditions in which they operate. These tools and conceptual
the possibility of achieving similar—or even better results—from models function to help us understand and adjust both the con-
the use of this intervention in their own contexts. text and the intervention so as to maximize learning. Dewey
warns that although general ideals and principles are of value in
Using Mixed Methods
the direction and enlargement of conduct, they are also danger-
DBR interventions are assessed on a wide variety of indices using ous: They tend to be set up as fixed things in themselves, apart
multiple methodologies. DBR is largely agnostic when it comes from reference to any particular case (Dewey, 1932, as cited in
to epistemological challenges to the choice of methodologies Boydston, 1971, p. 232). Dewey realized that new meanings,
used and typically involves mixed methods using a variety of values, and attitudes become encultured in schools only when
research tools and techniques. Most DBR researchers would con- they have become embodied and are sustained within real-life
cur with Maxcy (2003), who argued, “It is perfectly logical for contexts. This requirement to develop practical design principles
researchers to select and use differing methods, selecting them as is a key strength of DBR, and it disadvantages those types of
they see the need, applying their findings to a reality that is both research that unilaterally descend for testing in a classroom and
plural and unknown” (p. 59). Moreover, the choice of methods then disappear with the researcher once the experiment has been
and the focus on authentic and meaningful issues resonate with concluded.
the pragmatic philosophy and outlook associated with American Comparison to Action Research
pragmatism, associated with, notably, Charles Sanders Peirce,
John Dewey, and William James and later Abraham Kaplan and The focus on the evolution of design principles differentiates DBR
Richard Rorty. from much action research and formative evaluation designs in
that “the design is conceived not just to meet local needs, but to
Involving Multiple Iterations advance a theoretical agenda, to uncover, explore, and confirm
Design practice—whether in the manufacture of cars or of theoretical relationships” (Barab & Squire, 2004, p. 5). Cobb,
fashions—usually evolves through the creation and testing of Confrey, diSessa, Lehrer, and Schauble (2003), in a seminal article
prototypes, iterative refinement, and continuous evolution of the on DBR, identified five cross-cutting themes that mark and define
design, as it is tested in authentic practice. We often joke in our DBR. The fifth of these themes is that the research is both humble
work that the iterative adjustment and improvement of our inter- and accountable to the design, meaning that “the theory must
ventions could be characterized by “research through mistakes.” do real work. General philosophical orientations to educational
Design-based interventions are rarely if ever designed and imple- matters—such as constructivism—are important to educational
mented perfectly; thus there is always room for improvements in practice, but they often fail to provide detailed guidance in organiz-
the design and subsequent evaluation. This evolution through ing instruction” (p. 10). As an example, this differentiation between
multiple iterations is but one of the challenges of the methodol- grand theories and “theories that work” was picked up by
ogy in that it is difficult to know when (or if ever) the research Tiberghien, Vince, and Gaidioz (2009) as they derived a theoreti-
program is completed. cal model for DBR but restricted its dominion to very practical
constraints and the specific context of physics instruction.
Involving a Collaborative Partnership Between Researchers
Both practitioners and researchers often have trouble differen-
and Practitioners
tiating between action research and DBR—likely because they
In action research, the educator is both researcher and teacher share many epistemological, ontological, and methodological
(Kuhn & Quigley, 1997). The partnership in a design-based underpinnings. In a comparison of the two methods, Cole,
study recognizes that teachers are usually too busy and often ill Purao, Rossi, and Sein (2005) found that the defining qualities
trained to conduct rigorous research. Likewise, the researcher of both methodologies share a common “meta-paradigm”—
often is not knowledgeable of the complexities of the culture, pragmatism. They conclude their article by recommending the
technology, objectives, and politics of an operating educational integration of the two by adding a reflective step to DBR—
system to effectively create and measure the impact of an inter- although we see reflection as being an integral component of all
vention. Thus, a partnership is developed that negotiates the stages of DBR research. They also suggest the need for action
study from initial problem identification, through literature research to add a “build” phase in which the construction of
review, to intervention design and construction, implementa- theories, artifacts, models, and prototypes results in the instan-
tion, assessment, and to the creation and publication of theoreti- tiation, archiving, and distribution of the action research results.
cal and design principles. Finally, we repeat that action research is normally carried on by
the teacher alone, thus not benefitting from the expertise and
Evolution of Design Principles
energy of a research and design team that characterizes DBR.
Designs evolve from and lead to the development of practical The practical nature of DBR obviously places it (like action
design principles, patterns, and/or grounded theorizing. The research) in the camp of applied research. However, following
focus on theoretical understanding is implied in the fourth and Stokes (1997) and Stappers (2007), we reject the linear model
final phase of a DBR study, which Reeves (2000) referred to as that places basic and applied research at polar opposites. Rather,
“reflection to produce design principles and enhance solution good science often leads to very practical outcomes while con-
implementation” (p. 9). These principles are not designed to cre- tributing to theoretical and basic understandings. In many ways,
ate decontextualized principles or grand theories that function DBR is the educational instantiation of Stokes’s most productive
January/February 2012 17
fourth quadrant, which maximizes both generalization and 1997), but DBR also requires comradeship, enthusiasm, and a
insight with the production of practical applications. willingness to actively support the intervention. Thus, a certain
wisdom is needed to walk this narrow line between objectivity and
Practical Impact on Practice
bias. The personal skill to hold all of these attitudes simultane-
Reeves (2000) also has noted that the issue of implementation ously is a challenge and a defining feature of quality DBR.
and adoption is of critical importance and one in which educa- The challenge of bounding the temporal scope of a DBR proj-
tion research in general has had a very poor record. In our own ect is exacerbated by the requirement for multiple iterations.
keynote talks, we often challenge participants to think of one Herrington, McKenney, Reeves, and Oliver (2007) addressed this
research result that has made a difference in their educational concern head-on in an article demonstrating ways in which DBR
practice. It is both surprising and depressing that many educators can be used as a basis for a doctoral dissertation over a four-year
cannot think of a single research output or can think only of period. We also note that a partial solution is for established educa-
trivial outputs that meet this most practical and important out- tion researchers to develop multiyear DBR research agendas that
come of research. One could contrast this with the many results have legitimate space and roles for graduate students to undertake
of medical research and the way that treatments and practice have and “own” significant pieces of this larger agenda. Such large-scale
evolved and continue to evolve as a result of medical research. agendas are common in the natural and health sciences, but unfor-
This necessity for impact in real education settings was succinctly tunately education research has not, except in rare circumstances
captured by Barab and Squire (2004), who argued that “design- (notably the research led by Chris Dede on the River City project
based research that advances theory but does not demonstrate the at Harvard University), managed to achieve the financial support
value of the design in creating an impact on learning in the local or develop the necessary vision and leadership required by such
context of study has not adequately justified the value of the multiyear and multifaceted research agendas.
theory” (p. 6). From this review of characteristics and concerns, we conclude
that DBR has a number of significant characteristics that reso-
Problems With DBR
nate with the calls for educational reform and for improvements
Despite the numerous articles extolling the benefits of DBR, in education research. But how has that potential been realized in
there has been a number of useful critiques—notably Barab and the past decade? Has DBR been used, and, if yes, has it been most
Squire’s (2004) article. They argued that “if a researcher is inti- effective in particular disciplines or subjects? Does it seem most
mately involved in the conceptualization, design, development, compatible with the study of one or more of the levels of formal
implementation, and re-searching of a pedagogical approach, education? Is its use, or at least its recognition, centered in one
then ensuring that researchers can make credible and trustworthy country or region, or is it a global phenomenon? Are interest in
assertions is a challenge” (p. 10). This challenge is a familiar one and use of DBR growing, or did DBR peak with the special issues
to anthropological research and to many forms of qualitative
of the early part of the decade? Finally, are the types of innovation
research in that none of these methods can or do claim that the
that researchers have studied using DBR small scale, incremental,
researcher’s bias is removed from the research process. Indeed,
and sustaining, or is DBR also useful for creating and measuring
some qualitative proponents argue that the researchers them-
disruptive changes (Christensen, Horn & Johnson, 2008)?
selves (with their biases, insights, and deep understanding of the
To answer these questions, we turn to a systematic examina-
context) are the best research tool.
tion of the research literature.
A number of ways have been suggested to minimize this con-
cern in DBR as well as in other qualitative methods (Onwuegbuzie Method
& Leech, 2007), and we argue that this inside knowledge adds
as much as it detracts from the research validity. Good research To find the articles that used or focused on DBR, we used Harzing’s
demands “skepticism, commitment and detachment” (Norris, (2010) open-source tool, Publish or Perish, to perform a Google

Figure 1. Search screen from Publish or Perish.

18 educational Researcher
Scholar search of articles in the humanities, arts, and social sciences
that contained the keywords design-based research and education
(see Figure 1). Unfortunately, no one as yet has developed or imple-
mented a structured means of describing or coding research articles
such that the research method is very cleared labeled for semantic
search and retrieval. However, Google Scholar was able to identify
articles in which the term was mentioned. It is not feasible to
review all of the 1,940 articles identified by Google Scholar and so
we used a form of a “recommender system” in which the 5 articles
with the highest number of citations in each year from 2002 to
2011 were retrieved and examined. We choose to not count book Figure 2. Number of articles using or discussing design-based
chapters, conference proceedings, or other gray literature in this research.
analysis—even though some of these were cited by other scholars. Data from the 47 articles selected (5 articles each year from
Our selection criteria focused on choosing the 5 most cited articles 2003 to 2011 and 2 articles from 2002) were entered into a spread-
of each year that either explicitly used DBR or focused on the sheet; the graphs and analyses presented in the following sections
description, critique, or review of DBR methodology. Articles that are derived from this compilation. The spreadsheet is available for
merely cited a DBR research study were not included. The only further analysis (and reader comments; for further details, see sup-
year that we were unable to find 5 articles meeting our criteria was plemental document available on the journal website).
2002 (2 articles were selected).
Citation counts have long been used as the one of two primary Results
means of assessing the quality and/or impact of scientific research. Figure 2 shows the continuing interest and growth of research
The most common resource for citation analysis is Thomson articles about DBR. As previously mentioned in the Method sec-
Reuter (ISI) Web of Knowledge or, as it is now called, the Social tion, it was not feasible to attempt a review of such a large body
Science Citation Index (SSCI); however, any source of citation or work, so we selected 47 articles from 2002 to 2011 that met
data can be used. Although SSCI claims coverage of more than our criteria. The 47 articles included in our database were of two
11,400 journals, it covers only 2,929 journals in the social sciences distinct types. Sixteen (34%) were philosophical or expository
and humanities (Thompson SSCI, June 2010, http://science articles about DBR itself (of these 16 articles, 6 described one or
.thomsonreuters.com/cgi-bin/jrnlst/jlresults.cgi?PC=SS), and more DBR projects in detail to illustrate DBR in action). These
many important educational journals are not covered. In addition, articles often had a proselytizing nature, as could be expected
Anderson and McConkey (2009) claim that SSCI, as a commercial from proponents of DBR trumpeting its (claimed) potential to
venture of Thomson Reuters publishing, has a pecuniary bias significantly improve the quality and, most important, the
against coverage of open-access journals, although SSCI has impact of research in real educational contexts. The 31 remaining
increased the number of open-access journals cited in recent years. articles (66%) provided empirical evidence and results of DBR-
Scopius, owned by Elsevier, has slightly larger coverage of based research studies. Not surprisingly for an emerging research
social sciences and humanities than SSCI does, but it still covers framework, 88% (14 out of 16) of the philosophical or exposi-
much less scope than Google Scholar. Google Scholar, however, tory articles about DBR were written in 2002 through 2006,
is certainly not perfect—one of the reasons is the lack of transpar- with 74% (23 out of 31) of the empirical studies occurring in
ency in that Google Scholar does not publish where, how often, the second half of the decade, from 2007 to 2011. These data
and when it searches for scholarly articles. In an extensive discus- suggest that the design is moving from theoretical discussion to
sion of the advantages and disadvantages of the these three large practice.
index services Harzing (2010) noted four major advantages of The research tools employed in the empirical DBR studies
Google Scholar: (a) it is free, (b) it is easy to use, (c) it is quick, were primary mixed, with results of both qualitative and quanti-
and (d) it is comprehensive in its coverage. Harzing went on to tative measurements presented. For example,
show that none of the three systems are without error and that This paper reports multiple methods as a part of a larger
each has particular strengths and weaknesses. A number of stud- design-based research project. (Klopfer & Squire, 2008,
ies have shown no significant differences in citation rates among p. 209)
the subset of journals that are covered by all three systems. In this section, we will first present two brief qualitative analyses
Further, Harzing provided evidence of strong correlations of two GS [GroupScribbles] lessons and motivate how the peda-
between the citation ratings of Google Scholar, Thompson SSCI, gogical design of the GS activity is enacted with good teacher
and Scopius. facilitation. Next, we present some quantitative analysis of data
Finally, education (the focus of this review) has always placed which shows positive gains for GS classes compared with non-GS
premium value on access to both scholarly resources and educa- classes. (Looi, Chen, & Ng, 2010, p. 18)
tion resources. Harzing and van der Wal (2008) argued that The quantitative and qualitative methods used in this study com-
“the free availability of [Google Scholar] allows for a democrati- plement each other. While the quantitative method revealed
zation of citation analysis as it provides every academic access to broad patterns of design based discourse, the qualitative method
citation data regardless of their institution’s financial means” facilitated local clarification through observation, description and
(p. 71). interpretation of the features of interactions and the role of

January/February 2012 19
 the faculty, students, and tasks. (Koehler, Mishra, & Yahya, 2007, described as examples of DBR in the 16 expository/philosophical
p. 750) articles. Some articles discussed more than one teaching subject,
and we included in the data all teaching subjects that were identi-
There was no meta-analysis—which is not surprising given that the
fied in the research. Some studies worked with students from
studies generally lacked control group comparisons, in line with
more than one educational sector, and we included all stated sec-
DBR philosophy. There was also no “qualitative meta-analysis”
tors in our data. In addition, some studies clearly stated age group
(Timulak, 2009), which may be a promising area for further
or grade level, whereas others grouped participants into broad
research but also is problematic for DBR aggregation, given the
K–12 or postsecondary categories. This is indicated in Figure 5
mixed methodologies often employed.
by a broad group of K–12 for studies that did not specify age or
Geographic Focus grade, and elementary (Grades K–5, ages 5–10), middle (Grades
6–8, ages 11–13), and high (grades 9–12, ages 14–18). As most
Figure 3 shows the predominance of publications using DBR studies occurred in the United States, we used the U.S. school
that originated in the United States (we used the country affili- system for grades and ages. Figures 5 and 6 show that science is
ated with the first-stated author to indicate country of origin; the discipline most studied in DBR and K–12 the age group in
however, 6 studies were written by authors from more than one our top cited articles.
country). This is perhaps understandable, given that the major The empirical studies provided rich descriptions of the
original development work on DBR and the host country of the contexts, with authors both pointing out the limitations of their
journals with special issues on DBR was the United States. studies and offering suggestions on how their studies may be
Anderson (2005) argues that the practical nature of this method- applicable in other contexts. The following are some examples:
ology resonates with the pragmatic tradition of American educa-
tional philosophy originating with Dewey and James. The results are limited to such social institutions (e.g., universities),
non-profit organizations or companies. (Jahnke, 2010, p. 544)

Discipline and Curricular Focus Additional studies are needed in a variety of school contexts to
explore design strategies for enhancing effective opportunistic col-
All of the studies reviewed were conducted in educational con- laboration to determine how, and with what success, different
texts and primarily published in education research or discipline- teachers might engage students in more flexible and opportunistic
related journals (see Figure 4). arrangements. (Zhang, Scardamalia, Reeve, & Messina, 2009,
To more clearly differentiate the context of this work, we clas- p. 39)
sified the studies in which DBR was used by the teaching subject After the last cycle of fine-tuning, TM [technology mapping] is
(science, math, history, etc.) and by the stated educational sector. presented herein as a methodology for guiding teacher thinking
In addition to the data from the 31 articles that focused on DBR about the ill-defined problem of designing technology-enhanced
research studies, we included data from any empirical studies learning. TM can be used in teacher education departments to

Figure 3. Pie chart of first-stated authors’ countries of work (N = 47).

20 educational Researcher
 Figure 5. Educational sector of empirical design-based research
studies (n = 55).

(62), knowledge (44), study (44), science (36), development (34),

technology (28), and curriculum (26; see Figure 7). A search of the
term design-based in the abstracts resulted in 11 articles, design
research 8 articles, design experiment(s) 5 articles, and design
study(ies) 3 articles. As can be seen, the topics confirm the educa-
tional focus of the studies, with inclusion of many relevant topics
within our discipline.
Figure 4. Scholarly publication outlets for design-based research DBR studies always involve some type of intervention. We
studies (N = 47). clustered and tallied the 31 empirical studies based upon the type
of intervention, and we display the results in Figures 8 and 9.
Using a qualitative analysis package (Atlas.ti), we open-coded the
teach pre-service teachers how to teach with technology, and thus interventions described in the abstracts into two main groups:
to develop their ICT-TPCK [information and communication instructional method/model/strategy and technological, with 10
technologies–technological pedagogical content knowledge] interventions falling into the first coding category and 21 in the
knowledge; in teacher professional development programs to pre- second. We then grouped the two major codes into subcodes (see
pare in-service teachers in the pedagogical uses of technology; by
Figures 8 and 9 for summary data).
curriculum developers to decide how technology can be infused
in the different content domains; and, most importantly, by the Iterations
teachers themselves to design lessons with technology. (Angeli &
One of the challenges of DBR studies is that the iterative nature
Valanides, 2009, p. 160)
can exceed the resources or the time available to researchers or
The findings from this study contribute to the literature by pro- funding bodies. In Figure 10, we show how many studies indi-
viding support that spatial thinking can be learned, can be cated that they were describing the exploratory, initial stage of
taught formally to all students in an urban middle school, and DBR (6 studies), that they were part of a multi-iteration project
can be supported by appropriately designed curriculum and
(27 studies), or that they were reporting on the final stage
geospatial information technologies. This research also informs
curriculum developers with design strategies that can be used to
(1 study). A variety of terms and time measurements are used
promote spatial thinking with diverse urban classroom learners in DBR studies to discuss iterations (e.g., year, site, phase, itera-
to promote environmental sciences learning with geospatial tion, cycle, phase, case study), so it is challenging to depict a com-
information technologies that include virtual globe (Google parison of multi-iteration projects. We categorized the multi-
Earth) use and aerial and [remotely sensed] imagery. (Bodzin, iteration projects using the number given for the iteration term
2011, pp. 295-296) used. Figure 10 shows that over half of the DBR projects discussed
Content and Intervention in the empirical studies focused on projects that had progressed
through three or more iterations. The interested reader might also
A word analysis of the 47 abstracts gave the top-10 content words enjoy the blog post by Mike Rook (2010) in which he chronicles
as research (84 occurrences), design (73), students (67), learning four iterations of the River City DBR program.

January/February 2012 21
 Figure 6. Subject or program in which design-based research studies were undertaken (n = 45).

Major Results
Deriving generalizations about the results from the diverse types
and contexts of DBR study is a daunting task. We used a qualita-
tive analysis package to code the 31 abstracts of the empirical
studies, with a focus on the results of the intervention. We open-
coded the results described in the abstracts into four main groups:
potential for improved student learning (7 occurrences); major
new understandings about educational outcomes or context (12
occurrences); increased student learning took place (12 occur-
rences); and improvements in attitude/epistemology/motivation
(5 occurrences). Some abstracts’ results fell into more than one of
these categories. (See Figure 11 for summary data.)
Figure 7. Frequency of keywords in abstracts of empirical
Unlike quantitative studies, most DBR studies do not produce
design-based research studies (n = 31).
measureable effect sizes that demonstrate “what works.” However,
they provide rich descriptions of the contexts in which the studies
occurred, the challenges of implementation, the development pro-
cesses involved in creating and administrating the interventions,
and the design principles that emerged. An example of such rich
descriptions, taken from our database of 47 articles, is the work on
the multiuser virtual environment River City by Ketelhut (2007)
and Ketelhut, Nelson, Clarke, and Dede (2010) . Both articles
presented research on a scientific inquiry–based project, funded by
the National Science Foundation (NSF), focused on infusing sci-
entific inquiry into a standards-based curriculum. Ketelhut’s
(2007) exploratory study investigated self-efficacy in 100 seventh-
grade students who participated in the River City project, and she
highlighted that “this research on self-efficacy was embedded in a
larger, ongoing, NSF-funded project that has implemented River
City nationwide with nearly 8,000 students since 2000” (p. 101).
Ketelhut et al. (2010) discussed results from approximately 2,000
middle school students who were involved in three implementa-
tions of the project in 2004–2005. Both articles referred to other
research focused on River City and painted a picture of a multifac-
eted, ongoing, and complex DBR project.
Discussion
The most cited articles focusing on DBR in the past decade
Figure 8. Instructional method/model/strategy (n = 10). highlight the following trends in published DBR studies. The

22 educational Researcher
Figure 9. Technological or instructional design intervention Figure 11. Summary of results described in abstracts (n = 36).
(n = 21).

knowing when, if ever, the research project is completed, so it is not

surprising to only find one study that focused on the final write-up
of the research. However, even that study left the door open for
more research and refinement of the outcome of the multiyear
project: the ICT-TPCK framework.
We do hope that researchers, instructional designers, and practi-
tioners will find the conceptual framework discussed herein use-
ful and applicable in their respective contexts. Any future research
efforts that will be undertaken to validate, modify, or improve the
framework proposed for the conceptualization, development,
and assessment of ICT-TPCK will be important for both research
and practice. (Angeli & Valanides, 2009, p. 167)

Collins, Joseph, and Bielaczyc (2004) highlighted that DBR

projects often produce a large amount of data as they progress
through various iterations, and they called for the sharing of data:
Design experiments often lead to the collection of large amounts
of data that go unanalyzed. Hence, it makes sense for the design-
research community to establish an infrastructure that would
allow researchers at other institutions to analyze the data collected
Figure 10. Iteration and phase sequences (n = 34). in design studies, in order to address their own questions about
learning and teaching. This would require the community to
honor such reanalysis of data with the same status as original
research and it would require research journals and tenure com-
majority of the 47 articles had a primary author from the United
mittees to take such work seriously. (p. 40)
States; however, in the last four years reviewed, 2008 to 2011, 50%
of first-cited authors were not from the United States, suggesting Although a number of the 47 articles presented on different
that DBR is becoming accepted more globally as a popular research stages of the same intervention (e.g., River City: Ketelhut, 2007;
methodology. Half of all the empirical studies explored designing Ketelhut et al., 2010; Quest Atlantis: Barab, Sadler, Heiselt,
learning interventions for science, and 87% of studies focused on Hickey, & Zuiker, 2007; Barab, Thomas, Dodge, Carteaux, &
K–12 student populations. The majority (68%) of interventions Tuzun, 2005; Barab, Zuiker, et al., 2007), there was no evidence in
involved the use of online and mobile technologies, and all studies our small sample of 47 articles of the type of data pools suggested
indicated that they were part of a multi-iteration research project, by Collins et al. (2004) being shared among diverse research teams.
with 18% being in the exploratory cycle, 3% reporting the post- One of the characteristics of a quality DBR study is “a com-
cycle, and the rest falling into the space between initial piloting and mitment to theory construction and explanation while solving
final report. As we stated earlier, one of the challenges of DBR is real-world problems” (Reeves, Herrington, & Oliver, 2005,

January/February 2012 23
p. 103). Our qualitative analysis of the empirical studies’ abstracts Conclusion
indicated that the studies’ results focused on both what research-
From this study we learned that DBR is being utilized increas-
ers learned to further their knowledge about a phenomenon and
ingly in educational contexts and especially those in the United
what effect the intervention had on student learning and motiva-
States. It seems to be especially attractive for use in K–12 contexts
tion. This combined focus on theory building and practical,
and with technological interventions. The increasing number of
effective applications is a theme that is apparent in nearly all the
studies reported suggests that researchers and graduate students
articles we analyzed. For example, Jahnke (2010) stated that
are finding ways to meet the time demands of multiple iteration
“from a DBR perspective the results of this study produced both
studies.
practical educational interventions and theory generation includ-
Most of the articles conclude that their interventions have
ing measurable theses that can be investigated in follow-up
resulted in improved outcomes or student attitudes, and they
research” (p. 545), and Barab, Zuiker, et al. (2007) used the met-
offer rich clues as to the match between the successful testing of
aphor of having one’s cake in order to depict the affordance of
the intervention and the context of practice. It is unclear if the
DBR to build theory in a practical context:
results achieved are meeting the challenge of promoting wide-
Through our design framework, we were able to evolve theory spread adoption of the tested interventions. However, it seems
grounded not only in practice, but in a visionary frame—one that the expanding use of the methodology and these promising
allowed us to test not only what exists in schools but what could results provide evidence for cautious optimism. The claims of
exist. . . . In this way, as situativity theorists, we are working theo- early proponents are being realized in classrooms and online
retically to “have our cake,” and in the context of a standards
learning contexts.
emphasis in schools, be able to pedagogically “serve it to others” as
well. (p. 780)
The study allows us to concur with Dede, Ketelhut,
Whitehouse, Breit, and McCloskey’s (2009) claim that “DBR
Finally, it is interesting to question the role and efficacy of offers a ‘best practice’ stance that has proved useful in complex
DBR in bringing about large-scale and far-reaching systems learning environments, where formative evaluation plays a sig-
reforms or to ask if the methodology is more suitable in small- nificant role, and this methodology incorporates both evaluation
scale systems or for sustaining improvements in educational sys- and empirical analyses and provides multiple entry points for
tems when the research project is finished. There seems little various scholarly endeavors” (p. 16). However, as promising as
evidence in our sample of DBR studies that they are of the type the methodology is, much more effort in this and other areas of
of disruptive and wholesale change envisioned in extensive education research is needed to propel the type of education
reforms on the scale of John Dewey’s Laboratory School or A. S. innovation that many of us feel is required.
Neil’s Summerhill School (http://www.summerhillschool.
co.uk/). The interventions developed in these studies could be References
characterized as small improvements to the design, introduction,
and testing of sustaining technologies and practices in classroom Anderson, T. (2005). Design-based research and its application to a call
center innovation in distance education. Canadian Journal of Learn-
or distance education contexts. However, even small changes can
ing and Technology, 31(2), 69–84. Retrieved from http://auspace
have long-term effects. One of most active publishers in this first
.athabascau.ca:8080/dspace/bitstream/2149/741/1/design_based_
decade of DBR was Diane Ketelhut, who was a major researcher research.pdf
associated with the River City project. She remarked, Angeli, C., & Valanides, N. (2009). Epistemological and methodologi-
I don’t think you can develop a 2- to 3-week intervention in the cal issues for the conceptualization, development, and assessment of
classroom like River City and, by doing so, say you were having ICT–TPCK: Advances in technological pedagogical content knowl-
a direct impact on how teachers generally use emerging technolo- edge (TPCK). Computers and Education, 52(1), 154-168.
gies or integrate innovative educational practices in the class- Anderson, T., & McConkey, B. (2009). Development of disruptive open
room. However, I can give many examples of how individual access journals. Canadian Journal of Higher Education, 39(3), 71–87.
teachers changed their attitude and understanding of technology Retrieved from http://ojs.library.ubc.ca/index.php/cjhe/article/
long term . . . and changed their whole focus on teaching. view/477
(Personal communication, September 2011) Barab, S. A., Sadler, T. D., Heiselt, C., Hickey, D., & Zuiker, S. (2007).
Relating narrative, inquiry, and inscriptions: Supporting consequen-
This study was designed to provide an overview of the DBR tial play. Journal of Science Education and Technology, 16(1), 59-82.
landscape, and thus we performed a number of analyses using Barab, S., & Squire, B. (2004). Design-based research: Putting a stake
only the abstracts, as opposed to the full texts, of the 47 articles. in the ground. Journal of the Learning Sciences, 13(1), 1–14. Retrieved
Given the interesting preliminary results, a more detailed exam- from http://website.education.wisc.edu/kdsquire/manuscripts/jls
ination of the full texts of these 47 and the much wider set of -barab-squire-design.pdf
published DBR articles is both warranted and recommended. Barab, S., Thomas, M., Dodge, T., Carteaux, R., & Tuzun, H. (2005).
Thus, DBR seems have been used to make a difference—but Making learning fun: Quest Atlantis, a game without guns.
Educational Technology Research and Development, 53(1), 86-107.
mostly at the level of small-scale interventions and in the lives of
Barab, S., Zuiker, S., Warren, S., Hickey, D., Ingram-Goble, A., Kwon,
individual teachers and schools. It is interesting to speculate if the
E.-J., Kouper, I., Herring, S. C. (2007). Situationally embodied cur-
methodology could and will be used by researchers to investigate riculum: Relating formalisms and contexts. Science Education, 91(5),
today’s disruptive innovations such as massive open online courses, 750-782.
tuition-free universities (e.g., People’s University), open educa- Bodzin, A. M. (2011). The implementation of a geospatial information
tional resources, and other networked learning innovations. technology (GIT)–supported land use change curriculum with urban

24 educational Researcher
 middle school learners to promote spatial thinking. Journal of Research Handbook of mixed methods in social and behavioral research
in Science Teaching, 48(3), 281-300. (pp. 51–89). Thousand Oaks, CA: Sage
Boydston, J. (1971). John Dewey: The early works. Carbondale: Southern Mingfong, J., Yam San, C., & Ek Ming, T. (2010). Unpacking the
Illinois University Press. design process in design-based research. In Proceedings of the 9th
Brown, A. (1992). Design experiments: Theoretical and methodological International Conference of the Learning Sciences (Vol. 2). International
challenges in creating complex interventions in classroom settings. Society of the Learning Sciences.
Journal of the Learning Sciences, 2(2), 141–178. Norris, N. (1997). Error, bias and validity in qualitative research.
Christensen, C., Horn, M., & Johnson, C. (2008). Disrupting class: How Educational Action Research, 5(1), 172–176. Retrieved from http://
disruptive innovation will change the way the world learns. New York: portal.acm.org/ft_gateway.cfm?id=345004&type=pdf&CFID=2464
McGraw-Hill. 3476&CFTOKEN=60339809
Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Oha, E., & Reeves, T. (2010). The implications of the differences
Design experiments in educational research. Educational Researcher, between design research and instructional systems design for educa-
32(1), 9–13. Retrieved from http://www.aera.net/uploadedFiles/ tional technology researchers and practitioners. Educational Media
Journals_and_Publications/Journals/Educational_Researcher/3201/ International, 4(47), 263–275.
3201_Cobb.pdf Onwuegbuzie, A., & Leech, N. (2007). Validity and qualitative research:
Cole, R., Purao, S., Rossi, M., & Sein, M. K. (2005, December). Being An oxymoron? Quality and Quantity, 41(2), 233–249. Retrieved from
proactive: Where action research meets design research. Paper presented http://dx.doi.org/10.1007/s11135-006-9000-3
at the Proceedings of the Twenty Sixth International Conference on Reeves, T. (2000, April). Enhancing the worth of instructional technology
Information Systems. Retrieved from http://citeseerx.ist.psu.edu/ research through “design experiments” and other developmental strategies.
viewdoc/download?doi=10.1.1.83.956&rep=rep1&type=pdf Paper presented at the American Educational Research Association
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Annual Meeting. Retrieved from http://itech1.coe.uga.edu/~treeves/
Theoretical and methodological issues. Journal of the Learning AERA2000Reeves.pdf
Sciences, 13(1), 15-42. Reeves, T. C., Herrington, J., & Oliver, R. (2005). Design research: A
Conceicao, S., Sherry, L., & Gibson, D. (2004). Using developmental socially responsible approach to instructional technology research in
research to design, develop and evaluate an urban education portal. higher education. Journal of Computing in Higher Education, 16(2),
Journal of Interactive Learning Research, 15(3), 271–286. 96-115.
Harzing, A. W. (2010). Publish or perish. Melbourne, Australia: Tarma Rook, M. (2010, November 10.) Following a Design-Based Research
Software Research. Cycle. Retrieved from http://www.personal.psu.edu/mdm392/blogs/
Harzing, A. W., & van der Wal, R. (2008). Google Scholar: The democ- design/2010/11/following-a-design-based-research-cycle.html
ratization of citation analysis. Ethics in Science and Environmental Stappers, P. J. (2007). Doing design as a part of doing research. In R.
Politics, 8(1), 61–73. Michel (Ed.), Design research now (pp. 81-91). Basel, UK: Birkhauser.
Herrington, J., McKenney, S., Reeves, T. C., & Oliver, R. (2007). Retrieved from http://dx.doi.org/10.1007/978-3-7643-8472-2_6
Design-based research and doctoral students: Guidelines for preparing Stokes, D. (1997). Pasteur’s quadrant: Basic science and technological inno-
a dissertation proposal. Retrieved from www.editlib.org/d/25967/ vation. Washington, DC: Brookings Institution Press.
proceeding_25967.pdf Tiberghien, A., Vince, J., & Gaidioz, P. (2009). Design-based research:
Jahnke, I. (2010). Dynamics of social roles in a knowledge management Case of a teaching sequence on mechanics. International Journal of
community. Computers in Human Behavior, 26(4), 533-546. Science Education, 31(17), 2275–2314. Retrieved from http://www
Ketelhut, D. (2007). The impact of student self-efficacy on scientific .informaworld.com/10.1080/09500690902874894
inquiry skills: An exploratory investigation in River City, a multi-user Timulak, L. (2009). Meta-analysis of qualitative studies: A tool for
virtual environment. Journal of Science Education and Technology, reviewing qualitative research findings in psychotherapy. Psychotherapy
16(1):99–111. doi: 10.1007/s10956-006-9038-y Research, 19(4/5), 591–600.
Ketelhut, D. J., Nelson, B. C., Clarke, J., & Dede, C. (2010). A multi- Zhang, J., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs
user virtual environment for building higher order inquiry skills in for collective cognitive responsibility in knowledge-building commu-
science. British Journal of Educational Technology, 41(1), 56-68. nities. Journal of the Learning Sciences, 18(1), 7-44.
Klopfer, E., & Squire, K. (2008). Environmental detectives—The devel-
opment of an augmented reality platform for environmental simula- AUTHORS
tions. Educational Technology Research and Development, 56(2), TERRY ANDERSON is a professor at Athabasca University, Centre for
203-228. Distance Education, 10011 109th Street, Edmonton, Alberta T5J 3S8,
Koehler, M. J., Mishra, P., & Yahya, K. (2007). Tracing the development Canada; terrya@athabascau.ca. His research focuses on social network-
of teacher knowledge in a design seminar: Integrating content, peda- ing in distance education courses.
gogy and technology. Computers and Education, 49(3), 740-762.
Kuhn, G., & Quigley, A. (1997). Understanding and using action JULIE SHATTUCK is an instructional designer and assistant professor at
research in practice settings. In A. Quigley & G. Kuhne (Eds.), Frederick Community College, Centre for Distributed Learning, 7932
Creating practical knowledge through action research (pp. 23–40). San Opossumtown Pike, Frederick, MD 21702; jshattuck@frederick.edu. She
Francisco: Jossey-Bass. is a doctoral student at Athabasca University, and her research focuses on
Looi, C.-K., Chen, W., & Ng, F. K. (2010). Collaborative activities adjunct faculty training for online teaching.
enabled by GroupScribbles (GS): An exploratory study of learning
effectiveness. Computers and Education, 54(1), 14-26.
Maxcy, S. J. (2003). Pragmatic threads in mixed methods research in the Manuscript received July 7, 2011
social sciences: The search for multiple modes of inquiry and the end Revision received September 20, 2011
of the philosophy of formalism. In A.Tashakkori & C. Teddlie (Eds.), Accepted September 26, 2011

January/February 2012 25

View publication stats