Design-Based Implementation Research:

An Emerging Model for Transforming the

Relationship of Research and Practice
BARRY J. FISHMAN
University of Michigan

WILLIAM R. PENUEL
University of Colorado Boulder

ANNA-RUTH ALLEN
University of Colorado Boulder

BRITTE HAUGAN CHENG

SRI International

NORA SABELLI
SRI International

This chapter presents an introduction to design-based implementation research

(DBIR). We describe the need for DBIR as a research approach that challenges
educational researchers and practitioners to transcend traditional research/prac-
tice barriers to facilitate the design of educational interventions that are effective,
sustainable, and scalable. We examine antecedents to DBIR, including evalua-
tion research, community-based participatory research, design-based research, and
implementation research. The four core principles of DBIR are explained: (1) a
focus on persistent problems of practice from multiple stakeholders’ perspectives;
(2) a commitment to iterative, collaborative design; (3) a concern with developing

National Society for the Study of Education, Volume 112, Issue 2, pp. 136-156
Copyright © by Teachers College, Columbia University
 Design-Based Implementation Research 137

theory and knowledge related to both classroom learning and implementation

through systematic inquiry; and (4) a concern with developing capacity for sus-
taining change in systems. We close with an overview of the chapters contained
in this NSSE Yearbook on DBIR and explain how each chapter contributes to the
overall development of the DBIR approach.

This National Society for the Study of Education Yearbook presents an over-
view of an emerging model at the intersection of policy, research, and
practice called design-based implementation research (DBIR). DBIR applies
design-based perspectives and methods to address and study problems
of implementation. As the chapters in this volume illustrate, DBIR chal-
lenges education researchers to break down barriers between sub-disci-
plines of educational research that isolate those who design and study
innovations within classrooms from those who study the diffusion of in-
novations. It also aims to reconfigure the roles of researchers and prac-
titioners in bringing about systemic change in ways that make it more
likely that practitioners can adapt innovations productively to meet the
needs of diverse students and that durable research–practice partner-
ships can sustain innovations that make a difference.

THE BROADER CHALLENGE

We are motivated in this endeavor by a broader concern with excellence

and equity: We want to improve all learners’ access to powerful, effec-
tive learning opportunities. Furthermore, we believe that these improve-
ments must be scalable and sustainable, which requires changing the
underlying assumptions about how such change is created and studied.
We share these goals with a wide range of policy makers, social entre-
preneurs, researchers, and educators who seek to improve educational
systems and, in some cases, remake them entirely. We share with these
educational stakeholders a focus on innovation, that is, the need to de-
velop new designs that afford expanded pathways for young people to
learn, participate in civic life, and earn a productive, sustainable living.
Gathering and interpreting evidence of effectiveness are important
practices to integrate into the process of innovation. As with any design,
we want to know whether it can work when it is given the support and re-
sources necessary to have a positive effect on learning. A number of meth-
odologies for determining “what works” have become quite sophisticated
in recent years, thanks to federal investments in building the capacity of
the field to conduct randomized controlled trials and analyze their re-
sults. Less well developed, however, are methods for answering the ques-
tion of “what works where, when, and for whom” (Means & Penuel, 2005)
138 National Society for the Study of Education

to understand the conditions under which designs can improve learning.

Current conceptualizations of effectiveness and scaling up (e.g., McDon-
ald, Keesler, Kauffmann, & Schneider, 2006) give even less attention to
research that compares the efficacy of different approaches to creating
the conditions necessary for designs to be well implemented and work
effectively. Thus, we need to pose new kinds of questions related to prac-
tice and establish new methods within educational research to address
them—including some that borrow from fields outside education, such
as public health and medicine (Berwick, 2008).
It takes more than just sound evidence to bring about and sustain ex-
cellence and equity in systems. Studies of evaluation utilization going
back decades (Johnson et al., 2009), as well as more recent research on
evidence use in education (e.g., Coburn, Honig, & Stein, 2009), make
clear that just because research may be rigorous does not mean it will
be judged as relevant by practitioners and policy makers. Practitioners
and policymakers consider a wide range of concerns when making deci-
sions, and they often consider researchers’ questions irrelevant to their
own problems of practice (Coburn & Talbert, 2006). Similarly, the chal-
lenge of scaling up an efficacious intervention presents significant imple-
mentation challenges. Many programs that work on a small scale when
well-supported by researchers fail when they are tested in effectiveness
studies, in part because educators face many challenges in implementing
them well. The few innovations—for example, Cognitive Tutor (Aleven
& Koedinger, 2002; see also Means & Harris, 2013, this Yearbook)—that
have gone to scale show just how hard it is to replicate positive outcomes
underscore the importance of effective implementation and the need for
systemic perspectives on improvement.
The distinctiveness of DBIR as an approach lies in the way that the au-
thors in this volume conceptualize a new relationship between research
and practice that is mutually transformative. In their historical analysis
of a century of educational reform (now more than 15 years old, but
still accurate and relevant), David Tyack and Larry Cuban (1995) noted
the profound influence of Taylor’s industrial model of scientific man-
agement on instructional practice. This model created a sharp division
of labor between those who design innovations and those charged with
implementing them. DBIR seeks to establish partnerships and engage
in specific research projects that are based on a radically different con-
ceptualization of the relationships between research and practice, and
innovation developer and implementer. The authors in this Yearbook do
not view research and development as a linear process that leads from
design by researchers to scale up by practitioners. Instead, the relation of
research to practice is more two-way and recursive, as conceptualized by
Coburn and Stein (2010).
 Design-Based Implementation Research 139

ANTECEDENTS OF DBIR

We do not claim that DBIR is an entirely new form of research or a model

that has no parallels in educational research and evaluation. In fact, most
of the key elements of the approach can be found in existing lines of
research and in calls by researchers for new infrastructures for relating
research and practice. At the same time, our claim is that DBIR emerges
from the insights of past research and from both successful and unsuc-
cessful attempts to bring educational innovations to scale. We outline
some of these research traditions next.

EVALUATION RESEARCH

Three different models of evaluation research align closely with the aims
of DBIR. One is utilization-focused evaluation, which directs evaluation
researchers to consider intended uses of research by intended users in
the design and conduct of evaluation (Patton, 1997, 2000). Like DBIR,
utilization-focused evaluation emphasizes uses of research findings for
programs, and it highlights the ways that stakeholder involvement in the
process of evaluation can support the design and development of both
programs and organizations (Patton, 2000). A second strand of evalua-
tion research includes participatory models of evaluation (e.g., Cousins
& Earl, 1992; Fetterman, 2001), in which stakeholders are involved in
all aspects of evaluation research, from formulating questions to inter-
preting results. This model presumes that the knowledge people use to
guide decision making within organizations is socially constructed and
that participation in all aspects of the process of evaluation helps prac-
titioners develop shared understandings of program goals, effects, and
conditions for success (Cousins & Earl, 1992). Finally, theory-driven eval-
uation (Donaldson, 2007) emphasizes the importance of using a combi-
nation of social science and stakeholder theories in designing evaluation
studies and of using evaluation to support program planning. A prem-
ise behind this model is that to be effective, interventions need to be
grounded in theory and evidence from relevant social science disciplines
(e.g., learning sciences, psychology, public health). Supovitz (2013, this
Yearbook) explores the relation between DBIR and different models of
evaluation, pointing out both intersections and conflicts between DBIR
and traditional evaluation of educational innovations.

COMMUNITY-BASED PARTICIPATORY RESEARCH

Another antecedent of DBIR is community-based participatory research,

in which researchers partner with people outside the academy to conduct
joint research to advance local social change goals (Stewart & Shamdasani,
140 National Society for the Study of Education

2006; Strand, Marullo, Cutforth, Stoecker, & Donohue, 2003; Weinberg,

2003). In contrast to research conducted in partnership with a single
institution, community-based participatory research is often conducted
in partnership with participants in social movements or with institutional
stakeholders who are marginalized or have limited voice in institutions
(for examples in education, see Oakes & Rogers, 2006, and O’Connor,
Hanny, & Lewis, 2011). Community-based research is a hybrid practice:
It is neither traditional basic research nor traditional evaluation, but a
blend of research and action. As such, it is related to, and often thought
of as a form of, participatory action research (Whyte, 1991). In other
fields, notably social services and public health, community-based partic-
ipatory research models have played key roles in supporting the imple-
mentation of evidence-based practices (see Palinkas & Soydan, 2012, for
a review). In those fields, this model of research has been used to support
the co-design of interventions (Wallerstein & Duran, 2010), as well as the
scaling of interventions that have some evidence of efficacy—two prac-
tices we imagine that DBIR in education might support.

DESIGN-BASED RESEARCH

Within the learning sciences, design-based research offers a model for the
design and testing of innovations within the crucible of classrooms and
other contexts for learning (Cobb, Confrey, diSessa, Lehrer, & Schau-
ble, 2003; O’Neill, 2012). As a model that emphasizes iterative cycles of
design and testing, design-based research is particularly well-suited to
making evidence-based improvements to innovations, in which evidence
from both implementation and outcomes informs changes that design
teams make to innovations for learning (see, e.g., Fishman, Marx, Best,
& Tal, 2003). The potential utility of design research to support imple-
mentation also derives from its commitment to developing both theory
that guides design decisions and practical tools that can be used to sup-
port local innovation and solve practical problems (Cobb et al., 2003). As
in community-based participatory research, the collaborative nature of
much design research positions practitioners as codesigners of solutions
to problems, which can facilitate the development of usable tools that
educators are willing to adopt (Penuel, Roschelle, & Shechtman, 2007).

IMPLEMENTATION RESEARCH

Implementation research is the systematic study of the implementation of

innovations (Fixsen, Naoom, Blase, & Friedman, 2005; Werner, 2004). It
encompasses descriptive studies of fidelity (e.g., Davidson, Fields, & Yang,
2009) and variation in implementation, as well as analyses of the condi-
tions under which programs can be implemented effectively (e.g., Penuel
 Design-Based Implementation Research 141

& Means, 2004; Penuel & Yarnall, 2005). Implementation research is of-
ten conducted within larger outcome studies, with the aim of analyzing
how and how much variations in implementation matter for innovation
effectiveness (e.g., Furtak et al., 2008; Lee, Penfield, & Maerten-Rivera,
2009; O’Donnell & Lynch, 2008). Notably, the study of implementation
among policy researchers in education and sociologists of education has
given an important context for theory development in these fields, from
new institutionalism (Meyer & Rowan, 2006) to the diffusion of innova-
tions (e.g., Frank, Zhao, & Borman, 2004). Implementation research stud-
ies conducted as part of policy and evaluation studies have yielded impor-
tant practical insights over the years, notably about the inevitability of local
adaptation and the need to support local actors’ sense-making in shaping
implementation of innovations (e.g., Berman & McLaughlin, 1975; Dat-
now, Hubbard, & Mehan, 1998; Means, Padilla, DeBarger, & Bakia, 2009).

SOCIAL DESIGN EXPERIMENTS

The aim of a social design experiment is to develop new tools and prac-
tices that produce new learning arrangements, especially for students
from nondominant communities (Gutiérrez & Vossoughi, 2010). As in
other forms of design research, researchers work in close partnerships with
practitioners to develop these arrangements. In addition, consistent with
DBIR, a focus is on transforming learning arrangements across different
settings and levels. A key aim is to develop so-called third spaces, in which
hybrid cultural practices enable students to bridge everyday and academic
literacies (Gutiérrez, Baquedano-Lopez, & Tejada, 2000). Doing so often
requires engaging community members in partnerships to ensure that the
voices, tools, and practices of nondominant communities become integral
to new learning arrangements. These partnerships, moreover, may incor-
porate cultural practices and beliefs of members of the community as a
means to ensure that the design process becomes a third space (Bang, Me-
din, Washinawatok, & Chapman, 2010). A number of scholars are develop-
ing approaches to designing learning environments that engage commu-
nity members and that attempt to help students relate everyday cultural
practices to disciplinary ways of thinking and reasoning (Bang & Medin,
2008; DeBarger, Choppin, Beauvineau, & Moorthy, 2013, this Yearbook;
Kirshner & Polman, 2013, this Yearbook; Tzou & Bell, 2010).

NEW INFRASTRUCTURES FOR RESEARCH-PRACTICE

COLLABORATION

The National Research Council report Strategic Education Research Part-

nership laid out a vision for a new infrastructure to support more durable
partnerships between researchers and practitioners aimed at improving
142 National Society for the Study of Education

teaching and learning in America’s schools (Donovan, Wigdor, & Snow,

2003). It called for the funding of an intermediary organization (Strate-
gic Education Research Partnership [SERP]) that could broker connec-
tions between researchers and practitioners to identify and solve persis-
tent problems of practice in education. The vision was an ambitious one:
SERP was imagined as:

. . . a place where interested school districts and researchers can

link up around the SERP agenda; a place where new members of
the field of teaching and its related research disciplines can join
a program of research and development productively channeled
to improving teaching and learning (we will propose fellow-
ship and internship programs); and a place where many kinds
of funders of education research and development can become
part of an ongoing collaborative effort to improve student out-
comes. (Donovan et al., 2003, p. 11)
We share this vision and imagine DBIR as necessarily creating a kind
of “third space” (Gutiérrez, Rymes, & Larson, 1995), that is, a hybrid
culture and place that researchers and practitioners create together and
that is organized to be self-sustaining over time. Such a space depends
on adherence to some core principles for organizing research and de-
velopment efforts that build from the SERP Institute’s experiences in
attempting to build an infrastructure for collaboration in multiple school
districts. Though our vision expands beyond work with school districts to
encompass work with other kinds of educational institutions, community-
based collaboratives, and even activist groups, adherence to these prin-
ciples defines the boundaries of DBIR as an approach to research.

PRINCIPLES OF DBIR

The principles that we outline next are heuristics for guiding the orga-
nization of DBIR (Penuel, Fishman, Cheng, & Sabelli, 2011). We gave
the authors of the chapters in this Yearbook guidance to use, adapt, and
extend them to illustrate what DBIR is now and might be in the future.
To provide readers with a sense of where we began, however, it is useful
to describe the principles we think make a particular research project,
program of research and development, or infrastructure for collabora-
tion an example of DBIR. The four key principles are:

• A focus on persistent problems of practice from multiple stake-

holders’ perspectives;
• A commitment to iterative, collaborative design;
 Design-Based Implementation Research 143

• A concern with developing theory and knowledge related to both

classroom learning and implementation through systematic inquiry;
• A concern with developing capacity for sustaining change in sys-
tems.

In DBIR, Teams Form Around a Focus on Persistent Problems of Practice From

Multiple Stakeholders’ Perspectives

Design-based implementation research shares a commitment with other

forms of design research (e.g., formative experiments, design experi-
mentation) to using research to solve practical problems. In DBIR, mul-
tiple stakeholders help identify practical problems that are candidates
to become the focus of research and development. Depending on the
endeavor, participants may include researchers, organizational leaders,
practitioners, parents and community members, and young people. A
diversity of ideas on what problems should be taken up is likely not only
across these different groups but also within them.

To Improve Practice, Teams Commit to Iterative, Collaborative Design

Collaborative design research often focuses on the development and test-

ing of usable tools for improving teaching and learning in specific subject
matter domains and settings (Cobb et al., 2003). As with other forms of
design-based research, the design effort begins with a vision for learning
that sets the focus for design (Collins, Joseph, & Bielaczyc, 2004). Other
decisions that are critical within design-based research are important
within DBIR as well, such as deciding on the composition of the design
team, the process of iteration, and what evidence and rationales will form
the basis for making changes to designs (Penuel, Confrey, Maloney, &
Rupp, 2011; Penuel, Tatar, & Roschelle, 2004).
The objects of focus in design are a key distinction between DBIR and
most design-based research. A hallmark of design-based research in the
learning sciences has been a focus on improving learning environments
in classrooms. At the same time, some learning scientists do focus more
explicit attention on designing learning opportunities for teachers (e.g.,
Fishman et al., 2003), with an eye toward building capacity for teachers
to enact innovations. DBIR encompasses such efforts, as well as designs
for bringing about system-level improvements (Bryk, Gomez, & Grunow,
2011; Resnick & Spillane, 2006). The objects of design in DBIR may
also include supports for implementers of informal education programs
or for interventions that cross different settings. The key for all of these
is that DBIR focuses not only on the design of tools and practices for
learners but also on the design of the necessary means of support for
144 National Society for the Study of Education

implementing those tools. Further, iterative cycles of design include iter-

ation on and coordination of these different objects of design; that is, as
teams refine or change tools for learning, they also refine or change tools
for implementation in ways that reflect what is being learned through
research (Fishman et al., 2003; Penuel & Fishman, 2012).

As a Strategy for Promoting Quality in the Research and Development Process,

Teams Develop Theory and Knowledge Related to Both Classroom Learning and
Implementation Through Systematic Inquiry

Though DBIR draws from the models of evaluation research described

earlier, it diverges from them somewhat with respect to this principle.
Most evaluation studies are motivated by practical and policy questions
aimed at generating knowledge of specific programs’ worth or value. But
DBIR aims to develop knowledge that might be useful across a range of
settings, either to inform design or to develop deeper understandings of
learning of different kinds (e.g., student, teacher, organizational). DBIR
also aims to develop theory for guiding design teams (which include
practitioners) and future learning research in the ways that design-based
research typically does (Edelson, 2002) and in the ways that well-struc-
tured case study research does (see Flyvbjerg, 2006).
The kinds of theories that are relevant to DBIR relate specifically to
the objects of design, and they encompass theories of learning within and
across subject matter domains, informal or everyday learning, teacher
learning, organizational change, and leadership. In contrast to design-
based research that focuses only on theories related to student learning in
domains, DBIR can also contribute to theories of organizations and insti-
tutions that guide much contemporary policy research in education, par-
ticularly by pointing out how the deployment of new tools (e.g., curricula,
technologies) can bring to light new needs for coordination across differ-
ent system levels and for capacity building (e.g., Stein & Coburn, 2008).

Design-Based Implementation Research is Concerned With Developing Capacity for

Sustaining Change in Systems

One strategy for promoting sustainability of designs is to develop capac-

ity through efforts to develop organizational routines and processes that
help innovations travel through a system. The predominant approaches
to building capacity in education have long been focused on improving
human capital of implementers—whether through professional develop-
ment or by selecting and rewarding teachers on the basis of their stu-
dents’ test scores—and on developing and supplying improved material
capital (e.g., curriculum) to them. DBIR focuses not only on building
the skills of individual implementers—although this may be a positive
 Design-Based Implementation Research 145

outcome of DBIR—but also on developing the capacity of the entire sys-

tem to implement, scale, and sustain innovations. Moreover, we include
researchers as individual participants and research organizations and
universities as institutions that are part of the ecology of supports that
must be strengthened to improve excellence and equity.

DEVELOPING DBIR AS AN APPROACH

As an approach to research on innovation design, development, and im-

plementation, DBIR shows tremendous promise. To date, that promise
has been realized both through the application of antecedent research
approaches described earlier and through a range of programs, projects,
and partnerships described throughout this Yearbook. But key questions
and issues must be addressed for DBIR to be better understood and em-
ployed as a viable research approach for ongoing and future research
and development toward broad and lasting school improvement.
The introduction of any “new” research approach frequently leads to
two different reactions from the research community: the inclination to
reject the approach as “too risky,” or as “nothing new.” In our conver-
sations about DBIR since we first described it in Educational Researcher
(Penuel, Fishman, et al., 2011), we have encountered both reactions, and
the chapters in this Yearbook offer responses intended to clarify and ex-
amine DBIR from a range of perspectives. In our view, what is new about
DBIR lies not in any single principle but in their integration. And what
makes the development of DBIR worthwhile as an endeavor is that it
expands methods available for developing evidence related to the imple-
mentation, efficacy, and scaling of innovations.
This Yearbook emerges from an NSF-sponsored workshop held at the
Presidio of San Francisco, California, in 2011. The workshop convened
practitioners who have been engaged with researchers in creating sus-
tainable innovations, education scholars with a diverse range of theo-
retical and methodological expertise, and representatives of public and
private funding agencies with a history of supporting research to create
and sustain educational innovation. Participants in that workshop both
developed a preliminary shared understanding of DBIR and surfaced
many issues and tensions that are both inherent in DBIR work and that
exist between DBIR and other research approaches. The chapters in this
Yearbook were selected to capture and represent both those understand-
ings and challenges to help clarify what is unique about DBIR and to il-
lustrate how DBIR provides a pathway toward more productive research
on innovation.
The challenges that the chapters in this Yearbook take up are multiple,
and they stem from the needs of different stakeholders in educational
146 National Society for the Study of Education

improvement: (1) the need to identify theories and methods that are ap-
propriate for research that focuses simultaneously on classroom teaching
and learning and on the policies and systems that support those improve-
ments at scale; (2) the need to develop cross-setting, cross-sector perspec-
tives on improving teaching and learning; (3) the need to develop meth-
ods for working in partnership with practitioners to negotiate the focus of
their work and to organize design to include a wide range of stakeholders
in schools and communities; (4) the need to develop standards of evidence
appropriate to the questions we pose in research; and (5) the need for
policies and infrastructures that help sustain partnerships and grow our
capacity for continuous improvement and sustainable change.

THEORIES AND METHODS OF DBIR

All rigorous scientific inquiry in education is guided by theory and em-

ploys methods suitable to address questions posed (National Research
Council, 2002). But the kinds of questions we pose in DBIR are broader
than the basic questions, “What is happening?” “Is there a systematic ef-
fect? and “How or why is it happening?” (cf. National Research Council).
DBIR researchers ask such questions as, “What works when, for whom,
and under what conditions?” and, “How can we make this innovation
work under a wide range of conditions?” Whereas descriptive and ex-
perimental methods and theories of learning may be appropriate for the
first set of questions, they may, by themselves, not be adequate for ad-
dressing the second set of questions posed by DBIR researchers. In their
chapter, Jennifer Russell, Kara Jackson, Andy Krumm and Ken Frank
(2013, this Yearbook) present four examples of DBIR projects and how
each takes up theories of learning and implementation that guide design
and development. In their analysis of these examples, they describe the
research methods employed and how these methods address the ques-
tions taken up.

DEVELOPING A CROSS-SETTING PERSPECTIVE ON TEACHING AND

LEARNING

A key construct emerging within both the learning sciences and policy re-
search in education is the importance of considering learning as unfold-
ing over time across multiple settings (Banks et al., 2007). Learning is
not bounded by the classroom, and it does not take place only in formal
settings. Improving student learning outcomes depends on taking into
account learners’ history of engagement with particular content areas,
the forms their engagement with that content takes in different settings
of their lives, and the organization of learners’ access to opportunities to
engage with specific domains. This broadened perspective on where and
 Design-Based Implementation Research 147

how learning takes place is an important anchor for DBIR as a model for
developing partnerships to improve the design and implementation of
educational interventions.
The chapters in this section both outline this life-long, life-wide per-
spective on learning and present examples of research–practice part-
nerships in which cross-setting interventions are being developed and
their implementation is being studied and promoted through partner-
ship activity. In the first chapter in this section, Milbrey McLaughlin and
Rebecca London (2013, this Yearbook) describe the work of the Youth
Data Archive (YDA) at Stanford University’s John W. Gardner Center.
The YDA makes it possible for people from different sectors to orga-
nize knowledge from diverse sources in order to gain new insight into
persistent problems and to identify new problems that can be addressed
through coordinated cross-sector efforts. In their chapter, Ben Kirsh-
ner and Joe Polman (2013, this Yearbook) take up the question of what
happens when innovations themselves move across different settings. In
contrast to approaches that emphasize adherence or fidelity to program
components, Kirshner and Polman discuss the need to authorize local
adaptation and for designers to provide “signature tools” that enable
productive—as opposed to maladaptive—adaptations of innovations.

DESIGNING ACROSS LEVELS

In educational systems, people who are charged with implementing pro-

grams often operate at and across different levels. In schools, these lev-
els include the classroom, school, and district. In out-of-school systems,
these levels might include a local site, a regional chapter, and a national
headquarters organization. Most educational design, however, focuses
on one of these levels. DBIR, by contrast, requires simultaneous atten-
tion to multiple levels of systems in design. If the goal is to develop and
test a new mathematics curriculum, then designers must minimally at-
tend to the conditions needed to support student learning in the class-
room and to conditions for teacher learning (e.g., the mathematics they
need to know, the principles underlying the curriculum), and to aligning
materials for teachers and students with the policy context (e.g., stan-
dards, assessments; Roschelle, Knudsen, & Hegedus, 2010).
The chapters in this section describe efforts to design across multiple
levels of a system. In their chapter, Bill Penuel, Cynthia Coburn, and
Dan Gallagher (2013, this Yearbook) analyze how stakeholders negotiate
the problems of practice that become the focus of joint work in proj-
ects that are structured as collaborative endeavors among researchers,
district leaders, and teachers. In her chapter, Meredith Honig (2013,
this Yearbook) focuses on the role that school district central staff play
148 National Society for the Study of Education

in the implementation of interventions and considers what is required

to organize supports for their learning through an approach she calls
design-based leadership research. In their chapter, Hilda Borko and Ja-
nette Klingner (2013, this Yearbook) explore what it means to design
scalable professional development that can be implemented effectively in
a wide range of settings. Teachers are ultimately in control of classroom
instruction, and therefore, any proposed change to instructional practice
depends heavily on the ability to design and deliver professional devel-
opment that is scalable and in which the outcomes of that professional de-
velopment are predictable. In the last chapter in this section, Angela De-
Barger, Jeffrey Choppin, Yves Beauvineau, and Savitha Moorthy (2013,
this Yearbook) focus on professional development supports for teachers’
adaptations of discourse-intensive pedagogies in both math and science.
This chapter examines how teachers develop these adaptations in re-
sponse to local classroom contexts that could not be fully anticipated by
curriculum developers, as a means to develop criteria for what consti-
tutes “productive adaptations” of materials.

FORMS OF EVIDENCE IN DBIR

Establishing standards of evidence is critical for any scholarly commu-

nity, and DBIR is no exception. However, it is neither desirable nor
possible to establish a “one-size-fits-all” approach to such standards. In
this section of the Yearbook, authors present different approaches to
generating evidence within DBIR that can be used to evaluate the suc-
cess of change efforts and to inform design. Paul Cobb, Kara Jackson,
Thomas Smith, Michael Sorum, and Erin Henrick (2013, this Yearbook)
open this section by presenting the ongoing work of the Middle School
Mathematics and the Institutional Setting of Teaching (MIST) project
as an exemplar of DBIR. MIST engages researchers and district leaders
in an iterative design process aimed at developing a coordinated system
of learning supports for enacting high-quality mathematics instruction.
A key to this work has been the development of methods designed to
make visible and study the often-invisible work of instructional coordi-
nation. Barbara Means and Christopher Harris (2013, this Yearbook)
address the topic of standards of evidence for DBIR by situating DBIR
within broader conversations among policy makers and other research-
ers about the nature of evidence needed to inform decision making. The
authors take up specifically how DBIR fits within current conceptions
of what makes for quality research studies that can yield robust causal
inferences about the impacts of programs. Jonathan Supovitz (2013, this
Yearbook) interrogates the relationships between practitioners, program
developers, and researchers to surface ways that notions of impartiality
 Design-Based Implementation Research 149

and evidence change in a DBIR approach and what challenges this pres-
ents to established views of validity and evidence in program evaluation.
This chapter considers the utility of identifying what evidence is needed
for an intervention on the basis of its location of an intervention along a
hypothetical “intervention development curve” as a framework for judg-
ing the adequacy of evidence in DBIR.

INFRASTRUCTURES IN SUPPORT OF DBIR

Just as the DBIR approach seeks to foster the creation of scalable and
sustainable educational innovation through new combinations of re-
search and practice partnerships, DBIR itself requires a different way to
conceive of, fund, and support the ongoing conduct of research and de-
velopment. This is an infrastructural challenge. The current policy and
funding environment frames the research enterprise in particular ways,
and DBIR challenges that framing. For instance, policy makers will need
to broaden conceptions of evidence beyond focusing on “what works”
to encourage researchers to develop evidence related to what works for
whom, when, and under what conditions.
The final section of this Yearbook comprises chapters that present
examples of new organizations and organizational structures for pur-
suing the work of DBIR and that critically examine existing infrastruc-
tures for supporting research in terms of what is needed for DBIR to
be fully realized. The Strategic Education Research Partnership (SERP)
was chartered by the National Academy of Sciences specifically to de-
velop new working relationships between research and practice and to
address pressing educational problems, and in many ways it exemplifies
the DBIR principles. Suzanne Donovan, Catherine Snow, and Phil Daro
(2013, this Yearbook) describe the history and work of SERP, with a fo-
cus on how that organization identifies important problems of practice
and brokers relationships between practice and research to address those
problems. Jimmy Scherrer, Nancy Israel, and Lauren Resnick (2013, this
Yearbook) describe the Institute for Learning (IFL), an organization with
a long history of focus on school reform. They examine work on both
teacher effectiveness and student learning from the perspective of insti-
tutions, employing a metaphor of “nesting” to describe organizational
structure. IFL has worked to design supports for “boundary crossers”
who communicate new information across contexts, and the chapter
explores how IFL has evolved over time and, with each iteration, has
moved toward DBIR principles. Jon Dolle, Louis Gomez, Jennifer Rus-
sell, and Tony Bryk (2013, this Yearbook) describe a radically different
approach to cross-boundary collaboration and improvement focused on
networked improvement communities. They describe how the Carnegie
150 National Society for the Study of Education

Foundation for the Advancement of Teaching forms partnerships with

community colleges not only to focus on a specific problem of practice
but also to construct social infrastructure for coordinating learning and
improvement across different individuals, organizations, and contexts.
Finally, Nora Sabelli and Chris Dede (2013, this Yearbook), who have
both served as program officers at the National Science Foundation,
bring their experience with infrastructure at the federal level to bear on
an examination of the types of funding, policies and technical support
required to enable DBIR to achieve its goals. Underlying these consid-
erations is the issue of timescales and feedback loops in educational im-
provement and an examination of the ways that the structure of research
in higher education must shift to better support extended partnership
focused on implementation.

CONCLUSION

The chapters in this Yearbook unpack the principles that characterize

DBIR. They provide details necessary for developing DBIR as a produc-
tive approach to designing effective, scalable, and sustainable education-
al interventions. Each chapter on its own tells only a part of the story, and
the volume itself is a kind of prologue to developing DBIR.
DBIR is a means to an end, not a methodology to be promoted or
followed for its own sake. The goal we all share is the transformation
of educational systems so that all young people have access to power-
ful opportunities to learn. Designing and testing effective, scalable, and
sustainable innovations using a DBIR approach is just one way to achieve
that goal. There are other approaches for the design and evaluation of
innovations; in the past, major changes to educational systems have ac-
companied major changes in society as a whole, such as happened dur-
ing the industrial revolution (Tyack & Cuban, 1995) and in concert with
major social movements (Oakes & Rogers, 2006). Such forces are likely
to shape educational systems in the future, so we would be wise to re-
member that we cannot design practice; we can only design for practice
(Spillane & Coldren, 2010). Similarly, DBIR exists alongside other ap-
proaches to research and development. Its success needs to be judged
relative to these other approaches’ success in producing innovations that
can improve teaching and learning at scale.

Acknowledgments

The workshop and development of materials in this volume were funded by the
National Science Foundation under award no. 1054086. The authors would,
in particular, like to thank Janice Earle of the NSF for helping to shape the
ideas that led to the workshop, and Denise Sauerteig of SRI International for
 Design-Based Implementation Research 151

invaluable support of all aspects of this work. The views represented in this chap-
ter and in this volume are those of the authors and not of the funding agencies or
the authors’ respective institutions.

References
Aleven, V. A., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by
doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2),
147–179.
Bang, M., & Medin, D. (2008, February). Community-based design of science learning
environments: Engaging with and implementing relational epistemologies. Paper presented at
the American Association for the Advancement of Science Meeting, Boston, MA.
Bang, M., Medin, D., Washinawatok, K., & Chapman, S. (2010). Innovations in culturally
based science education through partnerships and community. In M. S. Khine & M.
I. Saleh (Eds.), New science of learning: Cognition, computers, and collaboration in education
(pp. 569–592). New York, NY: Springer.
Banks, J. A., Au, K. H., Ball, A. F., Bell, P., Gordon, E. W., Gutierrez, K. D., . . . Zhou, M.
(2007). Learning in and out of school in diverse environments: Life-long, life-wide, life-deep.
Seattle, WA: The LIFE Center (The Learning in Informal and Formal Environments
Center), University of Washington, Stanford University, and SRI International and
Center for Multicultural Education, University of Washington.
Berman, P., & McLaughlin, M. W. (1975). Federal programs supporting educational change: Vol.
4. The findings in review. Santa Monica, CA: RAND.
Berwick, D. M. (2008). The science of improvement. Journal of the American Medical
Association, 299(10), 1182–1184.
Borko, H., & Klingner, J. K. (2013). Supporting teachers in schools to improve their
instructional practice. National Society for the Study of Education Yearbook, 112(2), 274–297.
Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building
networked improvement communities in education. In M. Hallinan (Ed.), Frontiers in
sociology of education (pp. 127–162). Dordrecht, The Netherlands: Verlag.
Cobb, P. A., Confrey, J., diSessa, A. A., Lehrer, R., & Schauble, L. (2003). Design
experiments in educational research. Educational Researcher, 32(1), 9–13.
Cobb, P., Jackson, K., Smith, T., Sorum, M., & Henrick, E. (2013). Design research with
educational systems: Investigating and supporting improvements in the quality of
mathematics teaching and learning at scale. National Society for the Study of Education
Yearbook, 112(2), 320–349.
Coburn, C. E., & Stein, M. K. (Eds.). (2010). Research and practice in education: Building
alliances, bridging the divide. Lanham, MD: Rowman and Littlefield.
Coburn, C. E., & Talbert, J. E. (2006). Conceptions of evidence use in school districts:
Mapping the terrain. American Journal of Education, 112, 469–495.
Coburn, C. E., Honig, M. I., & Stein, M. K. (2009). What’s the evidence on districts’ use of
evidence? In J. D. Bransford, D. J. Stipek, N. J. Vye, L. M. Gomez, & D. Lam (Eds.),
The role of research in educational improvement (pp. 67–87). Cambridge, MA: Harvard
Education Press.
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design research: Theoretical and
methodological issues. Journal of the Learning Sciences, 13(1), 15–42.
Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational
Evaluation and Policy Analysis, 14(4), 397–414.
152 National Society for the Study of Education

Datnow, A., Hubbard, L., & Mehan, H. (1998). Educational reform implementation: A co-
constructed process (Technical report). Santa Cruz: University of California, Center for
Research on Education, Diversity, and Excellence.
Davidson, M. R., Fields, M. K., & Yang, J. (2009). A randomized trial study of a preschool
literacy curriculum: The importance of implementation. Journal of Research on
Educational Effectiveness, 2, 177–208.
DeBarger, A. H., Choppin, J., Beauvineau, Y., Moorthy, S. (2013). Designing for productive
adaptations of curriculum interventions. National Society for the Study of Education
Yearbook, 112(2), 298–319.
Dolle, J. R., Gomez, L. M., Russell, J. L., & Bryk, A. S. (2013). More than a network:
Building professional communities for educational improvement. National Society for the
Study of Education Yearbook, 112(2), 443–463.
Donaldson, S. I. (2007). Program theory-driven evaluation science: Strategies and applications.
Mahwah, NJ: Erlbaum.
Donovan, M. S., Snow, C. & Daro, P. (2013). The SERP approach to problem-solving
research, development, and implementation. National Society for the Study of Education
Yearbook, 112(2), 400–425.
Donovan, M. S., Wigdor, A. K., & Snow, C. E. (2003). Strategic education research partnership.
Washington, DC: National Research Council.
Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal
of the Learning Sciences, 11(1), 105–121.
Fetterman, D. M. (2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.
Fishman, B. J., Marx, R. W., Best, S., & Tal, R. (2003). Linking teacher and student
learning to improve professional development in systemic reform. Teaching and Teacher
Education, 19(6), 643–658.
Fixsen, D. L., Naoom, S. F., Blase, K. A., & Friedman, R. M. (2005). Implementation research:
A synthesis of the literature. Tampa: Louis de la Parte Florida Mental Health Institute,
National Implementation Research Network, University of South Florida.
Flyvbjerg, B. (2006). Five misunderstandings about case study research. Qualitative Inquiry,
12(2), 219–245.
Frank, K. A., Zhao, Y., & Borman, K. (2004). Social capital and the diffusion of innovations
within organizations: Application to the implementation of computer technology in
schools. Sociology of Education, 77(2), 148–171.
Furtak, E. M., Ruiz-Primo, M. A., Shemwell, J. T., Ayala, C. C., Brandon, P. R., Shavelson,
R. J., & Yin, Y. (2008). On the fidelity of implementing embedded formative assessment
and its relation to student learning. Applied Measurement in Education, 21, 360–389.
Gutiérrez, K. D., Baquedano-Lopez, P., & Tejada, C. (2000). Rethinking diversity:
Hybridity and hybrid language practices in the third space. Mind, Culture, and Activity,
6(4), 286–303.
Gutiérrez, K. D., Rymes, B., & Larson, J. (1995). Script, counterscript, and underlife in
the classroom: James Brown versus Brown v. Board of Education. Harvard Educational
Review, 65(3), 445–471.
Gutiérrez, K. D., & Vossoughi, S. (2010). Lifting off the ground to return anew: Mediated
praxis, transformative learning, and social design experiments. Journal of Teacher
Education, 61(1–2), 100–117.
Honig, M. I. (2013) Beyond the policy memo: Designing to strengthen the practice of
district central office leadership for instructional improvement at scale. National Society
for the Study of Education Yearbook, 112(2), 256–273.
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009).
Research on evaluation use: A review of the empirical literature from 1986 to 2005.
American Journal of Evaluation, 30(3), 377–410.
 Design-Based Implementation Research 153

Kirshner, B., & Polman, J. L. (2013). Adaptation by design: A context-sensitive, dialogic

approach to interventions. National Society for the Study of Education Yearbook, 112(2),
215–236.
Lee, O., Penfield, R., & Maerten-Rivera, J. (2009). Effects of fidelity of implementation
on science achievement gains among English language learners. Journal of Research in
Science Teaching, 46(7), 836–859.
McDonald, S.-K., Keesler, V. A., Kauffmann, N. J., & Schneider, B. (2006). Scaling up
exemplary interventions. Educational Researcher, 35(3), 15–24.
McLaughlin, M., & London, R. A. (2013). Taking a societal sector perspective on youth
learning and development. National Society for the Study of Education Yearbook, 112(2),
192–214.
Means, B., & Harris, C. J. (2013). Towards an evidence framework for design-based
implementation research. National Society for the Study of Education Yearbook, 112(2),
350–371.
Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed
decision making in schools: Teacher access, supports and use. Washington, DC: Office of
Planning, Evaluation and Policy Development, Policy and Program Studies Service,
U.S. Department of Education.
Means, B., & Penuel, W. R. (2005). Research to support scaling up technology-based
innovations. In C. Dede, J. Honan, & L. Peters (Eds.), Scaling up success: Lessons from
technology-based educational improvement (pp. 176–197). New York, NY: Jossey-Bass.
Meyer, H.-D., & Rowan, B. (Eds.). (2006). The new institutionalism in education. Albany: State
University of New York Press.
National Research Council. (2002). Scientific research in education. Washington, DC: National
Academy Press.
Oakes, J., & Rogers, J. (2006). Learning power: Organizing for education and justice. New York,
NY: Teachers College Press.
O’Connor, K., Hanny, C., & Lewis, C. (2011). Doing “business as usual”: Dynamics of voice
in community organizing talk. Anthropology and Education Quarterly, 42(2), 154–171.
O’Donnell, C. L., & Lynch, S. J. (2008, March). Fidelity of implementation to instructional
strategies as a moderator of curriculum unit effectiveness. Paper presented at the annual
meeting of the American Educational Research Association, New York, New York.
O’Neill, D. K. (2012). Designs that fly: What the history of aeronautics tells us about the
future of design-based research in education. International Journal of Research and Method
in Education, 35(2), 119–140. doi:10.1080/1743727x.2012.683573
Palinkas, L. A., & Soydan, H. (2012). Translation and implementation of evidence-based practice.
New York, NY: Oxford University Press.
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand
Oaks, CA: Sage.
Patton, M. Q. (2000). Utilization-focused evaluation. In D. L. Stufflebeam, G. F. Madaus,
& T. Kellaghan (Eds.), Evaluation models (pp. 425–438). Boston, MA: Kluwer Academic.
Penuel, W. R., Coburn, C. E., & Gallagher, D. J. (2013). Negotiating problems of practice in
research-practice design partnerships. National Society for the Study of Education Yearbook,
112(2), 237–255.
Penuel, W. R., Confrey, J., Maloney, A., & Rupp, A. A. (2011, April). Design decisions in
developing assessments of learning trajectories: A case study. Paper presented at the annual
meeting of the American Educational Research Association, New Orleans, LA.
Penuel, W. R., & Fishman, B. J. (2012). Large-scale intervention research we can use.
Journal of Research in Science Teaching, 49(3), 281–304.
Penuel, W. R., Fishman, B., Cheng, B. H., & Sabelli, N. (2011). Organizing research and
development at the intersection of learning, implementation, and design. Educational
Researcher, 40, 331–337. doi:10.3102/0013189X11421826
154 National Society for the Study of Education

Penuel, W. R., & Means, B. (2004). Implementation variation and fidelity in an inquiry
science program: An analysis of GLOBE data reporting patterns. Journal of Research in
Science Teaching, 41(3), 294–315.
Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). The WHIRL co-design process:
Participant experiences. Research and Practice in Technology Enhanced Learning, 2(1),
51–74.
Penuel, W. R., Tatar, D., & Roschelle, J. (2004). The role of research on contexts of
teaching practice in informing the design of handheld learning technologies. Journal of
Educational Computing Research, 30(4), 331–348.
Penuel, W. R., & Yarnall, L. (2005). Designing handheld software to support classroom
assessment: An analysis of conditions for teacher adoption. Journal of Technology,
Learning, and Assessment, 3(5). Available from http://www.jtla.org.
Resnick, L. B., & Spillane, J. P. (2006). From individual learning to organizational
designs for learning. In L. Verschaffel, F. Dochy, M. Boekaerts, & S. Vosniadou (Eds.),
Instructional psychology: Past, present and future trends. Sixteen essays in honor of Erik De Corte
(pp. 259–276). Oxford, England: Pergamon.
Roschelle, J., Knudsen, J., & Hegedus, S. J. (2010). From new technological infrastructures
to curricular activity systems: Advanced designs for teaching and learning. In M. J.
Jacobson & P. Reimann (Eds.), Designs for learning environments of the future: International
perspectives from the learning sciences (pp. 233–262). New York, NY: Springer.
Russell, J. L., Jackson, K., Krumm, A. E., & Frank, K. A. (2013). Theory and research
methodologies for design-based implementation research: Examples from four cases.
National Society for the Study of Education Yearbook, 112(2), 157–191.
Sabelli, N., & Dede, C. (2013). Empowering DBIR: The need for infrastructure. National
Society for the Study of Education Yearbook, 112(2), 464–480.
Scherrer, J., Israel, N., & Resnick, L.B. (2013). Beyond classrooms: Scaling and sustaining
instructional innovations. National Society for the Study of Education Yearbook, 112(2),
426–442.
Spillane, J. P., & Coldren, A. F. (2010). Diagnosis and design for school improvement: Using a
distributed perspective to lead and manage change. New York, NY: Teachers College Press.
Stein, M. K., & Coburn, C. E. (2008). Architectures for learning: A comparative analysis of
two urban school districts. American Journal of Education, 114(4), 583–626.
Stewart, D. W., & Shamdasani, P. N. (2006). Applied social research methods series. Newbury
Park, CA: Sage.
Strand, K., Marullo, S., Cutforth, N., Stoecker, R., & Donohue, P. (2003). Community-based
research and higher education. San Francisco, CA: Jossey-Bass.
Supovitz, J. (2013). Situated research design and methodological choices in formative
program evaluation. National Society for the Study of Education Yearbook, 112(2), 372–399.
Tyack, D., & Cuban, L. (1995). Tinkering toward utopia: A century of public school reform.
Cambridge, MA: Harvard University Press.
Tzou, C. T., & Bell, P. (2010). Micros and Me: Leveraging home and community practices
in formal science instruction. In K. Gomez, L. Lyons, & J. Radinsky (Eds.), Proceedings
of the 9th International Conference of the Learning Sciences (pp. 1135–1143). Chicago, IL:
International Society of the Learning Sciences.
Wallerstein, N., & Duran, B. (2010). Community-based participatory research contributions
to intervention research: The intersection of science and practice to improve health
equity. American Journal of Public Health, 100(S1), S40–S46.
Weinberg, A. S. (2003). Negotiating community-based research: A case study in the “Life’s
Work” project. Michigan Journal of Community Service Learning, 9(3), 26–35.
Werner, A. (2004). A guide to implementation research. Washington, DC: The Urban Institute Press.
Whyte, W. F. (1991). Participatory action research. Newbury Park, CA: Sage.
 Design-Based Implementation Research 155

BARRY FISHMAN is a faculty member in learning technologies in both

the School of Information and the School of Education at the University
of Michigan. His research focuses on the transformation of teaching and
learning with technology, teacher learning with and about technology,
the design of learning environments, and design-based implementation
research. Recent publications include “Organizing Research and Devel-
opment at the Intersection of Learning, Implementation, and Design”
(2011, Educational Researcher) with William Penuel, Britte Cheng, and
Nora Sabelli; “Examining Study Attrition: Implications for Experimen-
tal Research on Professional Development” (2012, Teaching and Teacher
Education) with Beth Kubitskey, Richard Vath, Heather Johnson, and
Spyros Konstantopoulos; and “What Makes Professional Development
Effective? Strategies That Foster Curriculum Implementation” (2007,
American Educational Research Journal), with William Penuel, Ryoko Ya-
maguchi, and Lawrence Gallagher.

WILLIAM PENUEL is a professor of educational psychology and learn-

ing sciences at the University of Colorado Boulder. His research focuses
on teacher learning and organizational processes that shape the imple-
mentation of educational policies, school curricula, and after-school pro-
grams. Recent publications include “Large Scale Intervention Research
We Can Use” (2012, Journal of Research in Science Teaching) with Barry
Fishman; “Preparing Teachers to Design Sequences of Instruction in
Earth Science: A Comparison of Three Professional Development Pro-
grams” (2011, American Educational Research Journal) with Lawrence P.
Gallagher and Savitha Moorthy; and “Using Large-scale Databases in
Evaluation: Advances, Opportunities, and Challenges” (2011, American
Journal of Evaluation) with Barbara Means.

ANNA-RUTH ALLEN is a research associate in the School of Education

at the University of Colorado Boulder. Her research interests include
sociocultural approaches to discourse, literacy and learning, after-school
programs, and youth development. Publications include “Learning as
the Organizing of Social Futures” (2010, National Society for the Study of
Education) with Kevin O’Connor; “‘Becoming’ a Teacher” (2007, Teachers
College Record) with Mary Louise Gomez and Rebecca Black; and “Lan-
guage, Class, and Identity: Teenagers Fashioning Themselves Through
Language” (2001, Linguistics and Education) with Jim Gee and Katherine
Clinton.
156 National Society for the Study of Education

BRITTE HAUGAN CHENG is an education researcher at SRI Inter-

national’s Center for Technology in Learning. Her research focuses on
the design of learning technologies, instruction and assessments in K–12
and how their use can be conceptualized within the system of learning
processes, educational practice, and policy. Recent publications include
“Organizing Research and Development at the Intersection of Learning,
Implementation, and Design” (2011, Educational Researcher) with William
Penuel, Barry Fishman, and Nora Sabelli; and “Unpacking the Black
Box of Engagement: Cognitive, Behavioral, and Affective Engagement
in Learning Mathematics” in The Future of Learning: Proceedings of the 10th
International Conference of the Learning Sciences (ICLS 2012) with Nicole
Shechtman, Patrik Lundh and Gucci Trinidad.

NORA SABELLI is a senior science advisor at SRI International’s Center

for Technology in Learning. Her research interests are science educa-
tion, the role of technology in education and educational policy infor-
matics. Recent publications include “Organizing Research and Devel-
opment at the Intersection of Learning, Implementation, and Design”
(2011, Educational Researcher) with William Penuel, Barry Fishman, and
Britte Cheng; and “Complex Systems and Educational Change: Towards
a new research agenda” (2008, Educational Philosophy and Theory) with Jay
Lemke.