I novation

Spotlights:
Nine Dimensions for
Supporting Powerful
STEM Learning with
Technology
Table of Contents
Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Nine dimesions of powerful STEM learning. . . . . . . . . . . . . . . . 5
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Appendix: Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Identification of relevant studies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Coding of eligible studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Results from study search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Innovation Spotlights 2
Acknowledgements
This work was developed under the leadership and guidance of
Bernadette Adams from the U.S. Department of Education, Office of
Educational Technology.
Vanessa Peters led the development of the systematic literature
review for Digital Promise with support from Emi Iwatani and
Barbara Means. Karen Cator and Susan Thomas served as
independent consultants.
The research team extends their thanks to a Technical Working
Group of leading STEM education researchers who provided
valuable guidance on the systematic literature review and provided
recommendations for potential spotlight schools: Nancy Butler
Songer, Drexel University; Jennifer Corn, RTI International; Bill
Penuel, University of Colorado at Boulder; and Joseph Krajcik,
Michigan State University.

Innovation Spotlights 3
Introduction
We live in a time of staggering
technological change, with disruptive
technology innovations continually
reshaping how we live and work. One
example is the indispensable role
technology now plays in science,
technology, engineering, and
mathematics (STEM) professions. To
prepare young people for citizenship
and a workforce with rapidly changing
requirements, we must design learning
environments that provide all students with unique and engaging
opportunities to master STEM skills using the technologies STEM
professionals use.

This is already occurring in our nation’s innovative digital technology in STEM and
schools. Coding courses and workshops computer science curricula and classrooms.
engage students in computational thinking. 3D For purposes of the review, we defined digital
modeling software and printers let students technology broadly to include various forms of
become “makers” solving real-world problems computing devices and software applications,
through design and engineering. Robotics including sensors connected to computing
competitions combine digital coding and devices, digital immersive environments,
design with physical building to help students cellular devices and mobile applications, as
learn collaboration and problem-solving skills. well as collaborative technologies.
With augmented reality, students can now
develop field research skills while exploring The research review identified nine dimensions
simulations of ecosystems in faraway places. of effective STEM learning practices enabled
by technology, which are described in this
Still, as with all educational innovations, report. The report also provides links to videos
bringing technology into classrooms alone developed by the Department and Digital
does not ensure better outcomes. Powerful Promise that highlight the research-based
learning experiences also require effective dimensions in action in 10 spotlight schools
teachers who make instructional decisions (Exhibit 1). The purpose of the STEM spotlights
based on evidence of enhanced student is not to advocate for technology for its own
learning. sake, but rather to illustrate meaningful,
research-based STEM learning experiences
To help teachers implement new, research- that can be supported with digital tools in
based approaches for leveraging technology classrooms across the country.
to improve STEM learning, the U.S. Department
of Education, in collaboration with Digital An appendix provides additional detail on the
Promise, conducted a systematic review of the methodology used for the research review.
research literature on the impact of integrating

Innovation Spotlights 4
Nine dimensions of powerful
STEM learning
The review of the literature surfaced nine dimensions of powerful
STEM learning that can be supported with digital technology. The
review looked at studies of supports for students’ development of
STEM and computational thinking knowledge and practices (skills)
that provided evidence of student learning outcomes, as measured
by an objective test and not just self-reported perceptions of
learning. The nine dimensions were checked for consistency with
conceptual framings and empirical findings from the Department
of Education’s Policy and Program Studies Service and the Institute
for Education Sciences, as well as key policy documents such as
the National Science & Technology Council’s 2018 report, Charting
a Course for Success: America’s Strategy for STEM Education1.
Our process also involved soliciting input from STEM education
research experts. Below we present top-line summaries of the nine
dimensions.

1 Dynamic Representations. Students learn or

master STEM concepts through interacting with digital
models, simulations, and dynamic representations of
mathematical, scientific, and engineering systems.

2 Collaborative Reasoning. Technology tools support

students’ collaborative reasoning around STEM concepts,
equalizing participation among group members and
helping individuals and groups improve their ideas.

3 Immediate and Individualized Feedback.

Digital tools provide students practicing or learning
STEM skills or concepts with immediate and
individualized feedback beyond right or wrong.

4 Science Argumentation Skills. Students use

technology that supports science argumentation
skills including presenting and evaluating evidence
about scientific or mathematical claims.
1
White House of Science and Technology Policy (2018, December)
Innovation Spotlights | 5
 5 Engineering Design Processes. Students
plan, revise, implement, and test problem
solutions using engineering design processes
and appropriate technologies.

6 Computational Thinking. Students use technology

to problem-solve using algorithms, data, and
simulations to investigate questions and develop
new understandings about phenomena.

7 Project-based Interdisciplinary Learning. Students

use digital technology tools in the context of project-
or challenge-based learning activities that integrate
multiple STEM fields (e.g., science and mathematics).

8 Embedded Assessments. Digital assessments

are embedded in STEM instruction to prompt
students’ reflection on the quality of their
explanations, models, or problem solutions.

9 Evidence-based Models. Students use technology

to develop models based on data and evidence.

The nine dimensions served as a framework to This process identified 100 schools, each of
identify 10 schools that could be spotlighted which were invited to submit an application
for implementing one or more of the STEM that asked them to describe in detail a
dimensions in their classrooms. Spotlight classroom lesson that used technology
schools were selected based on nominations for addressing one or more of the STEM
from a Technical Working Group of leading dimensions. Of these, 34 schools submitted
education researchers and other individuals applications and 15 were interviewed about
knowledgeable about technology use for their suitability as a spotlight school. During
powerful STEM learning. Also considered the selection process we sought to include
were schools recognized as models of STEM a broad representation of geographic areas
education such as Future of Education and a range of grade levels (i.e., elementary,
Technology’s STEM Excellence Award finalists middle, and high schools). Exhibit 1, below,
and winners, Intel Schools of Distinction Award shows the names and locations of the final 10
finalists and winners, and schools profiled in spotlight schools.
policy reports such as Monitoring Progress
Toward Successful K-12 STEM Education (NRC,
2013).

Innovation Spotlights 6
Exhibit 1. Names and locations of the ten spotlight schools.

Weaver Lake
Henrietta Lacks Health and Elementary School
Pine Grove
Bioscience High School Maple Grove, MN
Middle School
Vancouver, WA Schoo Middle School East Syracuse, NY
Lincoln, NE
South Side
Elementary School
WA Kendallville, IN
MT ND ME
MN
OR VT
NH
ID SD WI NY MA
WY MI RI
CT
Design Tech High School
IA PA Greene Central
Redwood City, CA
NE OH High School
NV
NJ

UT IL IN Snow Hill, NC
CA CO DL
WV
VA
ML

KS MO KY
NC
TN
AZ OK
NM AR SC
Walter Bracken MS AL Douglas L. Jamerson, Jr.
STEAM Academy GA Elementary School
Las Vegas, NV
TX St. Petersburg, FL

LA
FL

Cedars Intl. Next

Generation High School
Austin, TX

The following section provides more details about the nine dimensions and the 10 spotlight
schools. Each dimension is supported with examples of powerful practice from the research
literature and highlights an exemplary STEM lesson from one of the 10 spotlight schools.2

2
Information for school “Quick Facts” obtained from the Institute of Education Sciences National Center for Education Statistics
website.

Innovation Spotlights 7
 1. Dynamic Representations. Students learn or
master STEM concepts through interacting with digital
models, simulations, and dynamic representations of
mathematical, scientific, and engineering systems.

Dynamic representations such as digital physics learning, for example, found that
models, interactive simulations, and virtual there was a significant reduction in students’
environments are fundamental tools used by misconceptions about motion and force after
scientists, mathematicians, and engineers. using a simulation-based learning environment
Based on a computer model, dynamic that included such scaffolds.4 In high school
representations engage learners and help mathematics, the interactive software SimCalc
them construct more accurate mental models has been shown to significantly improve
of natural or engineered phenomena that how students learn core algebra concepts
would otherwise be difficult or impossible that include procedural and conceptual
to observe.3 Dynamic representations vary problems.5 In a meta-analytic study on
widely and differ in the levels of control and simulations, researchers found that students
interactivity they provide to students. To better using simulations for STEM learning performed
support learners, they are often augmented better on measures of learning than students
with pedagogical scaffolds that guide students engaged in equivalent activities that did not
when learning about STEM systems and involve simulations.6
processes. A study on middle school students’

Walter Bracken STEAM Academy

At Walter Bracken a teacher uses an interactive Quick Facts
simulation to teach third graders about weather Location: Lincoln, NE (rural, fringe)
and climate. Students use the online simulation to
take temperature readings of two different places FRPL: 44% Asian: 3%
in the United States. Using the technology tool, Black 4% White: 73%
students can go forward in time to see different
Hispanic: 11% Other: 9%
measurements of temperature, humidity, rain, and
wind direction. Students use tablets to graph the
data to observe patterns and share their findings with one another. Since Las Vegas has
limited weather variability, students are engaged and have fun when learning about weather
in different parts of the country, an activity that would not have been possible without the
use of digital technology.

Watch the Walter Bracken STEAM Academy video!

3
National Research Council (2011)
4
Huang et al. (2017)
5
Hegedus et al. (2015)
6
D’Angelo et al. (2014, March)
Innovation Spotlights 8
 2. Collaborative Reasoning. Technology tools
support students’ collaborative reasoning around
STEM concepts, equalizing participation among group
members and helping individuals and groups improve
their ideas.
Collaboration facilitates learning through facilitated richer collaboration and resulted in
students’ ongoing efforts to construct and stronger responses to open-ended questions.
maintain a shared understanding of a problem.7 Observations showed that the technology
When students work together to reason about also appeared to help equalize participation—
STEM ideas, they engage in negotiation and students were more encouraged to change
renegotiation around meaning. Technology their minds and were less likely to dominate
can enhance collaboration by expanding the the conversation or to acquiesce to other
kinds of exchanges students can have and by students dominating classroom interactions
increasing access to community knowledge. when using the collaboration platform.9 Fifth-
Research has shown that students benefit graders who used Mindtool to create science
more from collaborative activities when they concept maps that were shared and discussed
receive support and guidance for interacting with peers had higher achievement scores than
with one another.8 In a study of fourth graders students engaged in the same activity using
learning science, use of a digital collaboration hand-drawn maps.10
platform that included e-sticky notes

Philip H. Schoo Middle School

At Philip H. Schoo Middle School students learn how
Quick Facts
sound reaches far distances using a unit from the
Next Generation Science Storylines. Students begin Location: Lincoln, NE (rural, fringe)
by observing the perplexing phenomenon of sound FRPL: 44% Asian: 3%
being generated when a pin is dragged over a vinyl
Black 4% White: 73%
record. This observation leads students to generate
questions related to the causes of sound and how it Hispanic: 11% Other: 9%
reaches the ear. Students then create digital models
using laptops to answer their initial questions, leading to the generation of even more
questions. The Schoo teachers use a class consensus model for this unit, with students
sharing their questions using an online communication and collaboration platform
and on a driving question board. Students work in groups to answer questions and
develop new ideas as they learn what their peers are thinking. The teacher projects every
student’s digital model in front of the class and together they discuss how the different
representations are similar to or different from the scientific model. In this way, teachers
are able to co-construct with their students the questions that drive scientific inquiry.

Watch the Philip H. Schoo Middle School video!

7
Roschelle & Teasley (1995)
8
Dillenbourg (1999)
9
Lan et al. (2010)
10
Hwang et al. (2011)
Innovation Spotlights 9
 3. Immediate, Individualized Feedback. Digital tools
provide students practicing or learning STEM skills or
concepts with immediate and individualized feedback,
beyond right or wrong.
Feedback is considered one of the most desirable.15 For example, a quasi-experimental
powerful means for influencing student study involving 4,000 second grade students
learning.11 When accompanied by challenging from 129 classrooms found greater learning
goals, feedback informs learners about the gains in basic calculation skills when students
level of performance that is required so they practiced their arithmetic skills using a
can set reasonable goals and direct their computer and received immediate feedback
actions and effort accordingly.12 When learning about their response.16 Feedback on task-
goals are clear and students are committed, based activities can be provided at both the
feedback can help students reduce the gap individual and the group level. When provided
between their current level of performance at the group level, learners must decide if the
and the intended level.13, 14 Feedback can take feedback is relevant to them individually, to
a number of different forms depending on the the group as a whole, or to other individuals.17
learning goals of a lesson. For example, when A study of fifth-graders found that students
students are engaged in tasks such as test- learned fractions faster when the software
taking, some delay in feedback is beneficial. provided individualized feedback about
When students are engaged at the process their misconceptions beyond whether their
level, however, as when processing classroom response was correct.18
activities, immediate feedback is more

Greene Central High School

At Greene Central High School in Snow Hill, North Quick Facts
Carolina, students regularly receive individualized
Location: Snow Hill, NC (rural, distant)
and immediate feedback while learning
mathematics. Using tablets, students watch videos FRPL: 96% Asian: 0%
of their teacher working through a problem set and Black 43% White: 32%
then work through the problems themselves. The
Hispanic: 25% Other: 1%
technology enables the teacher to see what each
student is doing, so s/he can spot students who are
struggling with a particular kind of math problem. The teacher can provide just-in-time
tailored supports for a struggling student in the form of hints or worked examples. The
ability to access data about each student in real-time enables quick interventions that are
individualized based on each student’s needs. The technology also provides the teacher
with dashboards showing data about each student as well as about the entire class so s/he
can see where students need help and which math standards they are struggling with.

Watch the Greene Central High School video!

11
Hattie & Gan (2011)
12
Locke & Latham (1990)
13
Hattie & Timperley (2007)
14
Kluger & DeNisi (1996)
15
Kulik & Kulik (1988)
16
Beserra et al. (2014)
17
Nadler (1979) Innovation Spotlights 10
18
Chu et al. (2014)
 4. Science Argumentation. Students use technology
that supports science argumentation skills including
presenting and evaluating evidence about scientific or
mathematical claims.
Science argumentation is a thought process relevancy of the evidence for supporting the
that requires critical thinking to present and argument in a claim. The researchers found
defend evidence that explains an idea about a significant improvement among lower-
scientific phenomenon.19 Argumentation is achieving students’ ability to evaluate the
central to the practice of all fields of science. quality of scientific arguments after using
In the classroom, science argumentation the computer-based scaffolds. A different
enables students to construct knowledge study of 960 middle school science students
and empowers them to criticize plausible- found that students improved their science
sounding ideas that lack supporting argumentation skills through an online,
evidence.20,21 Technology can promote science multiplayer game that was instructionally
argumentation in a number of ways. A 2011 designed to integrate scientific argumentation.
study tested the impact of computer-based Students who played the game 10 times or
scaffolds for argumentation on middle school more improved in all aspects of argumentation,
students’ ability to evaluate evidence-based including understanding a claim, judging
arguments during problem-based learning. evidence about a claim, determining the
Components of arguments that were tested reasoning applied to the claim, and making
included making claims, providing evidence judgments.22
that supports the claim, and explaining the

Weaver Lake STEM Elementary School

AAt Weaver Lake STEM Elementary School in Quick Facts
Maple Grove, Minnesota, fourth graders engage
in a year-long study about lake health to learn Location: Maple Grove, MN (suburb, large)
science argumentation skills using data and FRPL: 35% Asian: 22%
evidence. Guided by the question, “Is the lake Black 26% White: 42%
healthy?” students pay monthly visits to a nearby
Hispanic: 4% Other: 7%
freshwater lake to collect environmental data
with digital tools used by real scientists. Students
use probes to record water and soil temperatures; collect samples of lake water; and
use a turbidity tube to record water clarity. Students also take digital photographs of the
lakefront to document seasonal observations at different times of the year. Back in the
classroom, students are tasked to create a website that lets the broader school community
know whether or not the lake is healthy based on their data. Technology permits students
to effectively display the evidence that they collected over the school year (including
graphs and photos), articulate their arguments, and revise their arguments as needed.
These data provide students with the opportunity and motivation to enhance and expand
the efficacy of their scientific argument and presentation.
(continued)

19
Berland & Reiser (2009)
20
Berland & Reiser (2010)
21
Ford (2008)
22
Ault et al. (2015)
Innovation Spotlights 11
 Third graders at Weaver Lake engage in scientific argumentation as part of a year-long
study of ornithology. In the Create-a-Bird project, students start by learning about
Minnesota birds and the variety of beaks, feet, habitat, and diets that are associated
with different bird species. Students then get to create their own species of a bird by
creating and animating a digital model. Students then have to craft an argument about
the appropriateness of their bird for a selected habitat. Students use facts about birds as
evidence to explain their thinking about why the feet and beak of their bird are right for
the habitat they choose.

Watch the Weaver Lake STEM Elementary School video!

5. Engineering Design Processes. Students plan,

revise, implement, and test problem solutions using
engineering design processes and appropriate support
technologies.
Engineering can be defined as the process of in STEM domain knowledge and higher
designing for the human-made world.23 Like order thinking skills, including problem
science, engineering involves both iterative solving abilities and procedural knowledge,
and systematic processes. Engineers test for students who completed four units of
and modify each new version of a design an engineering module that integrated
based on available knowledge, and then components of science and mathematics.
systematically undertake steps to improve the Students constructed a cam mechanism toy
design.19 Using data, engineers ask questions using lever scales, a gear-wheeled range
to understand the design problem, determine finder, and a gear sets and then used computer
criteria for a successful solution, and identify aided design (CAD) software to simulate and
constraints.24 Digital tools play a central analyze motion and the speed ratio of different
role in the engineering design process. In gears. In middle school, seventh-graders
education, technology tools provide increased knowledge gains in engineering and math
opportunities for students to apply scientific were higher when they used technology tools
and mathematical ideas to their designs. Fan for data analysis and measurement within a
and Yu’s (2017) quasi-experimental study of curricular unit on heat transfer.25
high school students found improvements

23
National Academy of Engineering (2008)
24
Feder et al. (2009)
25
Guzey et al. (2017)

Innovation Spotlights 12
 Douglas L. Jamerson, Jr. Elementary School
At Jamerson Elementary in St. Petersburg, Florida,
Quick Facts
children learn the engineering design process
of “Plan, Design, Check, and Share” starting in Location: St. Petersburg, FL (city, large)
kindergarten. By third grade, students learn FRPL: 53% Asian: 3%
about insulators and conductors and apply their Black 37% White: 48%
knowledge when planning and making insulators
Hispanic: 7% Other: 5%
with the goal of keeping ice cubes from melting. A
variety of technology tools support the students’
engineering design and evaluation activities. Drawing on knowledge from previous lessons
in earlier grades about insulators and conductors, students start by collaboratively planning
their ice incubators by drawing out prototypes and talking through their ideas. During the
design stage, students test the temperature of different materials, such as black and white
paper and aluminum foil, to see which has the lowest temperature. After adding ice cubes
to the insulators, the initial temperature is taken prior to leaving the insulators outside in the
sun for 30 minutes. Students then check their design by using probeware to take multiple
readings of the temperature, which are later graphed on a chart and used as data. Back in
the classroom, students take out their ice vessels and measure the millimeters of liquid to
see how much water had melted. As a class, students share out their design challenges to
investigate which materials made the best insulators.

Watch the Douglas L. Jamerson, Jr. Elementary School video!

Design Tech High School

At Design Tech High School in Redwood City,
Quick Facts26
California, students use different digital tools
and the engineering design process to propose, Location: Redwood City, CA (suburb, large)
plan, scope, and budget a semester-long FRPL: 11% Asian: 20%
engineering project. In small groups, students
Black 0.4% White: 50%
create a personalized, resin table, in one case
to reflect aspects of the local peninsula and Hispanic: 13% Other: 17%
the previous location of the school. Students
create digital drawings of their table, using the size dimensions to calculate the amount
of materials they need. They use a Gantt chart to plan out their work for each week of the
project, with students making adjustments as needed, for example, when they find out
some of their materials are unavailable. Students learn important skills throughout the
entire design process, initially working as a team and then separating into individual roles.
Students learn to use new tools to manage and implement a process in which mistakes and
solving unanticipated problems are both important and necessary.

Watch the Design Tech High School video!

26
Information for Design Tech High School “Quick Facts” obtained from the California Department of Education website.

Innovation Spotlights 13
 6. Computational Thinking. Students use technology
to problem-solve using algorithms, data, and
simulations to investigate questions and develop new
understandings about phenomena.
Computational thinking involves the conclusions from data.30 In the study, students
formulation and analysis of problems and developed a computer program from source
their solutions using methods and tools code based in algorithmic thinking and then
for reasoning abstractly and automating used the program to solve a mathematics
procedures through algorithmic thinking.27,28 problem set. The code was transferred into
In education, the practices learned through MATLAB to generate graphical representations
computational thinking are widely applicable of the solution that illustrated the correctness
to topics in science and mathematics and of the code and the solution of the exercise.
provide a real-world and applied context A study of rural and indigenous middle school
for working with abstract datasets.29 For students also showed learning gains from
example, a quasi-experimental study with using robotics and game design that relied
high school students found that teaching on abstraction skills, logical thinking, use of
computational thinking in conjunction with algorithms, and analyzing and implementing
computer programming resulted in greater solutions.31 Samples of student work were
improvements in students’ reasoning skills, analyzed for evidence of computational
including making conjectures and drawing thinking.

South Side Elementary School

At South Side Elementary School in Columbus,
Quick Facts
Indiana, STEM education includes computational
thinking, which begins in kindergarten and Location: Kendallville, IN (town,distant)
continues through fifth grade. For example, in FRPL: 64% Asian: 0.2%
second grade, students engage in a coding activity Black 0.5% White: 84%
to illustrate a butterfly’s life cycle. Older students
Hispanic: 12% Other: 3%
create robots or other machines, an activity
that requires them to think computationally and
procedurally as they solve problems in their design.
By working through their coding errors, students come to understand that computational
thinking involves ongoing trial-and-error and that persisting through mistakes or failures
is an important part of the process. Teachers value coding activities for their connection to
mathematics and the engineering design process, and for teaching students how to explain
their reasoning and thinking to other people.

Watch the South Side Elementary School video!

27
Basu et al. (2014)
28
Grover & Pea (2013)
29
Wing (2008)
30
Psycharis & Kallia (2017)
31
Leonard et al. (2016) Innovation Spotlights 14
 7. Project-based Interdisciplinary Learning. Students
use digital technology tools in the context of authentic
project- or challenge-based learning activities that
integrate multiple STEM fields (e.g., science and
mathematics).
Research shows that using an interdisciplinary students’ learning activities culminate in a
curriculum that integrates multiple STEM tangible and sharable product that adequately
subjects provides students with a more addresses the problem or question.34 The
relevant, more engaging, and less fragmented final product or artifact gives students an
learning experience.32 In the context of opportunity to publicly display their work and
project- or challenge-based learning, students to practice presenting and explaining their
can pursue solutions to real problems in the project’s outcomes to others. With project-
same way that professional scientists do.33 In based interdisciplinary activities, students
this context digital technologies can be used to not only learn important STEM content and
find, organize, and communicate information practices, but also practice applying them in
and ideas; to support task management and conjunction with other skills and knowledge in
productivity; and to create a final product. a realistic context.
When teachers use this instructional approach,

Henrietta Lacks Health and Bioscience

High School Quick Facts

Henrietta Lacks (HeLa) Health and Bioscience Location: Vancouver, WA (city, midsize)
High School in Vancouver, Washington provides FRPL: 41% Asian: 9%
students with interdisciplinary project- or Black 2% White: 66%
challenge-based learning through a project in
Hispanic: 15% Other: 8%
which students learn about the chemicals in
sunscreen and their impact on the environment.
The challenge starts by creating a model of a molecule that’s commonly found in over-
the-counter products including sunscreen. Through their research, students learn that
certain compounds such as oxybenzone are harmful for coral reefs and for your skin. In
their investigations students use the authentic and freely available PubChem database of
chemical information and 3D interactive chemical structure models. For their culminating
artifact, students create a project poster as well as a public information piece in the form of
a sunscreen brochure. Students present their research to the entire class and describe how
the active ingredients found in sunscreen affects coral reef systems and human health.

Watch the Henrietta Lacks Health and Bioscience High School video!

32
Frykholm & Glasson (2005)
33
Krajcik & Blumenfeld (2006)
34
Blumenfeld et al. (1991)

Innovation Spotlights 15
 8. Embedded Assessments. Digital assessments are
embedded in STEM instruction to prompt students’
reflection on the quality of their explanations, models,
or problem solutions.
When embedded in STEM instruction, digital peer and expert feedback on their knowledge
assessments can provide rich information products, which students then used to revise
about the nature and quality of students’ STEM and resubmit their products. Study findings
practices during scientific inquiry.35, 36 When showed that the formative assessment
used for formative purposes, assessments improved science learning for all students
provide a valuable opportunity for adapting but was particularly effective for students
instruction to better meet learners’ needs.37 from underrepresented groups. Moreover, a
Such assessments are especially powerful qualitative analysis of students’ knowledge
when they fuse core STEM ideas with artifacts revealed that students’ peer reviews
STEM practices. A study by Zaidi (2016), became increasingly elaborate and specific
for example, investigated the impact of over time, enabling them to produce higher
embedded formative assessments designed to quality integrated knowledge products.
promote integrated science knowledge. The
assessment allowed students to receive both

Pine Grove Middle School

At Pine Grove Middle School in Syracuse, New
York, students complete embedded assessments Quick Facts
during a science lesson about the impact of Location: East Syracuse, NY (rural, fringe)
organisms on the ecosystem. The lesson starts with
FRPL: 45% Asian: 2%
a check-in question about what would happen if
certain species were removed from the food chain. Black 4% White: 87%
The question sets the context of the lesson, which Hispanic: 2% Other: 4%
is driven by students’ curiosities about food chain
impacts. Students add their ideas to a collaborative
document where they receive feedback from their peers. Facilitated by the technology,
students reflect on and synthesize ideas which often results in the formation of new or
revised understandings about the topic. In the second part of the lesson students focus
on what would happen to the food chain if certain species were added. Students submit
their answers online where the teacher can also provide real-time feedback to individual
students. An important factor for the successful use of digital technology, a teacher notes,
is to ask meaningful questions that do not require a yes/no response.

Watch the Pine Grove Middle School video!

35
Songer & Gotwals (2012)
36
Songer et al. (2009)
37
Black (2003, April)

Innovation Spotlights 16
 9. Evidence-based Models. Students use technology to
develop models based on data and evidence.

Developing and testing evidence-based another technology tool to draw a model of

models is a core practice that scientists use how odor travels to the nose when learning
when investigating theories about the natural about the movement of gasses and liquids.
world. Engineers likewise rely on evidence- Students revise their model multiple times
based models when they design systems and during an eight-week unit, making adjustments
solutions that solve practical problems.38 To to the representation as they gain knowledge
develop a deep understanding of the practices through inquiry activities where they explore
that STEM professional use, students need the properties of gasses. As the unit progresses
to develop their own models based on data and students’ knowledge increases, their
and evidence and the capacity to explain models gradually become more consistent
the evidence base and limitations of those with the full scientific model. STEM instruction
models.39 A clear example of how middle that fuses science content and practices in this
school students might use technology for way supports students in developing necessary
developing and revising models is described skills to understand and engage in scientific
by Krajcik and Merritt (2012). Sixth-grade inquiry process.
students, for example, can use tablets or

Cedars International Next Generation High School

Cedar International’s Next Generation High School
Quick Facts
in Austin, Texas, embeds the development of
evidence-based models into its project-based Location: Austin, TX (city, large)
learning curriculum. Teachers often engage FRPL: 48% Asian: 5%
students with authentic design challenges, such as
Black 20% White: 15%
how to add additional space to the school building,
a real and immediate problem facing the school. Hispanic: 57% Other: 2%
Students start by researching the spaces in the
school, using abstract measuring tools and algebra to measure the dimensions of each of
the rooms. Students use the data measurements to create a model for the existing school
and then apply their algebraic skills to create another model for a new school design with
more usable space. Students create drafts of floor plans and share them with a graphic
design team. Together, both teams collaborate by asking questions back and forth about
the specifics of the floor plan. When doing this project in the past, students used their
hand sketches as models, but those models conveyed limited information. When students
started using digital technology to generate models, their products became much more
compelling, detailed, and articulate.

Watch the Cedars International Next Generation High School video!

38
National Academy of Engineering and National Research Council (2014)
39
Krajcik & Merritt (2012)

Innovation Spotlights 17
Conclusion
Technology plays a central role in STEM professions and our
everyday lives. So, while a strong STEM education supported by
technology is valuable for young people who intend STEM careers, it
is essential that everyone has access to STEM learning opportunities
enhanced by technology that prepare them for life and work in
today’s complex world.
Numerous opportunities for STEM learning with technology exist
in both formal and informal educational experiences at all levels of
our education system. However, as in all fields of study, technology
alone is not sufficient to make a difference in STEM. For powerful
learning to occur, teachers must be able to make instructional
decisions based on evidence that a particular approach will improve
outcomes.
A systematic review of the research literature on the impact of
technology in STEM learning, summarized in this report, identified
nine dimensions practices that result in powerful learning. The nine
dimensions, with examples of ways in which they are being used in
classrooms across the country, can help teachers make effective
decisions about how to support STEM learning with technology now
and in the future.

Innovation Spotlights 18
References
Note: Below is the complete list of references that informed the development of the nine dimensions. Only those
cited in the report include footnotes.

Abdulkadir, K., & Ercan, A. (2015). The effect of web-based instruction on elementary students’
academic achievement. Education and Science, 40(177), 217-231.

Abramovich, S., Schunn, C. & Higashi, R. M. (2013). Are badges useful in education? It depends
upon the type of badge and expertise of learner. Education Technology Research &
Development 61(2), 217-232. doi.org/10.1007/s11423-013-9289-2

22
Ault, M., Craig-Hare, J., Frey, B., Ellis, J. D., & Bulgren, J. (2015). The effectiveness of Reason
Racer, a game designed to engage middle school students in scientific argumentation. Journal
of Research on Technology in Education, 47(1), 21-40. doi.org/10.1080/15391523.2015.967542

27
Basu, S., Dukeman, A., Kinnebrew, J., Biswas, G., & Sengupta, P. (2014). Investigating student
generated computational models of science. In Proceedings of the 11th International
Conference of the Learning Sciences (pp. 1097-1101). Boulder, CO.

Belland, B. R., Glazewski, K. D., & Richardson, J. C. (2011). Problem-based learning and
argumentation: Testing a scaffolding framework to support middle school students’ creation
of evidence-based arguments. Instructional Science, 39(5), 667-694. doi.org/10.1007/s11251-
010-9148-z

19
Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science
Education, 93(1), 26-55. https://doi.org/10.1002/sce.20286

20
Berland, L., & Reiser, B.J. (2010). Classroom communities’ adaptations of the practice of
argumentation. Science Education, 95(2), 191–216. https://doi.org/10.1002/sce.20420

16
Beserra, V., Nussbaum, M., Zeni, R., Rodriguez, W., & Wurman, G. (2014). Practising Arithmetic
Using Educational Video Games with an Interpersonal Computer. Journal of Educational
Technology & Society, 17(3), 343-358. Retrieved from http://www.jstor.org/stable/
jeductechsoci.17.3.343

37
Black, P. (2003, April). Formative and Summative Assessment: Can they serve learning together?
Paper presented at the American Educational Research Association, Chicago, IL.

34
Blumenfeld, P. C., Soloway, E., Marx, R. W., Krajcik, J. S., Guzdial, M., & Palincsar, A. (1991).
Motivating project-based learning: Sustaining the doing, Supporting the learning. Educational
Psychologist, 26(3/4), 369–398.

Chen, C.-H., & Chiu, C.-H. (2016). Collaboration scripts for enhancing metacognitive self-
regulation and mathematics literacy. International Journal of Science & Mathematics
Education, 14(2), 263-280. doi.org/10.1007/s10763-015-9681-y

Innovation Spotlights 19
Chen, C.-H., & Hwang, G.-J. (2017). Effects of the team competition-based ubiquitous gaming
approach on students’ interactive patterns, collective efficacy and awareness of collaboration
and communication. Journal of Educational Technology & Society, 20(1), 87–98. https://www.
jstor.org/stable/jeductechsoci.20.1.87

Chen, C.-H., Liu, G.-Z., & Hwang, G.-J. (2016). Interaction between gaming and multistage guiding
strategies on students’ field trip mobile learning performance and motivation. British Journal
of Educational Technology, 47(6), 1032–1050. doi.org/10.1111/bjet.12270

Chu, H.-C., & Chang, S.-C. (2014). Developing an educational computer game for migratory
bird identification based on a two-tier test approach. Educational Technology Research &
Development, 62(2), 147–161. doi.org/10.1007/s11423-013-9323-4

18
Chu, Y., Yang, H., Tseng, S., & Yang, C. (2014). Implementation of a model-tracing-based learning
diagnosis system to promote elementary students’ learning in mathematics. Journal of
Educational Technology & Society, 17(2), 347-357. Retrieved from http://www.jstor.org/stable/
jeductechsoci.17.2.347

Colston, N., Thomas, J., Ley, M. T., Ivey, T., & Utley, J. (2017). Collaborating for early-age career
awareness: A comparison of three instructional formats. Journal of Engineering Education,
106(2), 326–344. doi.org/10.1002/jee.20166

8
Dillenbourg, P., (1999). Collaborative learning: Cognitive and computational approaches:
Advances in learning and instruction series. New York, NY: Elsevier Science

6
D’Angelo, C., Rutstein, D., Harris, C., Bernard, R., Borokhovski, E., & Haertel, G. (2014,
March). Simulations for STEM learning: Systematic review and meta-analysis.
Menlo Park, CA: SRI International. Available from https://pdfs.semanticscholar.
org/6bf2/15af93f56403e2fe8cb8affe798d65b1142d.pdf

Fan, S.-C., & Yu, K.-C. (2017). How an integrative STEM curriculum can benefit students in
engineering design practices. International Journal of Technology and Design Education, 27(1),
107-129. doi.org/10.1007/s10798-015-9328-x

24
Feder, M. A., Pearson, G., Katehi, L., & Committee on K-12 Engineering Education. (2009).
Engineering in K-12 Education: Understanding the Status and Improving the Prospects.
Washington, DC: National Academies Press. Retrieved from https://search-ebscohost-com.
ezproxy.snhu.edu/login.aspx?direct=true&db=nlebk&AN=414575&site=eds-live&scope=site

Feng, C.Y., & Chen, M.P. (2014). The effects of goal specificity and scaffolding on programming
performance and self-regulation in game design. British Journal of Educational Technology,
45(2), 285–302. doi.org/10.1111/bjet.12022

Fesakis, G. S., Christina; Mavroudi, Elisavet. (2011). Using the Internet for communicative learning
activities in Kindergarten: The case of the ‘Shapes Planet’. Early Childhood Education Journal,
38(5), 385-392. doi.org/10.1007/s10643-010-0422-0

Innovation Spotlights 20
21
Ford, M. J. (2008). Disciplinary authority and accountability in scientific practice and learning.
Science Education, 92(3), 404-423.

32
Frykholm, J., & Glasson, G. (2005). Connecting science and mathematics instruction:
Pedagogical context knowledge for teachers. School Science and Mathematics, 105(3), 127-
141.

Furner, J. M., & Kuma, D. D. (2007). The mathematics and science integration argument: A stand
for teacher education. Eurasia Journal of Mathematics, Science & Technology Education, 2007,
3(3), 185-189

Gómez, F., Nussbaum, M., Weitz, J., Lopez, X., Mena, J., & Torres, A. (2013). Co-located single
display collaborative learning for early childhood education. International Journal of
Computer-Supported Collaborative Learning, 8(2), 225–244. doi.org/10.1007/s11412-013-
9168-1

González-Calero, J., Antonio; Arnau, D., Puig, L., & Arevalillo-Herráez, M. (2015). Intensive
scaffolding in an intelligent tutoring system for the learning of algebraic word problem
solving. British Journal of Educational Technology, 46(6), 1189-1200. doi.org/10.1111/
bjet.12183

Gómez, F., Nussbaum, M., Weitz, J., Lopez, X., Mena, J., & Torres, A. (2013). Co-located single
display collaborative learning for early childhood education. International Journal of
Computer-Supported Collaborative Learning, 8(2), 225-244. doi.org/10.1007/s11412-013-
9168-1

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and
associated methodologies. Health Information and Libraries Journal, 26(2), 91-108. doi.
org/10.1111/j.1471-1842.2009.00848.x

28
Grover, S., & Pea, R. (2013). Computational Thinking in K–12: A Review of the State of the Field.
Educational Researcher, 42(1), 38–43.

25
Guzey, S.S., Harwell, M., Moreno, M., Peralta, Y., & Moore, T.J. (2017). The impact of design-based
STEM integration curricula on student achievement in engineering, science, and mathematics.
Journal of Science Education and Technology, 26(2), 207–222. doi.org/10.1007/s10956-016-
9673-x

Halpern, D., Aronson, J., Reimer, N., Simpkins, S., Star, J., & Wentzel, K. (2007). Encouraging Girls
in Math and Science (NCER 2007-2003). Washington, DC: National Center for Education
Research, Institute of Education Sciences, U.S. Department of Education. Retrieved from
http://ncer.ed.gov

Hattie, J., & Gan, M. (2011). Instruction based on feedback. In R. Mayer & P. Alexander (Eds.),
11

Handbook of research on learning and instruction (pp. 249-271). New York: Routledge

13
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1),
81–112. https://doi.org/10.3102/003465430298487

Innovation Spotlights 21
5
Hegedus, S., Dalton, S. & Tapper, J. (2015). The impact of technology-enhanced curriculum on
learning advanced algebra in US high school classrooms. Educational Technology Research &
Development, 63(2), 203-228. doi.org/10.1007/s11423-015-9371-z

Hsiao, H.-S., Lin, C.-C., Feng, R.-T., & Li., K.L. (2010). Location based Services for outdoor
ecological learning system: Design and implementation. Journal of Educational Technology &
Society, 13(4), 98–111. https://www.jstor.org/stable/jeductechsoci.13.4.98

4
Huang, K., Ge., X. & Eseryel, D. (2017). Metaconceptually-enhanced simulation-based inquiry:
effects on eighth grade students’ conceptual change and science epistemic beliefs.
Educational Technology Research & Development, 65(1), 75-100. doi.org/10.1007/s11423-
016-9462-5

10
Hwang, G., Shi, Y., & Chu, H. (2011). A concept map approach to developing collaborative
Mindtools for context-aware ubiquitous learning. British Journal of Educational Technology,
42(5), 778-789. doi.org/10.1111/j.1467-8535.2010.01102.x

Hung, C.-M., Hwang, G.-J., & Huang, I. (2012). A project-based digital storytelling approach
for improving students’ learning motivation, problem-solving competence and learning
achievement. Journal of Educational Technology & Society, 15(4), 368-379. Retrieved from
http://www.jstor.org/stable/jeductechsoci.15.4.368

Hwang, G.-J., & Chang, S.-C. (2016). Effects of a peer competition-based mobile learning
approach on students’ affective domain exhibition in social studies courses. British Journal of
Educational Technology, 47(6), 1217–1231. doi.org/10.1111/bjet.12303

Hwang, G.-J., & Lai, C.-L. (2017). Facilitating and bridging out-of-class and in-class learning: An
interactive e-book-based flipped learning approach for math courses. Journal of Educational
Technology & Society, 20(1), 184–197. https://www.jstor.org/stable/jeductechsoci.20.1.184

Kiger, D., Herro, D., & Prunty, D. (2012). Examining the influence of a mobile learning intervention
on third grade math achievement. Journal of Research on Technology in Education, 45(1), 61-
82. doi.org/10.1080/15391523.2012.10782597

Kim, P., Suh, E. & Song, D. (2015). Development of a design-based learning curriculum through
design-based research for a technology-enabled science classroom. Educational Technology
Research and Development, 63(4), 575-602. doi.org/10.1007/s11423-015-9376-7

14
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance:
A historical review, a meta-analysis, and a preliminary feedback intervention theory.
Psychological Bulletin, 119(2), 254–284.

Kollöffel, B., & de Jong, T. (2013). Conceptual understanding of electrical circuits in secondary
vocational engineering education: Combining traditional instruction with inquiry learning in a
virtual lab. Journal of Engineering Education, 102(3), 375-393. doi.org/10.1002/jee.20022

Innovation Spotlights 22
Kostaris, C., Sergis, S., Sampson, D. G., Giannakos, M. N., & Pelliccione, L. (2017). Investigating the
potential of the flipped classroom model in K-12 ICT teaching and learning: An action research
study. Journal of Educational Technology & Society, 20(1), 261–273. https://www.jstor.org/
stable/jeductechsoci.20.1.261

33
Krajcik, J., & Blumenfeld, P. (2006). Project-Based Learning. In R. Sawyer (Ed.), The Cambridge
Handbook of the Learning Sciences (pp. 317-334). Cambridge: Cambridge University Press.

39
Krajcik, J., & Merritt, J. (2012). Engaging students in scientific practices: What does constructing
and revising models look like in the science classroom? Understanding A Framework for K−12
Science Education. The Science Teacher, 79(3), 38-4. Available at https://search.proquest.
com/openview/4f29da8460ffb46a119972b904b8a7f6/1?pq-origsite=gscholar&cbl=40590

15
Kulik, J. A., & Kulik, C.-L. C. (1988). Timing of Feedback and Verbal Learning. Review of
Educational Research, 58(1), 79–97. doi.org/10.3102/00346543058001079

Laisney, P., & Brandt-Pomares, P. (2015). Role of graphics tools in the learning design process.
International Journal of Technology & Design Education, 25(1), 109–119. doi.org/10.1007/
s10798-014-9267-y

9
Lan, Y. J., Sung, Y. T., Tan, N.C., Lin, C. P., & Chang, K. E. (2010). Mobile-device-supported
problem-based computational estimation instruction for elementary school students. Journal
of Educational Technology & Society, Special Issue: Innovations in Designing Mobile Learning
Applications, 13(3), 55-69. Retrieved from http://www.jstor.org/stable/jeductechsoci.13.3.55

12
Locke, E.A., & Latham, G.P. (1990). A theory of goal setting and task performance. Englewood
Cliffs, NJ: Prentice-Hall.

Mainali, B., & Heck, A. (2017). Comparison of traditional instruction on reflection and Rotation in a
Nepalese high school with an ICT-rich, student-centered, investigative approach. International
Journal of Science & Mathematics Education, 15(3), 487-507. doi.org/10.1007/s10763-015-
9701-y

Miller, D.J., & Robertson, D.P. (2011). Educational benefits of using game consoles in a primary
classroom: A randomised controlled trial. British Journal of Educational Technology, 42(5),
850-864.

17
Nadler, D. (1979). The effects of feedback on task group behavior: A review of the experimental
research. Organizational Behavior and Human Performance, 23, 309–338.

23
National Academy of Engineering (2008). Changing the Conversation: Messages for Improving
Public Understanding of Engineering. Washington, D.C.: The National Academies Press.

38
National Academy of Engineering and National Research Council. (2014). STEM Integration in
K-12 Education: Status, Prospects, and an Agenda for Research. Washington, DC: The National
Academies Press. doi.org/10.17226/18612

Innovation Spotlights 23
National Academies of Sciences, Engineering, and Medicine. (2018). How People Learn II:
Learners, Contexts, and Cultures. Washington, DC: The National Academies Press. doi.
org/10.17226/24783

National Research Council. (2015). Guide to Implementing the Next Generation Science
Standards. Washington, DC: The National Academies Press. doi.org/10.17226/18802

3
National Research Council. (2011). Learning Science Through Computer Games and Simulations.
Washington, DC: The National Academies Press. doi.org/10.17226/13078

31
Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O.S., Hubert, T., & Almughyirah, S. (2016).
Using robotics and game design to enhance children’s self-efficacy, STEM attitudes, and
computational thinking skills. Journal of Science Education and Technology, 25(6), 860–876.
doi.org/10.1007/s10956-016-9628-2

White House Office of Science and Technology Policy (2018, December). Charting a Course
1

for Success: America’s Strategy for STEM Education. A Report by the Committee on STEM
Education of the National Science & Technology Council. Available from https://www.
whitehouse.gov/wp-content/uploads/2018/12/STEM-Education-Strategic-Plan-2018.pdf

Pellas, N., Kazanidis, I., Konstantinou, N., & Georgiou, G. (2017). Exploring the educational
potential of three-dimensional multi-user virtual worlds for STEM education: A mixed-method
systematic literature review. Education and Information Technologies, 22(5), 2235-2279. doi.
org/10.1007/s10639-016-9537-2

30
Psycharis, S. & Kallia, M. (2017). The effects of computer programming on high school students’
reasoning skills and mathematical self-efficacy and problem solving. Instructional Science, 45,
583-602. doi.org/10.1007/s11251-017-9421-5

Reeves, J. L., Gunter, G. A., & Lacey, C. (2017). Mobile learning in pre-Kindergarten: Using student
feedback to inform practice. Educational Technology & Society, 20(1), 37–44. https://www.
jstor.org/stable/jeductechsoci.20.1.37

Rihtaršič, D. A., Stanislav; Kocijancic, Slavko. (2016). Experiential learning of electronics subject
matter in middle school robotics courses. International Journal of Technology & Design
Education, 26(2), 205-224. doi.org/10.1007/s10798-015-9310-7

7
Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative
problem solving. In C. E. O’Malley (Ed.), Computer-Supported Collaborative Learning (pp. 69-
197). Berlin: Springer-Verlag.

Rummel, N., Mullins, D., & Spada, H. (2012). Scripted collaborative learning with the cognitive
tutor algebra. International Journal of Computer-Supported Collaborative Learning, 7(2), 307-
339. doi.org/10.1007/s11412-012-9146-z

Innovation Spotlights 24
She, H.-C., & Liao, Y.-W. (2010). Bridging scientific reasoning and conceptual change through
adaptive web-based learning. Journal of Research in Science Teaching, 47(1), 91–119. doi.
org/10.1002/tea.20309

Shih, S. C., Kuo, B. C., & Liu, Y. L. (2012). Adaptively ubiquitous learning in campus math path.
Journal of Educational Technology & Society, 15(2), 298-308. Retrieved from http://www.
jstor.org/stable/jeductechsoci.15.2.298

Shin, N., Sutherland, L. M., Norris, C. A. and Soloway, E. (2012), Effects of game technology on
elementary student learning in mathematics. British Journal of Educational Technology, 43(4),
540-560. doi.org/10.1111/j.1467-8535.2011.01197.x

Song, Y., & Kapur, M. (2017). How to flip the classroom – “Productive Failure or Traditional Flipped
Classroom” pedagogical design? Journal of Educational Technology & Society, 20(1), 292-
305. Retrieved from http://www.jstor.org/stable/jeductechsoci.20.1.292

Songer, N.B. & Gotwals, A.W. (2012). Guiding explanation construction by children at the entry
35

points of learning progressions. Journal for Research in Science Teaching, 49, 141-165.

36
Songer, N. B., Kelcey, B., & Gotwals, A. W. (2009). How and when does complex reasoning
occur? Empirically driven development of a learning progression focused on complex
reasoning about biodiversity. Journal of Research in Science Teaching, 46(6), 610–631. doi.
org/10.1002/tea.20313

Terrazas-Arellanes, F.E., Gallard M., A.J., Strycker, L.A., & Walden, E.D. (2018). Impact of interactive
online units on learning science among students with learning disabilities and English
learners. International Journal of Science Education, 40(5), 498-518. doi.org/10.1080/095006
93.2018.1432915

Tezcan, Gamze, T., Hülya, G., & Egitim ve Bilim, A. (2017). The effects of 4MAT teaching model
and whole brain model on academic achievement in science. Education and Science, 42(192),
303-325.

U.S. Department of Education. (2008). National Educational Technology Trends Study: Local-level
Data Summary. Washington, DC: Office of Planning, Evaluation and Policy Development,
Policy and Program Studies Service. Available from www.ed.gov/about/offices/list/opepd/
ppss/reports.html

U.S. Department of Education. (2009). Evaluation of the Enhancing Education Through

Technology Program: Final Report. Washington, DC: Office of Planning, Evaluation and Policy
Development, Policy and Program Studies Service. Available from www.ed.gov/about/offices/
list/opepd/ppss/reports.html

U.S. Department of Education. (2010). Evaluation of Evidence-Based Practices in Online Learning:

A Meta-Analysis and Review of Online Learning Studies. Washington, DC: Office of Planning,
Evaluation, and Policy Development. Available from www.ed.gov/about/offices/list/opepd/
ppss/reports.html

Innovation Spotlights 25
U.S. Department of Education. (2012, May). Technology Enhanced Elementary and Middle School
Science (TEEMSS): What Works Clearinghouse (WWC) Intervention Report. Institute of
Education Sciences. Retrieved from http://whatworks.ed.gov

U.S. Department of Education. (2014). National Assessment of Educational Progress (NAEP), 2014
Technology and Engineering Literacy (TEL) Assessment. Institute of Education Sciences,
National Center for Education Statistics. Retrieved from https://www.nationsreportcard.gov/
tel_2014/

Webb, A., & Rule, A. (2014). Effects of teacher lesson introduction on second graders’ creativity in
a science/literacy integrated unit on health and nutrition. Early Childhood Education Journal,
42(5), 351–360. doi.org/10.1007/s10643-013-0615-4

Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining
computational thinking for mathematics and science classrooms. Journal of Science
Education and Technology, 25(1), 127-147. doi.org/10.1007/s10956-015-9581-5

Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical

29

Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences,

366(1881), 3717–3725.

Yang, D.C., & Tsai, Y. F. (2010). Promoting sixth graders’ number sense and learning attitudes via
technology-based environment. Journal of Educational Technology & Society, 13(4), 112-125.
https://eric.ed.gov/?id=EJ909942

Yang, E. F. Y., Chang, B., Cheng, H. N. H., & Chan, T.-W. (2016). Improving pupils’ mathematical
communication abilities through computer-supported reciprocal peer tutoring. Journal of
Educational Technology & Society, 19(3), 157-169. Retrieved from http://www.jstor.org/stable/
jeductechsoci.19.3.157

Zaidi, S. Z. (2016). A novel approach to fostering Next Generation Science Knowledge in middle
school students: Introducing double-blinded reviews in classroom formative assessments
(Unpublished doctoral dissertation). The University of Michigan, Ann Arbor, MI. Available from
http://hdl.handle.net/2027.42/120773

Zaranis, N. & Synodi, E. (2017). A comparative study on the effectiveness of the computer assisted
method and the interactionist approach to teaching geometry shapes to young children.
Education and Information Technologies, 22(4), 1377-1393. doi.org/10.1007/s10639-016-
9500-2

Innovation Spotlights 26
Appendix: Methodology
When reviewing the scholarly literature, we focused our efforts on research that uses an
experimental or quasi-experimental design to test the impacts of an intervention (e.g., a curricular
unit or instructional approach) on student learning outcomes. We also included studies that
were suggested by recognized experts in the field of STEM education.40 A number of the Pathway
Objects described in the Charting a Course for Success report further supported the STEM
dimensions. These include “Make computational thinking an integral element of all education”
(Dimension 6), “Encourage transdisciplinary learning” (Dimension 7), and “Expand digital platforms
for teaching and learning” (Dimensions 1-9).

Identification of relevant studies

In our search we consulted the EBSCO, ERIC, and ProQuest databases along with the search
engine Google Scholar. The Stanford University librarian recommended these databases because
they are the largest resource collections relevant to education. Scholarly peer-reviewed work
and grey literature (e.g., conference proceedings, dissertations and theses, technical reports,
white papers, etc.) that were published in English between 2010-2018 and listed “education” as
the subject (keyword) were included in the search. Additional search criteria included one or
more keywords from each of the categories listed in Table 1, below. We also examined resources
produced by the Department of Education’s Policy and Program Studies Service (PPSS), the
Institute for Education Sciences (IES) and sources cited by key STEM policy documents. For PPSS,
every resource within the “Educational Technology” subsection within the “K-12 Instructional and
Improvement Strategies” section was reviewed for relevance. For IES, every resource from 2010-
2018 that contained “technology” in the title was reviewed for relevance.

40
We are grateful to the individuals who supported this work by reviewing and providing guidance on the nine STEM dimensions
and/or provided suggestions for potential spotlight schools: Nancy Butler-Songer (Drexel University), Jennifer Corn (NC State
University), Bill Penuel (University of Colorado, Boulder) and Joseph Krajcik (Michigan State University).

Innovation Spotlights 27
Table 1. Search keyword list

School level Subject area* Digital technology

• High school • STEM • Educational technology

• Secondary • Science • Digital learning/technology
• Middle school • Match • Simulation/animation
• Elementary • Mathematics • Digital games
• Kindergarten • Engineering • Immersive environments
• First grade, second • Computer science • Probeware
grade… • Computational thinking • Models or modeling
• Grade 1, grade 2, … • Technology • 2D or 3D design software
• 3D printer
• Programming
• Virtual labs/reality/field trips
• Video, video editing
• Podcast, audio recording
• Makerspace
• Mobile devices
• Applications/apps
• AI or Augmented reality
• Engineering technology/
design

*Among studies identified with these keywords, we will prioritize those that include the phrase
“project-based learning,” “problem-based learning” and/or “inquiry-based learning.”

Coding of eligible studies

Studies that were coded for the research review were first screened according to established
criteria. To be eligible for inclusion a study needed to:

• Examine students’ learning or mastery of STEM content or practices through using/

integrating one or more digital technologies in K-12 STEM and/or computer science
classroom contexts;

• Report learning outcomes in STEM and/or computer science using an objective measure
(see Table 2); and

• Provide enough statistical information to support a judgment of whether or not the study
would meet WWC standards with or without reservations.41

Studies that would not meet WWC standards were not discarded, but were flagged. Each study was coded as “Appears to meet
41

WWC standards,” “Appears to meet WWC standards with reservations,” or “Does not meet WWC standards.”

Innovation Spotlights 28
Studies that were retained for the review after screening were coded using a study review
template that was designed for this purpose. When selecting studies to include in the review,
attention was paid to the following:

• Inclusion of studies that represent a broad range of student subgroups including rural,
urban, minorities, girls, and historically underrepresented groups in STEM, etc.;

• Inclusion of studies with positive/successful outcomes when reviewing studies with an

objective assessment; and

• Studies where technology was used to directly support students’ development of STEM/
computer science knowledge or practices (see Table 2 for general categories).

Table 2. Descriptions of STEM/computer science learning outcomes

STEM dimension Sample outcomes

STEM/computer • Subject-specific knowledge and concepts

science knowledge (disciplinary core ideas and crosscutting concepts)
(includes mastery and • STEM/computer science interdisciplinary
competency-based knowledge and concepts
learning)
• Computational thinking
• Research-based STEM/computer science
STEM/computer practices, for example:
science practices
• Designing and conducting scientific experiments
• Designing solutions through engineering or
programming
• Quantitative and/or abstract reasoning
• Analyzing and interpreting data; arguing from
evidence
• Appropriate use of STEM/computer science tools
• Effectively communicating STEM information
• Modelling STEM phenomena

Innovation Spotlights 29
Results from study search
Our database and Google Scholar search yielded 264 articles (see Figure 1, below). The PPSS, IES,
and STEM policy document review added four references. A review of the studies based on our
eligibility criteria narrowed the research corpus to 64 studies that tested the effectiveness of a
digital intervention in supporting an objectively measures STEM learning outcome. These studies
were reviewed and discussed by the team to derive a tentative set of dimensions for the rubric.

Figure 1. Study search process for research synthesis

Studies identified Studies identified
through EBSCO through ProQuest
(n = 157) (n = 101)

Studies remaining after

duplicates removed
(n = 243)

Studies added from PPSS (7) and IES (17)

(n = 267)

Studies remaining after Reasons for excluding papers:

abstract screening
• Full-text paper not available
(n = 96)
• Not K-12
• Intervention not digital
• Comparison condition not described
Studies remaining after • Did not measure learning
full-text screening • Learning outcomes were student- or
(n = 66) teacher- reported

Studies recommended by
experts (2). Total studies
informing STEM Dimensions
(n = 68)

In our review we sought to identify studies that clearly described the link between technology
use and students’ STEM learning outcomes. Ideally, a study would provide enough description of
the treatment condition to explain the role that technology played in facilitating students’ STEM
learning. These descriptions informed a number of emergent themes that ultimately became the
first seven practices for supporting powerful STEM learning with technology. Feedback received
from experts resulted in the addition of dimensions 8 and 9. Together with the spotlight videos,
the nine dimensions provide teachers and school leaders with tangible examples of effective
technology use for supporting powerful STEM learning.

Innovation Spotlights 30