District Data Toolkit

District Data Team Toolkit
Helping districts establish, grow, and

maintain a culture of inquiry and data use.
MODULES
Introduction
Getting Ready
Inquiry
Information
Knowledge
Action
Results
MODULE 0: INTRODUCTION TO THE TOOLKIT
Table of Contents
Introduction
Introduction—1
What is the Toolkit?—1
How Can the Toolkit Help Our District?—2
What’s in the Toolkit?—3
How Should the Toolkit Be Used?—3
How Do I Navigate the Toolkit?—4
Where Should Our District Begin?—4
Tools and Resources for
the Introduction
0.1.1R: Objectives for All Modules

0.1.2R: Tools and Resources for All Modules
0.2.1T: District Data Team Self-Assessment
District Data Team Toolkit—Version 1.0 Module 0: Introduction
INTRODUCTION
Welcome to the District Data Team Toolkit. This Toolkit is designed to help a
district establish, grow, and maintain a culture of inquiry and data use that can
inform decisions that impact teaching and learning, and ultimately improve the
achievement of all students. This short introduction will help you understand and
navigate the tools and resources available to support this work.
WHAT IS THE TOOLKIT?

Ensuring the effective use of inquiry and data district-wide involves many tasks
and types of expertise. For that reason, this Toolkit is designed to assist in the
establishment of a District Data Team—a cadre of staff who is collectively
responsible for the technical, organizational, and substantive aspects of data use.
These Team members must be data literate and able to lead the collaborative
inquiry process with both district and school staff. Members of the District Data
Team work with district staff and school-level data teams to:
Craft questions about accountability, equity, and continuous improvement

Coordinate the collection, analysis, and dissemination of data displays
that are necessary to address these essential questions
Build action plans
Monitor progress of improvement initiatives
These activities can help build the capacity of a District Data Team to engage in
inquiry and use data to inform district-level decisions. Over time, the Team can
engage the entire staff in using multiple data sources to continuously improve
teaching and learning throughout the district. Districts that engage with the
Toolkit should plan for a multi-year commitment to increase and embed a
capacity for effective data use.
The Toolkit is designed around a theory of action, the Data-Driven Inquiry and
Action Cycle (see diagram) on the next page, which provides a foundation for
effective data use. The Cycle provides a structure that takes data use from
asking the right questions to getting results. It is an iterative process in which the
district uses data to target and support continuous improvement. A disciplined
application of this kind of data-driven approach can build a district and school
environment that is focused on continuous improvement grounded in evidence.
This Cycle is also the basis for the ESE Education Data Warehouse trainings,
which further provide excellent tools to access and analyze data. But analyzing
data alone will not result in continuous improvement. Concrete actions that are
grounded in evidence and continually monitored through the collection and
analysis of multiple forms of data are critical to achieve improved results.
Massachusetts Department of Elementary and Secondary Education 1

HOW CAN THE TOOLKIT HELP OUR DISTRICT?

Districts and schools gather enormous amounts of data throughout the school
year. These data have historically been collected merely to comply with external
requirements. Increasingly, educators are seeking ways to use data systemically
for their own learning and to inform decisions about curriculum, instruction,
resource allocation, and other vital functions at district and school levels.
The ESE District Data Team Toolkit can help district staff:
Organize a District Data Team to facilitate productive use of data

throughout the district
Learn and practice the steps of an effective data use model
Access and apply tools and resources to support a process of inquiry
Engaging with this Toolkit can help a district identify and/or refine a focus for
improvement, including determining if current improvement efforts are having the
desired effect on student learning outcomes. For example, a district may frame
an inquiry process around one aspect of an existing District Improvement Plan as
a means to delve deeply into questions about the impact of the related initiatives.
Once a District Data Team has built its own capacity for data use and a culture of
inquiry, it will be better poised to support such efforts with principals, teachers,
and other stakeholders in the district.
2 Massachusetts Department of Elementary and Secondary Education

WHAT’S IN THE TOOLKIT?

The Toolkit has seven modules – this Introduction, and six modules aligned to
the Data-Driven Inquiry and Action Cycle. Each module contains objectives,
detailed information, tools, and resources to help implement the work.
1. Getting Ready module provides guidance on forming a District Data Team

and taking steps to build a solid foundation for building a culture of inquiry
and systemic data use. The module addresses the need for systems to
inventory, collect, and disseminate the data. It also has information to
promote assessment literacy and help manage change.
2. Inquiry module launches a process of inquiry by identifying high-level

questions of importance to the district, generating clarifying questions to
focus the inquiry, and identifying data needed to answer them. The
module includes guidance for effective data displays and data overviews.
3. Information module guides the process of analyzing the data identified in

Inquiry, first making factual observations about the data and then
generating inferences about what the data mean.
4. Knowledge module helps place information (analyzed quality data) into

the context of research and practice to accurately define the problem and
identify possible solutions.
5. Action module provides frameworks for putting new knowledge to work by

developing a logic model and articulating clear measures that will guide
and focus the work, and then by creating an action plan (if necessary).
If the information and knowledge gained from quality data analysis are not
acted on, data collection efforts are wasted and improvement won’t occur!
6. Results module shares methods for monitoring the work, evaluating the
impact, making mid-course corrections if necessary, and communicating
the outcomes with stakeholders.
HOW SHOULD THE TOOLKIT BE USED?

The Toolkit is designed for district-level staff, to promote the skills and knowledge
necessary to build their capacity to effectively use inquiry and data to inform
district-level decisions. As a District Data Team gains comfort with the tools,
resources, and processes in the Toolkit, it can plan ways to share them with
school-and teacher-level data teams.
Each of the modules provides specific tools and activities to implement the steps
of the inquiry process. Some tools are best used electronically. It is important to
understand, however, that superimposing a process does not necessarily yield a
positive result. A district must be mindful of doing what it can to embed a culture
of inquiry and data use that goes beyond technical compliance with processes.

HOW DO I NAVIGATE THE TOOLKIT?

No matter the entry point taken, the Toolkit is designed to support ready access
to tools and resources as a District Data Team needs them, using a numbering
system to make finding the right tool quick and efficient.
3.2.1T
Tool or Resource
Module Activity Number
When an activity has multiple tools or resources associated with it, they are
numbered sequentially and designated with either a T (tool) or an R (resource).
Tools are templates, protocols, organizers, or other items that the Team
will work with to build its knowledge and expertise.
Resources provide further reading, documentation, or guidance to help

the Team use the tools or expand its understanding.
Activity 0.1 Overview of Toolkit Contents

These documents summarize elements of the Toolkit in order to assist a
district in engaging with ones that will be most useful to its work.
(0.1.1R: Objectives for All Modules)
(0.1.2R: Tools and Resources for All Modules)
WHERE SHOULD OUR DISTRICT BEGIN?

Districts should begin by engaging with the District Data Team Self-Assessment,
at the end of this Introduction module, which will help determine the district’s
strengths and areas of need in regards to data use and a culture of inquiry. The
Self-Assessment aligns with each of the modules and therefore can help the
district identify where to begin engaging with the Toolkit – which modules, tools,
and resources would most help the district move forward with its work.
If in doubt, a district might gain the most value from starting in Module 1: Getting
Ready and working through the Toolkit sequentially, committing to a multi-year
process of building robust use of data at the district level. If the District Data
Team has been in existence for several years and needs to work on refining
processes and policies that support data use, it may find it useful to go directly to
certain tools in the Toolkit.
Activity 0.2 District Data Team Self-Assessment
This self-assessment can help a district determine its strengths and needs,
and how best to use this Toolkit to support inquiry and data use.
(0.2.1T: District Data Team Self-Assessment)

Many thanks to all the individuals who contributed to the creation of this Toolkit,
including a number of personnel from:
Public Consulting Group (PCG);
MA ESE Offices of Urban District Assistance, School Improvement Grants
Management, Information Technology, Data Collection, and the Center for
Curriculum and Instruction; and
Brockton, Chelsea, Fitchburg, Holyoke, and Lynn Public School Districts.
For more information on this and other district support resources, or to share
feedback on this tool, visit http://www.doe.mass.edu/sda/ucd/ or email
districtassist@doe.mass.edu.

O B J E C T I V E S F O R A L L M O D U L E S 0 . 1 . 1 R
Purpose To provide an overview of the objectives for all Related Documents

modules in the District Data Team Toolkit. 0–Introduction
0.1.2R: Tools and
Description In conjunction with the Introduction and Self-
Resources for All
Assessment, this summary can help a District Data
Modules
Team identify the parts of the Toolkit that might be
most useful to its work.
Time N/A
Module 1 (Getting Ready) will help a district:

 Set the vision for data use across the district
 Build a culture of inquiry to promote systemic data use
 Establish a District Data Team to drive this work
 Build data literacy
 Understand types of data that inform inquiry
 Establish systems and policies to inventory, collect, and disseminate data
 Manage the change process
Module 2 (Inquiry) will help a District Data Team use the above roles and vision to:
 Formulate questions to drive an inquiry process
 Create and present effective data displays and data overviews
 Identify the data needed to answer the questions
Module 3 (Information) will help a District Data Team use the above questions and data to:
 Collect and organize data relevant to the inquiry process
 Distinguish between observations and inferences
 Make inferences from multiple sources of data
Module 4 (Knowledge) will help a District Data Team use the inferences generated above to:
 Clearly articulate a problem statement
 Identify and explore root causes of the problem
 Cross-reference solutions with research and local knowledge
 Begin to capture information on the district’s improvement efforts
Module 5 (Action) will help a District Data Team use the knowledge generated above to:
 Craft a logic model or theory of action to guide subsequent action and evaluation
 Articulate meaningful measures of implementation and change
 Develop action plans, if necessary, to implement new strategies or to implement existing
strategies more effectively
Module 6 (Results) will help a District Data Team use the action plan generated above to:
 Decide what to evaluate
 Develop an evaluation plan
 Analyze evaluation data
 Identify and develop a communication strategy
 Continue the process of inquiry
0.1.1R: Objectives for All Modules—Version 1.0 1/1

TOOLS AND RESOURCES FOR ALL MODULES 0.1.2R
Purpose To provide an overview of the tools and resources Related Documents

available in all modules of the District Data Team 0–Introduction
Toolkit. 0.1.1R: Objectives for All
Modules
Description In conjunction with the Introduction and Self-
Assessment, this list of tools and resources can help
a District Data Team identify the parts of the Toolkit
that might be most useful to its work.
Time N/A
Module 0: Introduction
0.1.1R: Objectives for All Modules
0.1.2R: Tools and Resources for All Modules
0.2.1T: District Data Team Self-Assessment
Module 1: Getting Ready (Organize for Success)

1.1.1T: Functions of a District Data Team 1.5.3R: ESE Data Resources
1.2.1T: Barriers to Effective Data Use 1.6.1T: Data Collection Self-Assessment
1.3.1T: Vision for Data Use 1.7.1T: Data Dissemination Schedule Template
1.4.1T: Norm Setting Protocol 1.7.2R: Data Dissemination Schedule Example
1.4.2T: Data Team Meeting Agenda 1.7.3R: ESE Policies for Data Access
1.4.3T: Data Team Meeting Minutes 1.8.1T: Data Literacy Training Catalog
1.5.1T: Data Inventory Template 1.8.2R: Assessment Glossary
1.5.2T: Data Inventory Template: SIMS and 1.9.1T: Managing Change and Understanding
EPIMS Data Concerns Protocol
Module 2: Inquiry (Ask the Right Questions)

2.1.1T: Question Formulation Protocol 2.4.2R: Data Display Rubric
2.2.1T: Inventory of District and School Initiatives 2.4.3R: Types of Data Displays
2.3.1T: Data Overview Checklist 2.4.4R: More Data Display Resources
2.3.2R: Data Overview Example 2.5.1T: Data Overview Brainstorming Protocol
2.4.1T: Building Data Displays Protocol 2.5.2T: Focusing Question Investigation Template
Module 3: Information (Collect and Analyze Data)

3.1.1T: Data Collection Planning Tool Also revisit
3.2.1T: Practice Making Valid Inferences 2.4.1T: Building Data Displays Protocol
3.3.1T: Data Analysis Protocol 2.4.2R: Data Display Rubric
2.4.3R: Types of Data Displays
2.4.4R: More Data Display Resources
0.1.2R: Tools and Resources for All Modules—Version 1.0 1/2

Module 4: Knowledge (Make Meaning of Data)
4.1.1T: Writing a Problem Statement
4.2.5T: Dimensions Bulls-Eye
4.2.1T: Why, Why, Why?

4.3.1T: Problem Investigation Plan
4.2.2T: 20 Reasons
4.3.2R: Educational Research Websites
4.2.3T: Fishbone Analysis

4.4.1T: Problem Catalogue Template
4.2.4T: Graphic Representation
Module 5: Action (Apply Meaning)

5.1.1T: Logic Model Template
5.2.2R: Elements of a Well-Written Measure
5.1.2T: Logic Model Checklist

5.3.1T: Action Plan Template
5.2.1T: Crafting Meaningful Measures Checklist

5.3.2T: Action Plan Checklist
Module 6: Results (Look for Improvement)

6.1.1T: Evaluation Plan Template
6.2.2T: Building Data Walls
6.2.1T: Communication Organizer

6.2.3T: Evaluation Report Template
For more information on this and other district support resources, or to share feedback on these tools,
visit http://www.doe.mass.edu/sda/ucd/ or email districtassist@doe.mass.edu.
0.1.2R: Tools and Resources for All Modules—Version 1.0

2/2
District Data Team Self-Assessment 0.2.1T
MA ESE District Data Team Toolkit
District Data Team Self-Assessment 1

District Data Team Self-Assessment
Background: This tool is designed to give a District Data Team an indication of its strengths and
challenges in a variety of areas related to promoting a district-wide culture of inquiry and data use. The
self-assessment is comprised of six short surveys which are aligned to each of the six modules in the
Toolkit and to the six steps in the Data-Driven Inquiry and Action Cycle. Each survey has a number of
selected-response questions grouped by Data Team practice. The possible responses are described in
the rubric below.
1. Print each page of this self-assessment (including this page so that the rubric is readily available to
anyone taking the survey) and provide a full copy to each member of the group.
2. Individually complete the survey, assigning a rating from the rubric below to each indicator.
3. As a group, discuss each page of the survey and agree on a rating for each indicator. It is not
necessarily best to average the individual scores to get this final rating. If responses among individuals
vary widely, engaging in a discussion about which rating best represents the level of practice can help
the Team begin the hard work of developing a common understanding of the work.
4. Enter the final rating for each indicator into the spreadsheet version of this survey.
5. Print out the Graphs page (or use a projector to display it on the wall), and as a group talk through the
discussion questions for each graphical display.
Rubric for Assessing Each Practice:

Respondent/Team has no knowledge about this indicator and cannot provide a
0 No Knowledge
judgment about its existence in the district.
1 No Evidence There is no evidence that this indicator is in place within the district.
There is some evidence of this indicator in the district, but the evidence
2 Emerging Evidence indicates that the practice is far from standard procedure and has clear room
for improvement in both quality and frequency.
This indicator has clear evidence of existence in the district and is consistently
3 Adequate Evidence practiced in many places. There is room for improvement in either quality or
frequency.
This indicator is evident in a variety of ways throughout the district. The

4 Exemplary Evidence practice described is clearly a part of the district culture and the way people
operate within the district.
0.2.1T: District Data Team Self-Assessment 2

Interpreting the Data: When the Team completes the six surveys, the data will be displayed in a series of
charts with questions to guide your review. There are two types of displays.
* A radar chart showing your Team's perceptions of strengths across each of the six modules.
* A horizontal bar chart for each step displaying the strengths within the step by practice.
A radar chart, also known as a spider chart or a star chart because of its appearance, plots the values of
each category along a separate axis that starts in the center of the chart and ends on the outer ring. In the
example below, each step of the Data-Driven Inquiry and Action Cycle is plotted on an axis. This makes it
possible to compare your survey results across the steps. When your surveys are complete, your results
will be displayed in a ring plotted on the chart. A district performing consistently across all of the steps will
be displayed in a near circle. A district performing higher in some steps than others will be displayed in a
more free form shape.

In the example below, this district is doing comparatively better in Inquiry, Information, and Knowledge
than it is in the other three steps. In this case, the district might find value in starting with the Results
module to get an idea of what processes are involved with monitoring action plans, and then shifting back
to the Getting Ready module to strengthen its processes across the whole cycle.
The horizontal bar charts will display your Team's perceptions of strength within each step. The questions
in each step are grouped by practice. The chart displays the averages of the responses within each step,
allowing you to view a disaggregated depiction of performance. When viewing these charts, you may find
it valuable to go to the modules themselves to find the tools, resources, and activities that contribute to the
effective implementation of that practice. Each of the practices surveyed in this instrument are supported
in the modules.
Getting Ready
Strengths by Practice
Vision f or Data Use 1.7
District Data Team 2.7
Assessment Literacy 2.7
Data Technology(ies) 3.0
Data Inventory 1.0
Data Collection 1.8
Data Access 3.0
Change Management 2.0
0.0 2.0 4.0

Getting Ready Incomplete
Rating Scale: 0 = No Knowledge; 1 = No Evidence; 2 = Emerging Evidence; 3 = Adequate Evidence; 4 = Exemplary Evidence
Vision for Data Use Rating:

The district has a vision for data use that aligns with and furthers the wider district mission and vision. 0
The district’s vision for data use is widely understood and accepted by all stakeholders. 0
The vision is supported by district policies and published expectations that support the use of inquiry and 0
data for instructional, school, and district improvement.
Practice Average: 0.0
District Data Team Rating:

The district has an established District Data Team or has designated another team to fulfill those 0
functions.
The District Data Team has a data champion, a data manager, and additional members with the range of 0
skills and perspectives needed to address the functions of the Team.
The District Data Team has a clear sense of its purpose and role in furthering the district's vision for data 0
use.
The District Data Team has a written plan that outlines the membership, roles, and responsibilities of the 0
Team, and this is publically communicated.
The District Data Team uses effective team practices (starting and ending on time, appointing a 0
moderator, following an agenda, appointing a note-taker, and clearly communicating regular meeting
times).
The District Data Team accomplishes its tasks effectively (action items are clearly noted with the people 0
responsible and timelines for completion; Team communicates with all stakeholders).
Assessment Literacy Rating:

District Data Team members are fully fluent in and have a shared understanding of contemporary, 0
standards-based assessment terms and concepts, e.g., summative assessments, formative
assessments, performance levels.
District Data Team members know what assessments are in use in the district and understand each 0
assessment's purpose.
Members of the District Data Team actively support the development of assessment literacy in district 0
colleagues, school data teams, and others through professional development offerings, coaching, and
modeling.
Data Technology(ies) Rating:

Resources, e.g., user manuals, job aids, IT support, for data technology(ies) exist and are easily 0
accessible to district and school users.
The district has technology(ies) for collecting and combining data from multiple sources, e.g., 0
demographics, grades, assessment data, attendance.
Data Inventory Rating:

The district has a comprehensive inventory of demographic, assessment, and other data available to 0
inform improvement plans and instructional practices.
The data inventory is published and available to all, along with a regularly updated schedule to 0
communicate how and when each type of data becomes available.

Data Collection Rating:
The District Data Team solicits feedback from stakeholders to identify opportunities for improvement in 0
methods of data collection, management, and reporting.
The District Data Team regularly uses established methods of communicating and coordinating with 0
stakeholders at all levels regarding data work.
Protocols or procedures are regularly used to ensure accuracy of data. 0
The District Data Team collects and reviews data from non-electronic sources, such as Learning 0
Walkthroughs, Common Planning Time (CPT) Self-Assessment, and interviews with principals and
other staff.
Data Access Rating:

The district has clear policies for data access that conform to state and federal requirements. 0
The District Data Team has knowledge of what data reports are disseminated to which stakeholders, 0
when, why, and what actions are taken as a result.
Change Management Rating:

District Data Team members are able to identify and address potential resistance to change before 0
problems occur.
Getting Ready Composite Average: 0.0

Inquiry Incomplete
Question Formulation Rating:

The District Data Team clearly articulates a focusing question to guide the inquiry process. 0
When determining a focus for the inquiry process, the District Data Team considers high-level data and 0
also solicits input from school and district leaders and other stakeholders regarding areas of priority
concern, e.g., dropout rates, programs to evaluate.
Data Displays
The IT staff and District Data Team create user-friendly data displays (charts, tables, and reports) that 0
facilitate meaningful conversations and promote new insights on the work of the district in service of
teaching and learning.
Data Overviews Rating:

The District Data Team creates effective data overviews (presentations to an audience) that are tied to 0
identified questions.
The District Data Team presents data overviews to appropriate audiences to introduce stakeholders to 0
the inquiry process.
Data overviews engage stakeholders in discussions of high-level data and solicit their input on the 0
formation of the questions that will guide the inquiry process.
Data overviews result in collaborative discussions about the meaning of the data, clarifying questions to 0
focus the inquiry, and a list of data potentially needed to address the questions.
Priority Identification Rating:

The District Data Team prioritizes among identified areas of need arising from the inquiry process. 0
The District Data Team examines new initiatives and priorities in the context of available resources and 0
ongoing initiatives to ensure alignment.
Inquiry Composite Average: 0.0
0 No Knowledge

Information Incomplete
Data Collection Planning Rating:

The District Data Team has a system to identify who will gather and organize data needed for analysis 0
(related to the specific focusing and clarifying questions), by when, and from where.
Data Analysis Rating:

The District Data Team understands the differences between factual observations and inferences 0
generated from analyzing data and is rigorous about distinguishing between the two when discussing
evidence.
Assessment data are analyzed in aggregate and disaggregate formats. 0

Assessment data is triangulated with other data, e.g., attendance, benchmark assessments, 0
demographics, data from Learning Walkthroughs, stakeholder surveys, or central office processes.
Data analysis at the district level is conducted collaboratively within and among departmental teams. 0
Data analysis results in the identification of specific problems or questions that need to be addressed, 0
e.g., problems at the student level, classroom level, school level, or district level.
Information Composite Average: 0.0
0 No Knowledge

Knowledge Incomplete
Root Cause Identification Rating:

District Data Team members know and implement multiple protocols for the safe discussion of root 0
causes.
Root cause analysis helps the Team decide on the one potential factor that, if addressed, would 0
eliminate or dramatically alleviate the problem.
Connections to Research and Local Knowledge

Potential root causes and proposed solutions are investigated through the consultation of research to 0
construct strong inferences about possible solutions/action steps.
Potential root causes and proposed solutions are also investigated through the consultation of local 0
knowledge or expertise to construct strong inferences about possible solutions/action steps.
Potential root causes and proposed solutions are also investigated through the consultation of 0
information on programs and practices (including data on instruction) to construct strong inferences
about possible solutions/action steps.
Shared Knowledge Base Rating:

The District Data Team keeps and references a problem log or meeting records documenting questions 0
raised to guide further inquiry.
The District Data Team encourages collection, dissemination, and active use of one or more forms of 0
documentation of lessons learned and promising practices from improvement efforts in a library of local
knowledge.
Knowledge Composite Average: 0.0
0 No Knowledge

Action Incomplete
Action Planning Rating:

The District Data Team uses a theory of action to focus action planning efforts. 0
The District Data Team uses a defined process for action plan development. 0
The action planning process considers ways to refine or reallocate existing resources, structures, and 0
initiatives before proposing brand new ones.
Action plans identify the available resources necessary to carry out the action steps. 0
The district can justify to stakeholders how it uses resources to achieve desired outcomes. 0
District personnel can articulate the district's program goals. 0
District personnel can articulate their role in achieving program goals. 0
Action Composite Average: 0.0
0 No Knowledge

Results Incomplete
Evaluating Results Rating:

District personnel have had formal training in program evaluation. 0
The district evaluates the efficacy and impact of programs based on student outcomes. 0
The district uses a defined process to evaluate programs and initiatives. 0
The district's evaluation plans include intermediate and long-term outcomes. 0
The district consults its evaluation plans throughout the year. 0
The district makes mid-course adjustments to the action plan as necessary, based on 0
formative/progress data.
The district typically achieves its program goals. 0

Communicating Results Rating:

The District Data Team has a process and tools for communicating interim and summative results to 0
stakeholders.
The district uses the results of program evaluations to inform the development of new programs. 0
The district has a process for codifying best practices at the district, school, or classroom level. 0
The process for communicating results creates opportunities to solicit feedback to inform the 0
development of new focusing questions.
Results Composite Average: 0.0
0 No Knowledge

District Data Team Self-Assessment Results
Guiding Questions
What observations do you have as you view the data displays?
What patterns do you see in the data?
Are there any discrepancies you notice in the data?
What additional questions do you have based on the data?
What additional information do you gain when looking at the responses to indicators in the survey?
Strengths by Steps
Getting Ready
4.0
3.0
Results 2.0 Inquiry
1.0
0.0
Action Information
Knowledge
Notes:

Notes:
Getting Ready
Vision for Data Use 0.0
District Data Team 0.0
Assessment Literacy 0.0
Data Technology(ies) 0.0
Data Inventory 0.0
Data Collection 0.0
Data Access 0.0
Change Management 0.0
0.0 2.0
Inquiry
Question Formulation 0.0
Data Displays 0.0
Data Overviews 0.0
Priority Identification 0.0
0.0 2.0
Information
Data Collection Planning 0.0
Data Analysis 0.0
0.0 2.0

Notes
Knowledge
Root Cause Identification 0.0
Connections to Research and Local Knowledge 0.0
Shared Knowledge Base 0.0
0.0 2.0
Action
Action Planning 0.0
0.0 2.0
Results
Evaluating Results 0.0
Communicating Results 0.0
0.0 2.0

MODULE 1: GETTING READY
Table of Contents
Introduction—1
Where Are We Now?—1
Module Objectives—1
Functions of a District Data Team—2

What Is a District Data Team?—2
Building a Culture of Inquiry and Data Use—4
What Is a Culture of Inquiry?—4
A Vision for Data Use—6
Establishing a District Data Team—8

Data Team Composition—8
Launching the District Data Team—11
Taking Stock—13
Taking Stock of Current Data and Processes—13
Types of Data That Inform Inquiry—13
Data Inventory—15
Data Collection—16
Data Dissemination and Access—17
Data Literacy—18
Managing the Change Process—20
Why Is Change Management Necessary?—20
What Can a District Data Team Do?—20
Module Summary—23
MODULE 1: GETTING READY
Getting Ready
1.1.1T: Functions of a District Data

1.6.1T: Data Collection Self-
Team
Assessment
1.2.1T: Barriers to Effective Data Use

1.7.1R: Data Dissemination Schedule
1.3.1T: Vision for Data Use

Example
1.4.1T: Norm Setting Protocol

1.7.2T: Data Dissemination Schedule
Template
1.4.2T: Data Team Meeting Agenda
1.7.3R: ESE Policies for Data Access
1.4.3T: Data Team Meeting Minutes
1.8.1T: Data Literacy Training Catalog
1.5.1T: Data Inventory Template
1.8.2R: Assessment Glossary
1.5.2T: Data Inventory Template:
SIMS and EPIMS Data

1.9.1T: Managing Change and
Understanding Concerns
1.5.3R: ESE Data Resources

Protocol
District Data Team Toolkit—Version 1.0 Module 1: Getting Ready
INTR ODUCTI ON
WHERE ARE WE NOW?

The District Data Team Toolkit is based on the Data-Driven Inquiry and
Action Cycle. The Cycle provides the structure that takes data use within
the district from asking the right questions to getting results. It is an
iterative process in which the district acts on data to support continuous
learning and improvement. The Toolkit uses the steps of the Cycle to
structure a progression through the model—you are now in Module 1:
Getting Ready.
Module 1 Module 2 Module 3 Module 4 Module 5 Module 6
Getting
Inquiry Information Knowledge Action Results
Ready
This module introduces district leaders to the purpose and role of a

District Data Team in building a district-wide culture of data use. It
provides guidance on how to establish a District Data Team and address
some of the functions that will help the Team organize for success. By
addressing these basic functions, the Team will then be able to engage in
a meaningful data-driven inquiry process (outlined in modules 2–6).
MODULE OBJECTIVES
The Getting Ready module will help a district:
 Set the vision for data use across the district

 Build a culture of inquiry to promote systemic data use
 Establish a District Data Team to drive this work
 Build data literacy
 Understand types of data that inform inquiry
 Establish systems and policies to inventory, collect, and
disseminate data
 Manage the change process

FUNCTI ONS OF A DIS TRI CT

DA TA TEA M
WHAT IS A DISTRICT DATA TEAM?

A data team can generally be defined as a group of educators
collaboratively using data to identify and understand opportunities for
improvement, then working together to make changes that get
measureable results. Using protocols for collaborative inquiry, the group
follows a process in which members prepare, implement, and reflect on
data-informed actionable goals.
This simple definition can be applied broadly at many levels within a

district. At the classroom level, teachers use data to identify student
learning problems and work together to plan instructional changes that
will yield improvements in learning. At the school level, principals and
school improvement teams use data to identify goals to drive
improvements in the ways teachers collaborate and learn, thereby
improving results for all students. Within a district office, many
departments and leaders use data to make decisions regarding the
management and efficiency of their particular responsibilities.
However, a District Data Team is one that has a unique

position from all of the others. By definition, district
leaders have unique perspectives on the operations of
the district and correspondingly distinct responsibilities
to make decisions that will have maximum benefit for
the entire system. For these reasons, it is wise to
establish a District Data Team that intentionally and
strategically analyzes data from a wide variety of
sources, triangulating and cross-referencing evidence
as much as possible to gain new insight on the work
taking place in service of teaching and learning. The
Team can then use this new knowledge to make
© 2009 Public Consulting Group informed decisions, while also setting the course for
inquiry and data use by all other teams in the district.
In addition to conducting its own inquiries, a District Data Team is

responsible for establishing the supports necessary for everyone
throughout the district to create and sustain a culture of inquiry and data
use. To do this, a District Data Team fulfills five essential functions.

Five Key Functions of a District Data Team
Vision and Creating and articulating the vision, setting and

Policy modeling expectations, and implementing and
Management upholding policies for data use in the district.
Data Identifying data to be collected, managing data

Management infrastructure and access, and designing
meaningful data displays.
Inquiry, Developing focusing questions and analyzing data

Analysis, and to make district-wide decisions about curriculum,
Action staffing, resources, and professional development.
Professional Providing training and professional development to

Development support district departments, principals, school data
teams, and teachers to use data.
Communication Communicating district-level focusing questions and

and Monitoring findings throughout the district. Monitoring the
school-level use of data, as well as goals and action
plans to identify trends and patterns.
These functions are interconnected. In order to build a vibrant culture of

inquiry and data use, a district must ensure that all functions are
addressed. For example, an effort to establish high quality common
planning time can be hampered by a lack of access to periodic
assessment data. Conversely, the rollout of a data warehouse or other
method for reporting data within a district can fall flat if the end users are
not trained on how to access the information. However, it is better for a
district to focus on building capacity in one area and do that well, rather
than attempting to launch work in all five realms at once. Part of the work
of a District Data Team is to determine the needs of the district and which
function(s) should be the immediate priorities for the Team to address.
The majority of the guidance in this Toolkit focuses on supporting a

District Data Team with Inquiry, Analysis, and Action and Communication
and Monitoring. Some aspects of the Toolkit provide initial support for
Vision and Policy Management and Data Management.
Activity 1.1 Functions of a District Data Team
This activity will help a district begin thinking about the role(s) the
District Data Team will fill and who should serve on the Team.
(1.1.1T: Functions of a District Data Team)

BUILDI NG A CULTURE OF
INQUIR Y A ND DA TA USE
WHAT IS A CULTURE OF INQUIRY?

Developing a culture that uses data to make decisions is a difficult task.
Simply providing data is not enough. Mandating meetings to analyze
benchmark test results rarely causes the improvements we hoped to get
by implementing the assessment in the first place.
To make the most of the data available within a district, there must be
something more. In highly successful data use initiatives, there is a
cultural shift that causes people to want to work differently, where teams
of educators will meet regularly to analyze data, ask questions, and dig
deeply to understand and fix problems. In all cases, there is a process
that drives this kind of work and collaboration.
In highly successful Having a culture of inquiry means having people within a district who are
data use initiatives, regularly asking questions about what all students should know and be
able to do, how best to teach content and skills, and what student
there is a cultural
demonstrations will be acceptable ways to measure learning. The
shift that causes leadership that a District Data Team can provide is central to creating this
people to want to district-wide culture of inquiry.
work differently.
The modules in this Toolkit will help a district establish or enhance its
District Data Team, as well as build the foundations to create a culture of
inquiry and data use. One key to creating this culture is to understand
what might be getting in the way of the district developing a thriving
culture of inquiry and data use. Debra Ingram (2004)1 and others
uncovered seven barriers to the use of data to improve practice:
Cultural Barriers:
1. Many teachers have developed their own personal metric

for judging the effectiveness of their teaching, and often this
metric differs from the metrics of external parties, e.g., state
accountability systems and school boards.
2. Many teachers and administrators base their decisions on
experience, intuition, and anecdotal information
(professional judgment), rather than on information that is
collected systematically.

3. There is little agreement among stakeholders about which

student outcomes are most important and what kinds of
data are meaningful.
Technical Barriers:
4. Some teachers disassociate their own performance and that

of students, which leads them to overlook useful data.
5. Data that teachers want about ― really important outcomes‖
are rarely available and usually hard to measure.
6. Schools rarely provide the time needed to collect and
analyze data.
Political Barriers:
7. Data have often been used politically, leading to mistrust of

data and data avoidance.
Understanding which of these barriers is most salient in the district can

help the Team strategize on the best way to engage more stakeholders.
However, the Team must also consider whether these are actual
barriers, or symptoms of something else. The section on Managing the
Change Process provides additional guidance for engaging stakeholders
in an initiative to increase use of inquiry and data in the system.
Activity 1.2 Barriers to Effective Data Use
Use this activity to begin thinking about the challenges the Team will
address to improve data use in the district.
(1.2.1T: Barriers to Effective Data Use)

A VISION FOR DATA USE

In order for a district to shift from being an efficient generator and
collector of data to an organization that translates data into information to
guide improvement, a broad effort and a clear, shared vision are required.
A shared vision sets a common goal and direction for the work of all
involved. A vision that specifically addresses data use in the district can
serve to connect the various functions that a District Data Team fills, as
well as connect the Team to other district efforts, so they don’t stand as
silos, but rather work together as pillars to sustain a culture of inquiry and
Without a clear vision data use. Without a clear vision for inquiry and data use that is broadly
for inquiry and data use understood throughout the district, the related supports and tools may still
be viewed as disconnected from other district initiatives, and from
that is broadly
important work at the school and classroom levels.
understood throughout
the district, the related A vision statement is one that takes into account the mission of the district
supports and tools may and describes how things will look in the future if the Team’s work is
successful. Hallmarks of a clear and vibrant vision2 include:
still be viewed as
disconnected from The vision is reflected in the district’s strategy and corresponding
other district initiatives, use of resources
and from important Everyone in the system gives the same responses to important
work at the school and questions, like ―
why is data use important to teaching and
learning?‖
classroom levels.
Stakeholders demonstrate shared belief, collective clarity,
ownership, and energy for the work at hand
People are inspired to look beyond quick fixes and dig into real
challenges
People are able to look beyond smaller challenges and focus on
what is really important
The district likely has a mission statement that answers the question,
―Why do we exist?‖ and serves as a clear statement of purpose for
everyone in the district. At its core, the statement puts a stake in the
ground and declares why you exist—to educate children.
A district vision statement for data use should derive from the district’s
overarching mission and vision. It will have a slightly different tone that
focuses on data use, while still connecting in some way to improving
performance, taking action, or doing things differently than they have been
done in the past. The vision statement should define the future so it can
serve as a guidepost for all data use efforts.

If there are people anywhere in the district who aren’t sure of what data
are available, what actions data should inform, or why certain data are
even collected, this may be a sign that the district lacks a clear and
shared vision for data use. Consider for a moment the data use in your
own district.
How prepared are principals to use inquiry and data to inform their
own work?
How prepared are principals to lead their staff and teachers in
inquiry and data use? To what extent do they actually do this?
How prepared are staff and teachers to use inquiry and data to
inform their work? To what extent are they actually engaged in
data use?
To what extent can principals, teachers, and others in the district
articulate how data inform their practice and further the district’s
mission for educating its students?
Activity 1.3 Vision for Data Use
Open the Vision for Data Use document and complete the activities to
either assess the district’s existing vision for data use, or craft a new
one.
(1.3.1T: Vision for Data Use)

ESTAB LIS HING A DIS TRI CT

DA TA TEA M
DATA TEAM COMPOSITION

In conducting the work to this point, several people have probably
contributed who will serve as members of the District Data Team in a
sustained capacity. Now that the functions of the Team are understood
and there is a strong vision for data use across the district, it is time to
formalize the Team into a functioning entity.
Using the results of 1.1.1T: Functions of a District Data Team and 1.3.1T:
Vision for Data Use as a guide, identify the departments and people who
will be essential in helping the District Data Team fulfill all five key
functions necessary to support data use in the district, which were
previously noted in this module.
Depending on the size and composition of the district departments, you

may need to involve people from several different departments. Finding
the right people to fill each post on the Team may be challenging if they
have not participated in the process to this point. As potential members or
department heads who have not participated previously are
communicated with, consider sharing the District Data Team’s vision
statement to help them understand the importance of their participation.
When assembling a District Data Team, it is important to think

strategically about who will be on the team and why. The following
questions provide some guidance:
What are the perspectives and expertise needed to fulfill the

District Data Team’s vision and priority functions?
Who is familiar with and/or supportive of using inquiry and data to
inform decisions?
Who has solid skills in analyzing and explaining data?
Who has credibility with stakeholders and can champion inquiry
and data use with others?
Whose participation would help the Team address current barriers
to effective data use? (Note that this could lead the Team to
include individuals who might be hesitant or resistant to processes
of inquiry and data use, not just those who are already on board)

Who thinks creatively and can share a fresh out-of-the-box

perspective on both analysis and action planning?
Who has a solid understanding of programs, initiatives, and other
efforts taking place across the district?
Who has a deep commitment to improving the learning of all
students and the practice of all adults involved in educating them?
Who understands the concepts of team, consensus, and unified
messages?
Who is likely to be able to commit time and energy to a multi-year
effort to establish and maintain a district-wide culture of inquiry?
The most effective
The most effective District Data Teams have members who want to District Data Teams
support the inquiry process through the use of data and are broadly have members who
representative from a district perspective.
want to support the
The District Data Team must be led by a data champion. This individual inquiry process
should have the positional authority and credibility to ensure: through the use of
data and are broadly
The District Data Team has the resources and supports necessary
to function effectively representative from
a district
The work of the District Data Team is understood and visible to
others in the district perspective.
The work of the Team will be acted upon
The Team must also have a data manager who is in charge of the more
technical aspects of the work, such as:
Coordinating data use throughout the district

Establishing systems to ensure the cleanliness and quality of the
data
Integrating different data systems
Ensuring all users are using the same data dictionary and
terminology
It is also critical that the superintendent shows support for the inquiry
process and the work of the District Data Team by modeling data use and
visibly responding to the needs of the Team.
Core Data Team members will be determined within the context of the
local setting, but could include:
Data champion (chairperson)

District-level data manager

Director of pupil personnel services/special populations
Assistant superintendent for instruction
Director of assessment, research, and evaluation
Directors of elementary and secondary education
Literacy and mathematics coaches
Special projects coordinators, e.g., SLCs
Principals, lead teachers, or other school-based faculty
While broad representation is important, it is also essential that the Team

not be too large. One approach is to form a core Data Team of six to eight
members who will do most of the collection, analysis, dissemination, and
coordination work, and establish a more broadly representative group of
adjunct Data Team members who will provide input in specific areas and
general feedback to the core group.
The Adjunct Data Team members could include:
Subject-area directors
School-level data team chairs
Principals
Grants director
Union leadership
School board member
Parent association representative
Teachers and other school-based faculty
Remember to provide administrative support for the Team’s work, which

might include taking notes at Team meetings, producing materials, and
generating specific data displays.

LAUNCHING THE DISTRICT DATA TEAM

Prior to beginning the district-wide inquiry process, the District Data Team
must get organized and begin to build its capacity to drive inquiry
throughout the district. Some initial steps many districts have found useful
are noted here.
1. Obtain clearly stated and visible support from the

superintendent–in both written and oral communications.
2. Meet with district administrators to clarify the purpose of the
initiative, how it relates to the district’s mission and goals, the role
of the Data Team, and the Team’s decision making authority.
3. Establish clear relationships and lines of communication among
the Data Team and other teams at the district and building levels,
e.g., district leadership team, school improvement teams,
departmental teams, grade-level teams, or professional
development teams.
4. Organize itself to do the work by:
Agreeing to always set an agenda for Team meetings that
clearly delineates intended outcomes or products expected as
a result of the meeting
Establishing group norms and using protocols to structure
conversations
Understanding that there will be a learning curve for the Team
and that the Team shouldn’t address too many essential
questions at the outset
Agreeing to delegate tasks and expect timely completion
Expecting members to work between meetings to complete
tasks
5. Build the District Data Team’s capacity before building the
capacity of the school-level data teams and the district-level staff.
The District Data Team should:
Continue to build shared values and refine a common vision
for the inquiry process and data use at the district and school
levels
Participate in ongoing professional development activities to
build its capacity to use data and function constructively as a
team

6. Provide adequate time for the Team to understand, develop, and

complete its work. Time is the dearest resource in a school or
district, and insufficient time will sorely limit the Team’s
effectiveness.
Data Team members need enough time to share information,
generate understanding, and determine next steps
Data Teams need uninterrupted, protected time for
collaboration, in addition to time for capacity building,
professional development, and collaboration with the school-
level data teams (more time will be necessary during the
launching phase than in subsequent phases)
Activity 1.4 Data Team Meetings
The Norm Setting Protocol will help the Team articulate and agree on
ways of working together in order to foster risk-tasking and effective
communication during tricky conversations. The templates provide
models for agendas and capturing meeting minutes, in order to assure
productivity and high-quality communication during and after meetings.
(1.4.1T: Norm Setting Protocol)

(1.4.2T: Data Team Meeting Agenda)
(1.4.3T: Data Team Meeting Minutes)

TA KING STOCK
TAKING STOCK OF CURRENT DATA AND

PROCESSES
Before a newly established District Data Team dives into a process of
inquiry, it should first take stock of a few key elements related to data use
in the district. Given the Team’s functions of data management and
professional development in particular, it should consider what is being
done to promote data literacy district-wide, as well as document what
data are available to whom and when.
TYPES OF DATA THAT INFORM INQUIRY

Often when people think of data, they think of numbers, and particularly
student assessment data. However, data refer to a much wider range of
evidence and include any factual information (as measurements or
statistics) used as a basis for reasoning, discussion, or calculation.3
Any data team, but a District Data Team in particular, has a responsibility
to consider data from multiple sources in order to gain an understanding
of the quality of work being done in service of teaching and learning—not Four important
only in the classrooms, but in all areas of the district. The Team can types of data:
increase the validity of its inferences and conclusions if it taps a variety of

data sources to provide more information about the question being Student Outcomes
investigated. This may involve comparing different forms of the same type Demographics
of data, such as results from different types of assessments. Alternately, Perceptions
the Team may compare two entirely different types of data, such as Processes
comparing achievement to the length of time a student has been enrolled

in the district, or the length of time the student spends travelling to school.
The graphic below, based on the work of Victoria Bernhardt4, outlines four
primary domains of data: student outcomes, demographics, perceptions,
and school (or district) processes. This lens highlights the fact that
student achievement data provide only one view on the work of a district.
The Data Team must also analyze data related to processes such as
hiring, procurement, and even facilities maintenance, or perceptions of
stakeholders, in order to gain new insight on the supports needed from
the district to take teaching and learning to the next level. This may mean
looking for data in forms other than numbers that can be easily counted,

and also considering data generated by what one sees (such as through
Learning Walkthrough site visits) or hears (such as through stakeholder
surveys and focus groups).
This diagram also describes the interaction of data from the four primary
domains and the kinds of inferences that can be drawn from the
intersections.
Adapted from: Bernhardt, V. L. (2004). Data Analysis for Continuous School

Improvement. Larchmont: Eye on Education
It is important to note that of these four domains, only one can be directly
modified by a District Data Team (or anyone else, for that matter), and
that is processes. It is only by changing the way adults interact and
conduct business that a district can hope to shift the evidence it sees in
the realms of demographics, perceptions, and student outcomes.

DATA INVENTORY
With this lens of the four domains, the District Data Team can inventory
the data available in the district, when they are available, how readily they
can be accessed by the Team for consideration in the inquiry process,
and how they are being used in service of teaching and learning.
Completing this inventory serves multiple functions. It can help a district:
Gain a clearer picture of the data currently available to guide

inquiry at all levels in the district, and how they are being used
Identify data that are being collected, but that are not necessarily
well-used
Identify redundancies in data collection that could be eliminated
Identify additional data elements needed to address district and
school improvement and inquiry processes
Communicate expectations for what to do with particular data
The data inventory provides a framework to collect both district-wide and

school-based data. School-based data are not necessarily collected in
other schools in the district. These data, such as those generated through The data inventory
use of a commercial off-the-shelf assessment or screening tool, or a provides a
school-designed survey for families, can provide some value to district
framework to
inquiry processes since they can shed light on how a particular school
functions. Yet school-based data also have their limitations, most collect both district-
significantly that they do not enable ― apples to apples‖ comparisons to wide and school-
other schools. based data.
In promoting a district-wide culture of inquiry and data use, a district
should consider the benefit of promoting and analyzing common district-
wide data. For example, student assessment data are only useful for
district-wide inquiry and analysis if they represent every student across
the district in the same relative time span. Examples of common
assessments include:
Statewide tests such as MCAS or MELA-O

Commercial benchmark assessments or screening tools that are
used with all students in a particular population district-wide
Locally developed assessments such as common mid-terms and
finals created by district personnel
Other forms of common district-wide data include:
District financial statements

Human resources tracking systems

Student transportation information
Common district-wide data allow the Team to examine a snapshot of a

given population at a given time, and also help get various stakeholders
talking in the same language.
A data inventory, coupled with expectations for what to do with particular

data, can help the district establish standard procedures or processes for
what to do with certain data like a benchmark assessment. For this
reason, the district might consider sharing this inventory widely with
school leaders, teachers, and other stakeholders, as well as referencing it
for its own use. The district might also consider having each school
complete the inventory, then compare the results across schools. For
example, the district may learn that some schools are using a particular
assessment that the district would want to expand. Or the district might
notice a pattern between the types of assessments used at a school and
the achievement results of its students.
Activity 1.5 Conducting a Data Inventory
This activity will help determine current availability and use of data and
will identify additional data elements needed to further the inquiry
process. 1.5.2T: Data Inventory Template: SIMS and EPIMS Data is
pre-populated with information on these state-wide data elements.
1.5.3R: ESE Data Resources may be useful to reference for
information on this and data the state provides for districts.
(1.5.1T: Data Inventory Template)
(1.5.2T: Data Inventory Template: SIMS and EPIMS Data)
(1.5.3R: ESE Data Resources)
DATA COLLECTION
For the available data to further the inquiry process, they must be
complete, accurate, and timely. Collection and distribution tools and
processes need to be efficient and effective to ensure that these criteria
are met.
Data collection is people-centered. In order to have complete and

accurate information provided in a timely manner, it is essential that the
people who are responsible for data collection are well trained and well
supported. Successful data collection occurs when those responsible for
collecting data:

Understand and are invested in what the data will be used for
Understand how the data they collect will be integrated into other
systems
Participate in the creation of and agree to the use of a common
Data Collection Practices handbook
Are adequately trained to complete the task
Have appropriate tools to support the collection process
Work in an environment free from distraction
Are provided the time to collect the data and ensure the data’s
integrity
Without this support, it is highly likely that the district will not get valid
information, which in turn would detract from its ability to make quality
evidence-based decisions.
The District Data Team can contribute to the effective collection and
distribution of data by continually monitoring the needs of the district; the
effectiveness of the tools in place for data collection, storage, and
dissemination; and the training of those who are responsible for data
collection and input. One of the most important things that members of
the Team can do is listen and respond to the needs of the staff in charge
of the data collection process.
Activity 1.6 Data Collection Self-Assessment

This activity will assist you in describing the tools and systems in place
for the collection, storage, and dissemination of data, and in evaluating
the effectiveness of these tools and systems.
(1.6.1T: Data Collection Self-Assessment)
DATA DISSEMINATION AND ACCESS

Collecting complete and accurate data in a timely manner means little if
the data are not disseminated for use by stakeholders. It is important for
all members of the district community to know what data are available,
which data will be disseminated to whom, how the data are to be used,
and when the data will be refreshed, e.g., a new set of data produced.
Now that the District Data Team has a firm understanding of those
factors, the Team can publish this information through a data
dissemination schedule.

However, the Team must also pay close attention to who is given access
to what data, and why. Federal, state, and local regulations determine
who can have access to personally identifiable data. The Massachusetts
Department of Elementary and Secondary Education has published
guidelines regarding access to data that comply with these regulations.
Each district also has privacy policies to inform decisions regarding
access to data. Specific access guidelines have been developed by the
ESE for the ESE Education Data Warehouse. These guidelines can serve
as a model for the development or critique of locally developed guidelines
for data access.
Beyond compliance with federal, state, and local data access regulations,
the District Data Team must consider the logistics involved in providing
appropriate data in a user-friendly and timely manner to those who need
it. Faithful use of the data dissemination schedule will ensure that a large
segment of the community will be provided with the data that it needs.
Some may have access to data through the student information system,
while others will gain access through use of the ESE Education Data
Warehouse. It is important for the District Data Team to be sensitive to
the data needs of the district as a culture of systemic data use evolves,
and to act to meet those needs.
Activity 1.7 Data Dissemination and Access
The data dissemination activity can help a District Data Team construct
and publish a schedule for the distribution and use of major data
elements.
(1.7.1R: Data Dissemination Schedule Example)

(1.7.2T: Data Dissemination Schedule Template)
(1.7.3R: ESE Policies for Data Access)
DATA LITERACY
To effectively use the data available to them, principals, teachers, district-
level staff, and the community need certain knowledge and skills. It is
particularly important that the members of the District Data Team have
competencies in data and assessment literacy. Additionally, each of these
stakeholders needs to develop a shared understanding of the purposes
and uses of various data as they pertain to their roles in serving students.
Stakeholders must understand what data to use when, the uses and limits
of specific assessments, ways to interpret and use the various reports
produced by those assessments, and specific statistical terminology and
calculations used in those reports. A successful District Data Team will

take time to ensure that its members have a degree of assessment

literacy that sets them up for success not only in their own inquiry
processes, but also as they begin to model inquiry and data use for
others.
For each standardized assessment used in the district, there are unique
details about test and item construction that must be communicated to
and understood by all consumers of the test. This includes teachers,
principals, and other staff as they analyze results in preparation to take
action, as well as parents and students as they receive reports designed
to inform them of specific areas of strength, challenge, and progress
toward attaining proficiency in core curriculum standards.
Methods by which each of these consumers will gain this specialized

knowledge need to be well planned and implemented to ensure that the
data are used properly and safely in service of students.
The following tools and sources of information can be helpful in planning

ways to improve the level of data literacy in a District Data Team, as well
as across the district.
Activity 1.8 Assessment Literacy
These resources will help a District Data Team develop its assessment
literacy, as well as that of other stakeholders in the district.
(1.8.1T: Data Literacy Training Catalog)

(1.8.2R: Assessment Glossary)

MANAGING THE CHANGE

PROCE SS
WHY IS CHANGE MANAGEMENT NECESSARY?

An earlier section in this module provided guidance to reflect on a
district’s barriers to the effective use of data to improve practice. These
barriers are often symptoms of something else. Richard Sagor (1992)5
T he challenge is
suggests that some resistance may stem from the fact that teachers are,
to both implement
for the most part, already doing what they believe is best for their
the change while
students. Teachers and other district personnel may be slow to adopt a
also managing the new initiative because they cannot see how it would benefit their students
change process. or others for whom they are responsible. They may also feel a sense of
loss—for example, of competence—if they are asked to approach their
work differently than they have for the last 5, 10, or 20 years.
When presenting a new initiative designed to develop or enhance a

culture of inquiry and the use of data to inform educational decision
making, the District Data Team and others who are supporting the
initiative must be prepared to manage the stress and push back that will
naturally occur.
The challenge is to both implement the change while also managing the
change process. When introducing or enhancing a cultural norm of a truly
collaborative learning community—one where all members regularly ask
questions about their practice and what more can be done in service of
student learning and achievement—the Team must pay attention to the
human element, the students and adults who are being asked to
approach work differently in order to achieve new outcomes.
WHAT CAN A DISTRICT DATA TEAM DO?

To effectively meet the challenge posed by the change process, the
District Data Team must acknowledge that resistance is a natural process
and that the District Data Team and others have the power and
responsibility to mitigate the negative impact this phenomenon can
produce.
A District Data Team can use the following framework as a guide when
introducing the inquiry process district-wide. The guidelines suggest steps
that the Team can follow to support school-level data teams and other
teams within the district as it initiates the first steps in the collaborative
Data-Driven Inquiry and Action Cycle.

Build Awareness
Build a vision for data use that is grounded in positive student

outcomes
Articulate the vision for district-wide systemic data use clearly and
repeatedly with all stakeholders to paint an evident image of how
the future will be better if all engage in this work
Develop and communicate a sense of positive urgency
Share the structure and function of the District Data Team with
school-level teams
Understand Concerns
Talk openly with staff at all levels in the district about stress they
may experience as change is implemented
Actively listen: solicit and act upon the concerns of staff members
to facilitate the change process
Acknowledge losses that people may feel as they shift established
habits and approach their work in new ways
Model the Process

Lead by example, not by edict
Publicly demonstrate how the District Data Team is moving toward
the vision
Present the district-level data overview with school-level
participants and other district stakeholders
Design district-level action plans using the Data-Driven Inquiry
and Action Cycle
Manage the Process

Conduct and maintain a data inventory (2.2.1T) that includes
school-level data
Coordinate the upload of local data to the ESE Data Warehouse
Maintain an up-to-date data dissemination schedule (3.2.2T)
Disseminate relevant data sets and displays for school-based
action

Monitor the Process
Coordinate district- and school-level focusing questions

Provide feedback to school-level teams on the effectiveness of the
data displays that they construct
Work with all teams within the district to support the monitoring of
their action plans
Build Capacity
Develop a broad base of support among all stakeholders
Lead a discussion of how the vision can be realized through the
action of school-level teams
Involve staff in collaborative and objective analysis of data to
answer the high-interest questions that they have developed
Help schools and district offices form data teams
Provide support to school-level teams as they utilize the resources
of the Toolkit
Provide professional development to help district personnel build
assessment literacy and use relevant data warehouses
Assist schools as they learn to prepare local data for upload to
centralized data warehouses
Provide professional development activities to build assessment
literacy
Celebrate Success
Positively reinforce movement toward desired goals for a culture
of inquiry and data use as well as improved student achievement
Over time, with patience, perseverance, and strategic action, the District
Data Team can help the district as a whole establish and/or enhance a
cultural norm in which inquiry and data use is a regular part of everyone’s
work, where data are regarded as impartial evidence that can spark a
question, trigger an idea, or measure a result.
Activity 1.9 Managing and Understanding Change
This protocol can help a District Data Team gain a better understanding
about the concerns of stakeholders as it engages in this work.
(1.9.1T: Managing Change and Understanding Concerns Protocol)

MODULE SUMMARY
This module explores the roles and functions of a District Data Team to
set the course for data use in the district and support the establishment of
a culture of inquiry. It discusses the value of a vision statement for data
use and provides guidance on how to create or refine one. Setting and
communicating the vision for how the district will use data to make
decisions is key to success with the inquiry process outlined in the
remainder of the Toolkit’s modules.
The module also addresses data management activities that provide a

foundation for a culture of inquiry and data use, such as tracking the data
that are being collected in the district, who is using the data and how.
Any new initiative or way of approaching the work of teaching and

learning will likely take some time to gain traction. Taking some time to
consider how the district will manage the change process and address
the uncertainties felt by stakeholders can go a long way toward promoting
the success of the District Data Team’s efforts.
If you have not yet done so, consider administering the District Data
Team Self-Assessment at this time (0.2.1T). It will help the Team identify
its strengths and challenges related to an inquiry process, providing
guidance on how to use the remaining resources in the Toolkit.
REFERENCES
1
Ingram, D. S. (2004). Accountability policies and teacher decision making:
Barriers to the use of data to improve practice. Teachers College
Record, 106(6), 1258–1287.
2
Curtis, R. E. and E. A. City. (2009). Vision: Keeping the end in mind. Chapter 4
in Strategy in Action. Cambridge, MA: Harvard Education Press.
3
http://www.merriam-webster.com/dictionary/data (December 24, 2009)
4
Bernhardt, V. L. (2004). Data Analysis for Continuous School Improvement.
Larchmont: Eye on Education.
5
Sagor, R. (1992). How to conduct collaborative action research. Alexandria, VA:
Association for Supervision and Curriculum Development (ASCD).
For more information on this and other district support resources, or to share feedback on
this tool, visit http://www.doe.mass.edu/sda/ucd/ or email districtassist@doe.mass.edu.

FUNCTI ONS OF A DIS TRI CT DA TA TEA M 1 . 1 . 1 T
Purpose To understand the role and functions your District Data Team fulfills to support a culture of Related Documents
data use. 1–Getting Ready Module
Description Team members will review the functions of a District Data Team and think specifically
about how these tasks are accomplished within its district. The Team will also identify gaps
that might exist on the Team and begin thinking about how to address them.
Time 45 minutes to an hour.
―A District Data Team is responsible for establishing

the supports necessary for everyone throughout the
district to create and sustain a culture of inquiry and
data use. To do this, a District Data Team fulfills five
essential functions.‖
—Module 1: Getting Ready
© 2009 Public Consulting Group
1.1.1T: Functions of a District Data Team—Version 1.0 1/3

FUNCTIONS OF OUR DATA TEAM
S T E P 1 ) Use the table below to brainstorm the specific tasks already being performed in the district within each function. Allow
individual think/work time before sharing and charting everyone’s ideas. (See Module 1: Getting Ready for explanations of the five
functions)
Vision and Policy Inquiry, Analysis, Professional Communication and

Data Management
Management and Action Development Monitoring

FUNCTIONS OF OUR DATA TEAM
S T E P 2 ) Reflect on the results of the brainstorm by discussing the following questions.

It may be useful to have for reference the results of the District Data Team Self-Assessment.
1) In which functional area(s) is the district performing particularly well? What is the
evidence?
2) Which function is currently the district’s biggest challenge? What is the evidence? What
is getting in the way of success in those areas?
3) What key tasks, if any, are missing from the list of tasks currently being performed?
4) What tasks are the greatest priority for the coming year, given the district’s strategic
priorities and improvement plan?
5) Which of these tasks are dependent on cross-departmental cooperation? (Note which

departments).
6) Which tasks are currently being performed exclusively within one department? (Note
which departments).
7) If the District Data Team is going to fill all of these key tasks and functions, whom does
the Team need to have as members? (Identify by name and/or role).

BARRIERS TO E FFE CTIVE DATA USE 1 . 2 . 1 T
Purpose To identify barriers or problems your Team might face regarding data use. Related Documents
1–Getting Ready Module
Description The District Data Team will make a list of possible barriers or problems that might slow its
progress. You will also begin to think about solutions to them.
Time 30 minutes.
As a team, brainstorm a list of the barriers the district currently faces in creating and/or maintaining a culture of inquiry that is
embedded in everyone’s work. Try to identify a range that includes cultural, technical, and political barriers. As a team, identify which
barriers are the most significant and that, if addressed, would result in the greatest shift toward an embedded culture of inquiry. For
each of these prioritized barriers, identify possible strategies and people who can help address these barriers. Try to think out of the
box in identifying these people, looking beyond titles and positions. Keep this list accessible as the Team works through this module
and the rest of the Toolkit. These barriers will become areas of focus for the District Data Team.
Cultural,
Technical, People to Involve
Barrier or Political? Possible Strategy to Address in the Solution
1.2.1T: Barriers to Effective Data Use—Version 1.0 1/1

V I S I O N F O R D A T A U S E 1 . 3 . 1 T
Purpose To develop a shared vision for data use that will guide Related Documents
the District Data Team’s work. 1–Getting Ready Module
Description Team members will develop a shared vision for data
use in the district and craft a vision statement to drive
the Team’s work.
Time About 2 hours. (Can be done in two blocks).
If the district already has a vision statement that incorporates data use, locate it and use this
guide to assess and revise it if necessary. If there is no reference to data use in the existing
district vision, use this guide to draft a vision statement to guide the work of the Team.
If the Team has completed 1.2.1T: Barriers to Effective Data Use, it may want to have those
notes available for reference.
FINDING A SHARED HORIZON

A vision statement looks to the future and defines how things will be improved over their current
status. Some vision statements have a fairly long view (for instance, 3–5 years in the future),
and some seek to define changes that will be in place in less than a year.
1. Let the Team know that the purpose of the activity is to begin to articulate a vision for the
work of the Team. The actual writing of the vision statement will come later.
2. Provide each person on the Team with a large note card or sticky note and instruct them to
individually write a vision for data use in the district. Guide them by asking a specific
question such as:
If this team were to be successful in promoting data use, what would that look like?
What do we want the future of data use in the district to look like?
You might also ask members to consider:
The data use practices needed to fulfill the district’s mission

The functions of a District Data Team identified in 1.1.1T
The barriers to effective data use identified in 1.2.1T
What things would look like if an important problem were resolved
Provide about 5 minutes of silent work time and let them know that responses will be shared.
3. Once everyone has had a chance to write his or her vision statement, let the group know
that the next step is to work together to sort the notes in relative order, so that the most
immediate aspirations come early (Vision 1 or 2), while the longer-term aspirations are near
the end of the horizon line (Vision 3 or 4).
In preparation for this, each individual should review what he or she wrote and write each
separate thought onto a different card or sticky note.
1.3.1T: Vision for Data Use—Version 1.0 1/3

4. Draw a diagram similar to the one below on chart paper or a white board, or label sections
on a wall.
Vision 3 Vision 4
Vision 2
Vision 1
The curved line represents the future, while under the person’s feet is current reality.
5. Once all Team members have completed visions and separated statements onto separate
cards (if necessary), they attach their note to the diagram.
6. Review all the statements, discuss and arrange the notes until all members of the Team are
satisfied with the order.
7. As a Team, review the assembled statements and add any key ideas that seem to be
missing. Also ask if anyone has any questions or concerns about any of the ideas, and
whether anyone would have a hard time getting ‘on board’ with them.
The Team now has a view of a shared strategic focus. The diagram outlines priority areas of
need to be addressed by the District Data Team and is beginning to paint a picture for data
use in the district.
If the district has a vision for data use already written, compare it to the array of ideas the
Team has just created. Determine if the existing vision is in alignment with the shared
strategic focus the Team just developed. If there is not alignment, consider whether it is the
existing vision or the Team’s strategic focus that may need revision.
Note: This process can be modified for use in other settings, such as crafting or revising a
district’s vision for education. The key is to articulate a clear guiding question to focus the initial
work in step 1.

CRAFTING A VISION STATEMENT
If the district does not have an existing data use vision statement, follow the steps below to craft
a vision statement.
1. On a new piece of chart paper, write the following sentence starter:
In ___ years, our district will accomplish ___________ by doing ____________.
2. Ask each member of the Team to write a statement that incorporates the Team’s shared
strategic focus using the sentence starter as a guide.
3. Record each person’s draft vision statement on chart paper (or an electronic document
displayed with a projector).
4. Review the statements as a Team. Look for opportunities to combine similar ideas and
identify unique ideas.
5. Merge all of the ideas into a clear statement of the district’s vision for data use. The
statement may be multifaceted or bulleted, but it should include the essential elements
of the original sentence starter.
a. A timeframe
b. Accomplishments or goal statements
c. Methods or strategies that will be used to achieve the vision
6. Refine the statement until all members of the Team are satisfied that it captures the
Team’s priorities and vision for data use in the district.
7. Consider the authority with which the District Data Team has been charged. Does the
vision need to be approved by another team? How will this vision be finalized and
communicated to district leadership, schools, and other stakeholders?

NORM SETTI NG PR OTOCOL 1 . 4 . 1 T
Purpose Tools for launching or supporting the work of a District Related Documents
Data Team. 1–Getting Ready Module
1.4.2T: Data Team
Description This protocol will establish the norms under which the
Meeting Agenda
District Data Team will operate. Setting norms helps
1.4.3T: Data Team
keep unproductive behaviors in check while fostering
Meeting Minutes
risk-taking and effective communication during tricky
conversations.
Time 15–30 minutes.
3–5 minutes
Invite people to reflect in writing on the response to this question “In order to reach our
vision, what norms will we need?” Explain that norms are guidelines for an interaction or
meeting, and can include both process (e.g., start and end on time) and content (e.g., taking
risks with our questions and ideas).
5–10 minutes
Invite people to share norms. It’s sometimes best to do this round-robin style so that you hear
one from each person, and then open it up for other ideas. Record the norms on chart paper
or using a computer and projector. You don’t need to write these exactly as stated—just
capture the idea.
10–20 minutes
Ask if there are any norms people have a question about (sometimes people will ask a
clarifying question about what something means) or couldn’t live with during future meetings.
You may need to rephrase or reframe norms to pose them in a way that everyone is
comfortable with. When everyone seems clear and comfortable with the list, ask if there is
anyone who can’t live with and support these norms.
Note: Norms are only valuable if the Team regularly references them and holds each other
accountable for upholding them. Consider establishing a few rituals to keep the Team’s norms
alive, such as:
Posting norms and/or including them in any printed agenda

Building in time at the end of each meeting, or at periodic times in the year, to reflect on
the extent to which the Team is upholding norms, and whether any norms need to be
added, modified, or removed
Rotating the role of process observer, whose job it is to pay attention to whether norms
are followed
1.4.1T: Norm Setting Protocol—Version 1.0 1/1

DA TA TEA M MEETI NG A GE NDA 1 . 4 . 2 T
1.4.1T: Norm Setting
Description This template is a good model for meeting agendas
Protocol
that lead to productive meetings.
1.4.3T: Data Team
Time Ongoing. Meeting Minutes
Location:
Meeting Date:
Agenda
Item # Subject Presenter
Resources
Items/Resources to Bring to Meeting Items/Resources to Be Distributed at Meeting
Data Team Norms: (List all norms established and recorded by the Data Team—this list
should appear on all meeting agendas.)
1.4.2T: Data Team Meeting Agenda—Version 1.0 1/1

DA TA TEA M MEETI NG MI NUTES 1 . 4 . 3 T
1.4.1T: Norm Setting
Description To improve the effectiveness and efficiency of quality
Protocol
communication, it is a good idea to capture meeting
1.4.2T: Data Team
minutes accurately and efficiently. This template can
Meeting Agenda
serve as a good model to follow.
Time Ongoing.
Location
Meeting Date
Submitted by (name)
Submitted date
Members present Name Role

(list names) (list roles)
Agenda Item #
Subject
Discussion
Decisions/Action Steps Person Responsible Timeline
1.4.3T: Data Team Meeting Minutes—Version 1.0 1/2

Agenda Item #
Subject
Discussion
Agenda Item #
Subject
Discussion
Insert rows for additional agenda items as needed.
1.4.3T: Data Team Meeting Minutes—Version 1.0 2/2

D A T A I N V E N T O R Y T E M P L A T E 1 . 5 . 1 T
Purpose To develop an inventory of currently available data and how the data are being used in service Related Documents
of teaching and learning. 1–Getting Ready Module
1.5.2T: Data Inventory
Description Complete the attached templates to determine current availability and use of data in the district.
Template: SIMS
Time 1–2 hours to review template; 1–2 weeks to gather information, with ongoing upkeep. and EPIMS Data
1.5.3R: ESE Data
Resources
NOTES:
The sections of this Data Inventory align to the four domains of data described in the text of the Getting Ready module:
demographics, district and school processes, stakeholder perceptions, and student outcomes.
1.5.2T has been pre-populated with all the data elements collected for SIMS and EPIMS.
If the Team has completed 1.8.1T Data Literacy Training Catalogue or 2.2.1T Inventory of District and School Initiatives, it may want
to have those results available for reference for this process.
A Team might want to copy and paste these tables into Excel in order to be able to sort and group the information.
DIRECTIONS:
1. Organize data elements into district-wide and school-based data:
a. District-wide Data is common across all schools, a set of grade-alike schools (e.g. elementary) or at least across a given
population (e.g. all 5th graders, all English Language Learners, or all students who receive free or reduced lunch)
b. School-based Data is that which is not necessarily collected in other schools in the district, such as data a principal decides on his
or her own to collect and use with school personnel, or data for unique programs such as Expanded Learning Time or pilot schools.
2. For each assessment or element, provide the indicated information in the columns to the right.
a. Location/Owner of Data refers to the physical location of the data, e.g., Education Data Warehouse or school paper files.
b. Access refers to the degree to which the data are available to District Data Team members. (1 = hard to access; 4 = easily accessible).
c. Current Data Use describes how the data are used to inform decisions at the district, school, and/or classroom level.
3. Consider involving others in the data collection:
a. Ask personnel such as the SIS data manager, assessment coordinator, or guidance director to contribute information.
b. Consider having each school complete sections A2, B2, C2, and D2, in order to learn what schools collect and how the data are
used.
1.5.1T: Data Inventory Template—Version 1.0 1/9

<<District Name>> Data Inventory
SECTION A1— Demographic Data: District-wide Measures

Instructions: In the table below, list all the district-wide data elements currently collected that relate to the demographics and
indicators of all students, teachers, and other district staff, e.g., race, gender, special education, ELL, grade level, lunch status,
program participation, cost per pupil, average teacher salary, or time on learning. Not all columns may apply to all elements.
Program/ Location/ Access

Grade Date Data
Data Element Department/ Owner of Current Data Use
Level(s) Available (1–4)
Content Area(s) Data
(add more rows as

needed)

<<School Name>> Data Inventory
SECTION A2— Demographic Data: School-based Measures

Instructions: In the table below, list all the data elements currently collected related to the demographics and indicators of only
some students, teachers, and other district staff, e.g., race, gender, special education, ELL, grade level, lunch status, program
participation, cost per pupil, average teacher salary, or time on learning. Not all columns may apply to all elements.

Grade Date Data
Data Element School Department/ Owner of Current Data Use
(add more rows as

needed)

SECTION B1 — District and School Processes: District-wide Measures

Instructions: In the table below, list all district-wide data elements currently collected that relate to the many processes that take
place in the district. This includes information on (or generated by) specific programs, instruction, curriculum, professional
development, hiring, finances, facilities, technology, and district policies. Not all columns may apply to all elements.

Date Data
Available (1–4)
(add more rows as

needed)

SECTION B2— District and School Processes: School-based Measures

Instructions: In the table below, list all the data elements currently collected that relate to the many processes that take place in the
schools. This includes information on (or generated by) specific programs, instruction, curriculum, professional development, hiring,
finances, facilities, technology, and district policies. Not all columns may apply to all elements.

Date Data
Data Element School Department/ Owner of Current Data Use
Available (1–4)
(add more rows as

needed)

SECTION C1— Stakeholder Perception Data: District-wide Measures

Instructions: In the table below, list all the district-wide data elements currently collected that relate to the values, beliefs, and
perceptions that teachers, students, parents, and other stakeholders have of the working and learning environment of the district.
This includes data from surveys, focus groups, program evaluations, and other formal feedback systems. (Not all columns may apply
to all elements).
Program/ Location/ Date Access

Grade
Data Element Department/ Owner of Data Current Data Use
Level(s) (1–4)
Content Area(s) Data Available
(add more rows as

needed)

SECTION C2— Stakeholder Perception Data: School-based Measures

Instructions: In the table below, list all the data elements collected by schools that related to the values, beliefs, and perceptions
that teachers, students, parents, and other stakeholders have of the working and learning environment of the district. This includes
data from surveys, focus groups, program evaluations, and other formal feedback systems. (Not all columns may apply to all
elements).
Program/ Location/ Date Access

Grade
Data Element School Department/ Owner of Data Current Data Use
Level(s) (1–4)
Content Area(s) Data Available
(add more rows as

needed)

SECTION D1— Student Outcome Data: District-wide Measures

Instructions: In the table below, list the name of each student outcome measure that is currently collected for all students in a given
grade level district-wide. Examples include common assessments (such as MCAS, MELA-O, or common mid-terms), as well as other
outcome data (mobility, course grades, GPA, attendance and graduation). For each assessment, provide the indicated information in
the columns to the right. (Not all columns may apply to all elements). If an element is not collected for all students in a given grade,
record it in Section A2.
Date Location/
Grade Content Date Data Access
Data Element Administ- Owner of Current Data Use
Level(s) Area(s) Available (1–4)
ered Data
Attendance rate All N/A? Daily Student Monthly 2 Monthly principal attendance
Information reports.
Management
System (SIS)
(add more rows

as needed)

SECTION D2— Student Outcome Data: School-based Measures

Instructions: In the table below, list the data elements related to student outcomes that are currently collected for only some
students at the same grade-level district-wide. For example, some schools may implement commercial assessments that others do
not. For each student outcome data element, provide the indicated information in the columns to the right. (Not all columns may
apply to all elements).
Date Location/
Grade Content Date Data Access
Data Element School Adminis- Owner of Current Data Use
Level(s) Area(s) Available (1–4)
tered Data
(add more rows

as needed)

D A T A I N V E N T O R Y T E M P L A T E : S I M S A N D E P I M S D A T A 1 . 5 . 2 T
Purpose To develop an inventory of currently available data and how the data are being used in service Related Documents
of teaching and learning. 1–Getting Ready Module
1.5.1T: Data Inventory
Description The attached template is pre-populated with the data elements collected for SIMS and EPIMS.
Template
Districts can use this as a starting point to determine current availability and use of this data in
1.5.3R: ESE Data
the district, in conjunction with other data identified in the district’s Data Inventory.
Resources
Time 1–2 hours to review template; 1–2 weeks to gather information, with ongoing upkeep.
Notes:
This tool is meant to be used in conjunction with 1.5.1T: Data Inventory Template, which has further guidance and directions.
o The DOExxx series represents data elements from the Student Information Management System (SIMS)
o The IDxx, SRxx, and WAxx series represent data elements from the Education Personnel Information Management
System (EPIMS)
For descriptions and more information on these data elements see:
o SIMS Version 2.1 Data Handbook http://www.doe.mass.edu/infoservices/data/sims/DataHandbook.pdf
o EPIMS Data Handbook: http://www.doe.mass.edu/infoservices/data/epims/
Directions:
1. For each data element identified, confirm the information that is pre-populated, and complete the remaining columns.
a. Location/Owner of Data refers to the physical location of the data, e.g., Education Data Warehouse or school paper files, and/or
the person or department who is responsible for collecting the data and ensuring their quality.
b. Access refers to the degree to which the data are available to District Data Team members. Rate Access on a scale of 1–4
(1 = hard to access; 4 = easily accessible).
c. In the Current Data Use column, describe how the data are currently used to inform district-level decisions. The Team can
decide if it also wants to describe how the data are used to inform decisions at the school or classroom levels.
1.5.2T: Data Inventory Template: SIMS and EPIMS Data—Version 1.0 1/11
SECTION A1— Demographic Data: District-wide Measures (SIMS and EPIMS Data)
Grade Date Data
The DOExxx series represents data elements from the Student Information Management System (SIMS)
DOE001 – All
Locally Assigned All
Student Identifier
(LASID)
DOE002 – All All
State Assigned Student
Identifier (SASID)
DOE003 – All All

Student First Name
DOE004 – All All

Student Middle Name
DOE005 – All All

Student Last Name
DOE006 – All All

Student Date of Birth
DOE007 – All All

Date of Birth Format
DOE008 – All All

City/Town of Birth
Grade Date Data
DOE009 – All All
Gender
DOE010 – All All

Race/Ethnicity
DOE011 – All All

Reason for Reporting
DOE012 – All All

Enrollment Status at
Time of Data Collection
DOE013 – All All

Reason for Enrollment
DOE014 – All All

City/Town of Residence
– Student
DOE015 – All All

School Identification
Number
DOE016 – All All

Grade Level
DOE017 – All All

Days in Attendance
DOE018 – All All

Days in Membership
Grade Date Data
DOE019 – All All
Low-Income Status
DOE020 – All All

Title I Participation
DOE021 – All All

LEP Students in their
First Year in U.S.
Schools
DOE022 – All All

Immigration Status
DOE023 – All All

Country of Origin
DOE024 – All All

First (Native) Language
DOE025 – All All

Limited English
Proficiency
DOE026 – All All

English Language
Learners Program Status
DOE027 – All All

Alternative Education
Grade Date Data
DOE028 – All All
Title I School Choice
Participation
(DOE029 & DOE030 All All

were discontinued)
DOE031 – All All

Career/Vocational
Technical Education –
Competency Attainment
DOE032 – All
Special Education Pre-K to
Placement, ages 3–5 1st
DOE033 – All
High School Completer 12
Plans
DOE034 – All
Special Education 1–12
Placement, ages 6–21
DOE035 – All
Career/Vocational 6–12
Type of Program
DOE036 – All All
Special Education –
Nature of Primary
Disability
DOE037 – 12 All
Graduate, Completed
Massachusetts Core
Curriculum
Grade Date Data
DOE038 – All All
Special Education –
Level of Need
(DOE039 has been

discontinued)
DOE040 –
Special Education All All
Evaluation Results
(DOE041 has been

discontinued)
DOE042 – 6–12 All

Career/Vocational
Special Population
DOE043 – 6–12 All
Career/ Vocational
Technical Education -
Chapter 74–Approved
VTE Program
Participation
DOE044 – 12 All
Career/Vocational
Non-Chapter 74 Career
and Technical
Education Program
Participation
DOE045 – All All

Number of In-School
Suspensions
Grade Date Data
DOE046 – All All
Number of Out-of-
School Suspensions
DOE047 – All All

Advanced Placement
Course 1
DOE048 – All All

Advanced Placement
Course 2
DOE049 – All All

Advanced Placement
Course 3
DOE050 – All All

Advanced Placement
Course 4
DOE051 – All All

Advanced Placement
Course 5
DOE052 – All
Student Truancy All
The IDxx, SRxx, and WAxx series represent data elements from the EPIMS
ID01, ID02, ID03 – All

Staff First, Middle, and
Last Names
ID04 – All
Staff Date of Birth
ID05 – All
Staff Gender
Grade Date Data
ID06 – All
License / Certification
Number
ID07 – All
Local Employee
Number
SR01 – All
Massachusetts
Education Personnel
Identifier (MEPID)
(SR02 & SR03 align

with ID07 and ID06,
respectively)
SR04 – SR07 align with

ID01–ID04,
respectively)
SR08 – All
Race-Ethnicity
SR09 – All
Employment Status at
Time of Data Collection
SR10 – All
Reason for Exit
SR11 – All
Date of Hire
SR12 – All
Federal Salary Source 1
Grade Date Data
SR13 – All
Percent of Federal
Salary Source 1
SR14 – All
SR15 – All
Percent of Federal
Salary Source 2
SR16 – All
SR17 – All
Percent of Federal
Salary Source 3
SR18 – All
Degree Type 1
SR19 – All
Degree Institution 1
SR20 – All
Degree Subject 1
SR21 – All
Degree Type 2
SR22 – All
Grade Date Data
SR23 – All
Degree Subject 2
SR24 – All
Degree Type 3
SR25 – All
SR26 – All
Degree Subject 3
WA01 – All
Massachusetts
Education Personnel
Identifier (MEPID)
WA02 – (see ID07– All

Local Employee
Number )
WA03–WA05 (see All

ID01–ID03)
WA06 – All
District / School
Identification Number
WA07 – All
Job Classification
WA08 – All
Teacher /
Paraprofessional
Assignment
Grade Date Data
WA–09 – All
Grade
WA10 – All
Subject Area-Course
Code
WA11 – All
Class Section
WA12 – All
Full Time Equivalent
(FTE) (as per DSSR)
WA13 – All
NCLB Instructional
Paraprofessional
Requirements
WA14 – All
Highly Qualified
Teacher Status
WA15 – All
Subject Matter
Competency
ESE DA TA RE SOUR CES 1 . 5 . 3 R
Purpose This group of resources and tools can help a District Data Team understand Related Documents
what data is made available to districts by the Massachusetts Department of 1–Getting Ready Module
Elementary and Secondary Education, what it means, how to access it, and 1.5.1T: Data Inventory Template
how to use it effectively. 1.5.2T: Data Inventory Template: SIMS
and EPIMS Data
Description Website links to the most current information on a variety of data sources.
Time Ongoing.
The majority of data available from the ESE resides in one of two locations:
1. Education Data Warehouse: A collaborative effort of ESE and local school districts to centralize K–12 educational performance
data into one state-coordinated data repository hosted by the Department. It contains the SIMS and MCAS data for every district
in the state and will soon contain the EPIMS data for every district in the state. Data are available at the level of the state, district,
group, and individual student. Over 30 reports exist to compare data from individual schools and districts to state totals. After
receiving appropriate training, districts can load local data into the EDW and write their own reports. EDW training materials and
data can be accessed via the security portal. EDW Quick Tips and the EDW User Guide from the Information Service’s
Education Data Warehouse are available for download on the EDW webpage: http://www.doe.mass.edu/infoservices/dw/.
2. School/District Profiles: Sortable data reports on a variety of information, including enrollment, teacher data, and MCAS results.
Data are available at the level of the state, district, and group, but not at the level of individual student. Directories and reports
from individual organizations can also be found here: http://profiles.doe.mass.edu/.
Other sources of data are indicated as appropriate.
The MA ESE also publishes an annual data collection schedule which includes forms, descriptions of data, technical guidance, and
access information for data that are transmitted from districts to the state:
http://www.doe.mass.edu/infoservices/data/schedule.html.
1.5.3R: ESE Data Resources—Version 1.0 1/4

ESE Data Resources
Data Element Release Source of User Guides and Other Resources

and Brief Description Schedule Data
http://www.doe.mass.edu/mcas/
Massachusetts Comprehensive
Assessment System (MCAS): Designed
Information on MCAS participation and graduation
to test all public school students in the
Data requirements, testing schedules, test administration
state, measure performance based on the
Warehouse resources, test design and development, sample
Massachusetts Curriculum Framework
Annually, questions, scoring guides, technical reports and
learning standards, and report on the
in June results, as well as information on training sessions for
performance of individual students, schools, School/District the above.
and districts. Profiles
Additional guidance is available in the Data
Warehouse trainings, which can be accessed through
the security portal.
Adequate Yearly Progress (AYP): A http://www.doe.mass.edu/sda/ayp/
measure of the extent to which a student
group demonstrates proficiency in English Access to reports on AYP data as well as the review
language arts and mathematics based on and release schedules, baseline and improvement
Annually, See link
MCAS results. AYP Reports are issued data, student performance goal spreadsheet, and
in October in next column
each year and show the progress schools interpretive materials. Includes information on the
and districts are making toward the goal of Composite Performance Index (CPI).
having all students reach proficiency by the
year 2014.
Student Growth Percentile: The growth Data http://www.doe.mass.edu/mcas/growth/

model complements the MCAS year-by- Warehouse
Annually,
year test scores, since it reports change Links to student growth scores at the level of district,
in
over time rather than grade-level School/District school, and student group, as well as resources for
September
performance results in any one year. Profiles understanding and using the data.

ESE Data Resources
Data Element Release Source of Data User Guides and Other Resources
and Brief Description Schedule
Massachusetts English Proficiency http://www.doe.mass.edu/mcas/mepa/

Assessment–Reading/Writing (MEPA-
Reports of results as well as information on
R/W): Assesses LEP students' proficiency in
English Language Proficiency Benchmarks and
reading and writing at grades K–12. See link
Early Outcomes (ELPBO), student participation
Massachusetts English Language
June in next column requirements, sample student work and scoring
Assessment–Oral (MELA-O) assesses LEP
guides, MEPA test administration resources, and
students' proficiency in listening
information on training for the above.
(comprehension) and speaking (production)
at grades K–12.
EPIMS: Collects demographic data and work http://www.doe.mass.edu/infoservices/data/epims/

assignment information on individual public Three Data collection training materials, data handbook,
school educators, enabling Massachusetts to times a Education Data list of support specialists, maintenance tutorials,
comply fully with the No Child Left Behind year Warehouse FAQs, and a list of vendors.
Act by accurately reporting on highly (starting (beginning in
qualified teachers. EPIMS replaced the Note: Beginning in SY2011, EPIMS data will be
in 2010)
DSSR (District School Staffing Report). collected in October, March, and at the end of the
SY2011)
year, and are generally available in the EDW 2–3
months later.
SIMS: A student-level data collection system http://www.doe.mass.edu/infoservices/data/sims/

that allows the Department to collect and
SIMS user guide, data handbook, and other
analyze more accurate and comprehensive
training guides, as well as information on SIMS
information, to meet federal and state Three expansion, SSDR, reporting schedule, reporting
reporting requirements, and to inform policy Education Data
times a guidelines, FAQs, and information on vendors.
and programmatic decisions. Includes a Warehouse
year
unique student identifier for all students Note: SIMS data is collected via the security portal
receiving a publicly funded education in in October, March, and at the end of the year, and
Massachusetts. are generally available in the EDW 2–3 months
later.

ESE Data Resources
Data Element Release Source User Guides and Other Resources

and Brief Description Schedule of Data
http://www.doe.mass.edu/infoservices/reports/
Statistical Reports. The ESE provides Links to other ESE web pages with data and
statistical reports in the following areas: additional information on each type of statistical
Graduation rates report.
Grade retention reports Note: The Select Report list for School and District
Dropout rates See link in next Data Reports has some but not all of the same
Varies
Educator data column reports available in the Quick Statewide Reports
Enrollment data list on the School and District Data Reports page
Plans of high school graduates itself.
Student exclusions
School and district data reports
State profile
http://finance1.doe.mass.edu/statistics/
Statistical Comparisons. The ESE
provides statistical comparisons for districts
in the following areas: Links to data reports and supporting resources.
Some data sets allow for easy comparison to
Per pupil expenditure reports similar districts, and can be easily downloaded as
Enrollment trends See link in next Excel files.
Varies
Average teacher salaries column
Special education direct expenditure
trends
School and district data reports
State profile

D A T A C O L L E C T I O N S E L F - A S S E S S M E N T 1 . 6 . 1 T
Purpose To assess the effectiveness of data collection systems. Related Documents

Description For data to effectively provide information to further the inquiry process, they must be
complete, accurate, and disseminated in a timely manner. Within a busy district or school
environment, this is often difficult to achieve. This activity provides an opportunity for the
District Data Team to identify strengths and areas for improvement in the district’s data
collection, storage, and dissemination systems.
Time 30 minutes to develop a distribution plan.
Variable for data survey completion and tabulation.
One hour for survey analysis.
Directions
1. The District Data Team should familiarize itself with the self-assessment instrument and delete any items necessary to adapt
it to the Team’s local situation.
2. As a team, determine who should participate in the survey process.
3. Distribute the instrument to the target audience.
4. Collect and tabulate the results.
5. Analyze the results to determine the effectiveness of the data collection, storage, and dissemination systems.
6. Recommend changes to improve the system as necessary.
1.6.1T: Data Collection Self-Assessment—Version 1.0 1/5

Data Collection, Storage, and Dissemination Self-Assessment
This survey is designed to gather your perception of the efficiency and effectiveness of data collection, storage, and dissemination in
your district. Please share your perceptions by indicating your degree of agreement with the following statements.
Strongly Strongly Not

Data Task Statement about Data Disagree Agree
Disagree Agree Applicable
Data Policies and protocols are in place to guide data 1 2 3 4 N/A
Collection collection in the district.
A schedule is in place that indicates when various 1 2 3 4 N/A

data elements should be collected.
There are staff charged with collecting data. 1 2 3 4 N/A
Staff charged with collecting the data have a clear 1 2 3 4 N/A

understanding of what the data will be used for.
Staff charged with collecting the data know the 1 2 3 4 N/A

protocols for inputting the data, e.g., field names
and locations.
Staff charged with collecting the data are provided 1 2 3 4 N/A

with an environment that promotes the accurate
input of data, e.g., free of distractions, no
conflicting tasks.
Staff charged with collecting data have adequate 1 2 3 4 N/A

time to complete their tasks.
Staff charged with collecting the data have been 1 2 3 4 N/A

trained in data input techniques and data use
concepts.

Appropriate hardware is available to expedite the 1 2 3 4 N/A
collection of data.
Appropriate software applications are available to 1 2 3 4 N/A

facilitate the collection of data.
Protocols are in place to monitor the accuracy and 1 2 3 4 N/A

completeness of the data inputted.
Staff charged with collecting data adhere to district 1 2 3 4 N/A

guidelines for recording data.
State and federal confidentiality regulations are 1 2 3 4 N/A

followed by those responsible for collecting data.
Systems are in place to ensure that complete and 1 2 3 4 N/A

accurate data are recorded.
Staff who are responsible for data collection are 1 2 3 4 N/A

included in establishing data collection protocols
and policies.
Staff charged with the collection of data are 1 2 3 4 N/A

consulted to determine changes that need to be
made to improve data collection processes, e.g.,
accuracy, completeness, security.
Data Storage Data are added to the student information system 1 2 3 4 N/A
in a timely manner.

Data stored in the student information system can 1 2 3 4 N/A
be easily uploaded to the data warehouse.
Data from various sources can easily be uploaded 1 2 3 4 N/A

to the central district storage medium.
Web-based applications are in place to facilitate 1 2 3 4 N/A

the uploading of data to the central district storage
medium.
Data are archived to provide the basis for 1 2 3 4 N/A

longitudinal analysis.
Data Data can easily be retrieved from the student 1 2 3 4 N/A

Dissemination information system and/or data warehouse to
provide reports that answer specific questions.
A system exists to facilitate the acquisition of data 1 2 3 4 N/A

by staff to answer questions to improve teaching
and learning.
Reports are routinely generated and disseminated 1 2 3 4 N/A

to key staff to answer questions related to
improving teaching and learning.
Staff members know how to access data that they 1 2 3 4 N/A

need to answer questions to improve teaching and
learning.

Reports generated through the district data 1 2 3 4 N/A
information systems are easy for staff to
understand.
Reports are disseminated in a timely manner. 1 2 3 4 N/A

/

DA TA DI SSEMINATI ON S CHEDULE E XA MPLE 1 . 7 . 1 R
Purpose To communicate data availability and use broadly throughout the district. Related Documents
Description This is an example of a Data Dissemination Schedule. 1.7.2T: Data Dissemination
Time Ongoing. Schedule Template
1.7.3R: ESE Policies for Data
Access
Scenic Cove School District
Sample Data Dissemination Schedule
2009–2010
Time Frame Data Displays Disseminated To Purpose Action

August Longitudinal Trends Data overview Collaborative 1. Articulate problem
MCAS (2002–2009) presentation to the problem 2. Craft clarifying questions
Subtests district leadership identification and 3. Identify additional data needed
team clarifying question 4. District Data Team collects and
formulation analyzes data and builds data
displays for next meeting
Opening of school Principal Overview of Balance teams

demographics and program Team leaders composition of
participation school and teams
By grade level
By team
September Middle school performance of Principal Identify populations Conduct a data overview for each
incoming ninth graders by team Team leaders in need of school-level team
Grade 9 teachers intervention
Lists of students in at-risk Grade 9 team leaders Identify specific Develop interventions as
populations, e.g., Grade 8 and teachers students for necessary for identified students
Failures; Grade 8 Poor intervention at the team or grade level
Attendance.
1.7.1R: Data Dissemination Schedule Example—Version 1.0 1/1

DA TA DI SSEMINATI ON S CHEDULE TEMPLATE 1 . 7 . 2 T
Purpose To communicate data availability and use broadly throughout the district. Related Documents
Description In this activity, a District Data Team will construct and publish a schedule for the 1.7.1R: Data Dissemination
distribution and use of major data elements. Schedule Example
Time Ongoing. 1.7.3R: ESE Policies for Data
Access
1.7.2T: Data Dissemination Schedule Template—Version 1.0 1/2

<<District Name>>
Data Dissemination Schedule
<<School Year>>
Time Frame Data Displays Disseminated To Purpose Action
August
September
October
November
December
January
February
March
April
May
June
July
1.7.2T: Data Dissemination Schedule Template—Version 1.0 2/2

ESE P OLI CIES FOR DA TA A CCESS 1 . 7 . 3 R
Purpose To connect districts to current ESE policies for data Related Documents
access in order to inform district policies on data 1–Getting Ready Module
access and dissemination. 1.7.1R: Data Dissemination
Schedule Example
Description These documents should be reviewed, and
1.7.2T: Data Dissemination
corresponding district policies developed, prior to
Schedule Template
disseminating data within the district.
Time N/A.
These ESE Education Data Warehouse resources provide background information necessary
for a District Data Team, in conjunction with the district’s Information Technology Department, to
assign data access consistent with federal, state, and local regulations.
Website Brief Description
The Statewide Educational Data

Warehouse Project Policy Statement
outlines the legal authorities at both the
federal and state levels ― that govern the
information exchange and access, and
confidentiality of student records and
1 http://www.doe.mass.edu/infoservices/dw/accesspolicy.pdf personally identifiable information.‖ It
provides information on:
User agreements
Confidentiality policies
Third party access to information
in the Data Warehouse
The Security and Training in Chapter Two

of the Education Data Warehouse User
Guide provides information on:
Security and user administration
Maintaining confidentiality
2 http://www.doe.mass.edu/infoservices/dw/chapter2.pdf Suppressing aggregate data for
small groups
Responsibilities of the directory
administrator
District data access policies
Training district users
1.7.3R: ESE Policies for Data Access—Version 1.0 1/1

D A T A L I T E R A C Y T R A I N I N G C A T A L O G 1 . 8 . 1 T
Purpose This group of resources and tools will help a District Data Team build district-wide Related Documents
capacity to use data. 1–Getting Ready Module
1.8.2R: Assessment Glossary
Description This template will help the Team identify the training and support the district currently
provides for data literacy.
Time Ongoing.
Directions
1. Begin by listing the data used in your district. If you have completed 1.5.1T: Data Inventory, you can use that list of data elements. If you
have not yet completed a data inventory, you may want to begin by focusing this activity on the most commonly used data elements.
2. After you have listed the data, complete the next three columns.
3. As you consider each data element, make note of any opportunities for improvement.
Training or Resource
Data Element Provided Audience Department Responsible
1.8.1T: Data Literacy Training Catalog—Version 1.0 1/2

Training or Resource
Data Element Provided Audience Department Responsible
1.8.1T: Data Literacy Training Catalog—Version 1.0 2/2

ASSESSMENT GLOSSARY 1.8.2R
Purpose This group of resources and tools will help a District Related Documents
Data Team build district-wide capacity to use data. 1–Getting Ready Module
1.8.1T: Data Literacy
Description This document outlines general assessment
Training Catalog
terminology.
Time Ongoing.
Note: These terms apply primarily to student assessment data, but could be extrapolated to apply to
other forms of data, such as those related to adult practice or district systems and processes.
Aggregated Data: Data that are presented in summary (as opposed to student-level data or data broken
down by student group).
Alignment: Judgmental procedures undertaken to ensure the content of state tests appropriately reflects
the knowledge, skills, and abilities articulated in the state’s content standards for each grade level and
subject area.
Benchmark: A standard against which something can be measured or assessed.
Cohort: A group of individuals sharing a particular statistical or demographic characteristic.
Decile: One of ten segments of a distribution that has been divided into tenths. The ninth decile shows
the number (or percentage) of the norming group that scored between 80 and 90 NCE.
Disaggregation: Summary data split into different subgroups, e.g., gender, race, ethnicity, lunch status.
Distractor: An incorrect option in a multiple choice test item.
Equating: A set of statistical procedures undertaken in order to a) adjust for differences in the difficulty of
different test forms for the same subject area and grade level from year-to-year (horizontal equating), or
b) scale test scores (and/or performance levels) so they have a consistent meaning across adjacent
grade levels (vertical equating, vertical scaling, vertical articulation or moderation).
Formative: Assessments at regular intervals of a student’s progress designed to provide information to
improve the student’s performance.
Gain Score: The difference between two administrations of the same test. A student can have either a
positive or negative gain.
Generalization: Application of inference to a population greater than the sample.
Inference: A conclusion that is drawn from a data set. The process of using data from a sample of
students to generalize to other similar groups of students, such as assuming the observed three-year
th
upward trend in 10 grade mathematics achievement will continue next year.
Item: An individual question or exercise in an assessment or evaluative instrument.
Mean: The average of a set of scores.
Measure: Outcome data that can be used to measure the performance of a student or group of students.
Includes test scores, attendance, discipline, grades, and credits earned.
Median: The score that is the midpoint in a series of scores; half of the data values are above the
median, and half are below.
Mode: The score that occurs most frequently in a series of scores.
1.8.2R: Assessment Glossary—Version 1.0 1/3

Norm Group: A group of students with similar characteristics, i.e., age, number of months from the start
of the school year, number of years in school, selected to take a test to provide a range of scores and
establish the percentiles of performance for use in establishing scoring standards.
Normal Curve: The bell-shaped curve of the normal distribution.
Normal Curve Equivalent: A score that ranges from 1–99 often used to compare different tests for the
same student or group of students on the same test. Mathematically, an NCE is a normalized test score
with a mean of 50 and a standard deviation of 21.06.
Normal Distribution: A distribution of scores where the scores are distributed symmetrically above and
below the mean.
Norm-Referenced Test: Standardized tests designed to compare the scores of children to scores
achieved by children the same age who have taken the same test.
Percent Correct: A percentage that expresses the number of raw points earned by a test taker divided
by the number of raw points possible on the test.
Percent Proficient: The percentage of students who scored higher than the cut score defined by the test.
Percentile: A score that indicates the percentage of a reference or norm group obtaining scores equal to
or less than the test-taker’s score.
Performance Assessment: Assessments that measure skills, knowledge, and ability directly through a
performance or demonstration by the student.
Population: Every student who is eligible to become a member of a specific sample of students. For
th th
example, the population of 10 graders is all 10 graders who may be enrolled in the district.
th
Quartile: A division of percentile scores into four equal groups. For example, Q1 = 0 to 25 percentile
scores.
Range: The difference between the highest and lowest score in a distribution of scores.
Raw Score: The number of points earned on a test or subtest.
Reliability: The degree to which the results of an assessment are dependable and consistently measure
particular student knowledge and/or skills. Reliability is an indication of the consistency of scores across
raters, over time, or across different tasks or items that measure the same thing. Thus, reliability may be
expressed as a) the relationship between test items intended to measure the same skill or knowledge
(item reliability), b) the relationship between two administrations of the same test (or comparable tests) to
the same student or students (test/retest reliability), or c) the degree of agreement between two or more
raters (rater reliability). An unreliable assessment cannot be valid.
th
Sample: Group of students included in a data set. For example, the group of 10 graders in a district for
th
any one school year is a sample of the entire population of 10 graders who may be enrolled in the
th
district. The extent to which that group of 10 graders is representative of the entire population is the
th
extent to which generalizations can be made to 10 graders in the future.
Sampling Error: Statistical terminology for the possibility that a particular sample chosen to study may be
unusual in some way, leading to invalid or inaccurate inferences about the characteristics of the larger
th
population from which the sample was drawn. For example, when comparing the performance of 10
th
graders in one year to 10 graders in the next, it is important to bear in mind that the performance is
th
based on two different groups (samples) of 10 graders who may have different characteristics.
Scaled Scores: In the same way that the centigrade thermometric scale can also be expressed on the
Fahrenheit scale, student raw scores can be converted to scaled scores. Equating adjustments may
result in different raw score ranges for performance levels from year-to-year. Raw scores can be scaled
so that scaled score ranges for performance levels stay the same from year-to-year.
Scoring Rubrics: Guidelines for judgmental procedures for assigning values to student performance
such as checklists, yes or no, numerical rating scales, i.e., 1-6, or descriptive, i.e., the student presented
multiple points of view to support her essay.

Standard Error of Measurement (SEM): Based on the reliability of a test—the higher the reliability, the
lower the SEM. SEM can be used to put an ―
uncertainty‖ band around individual raw scores and scaled
scores.
Standardization: A consistent set of procedures for designing, administering, and scoring an
assessment. The purpose of standardization is to assure that all students are assessed under the same
conditions so that their scores have the same meaning and are not influenced by differing conditions.
Stanine: A normalized score that describes pupil performance on an equally distributed nine-point scale
ranging from 1 to 9.
Subtest: A group of test items that measure a specific area, i.e., mathematics calculation and reading
comprehension. Several subtests make up a test.
Summative: Assessments used to provide information in order to make a judgment about a student’s
achievement at the end of a period of instruction.
Validity: The extent to which an assessment measures what it is supposed to measure and the extent to
which inferences and actions made on the basis of test scores are appropriate and accurate. For
example, if a student performs well on a reading test, how confident are we that that student is a good
reader? A valid standards-based assessment is aligned with the standards intended to be measured,
provides an accurate and reliable estimate of students' performance relative to the standard, and is fair.
An assessment cannot be valid if it is not reliable.
Definitions were compiled from the following sources:
CRESST Assessment Glossary, found at http://www.cse.ucla.edu;
Slaughter, R. (2008). Assessment literacy handbook: A guide for standardized assessment in public
education. Portsmouth, NH: Public Consulting Group, Inc.
Massachusetts Department of Elementary and Secondary Education. (2008). Data Warehouse 102
Handbook: Understanding MCAS Reporting. Malden, MA: Author.

MANA GI NG CHANGE A ND UNDER STA NDING
CONCERNS PROTOCOL 1.9.1T
Purpose To enable the District Data Team and others to gain a Related Documents
better understanding about its concerns and the 1–Getting Ready Module
concerns of other staff as the change to a district norm
of inquiry and data use moves forward.
Description In this activity, the Team will review common concerns
associated with change and brainstorm ways to
mitigate concerns related to the implementation of the
Data-Driven Inquiry and Action Cycle.
Note: Keep in mind that this activity is a good faith attempt to take into account the concerns of
constituents, but there is no way to know for sure without asking them directly. A District Data
Team may choose to follow in this protocol on its own, or to engage different stakeholders in the
process through surveys or focus groups. While the former approach may take less time, the
latter could generate valuable perspectives and ideas that the Team may not think of on its own.
Directions:
1. As a group, identify the stakeholders who will likely be impacted by the district’s
increased focus on data use, inquiry, and action.
2. Individually review the seven stages of concern that individuals commonly experience in
response to a change effort. Record specific concerns the various stakeholders may
have for each of the stages. Be sure to include yourself and your own concerns.
Individuals may ask themselves ― What am I hearing from the field?‖
Stage General Concern Potential Specific Concerns of

Stakeholders
Awareness What is this change I’ve
been hearing about?
Information Tell me everything I need
to know.
Personal What does this mean for
me?
Management How will I manage all of
this?
Consequence What will happen if I do
implement the change?
What will happen if I don’t?
Collaboration How can we help each
other through the change?
Refocusing How can I make it even
better?
1.9.1T: Managing Change and Understanding Concerns Protocol—Version 1.0 1/2

3. Work as a group to record everyone’s responses for each of the stages. Use chart paper
or project typed notes so all can see and read the responses.
Note: If the Team has completed 1.2.1T: Barriers to Effective Data Use, it may want
to revisit and compare the responses. Does this make the group think of any
additional stakeholders that need to be considered, or concerns to address?
4. As a group prioritize* which concerns are most important for the District Data Team to
address as it pursues this work, the group should narrow the list to 2–5 specific
concerns. In doing this, the team might consider:
Which stage, as a whole, best represents the general sentiment of the district?
Which specific concerns, if resolved, would result in the greatest shift toward an
embedded culture of data use, inquiry, and action?
Which concerns does the District Data Team have the greatest potential and
leverage to address?
*One quick strategy for prioritizing is multi-voting. Give each team member a
number of votes that represents 1/3 to 1/2 of the total ideas generated, e.g., if 21
ideas were generated, each member could get 8 votes. Members cast one vote for
each idea they see as a priority. Tally the votes to determine which ideas are seen
as the greatest priorities.
5. As a group, brainstorm ways to mitigate the impact of each of the prioritized concerns.
Record the suggested strategies on a new sheet of chart paper.
Note: Again, it may be useful to reference the strategies generated in 1.2.1T:
Barriers to Effective Data Use
Prioritized Concerns Strategies for Mitigation or Resolution
6. Discuss, prioritize, and come to agreement on the strategies that make the most sense
to pursue at this time. Document these strategies and revisit them periodically, noting
concerns that get resolved, and new ones that may emerge. The group may want to
retain all of the notes from this discussion for future reference as well.
1.9.1T: Managing Change and Understanding Concerns Protocol—Version 1.0 2/2

MODULE 2: INQUIRY
Table of Contents
Introduction—1
Culture of Inquiry—2
The Data-Driven Inquiry and Action Cycle—2
Asking the Right Questions—3
Types of Questions—3
Question Formulation—4
Alignment With District Priorities—7
Data Overview Process—8

Anatomy of a Data Overview—8
Preparing a Data Overview—11
Building a Data Display—11
Designing the Data Overview—12
Module Summary—13
Inquiry
2.1.1T: Question Formulation Protocol 2.4.2R: Data Display Rubric

2.2.1T: Inventory of District and School 2.4.3R: Types of Data Displays
Initiatives 2.4.4R: More Data Display Resources
2.3.1T: Data Overview Checklist
2.5.1T: Data Overview Brainstorming Protocol
2.3.2R: Data Overview Example
2.5.2T: Focusing Question Investigation
2.4.1T: Building Data Displays Protocol Template
District Data Team Toolkit—Version 1.0 Module 2: Inquiry
INTR ODUCTI ON
WHERE ARE WE NOW?

improvement. The Toolkit uses the steps of the Cycle to structure a
progression through the model—you are now in Module 2: Inquiry.
Getting
Ready
Module 2: Inquiry explores the inquiry process first introduced in Module

1: Getting Ready. The activities in this module build the capacity of the
District Data Team and key stakeholders to formulate hypotheses about
problems identified through an initial review of data displays and develop
clarifying questions to dig deeper into the data. Techniques for the
acquisition of these data and their analysis are addressed in Module 3:
Information and Module 4: Knowledge.
MODULE OBJECTIVES
The Inquiry module will help a District Data Team:
 Formulate questions to drive an inquiry process
 Create and present effective data displays and data overviews
 Identify the data needed to answer the questions

CULTURE OF INQUIRY
THE DATA-DRIVEN INQUIRY AND ACTION CYCLE
The Data-Driven Inquiry and Action Cycle drives the effective use of data
to answer critical questions about teaching and learning that result in
school improvement and higher achievement for all students. If the Team
asks the right questions, collects and analyzes appropriate data to
address the questions, views the information it has gathered in the
context of findings on research and practice to form an appropriate
knowledge base, and takes action on the knowledge it has gained, the
district and its schools will improve and its students will perform at higher
levels.
The modules in this Toolkit take you step-by-step through the Data-Driven
Inquiry and Action Cycle and provide you with the tools and resources
necessary to effectively implement this collaborative data use framework.
Module 2: Inquiry initiates this activity.
Data-Driven Inquiry and Action Cycle

ASKIN G T HE R IGH T QU ESTI ONS
TYPES OF QUESTIONS
Educators ask questions about their district, schools, and students all the
time. The questions are based on their observations, experience, gut, and
hopefully, on data. The challenge is to craft meaningful questions to drive
the inquiry process that are based on all of these sources and that, if
answered, will significantly improve teaching and learning in the district.
As the Team formulates questions that will have a direct impact on

teaching and learning in the district, it should consider two things.
1. Does the question relate to something over which the district or

school has control?
2. Does the question relate to something which, if improved, will
have a significant impact on teaching and learning?
Questions about factors that districts and schools can influence form the Questions about
basis for the action step of the Data-Driven Inquiry and Action Cycle. factors that districts
and schools can
The Team may also ask questions about factors that can have an effect
on teaching and learning, but that cannot be influenced or changed by influence form the
districts and schools. These questions are more descriptive in nature and basis for the action
help educators develop a better understanding of their students. This step of the Data-
understanding can provide insight into structures and strategies that can
Driven Inquiry and
be implemented which place teaching and learning in the context of
students’ experiences. Action Cycle.
For example, the size and distribution of the low-income student

population in a district is not a factor a district can control. However,
knowing the size and distribution of this population may suggest that
professional development activities are needed to help teachers better
understand the student population and what support they need to learn.
As the Team goes through the question formulation process, it will

discover that starting with broad questions to focus its inquiry will lead to
the development of narrower questions that will deepen the inquiry
process. The broad questions are called focusing questions, while the
narrower questions are called clarifying questions. Focusing questions
provide a starting point to help the Team identify the data it will need to

begin its inquiry. Clarifying questions are generated in response to the

analysis of the initial data set and often require the collection and analysis
of additional data. In turn, based on this subsequent data collection and
analysis, original clarifying questions can become focusing questions for
the next phase of inquiry.
QUESTION FORMULATION
Questions that a Team might want to explore can be formulated based on
some of the following considerations: demographics, perceptions, school
processes, and student outcomes. Each category provides the district a
framework for which it can begin to craft both focusing and clarifying
questions from the data gathered. Districts can then use those questions
to guide the next steps in the data review process.
Demographics Perceptions School Processes Student Outcomes

Race Values Programs Assessments
Gender Beliefs Instruction Course grades
Special education Perceptions Curriculum GPA
Grade level teachers, parents, Processes Teacher observations
and students have
Lunch status Policies Attendance
of the learning
environment Procedures Dropout rate
Practices

Focusing Questions
Broad questions are called focusing questions. Focusing questions
provide a starting point to help a Team identify the data it will need to
begin its inquiry. By beginning with the broad categories above, a district
can begin the process of looking at data across sets of schools.
Sample Focusing Questions

Are the district’s teachers utilized in the most effective and efficient
manner to meet the needs of its students?
Are teachers throughout the district committed to high levels of

learning for all types of learners?
Are the programs for special populations effectively meeting their goals?
What are the characteristics and performance levels of students with

high absence rates?
Activity 2.1 Question Formulation Protocol
Throughout this Toolkit, the Team will use protocols to guide productive
discussions on a variety of topics. The Question Formulation Protocol
will help the District Data Team develop, organize, and prioritize
questions to structure its inquiry.
(2.1.1T: Question Formulation Protocol)

Clarifying Questions
Narrower questions are called clarifying questions. Focusing questions
provide a starting point to help a Team identify the data it will need to
begin its inquiry. Clarifying questions are generated in response to the
analysis of the initial data set and often require the collection and analysis
of additional data. In turn, based on this subsequent data collection and
analysis, original clarifying questions can become focusing questions for
the next phase of inquiry.
Sample Clarifying Questions

Focusing Question #1: Are the district’s teachers utilized in the most
effective and efficient manner to meet the needs of its students?
Potential Clarifying Questions:

 What is the ratio between special education students and special
education teachers in each of the district’s schools?
 Are the “highly qualified teachers” equitably distributed across
schools in the district?
 Are teachers assigned to classes in their area of certification?
 Are the district’s neediest students taught by the most effective
teachers?
Focusing Question #2: Are teachers throughout the district committed to

high levels of learning for all types of learners?

 Are teachers asking all students questions that would foster higher
level thinking skills?
 Are students placed in least restrictive environments with
modifications and accommodations being followed?
 Do students receive instruction in varied ways that meet their
individual learning styles?
Focusing Question #3: Are the programs for special populations

effectively meeting their goals?

 Are students in inclusion programs and substantially separate
programs achieving proficiency on the state assessment?
 Over the past three years, has the performance of English
language learners improved by the end of grade 6?

 Has growth within special populations equaled the growth of the

general population?
Focusing Question #4: What are the characteristics and performance
levels of students with high absence rates?

 What is the relationship between absenteeism and performance
on state assessments?
 Which subgroups and grade levels have the highest absence
rates? Lowest?
 When does high absenteeism occur throughout the school year?
 How does the district’s absence rate compare to the state?
ALIGNMENT WITH DISTRICT PRIORITIES

The Question Formulation Protocol from the previous activity has helped
the Team to define the high-priority starting point for its inquiry. But before
the Team begins gathering data to answer its questions, it is important to
determine if there are initiatives currently underway in the district that
relate to the Team’s focusing question, and how further investigation of
these questions can be coordinated with these existing initiatives. The
Inventory of District and School Initiatives tool will help the Team identify
and create an inventory of district and/or school initiatives. The Team
should follow the instructions in the tool to help it relate those initiatives to
its focusing question. This information can serve three key purposes:
 Help the Data Team coordinate efforts with other existing teams
 Help the Team identify data that might be available to inform the
inquiry process
 Help the Team avoid redundancy when it gets to the point of

developing strategies and action steps (Module 5)
Activity 2.2 What Current Initiatives Relate to the Focusing

Question?
Districts and schools have many initiatives in place at one time. Adding
a new initiative that addresses a focusing question may be redundant if
the question is already being effectively addressed by an existing
initiative. The Inventory of District and School Initiatives will identify
current initiatives and will provide data on the effectiveness of the
implementation of those initiatives.
(2.2.1T: Inventory of District and School Initiatives)

DA TA OVE RVI EW PR OCE SS
After completing the Inventory of District and School Initiatives activity in

the previous section, the Team should be ready to proceed with its
inquiry. The next step in the collaborative inquiry process is to share the
focusing question and related data with an appropriate audience by
creating and presenting a data overview.
The primary objective of a data overview is to enable stakeholders to

collaboratively interact with data related to the focusing question in order
to generate clarifying questions that will drive the inquiry process forward.
These clarifying questions will serve to focus subsequent efforts in data
collection and analysis.
To achieve this objective, the District Data Team must build user-friendly
data displays that tell a valid and interesting story about the focusing
question. The District Data Team must then involve stakeholders in the
collaborative analysis of the data and the creation of clarifying questions.
ANATOMY OF A DATA OVERVIEW

The data overview is a presentation (usually accompanied by
PowerPoint) designed to introduce to stakeholders preliminary data
related to a focusing question.
The specific content of a data overview will vary based on the audience
(administrative team, full faculty, department faculty, specialists), but the
purpose and structure remain constant. A typical data overview meeting
will contain the following sections.
Typical Agenda of a Data Overview Meeting
Welcome and  Outline the data overview presentation so

Introductions that the audience understands the purpose,
structure, and outcomes of the meeting.
 A quick roll call or introductions so everyone

knows who is in the audience, e.g., who
here is a teacher, family member, or district
administrator.

Purpose  Clearly state the purpose for the data

overview in terms of the focusing question(s)
that is being explored.
Data Displays  Use well-developed data displays that

clearly tell a story related to the focusing
question(s) and stimulate constructive
conversations.
 Collectively make factual observations (no

inferences) about each data display.
Brainstorming  Formulate hypotheses that might explain the

Session data.
 Pose clarifying questions to guide the

exploration of the hypotheses.
 Identify the kinds of data needed to answer

the questions and suggest ways to collect
the additional data.
Next Steps  Discuss next steps, such as action items

from the meeting, e.g., who will collect
additional data and by when, and the date
and time of the next meeting.
When planning a
When planning a data overview, it is important for the Data Team to data overview, it is
consider the audience and context and adjust the presentation important for the
accordingly. To do this, the Team might consider:
Data Team to
 What is the level of data literacy of the audience, e.g., principals, consider the
teachers, students, families, union representatives, school audience and
committee, or community members? context and adjust
 What questions might these stakeholders already be considering the presentation

in relation to the focus of inquiry? accordingly.
 What barriers to data use (from 1.1.1T) might be relevant to this

audience, and how could the data overview help address them?

The data overview should result in at least two specific outcomes. The set
of clarifying questions developed through the brainstorming protocol and
the identification of related data help guide the next steps in the inquiry
process. Additionally, as the group engages with the data and formulates
hypotheses and clarifying questions, they increase their capacity for
inquiry and become invested in the process. This buy-in is critical for
subsequent processes and is crucial toward creating a district-wide
culture of inquiry.
Activity 2.3 Anatomy of a Data Overview
In this activity, you will review and critique a sample data overview
presented by the Scenic Cove District Data Team. Review the
PowerPoint presentation and use the Data Overview Checklist to
determine if all of the essential elements are present. As a District Data
Team, discuss how the Scenic Cove School District ELA Data
Overview could be improved.
(2.3.1T: Data Overview Checklist)

(2.3.2R: Data Overview Example)

PREPARIN G A DAT A OVERVI EW
Building a data overview starts with articulating a focusing question. To

prepare, return to the Team’s work in 2.1 Question Formulation Protocol.
Select one of the focusing questions the Team developed and identify
and gather high-level data necessary to begin exploring it. This focusing
question will guide the creation of a data display. By creating a well
thought out focusing question, engaging in collaborative discussion
around that question, and by skillfully using data, the Team will begin to
build a story, and ultimately, this should also help the Team identify and
refine the clarifying questions and accompanying data displays.
BUILDING A DATA DISPLAY

At the center of any data overview sit quality data displays. Constructing
these displays requires careful thought and effort to ensure that the data
are displayed in a way that connect to the focusing question and inspire
clarifying questions that will drive deeper inquiry by the audience.
Quality data displays need to:
 Tell the whole story

 Have complete, accurate, and timely data
 Contain all relevant and pertinent data
 Be readable and understandable
Activity 2.4 Building a Data Display
The Building Data Displays Protocol enables District Data Team
members to apply the principles of data display construction to tell a
story related to a focusing question. The Data Display Rubric provides
a framework for the Team to assess the quality of the data displays it
creates. The Types of Data Displays and More Data Display Resources
provide some ideas for different ways that data can be represented.
(2.4.1T: Building Data Displays Protocol)
(2.4.2R: Data Display Rubric)
(2.4.3R: Types of Data Displays)
(2.4.4R: More Data Display Resources)

DESIGNING THE DATA OVERVIEW

Once the data displays are built and designed, the data overview will
begin to take shape and the Team will be ready to craft the rest of the
presentation. The agenda should follow the model outlined earlier.
 Purpose
 Agenda
 Data displays driven by a focusing question and additional
clarifying questions that tell a story
 Make observations about the data (no inferences)
 Record observations on chart paper

 Structured brainstorming session
 What might lie behind what the group is seeing in the data?
(careful inferences)
 What additional questions does the group have?
 What additional data are needed to answer those questions?

 Identify next steps
 What data need to be collected?
 Who will collect it?
 When will a follow-up meeting occur?

Working with the Team, construct a presentation in PowerPoint using the
data displays that have been built. It may be helpful to first storyboard the
presentation on chart paper with the Team. Return as necessary to
Activities 2.3 and 2.5 to create and refine data displays that support the
Team’s central theme.
With the data displays freshly made and assessed and the presentation
assembled, the Team is ready to engage a larger audience in the inquiry
process. Plan the meeting and deliver the data overview that the Team
has created.
Activity 2.5 Delivering the Data Overview
These tools will help the Team deliver a data overview and follow up
afterwards. The results of this work will lead the Team into Module 3:
Information and serve as the foundation for the rest of the inquiry cycle
throughout this Toolkit.
(2.5.1T: Data Overview Brainstorming Protocol)

(2.5.2T: Focusing Question Investigation Template)

MODULE SUMMARY
This module provides an overview of the Data-Driven Inquiry and Action

Cycle, which can serve as the foundation for conducting an inquiry
process. It launches the inquiry process by helping a District Data Team
formulate a meaningful focusing question that is of high interest to the
district and that will guide its inquiry process.
The module includes tools to help a District Data Team build its capacity
to design meaningful data displays and present them in an effective data
overview to targeted audiences. This approach can help engage
stakeholders in the inquiry process, as well as inform the process of
generating clarifying questions that refine the focus of the inquiry and
identifying the data needed to provide insight on those questions.
A District Data Team should emerge from this stage of the process with
clearly articulated focusing and clarifying questions, as well as a list of
data it plans to collect and analyze to answer those questions.
Answers to the clarifying questions generated through this process will

require the collection and analysis of additional data—the subject of the
next module in the District Data Team Toolkit—Module 3: Information.
REFERENCES
Sagor, R. (1992). How to conduct collaborative action research.

Alexandria, VA: Association for Supervision and Curriculum Development
(ASCD).

QUESTI ON FORMULA TION PROTOCOL 2 . 1 . 1 T
Purpose To formulate a focusing question derived from an Related Documents

issue of importance to your district. 2–Inquiry Module
Description This protocol will help the District Data Team develop,
organize, and prioritize questions to structure inquiry.
Time About 30 minutes.
Directions
1. Identify an issue in your district that you as a District Data Team wish to address. Write
the issue on the top of a piece of chart paper. It can be formulated as a statement or
question. Your issue/question should be related to student outcomes.
5 minutes
2. As a Team, brainstorm questions that stem from the original question/statement. Write
the questions as stated on the chart paper. All items must be phrased as questions.
Your questions should be related to student outcomes.
15 minutes
3. From this group of questions, identify three questions that deal with issues that the
district has control over and which, if positively resolved, will have a significant impact on
teaching and learning. Out of these three, identify the top priority question.
10 minutes
4. Your top priority question should serve as the focusing question to initiate the Data-
Driven Inquiry and Action Cycle.
2.1.1T: Question Formulation Protocol—Version 1.0 1/1

I N V E N T O R Y O F D I S T R I C T A N D S C H O O L I N I T I A T I V E S 2 . 2 . 1 T
Purpose To identify data sources and help avoid redundancy in relation to district/school initiatives Related Documents
that address the focusing question the District Data Team has decided to investigate. 2–Inquiry Module
Description Districts and schools may have multiple initiatives in place at one time. Adding a new
initiative that is potentially related to or has an impact on the focusing question may be
redundant if the question is already being effectively addressed by an existing initiative.
The Inventory of District and School Initiatives will identify current initiatives and will provide
data on the effectiveness of the implementation of those initiatives.
Time Approximately 1 hour.
Directions
As a District Data Team, think about the initiatives/programs that are currently part of the improvement efforts in your district. List
each initiative/program in the Inventory of District and School Initiatives on the following page. For each, provide the information
indicated in the columns to the right of the initiative name. The District Data Team may have to call upon others in the district to help
provide the required information.
 After you have gathered the required data on each initiative, determine which initiative/program(s) is directly related to your
focusing question.
 For the related initiatives/programs, consult the Effectiveness of Implementation and Desired Outcomes columns (3–5) to
determine which appear to be addressing your focusing questions effectively.
 If, as a District Data Team, you feel you need to gather more data to determine effectiveness, collect the data and re-evaluate
the initiatives.
 If the consensus of all relevant parties is that the initiative is achieving the desired result, select a new focusing question. If
not, move forward with the inquiry.
2.2.1T: Inventory of District and School Initiatives—Version 1.0 1/2

Instructions: Think about the initiatives/programs you currently have running as part of the school improvement efforts in your
district. Provide information about each initiative in the table below.
District Name:
Inventory of Instructional Initiatives
Effectiveness of Implementation and Desired Outcomes
Teachers Extent of
Implementing Implementation
4 = All (100%) 4 = Complete
Staff 3 = Most (>75%) 3 = Progressing Other Evidence that
Name of Instructional Responsible for 2 = Some (25–75%) 2 = Partially/Weak Evidence of Desired Would be Helpful to
Initiative Implementation 1 = Few (<25%) 1 = Just beginning Outcomes Collect
2.2.1T: Inventory of District and School Initiatives—Version 1.0 2/2

DA TA OVE RVI EW CHE CKLIS T 2 . 3 . 1 T
Purpose To provide the District Data Team with an example of a data overview presentation. Related Documents
2–Inquiry Module
Description In this activity, you will review and critique a sample data overview presented by the Scenic 2.3.2R: Scenic Cove School
Cove District Data Team. Review the PowerPoint presentation and use the Data Overview District ELA Data
Checklist to determine if all of the essential elements are present. As a District Data Team, Overview
discuss how the Scenic Cove School District ELA Data Overview could be improved.
DATA OVERVIEW CHECKLIST
District/School Name: Date:
Format & Structure Y/N

Does your Data Overview:
Identify the audience that will participate in the overview?
Have a purpose?
Have an agenda?
Contain data displays driven by a focusing question?
Include a structured brainstorming session?
Identify next steps?
Will the format and structure of your data overview result in specific outcomes that will move inquiry forward?
Agenda Y/N
Does your Agenda:
State the purpose of the data overview session?
List the data displays to be reviewed?
List the steps in the brainstorming process?
Include identifying next steps?
2.3.1T: Data Overview Checklist—Version 1.0 1/2

Data Displays Y/N
Do the Data Displays:
Contain the attributes of a good chart?
Appear free of unnecessary detail and extraneous features?
Use the most appropriate chart style to display the data?
Tell the story that you want to convey about the data?
Brainstorming Y/N
Will the structure of the Brainstorming activity result in:
The identification of problems evident in the data?
The identified problems being listed in priority order?
The formulation of hypotheses to explain the problem?
Clarifying questions to further direct the inquiry?
The identification of additional data needed and potential data sources?
Next Steps Y/N
Do the identified Next Steps:
Logically follow from the outcomes of the brainstorming session?
Contain action items?
State the date and time of the next meeting?
Identify the audience and/or participants in the next meeting?
2.3.1T: Data Overview Checklist—Version 1.0 2/2

DATA OVERVIEW EXAMPLE 2.3.2R
Purpose To provide the District Data Team with an example of a data Related
overview presentation. Documents
2–Inquiry Module
Description In this activity, you will review and critique a sample data 2.3.1T: Data
overview presented by the Scenic Cove District Data Team. Overview
Review the PowerPoint presentation and use the Data Checklist
Overview Checklist to determine if all of the essential
elements are present. As a District Data Team, discuss how
the Scenic Cove School District ELA Data Overview could be
improved.
2.3.2R: Data Overview Example—Version 1.0 1

Data Overview Presentation, September 2009
Presented by the District Data Team
2.3.2R: Data Overview Example—Version 1.0

2
Agenda
1. Purpose of this presentation
2. Presentation of relevant data displays
3. Collaboratively brainstorm to:
• Identify and prioritize a problem evident in the data
• Formulate hypotheses to explain the problem
• Generate clarifying questions to direct further inquiry

• Identify additional data needed and potential data
sources
4. Discuss next steps
3
The Issues
 The academic performance of English language
learners (ELLs) in the Scenic Cove School District is
lower than the statewide population of ELLs at several
grade levels on 2009 MCAS English Language Arts
(ELA) tests.
 The academic performance of ELLs in the district lags
behind their native English-speaking peers.
 The academic performance of ELLs in the district falls
below established NCLB performance and improvement
targets.
4
Purpose
To begin the process of collaborative
inquiry that will address the focusing
questions:
1. What does the performance of ELLs in the district look like over
time on MCAS ELA tests?
2. How does the performance of ELLs in the district compare to that of

ELLs state-wide on the 2009 MCAS ELA test?
3. Why is the performance of ELLs in some grades closer to the state

average than others on the 2009 MCAS ELA test?
4. What is the performance of ELLs in targeted grades in district
schools on the 2009 MCAS ELA test?
5. How does the performance of ELLs in the district compare to the
state on the 2009 MEPA test?

5
Tests Administered 2006–2009
2006 2007 2008 2009
as values Change
3 A/P 25.3% 23.3% 31.5% 27.1% 1.8%
NI 51.6% 47.8% 41.6% 38.6% -13.0%
What does the W/F 23.2% 28.9% 27.0% 34.3% 11.1%

4 A/P 3.7% 11.5% 11.8% 14.6% 10.9%
performance of NI 53.2% 39.7% 38.2% 43.9% -9.3%
ELLs in the W/F 43.1% 48.7% 50.0% 41.5% -1.6%
district look 5 A/P 2.6% 8.0% 1.7% 9.6% 7%
like over time NI 40.2% 27.6% 14.8% 33.7% -6.5%
on MCAS ELA W/F 57.3% 64.4% 83.5% 56.6% -0.7%
tests? 6 A/P 5.0% 6.3% 15.9% 11.5% 6.5%
NI 26.7% 33.3% 36.5% 32.8% 6.1%
W/F 68.3% 60.3% 47.6% 55.7% -12.6%
7 A/P 10.0% 1.5% 9.4% 9.7% -0.3%
NI 20.0% 27.3% 25.0% 29.2% 9.2%
Source: Data Warehouse > Public W/F 70.0% 71.2% 65.6% 61.1% -8.9%
Folders > ESE Cubes > MCAS
Official Release 2009 8 A/P 4.5% 7.8% 7.4% 7.8% 3.3%
NI 22.7% 18.9% 25.3% 20.7% -2.0%
W/F 72.7% 73.3% 67.4% 71.6% -1.1%
10 A/P 16.3% 4.7% 5.9% 3.0% -13.3%
NI 46.5% 31.3% 35.3% 19.7% -26.8%
W/F 37.2% 64.1% 58.8% 77.3% 40.1%
6
District State Difference
Tests Administered
2009 2009
as values
3 A/P 27.1% 26.2% 0.9%
How does the (N = 70)
NI 38.6% 40.7% -2.1%
performance of ELLs W/F 34.3% 33.0% 1.3%
in the district 4 A/P 14.6% 18.0% -3.4%
(N = 82)
compare to that of NI 43.9% 46.8% -2.9%
ELLs state-wide W/F 41.5% 35.2% 6.3%
on the 2009 MCAS 5

(N = 83)
A/P 9.6% 16.3% -6.7%
NI 33.7% 40.5% -6.8%
ELA test? W/F 56.6% 43.2% 13.4%
6 A/P 11.5% 18.7% -7.2%
(N = 61)
NI 32.8% 34.3% -1.5%
W/F 55.7% 47.0% 8.7%
7 A/P 9.7% 13.1% -3.4%
Source: Data Warehouse > Public
(N = 72)
NI 29.2% 34.7% -5.5%
Folders > ESE Cubes > MCAS Official
Release 2009
W/F 61.1% 52.2% 8.9%
8 A/P 7.8% 12.9% -5.1%
(N = 116)
NI 20.7% 27.0% -6.3%
W/F 71.6% 60.1% 11.5%
10 A/P 3.0% 21.8% -18.8%
(N = 66)
NI 19.7% 36.8% -17.1%
W/F 77.3% 41.4% 35.9% 7
District State Difference
Tests Administered
2009 2009
as values
3 A/P 27.1% 26.2% 0.9%
Why is the (N = 70)
NI 38.6% 40.7% -2.1%
performance of ELLs W/F 34.3% 33.0% 1.3%
in some grades 4 A/P 14.6% 18.0% -3.4%
(N = 82)
closer to the state NI 43.9% 46.8% -2.9%
average than others W/F 41.5% 35.2% 6.3%
on the 2009 MCAS 5

(N = 83)
A/P 9.6% 16.3% -6.7%
NI 33.7% 40.5% -6.8%
ELA test? W/F 56.6% 43.2% 13.4%
6 A/P 11.5% 18.7% -7.2%
(N = 61)
NI 32.8% 34.3% -1.5%
W/F 55.7% 47.0% 8.7%
7 A/P 9.7% 13.1% -3.4%
Source: Data Warehouse > Public (N = 72)
NI 29.2% 34.7% -5.5%
Folders > ESE Cubes > MCAS Official
Release 2009 W/F 61.1% 52.2% 8.9%
8 A/P 7.8% 12.9% -5.1%
(N = 116)
NI 20.7% 27.0% -6.3%
W/F 71.6% 60.1% 11.5%
10 A/P 3.0% 21.8% -18.8%
(N = 66)
NI 19.7% 36.8% -17.1%
W/F 77.3% 41.4% 35.9% 8
What is the performance of ELLs in targeted grades in
district schools on the 2009 MCAS ELA test?*
Sea Breeze Golden

Sea Gull ES Rip Tide ES Coral ES
ES Sands ES
Tests Administered
2009 2009 2009 2009 2009
as Values
3 N 4 21 0 0 32
A/P 100.0% 23.8% 21.9%
NI 0.0% 23.8% 56.3%
W/F 0.0% 52.4% 21.9%
4 N 6 2 4 2 49
A/P 16.7% 0.0% 0.0% 0.0% 18.4%
NI 83.3% 0.0% 100.0% 100.0% 34.7%
W/F 0.0% 100.0% 0.0% 0.0% 46.9%
5 N 3 5 15 42 0
A/P 100.0% 0.0% 0.0% 2.4%
NI 0.0% 0.0% 60.0% 31.0%
W/F 0.0% 100.0% 40.0% 66.7%
*Minimum 10 Students
Source: Data Warehouse > Public Folders > ESE Cubes > MCAS Official Release 2009

9
Coastal MS Rock MS
Tests Administered
2009 2009
as Values
6 N 53 8
A/P 13.2%
NI 34.0% 25.0%
W/F 52.8% 75.0%
7 N 70 2
A/P 10.0%
NI 30.0%
W/F 60.0% 100.0%
8 N 95 18
A/P 2.1% 27.8%
NI 20.0% 22.2%
W/F 77.9% 50.0%
Source: Data Warehouse > Public Folders > ESE Cubes > MCAS Official Release 2009

10
Ebb Tide HS
Tests Administered
2009
as values
10 N 66
A/P 3.0%
NI 19.7%
W/F 77.3%
Source: Data Warehouse > Public Folders > ESE Cubes > MCAS
Official Release 2009

11
Spring 2009 MEPA Results:
Percentage of Students at Each Performance Level
by Years of Enrollment in the State and Scenic Cove
All Students - State 4 6 21 48 20
All Students - SC 20 18 32 23 7
First Year - State 32 22 21 21 4
First Year - SC 45 23 22 6 3
Second Year - State 5 15 30 37 13
Second Year - SC 10 19 34 27 10
Third Year - State 1 7 26 51 15
Third Year - SC 4 16 38 35 8
Fourth Year - State 1 3 21 55 20
Fourth Year - SC 3 6 38 38 16
Fifth Year or More - State 12 17 52 28
Fifth Year or More - SC 7 10 45 38 0
0% 20% 40% 60% 80% 100%
Level 1 Level 2 Level 3 Level 4 Level 5
Source: Massachusetts English Proficiency Assessment (MEPA) Statewide Results: Spring 2009; Spring 2009 MEPA Results by District
http://www.doe.mass.edu/mcas/mepa/results.html
Brainstorm Groups
Group A Group B Group C Group D
Jerry Jill Colleen Kerry
Maria Saleem Michelle Alex
Jose Jeff Clarence Jordan
Roger Patricia Cindy Ann

13
Brainstorm!
Purpose To collaboratively investigate the

focusing question.
What you will Using the Data Overview Brainstorming
do? Protocol, you will make observations,
identify problems, form hypotheses, and
craft clarifying questions related to the
focusing question.
Estimated About 30 minutes.

Time

14
Next Steps
 What additional data do we need to collect?
 Who will collect the data?
 How will the data be collected?
 When will we have collected the data?
 Who will analyze the data?
 What materials should we have for the next meeting?
 Next meeting date and time?

15
BUILDI NG DA TA DIS PLAYS PROTOCOL 2 . 4 . 1 T
Purpose To build data displays based on data related to the Related Documents
focusing question. 2–Inquiry Module
2.4.2R: Data Display Rubric
Description This activity enables District Data Team members to
2.4.3R: Types of Data
apply the principles of data display construction to tell a
Displays
story related to the focusing question.
2.4.4R: More Data Display
Time Approximately 1 hour. Resources
During this activity, the District Data Team will work collaboratively to create a strong data display for
the selected focusing question. Before beginning, review and become familiar with the related tools
and resources listed above. These should be used each time you prepare a data display.
Directions
1. As a District Data Team, restate the focusing question you crafted in 2.1.1.T: Question
Formulation Protocol and post it for all to see.
For example: How did the achievement of the population of English language learners
vary across all schools in the Scenic Cove School District?
2. Examine a few sources of high-level district data, such as MCAS/AYP reports or student
growth data. Think individually, then discuss as a Team: What do you see in the data that
relates to the focusing question? As a Team, brainstorm and chart the collective observations
for all to see. Be sure to make only factual observations and interpretations about what the
data appear to say—don’t make inferences from the data.
3. Each member of the District Data Team should now sketch on a piece of chart paper a data
display that illustrates what s/he thinks are the most important observations. Refer to 2.4.3R:
Types of Data Displays for guidance regarding the construction of data displays. Post the data
displays for the whole District Data Team to review.
Note: This next section works best in groups of 3–4.
4. Each District Data Team member should present one data display to the balance of the Team
or to one of the small groups. Number the data displays to identify them later.
5. Each presenter should ask group members what they see in the data (observations, not
inferences). Presenters should record observations on chart paper for each display. (5–10
minutes).
6. Then each member should explain to the group:
Why s/he chose a particular display type
What story s/he wanted to tell with the display that s/he selected
What clarifying questions the display elicits for him/her (5–10 minutes)
7. After each presentation, each person fills out the Data Display Rubric for each data display,
including the presenters.
2.4.1T: Building Data Displays Protocol—Version 1.0 1/2

8. Repeat this process until all have presented his or her display.
9. Think individually and discus as a Team: How do the sketches compare? Be sure to record
answers for future reference.
10. Regroup as a District Data Team. Review the feedback on each data display. Spend about
5–10 minutes digesting and summarizing the feedback. Note common themes across data
displays. Discuss the various sketches that Team members created and reach consensus as
a District Data Team on the data displays that best communicate the story to be told about the
data.
11. Save all of the sketches for future reference.
Alternative Approach
1. Have District Data Team members work in pairs to collaboratively develop each data display.
2. Often there is more than one story to be told by a set of data. See how many different valid
and interesting stories about the data can be told using different data displays.
2.4.1T: Building Data Displays Protocol—Version 1.0 2/2

DA TA DI SP LA Y R UBRI C 2 . 4 . 2 R
Purpose To assess the quality of data displays and gain Related Documents
feedback to improve them. 2–Inquiry Module
2.4.1T: Building Data
Description This rubric can be used to assess the quality of a data
Displays Protocol
display. It can be used with the Data Display Feedback
2.4.3R: Types of Data
Protocol to gain group input, or can be used as a tool
Displays
for individual reflection.
Time 15 minutes. Resources
Focusing Question Driving the Data Display:
Data Display Number:
Use the scale provided below to rate each of the following statements about the data display.
4 = Excellent: No 3 = Good: Some 2 = OK: Moderate 1 = Not So Good:
change needed changes needed changes should Needs extensive
be made rework
Question Rating Comments

1. The data display contains attributes of an
effective data display: all axes are labeled
and the display includes an informative title;
population assessed; number and percent
of students; subject and test; and when
they were assessed.
2. The data display is uncluttered and free of

unnecessary detail and extraneous
features.
3. The data display uses an appropriate

choice of chart style, e.g., clustered bar
chart, correlation chart, scatter chart.
2.4.2R: Data Display Rubric—Version 1.0 1/2

Answer the following questions with an open response.
Question Response
4. What do you like about this data
display?
5. Is there anything that makes it

difficult to understand?
6. What are some concrete

suggestions that could make this
display more effective or easier to
understand?
2.4.2R: Data Display Rubric—Version 1.0 2/2

TYP ES OF DA TA DIS PLAYS 2 . 4 . 3 R
Purpose To understand various types of data displays and Related Documents

their potential uses. 2–Inquiry Module
2.4.1T: Building Data
Description This resource can help a Team choose data displays
Displays Protocol
that will be most useful for engaging in thoughtful
data analysis or communication with stakeholders.
Time Ongoing. Resources
Simple Bar Chart

A simple bar chart shows a frequency distribution for a single variable, e.g., percent Proficient,
on a specific measure for components within a
single category, e.g., grade-level populations.
Each bar displays the results for each
individual category component (as opposed to
relative distribution, as in a pie chart). A simple
bar chart can answer questions such as:
What percent of students in each grade
level achieved Proficiency for a particular
school year?
How do the results for one population

subgroup compare to those of other subgroups?
Clustered Bar Chart

A clustered bar chart allows you to disaggregate
data by a category or subgroup. For example,
you would use a clustered bar to look at
performance across years, between subgroups
of students, e.g., gender, lunch status, or across
grades. A clustered bar chart can answer
questions such as:
How did students who are eligible for free- or
reduced-price lunch (FRPL) perform
compared to students who are not?
Which grade level achieved the highest
percentage of correct items? The lowest?
What was the performance of our students across subject areas or strands?
What subject or curriculum areas show the greatest need for improvement?
2.4.3R: Types of Data Displays—Version 1.0 1/6

Stacked Bar Chart
A stacked bar chart allows you
to see the trend across a given
category (performance
category in this example), and
then within each category
component. It allows you to see
the relative distribution of
results across another
category, e.g., FRPL.
A stacked bar chart can be
oriented on the vertical or
horizontal axis.
Either form of stacked bar chart can help answer questions such as:
Which performance category has the highest concentration of students receiving FRPL?
Which grade level has the highest concentration of lower-performing students?

Simple Line Chart
A simple line chart is similar to a simple bar chart,
except the data are represented with a line rather
than a bar. Some people like to use line charts
when representing data across a time scale (as in
the example). Some prefer to use line charts only
when the data represent the same group of
students over time (a cohort) because the line
suggests movement.
A simple line chart can help answer questions
such as:
What are the CPI results for the fourth grade for the last six years?
Multiline Chart
A multiline chart is similar to a clustered bar chart,
except that the data are represented with lines
rather than bars. As with the single line chart, some
people like to use multiline charts when
representing data across a time scale (as in the
example).
A multiline chart can help answer questions like:
Are we closing the achievement gap
between two student groups over time?
Correlation Chart
A correlation chart allows you to
examine the relationship between two
different measures using two different Y
axes. The first measure appears as a
bar chart whose scale is on the left Y
axis. The second measure appears as a
line chart whose scale is on the right Y
axis.
A correlation chart allows you to answer
questions such as:
What is the distribution of % Correct compared to the number of tests administered across
grade levels?
What is the relationship between the number of correct items and the number of possible
items?

Scatter Chart (or Bubble Chart)
A scatter chart allows you to look
at the relationship between two
different measures using the X and
Y axes. The first measure is
represented on the Y axis, and the
second measure is represented on
the X axis. In the sample graph to
the right, each circle represents
one student. In the sample graph
below, each circle represents one
school.
A scatter chart can help answer
questions such as:
What is the relationship between a student’s scaled MCAS score and his or her student
growth percentile?
What is the correlation between local district assessments (or grades) and state assessment
scores?
A third dimension of
information can be added by
using different colors or sizes
to differentiate the data points.
For example, different colors
can distinguish membership in
different groups, e.g., income
status in the chart above.
Different size points can show
values, such as the age of the
individual, years of service, or
number of members, e.g., size
of school in the chart below.
Adding this third level of data
can be valuable for answering
questions such as:
Is there a relationship
between income status
and performance?
Is there a correlation between the size of a school and student proficiency and growth rates?
Scatter charts are usually accompanied by a table that summarizes the data and helps label
each individual point. Because each point represents a person or group (such as a school),
individuals can engage with the chart by locating themselves in the chart and asking:
Where do I (or my school, or my student) fall in relation to others?

Radar Chart (or Spider Graph)
A radar chart, also known as a spider chart
or a star chart because of its appearance,
plots the values of several measures along a
separate axis that starts in the center of the
chart and ends on the outer ring. This makes
it possible to compare data across
measures. Data that are consistent across all
of the measures will be displayed in a near
circle. Data that are higher in some
measures than others will be displayed in a
more free form shape.
A radar chart can help answer a question

such as:
How does student performance on

one strand of a test compare to
several other strands?

Consumers of data should be familiar with these last two types of data displays. However, in
general there are more informative ways to display data. These types of displays are rarely, if
ever, used in the Department’s Education Data Warehouse.
Pie Chart
A pie chart shows part-to-whole relationships.
Pie charts show the relative distribution of
performance for a specific population across
performance categories, which sum to 100%.
Pie charts can answer questions such as:
What was the relative distribution of student
scores across performance levels for a
specific subgroup?
Which subgroup had the highest proportion
of students achieving Proficiency?
3D Bar Chart
A 3D bar chart is helpful when you want to visually represent a data set across multiple
categories. It allows you to see the relationships and trends in a data set across three
dimensions.
A 3D bar chart allows you to answer
questions such as:
Where are our greatest
achievement gaps?
What do year-to-year trends tell
us about the learning needs of
different subgroups of students?
In which subject areas and grade
levels do we have the greatest
concentration of lower performing
students?

MOR E DA TA DISPLA Y RES OUR CES 2 . 4 . 4 R
Purpose To connect districts to additional resources on Related Documents

creating effective data displays. 2–Inquiry Module
Description This list can serve as a starting place for learning 2.4.1T: Building Data
more about how to display data in meaningful ways. Displays Protocol
Time N/A. 2.4.3R: Types of Data
Displays

― The National Forum on Education Statistics
develops free resources on a variety of issues that
1 http://nces.ed.gov/forum/publications.asp affect schools, school districts, and state education
agencies.‖ A number of these documents are
available for download at this website.
Perceptual Edge, founded by Stephen Few, is a
company ― that was established to help organizations
like yours learn to design simple information displays
http://www.perceptualedge.com/files/GraphDesignIQ
2 .html
for effective analysis and communication.‖ This short
IQ test can help you ― determine how well you
understand the principles of good table and graph
design.‖
Selecting the Right Graph for Your Message. In
this article, Stephen Few outlines some general
http://www.perceptualedge.com/articles/ie/the_right_
3 graph.pdf
principles that can be applied to a wide range of data
displays. He presents seven quantitative message
types, along with brief descriptions and examples.
Table and Graph Design for Enlightening
Communication is a very long but informative
PowerPoint that was presented by Stephen Few at
the National Forum on Education Statistics’ Summer
2009 Forum on Education Statistics. ― Mr. Few offered
three fundamental steps in the table and graph design
process:
1. Determine your message.
2. Select the best medium to display your
message.
3. Design all components of the display to show
http://nces.ed.gov/forum/pdf/NCES_table_design.pdf the data.”
4
The PowerPoint addresses several points:
Page Topic
1–12 Context
13–36 Examples of Good and Bad Graphs
37–41 Grice’s Maxims: Quantity, Quality,
Relevance, Manner
42–48 Fundamentals of Data Presentation
49–67 Transforming a Poor Graph
68–79 Parts of Displays
80–92 Design Principles
93–102 Table Design
103–178 Graph Design
2.4.4R: More Data Display Resources—Version 1.0 1/1

DATA OVERVIEW BRAINSTORMING PROTOCOL 2.5.1T
Purpose To provide a structure that enables all members of the Related Documents
target audience to become familiar with the focusing 2–Inquiry Module
question, engage with relevant data, and help further 2.5.2T: Focusing Question
the inquiry process. Investigation
Template
Description Use this protocol to facilitate the center of your data
overview presentation. The brainstorming activity
provides an opportunity for the target audience to
collaboratively interact with the data displays
associated with the focusing question. Through this
collaborative inquiry, the audience will identify
problems revealed by the data, develop hypotheses
about the cause of the problem, craft clarifying
questions to extend the inquiry process, and identify
data needed to address those questions.
Directions
Divide the target audience into groups of 4–5 people. Provide sticky note pads, chart paper,
markers, and large copies of the data displays for each group. Provide a facilitator for each group.
1. Write the focusing question on the top of a sheet of chart paper. Check to make sure
each person understands the question.
2. Post the large copy of the data display (or displays) for the group to view. These may
have been created in 2.4.1T: Building Data Displays Protocol.
3. Ask individuals to silently observe the data and record objective, factual observations
about what the data say in the data display. Ensure that all have adequate time to
process the information and ask clarifying questions if necessary.
4. Ask individuals to share their observations with the group. Record the observations on
chart paper with the focusing question next to the display. Highlight observations that
represent ―problems‖ revealed by the data.
5. On a new sheet of chart paper, the group should write the title–Hypotheses about
Possible Causes. They then brainstorm hypotheses about the causes of the ― problem(s)‖
revealed by the data and record them on the chart paper.
6. As a group, then write the title—Clarifying Questions—at the top of a new sheet of chart
paper.
2.5.1T: Data Overview Brainstorming Protocol—Version 1.0 1/2

7. Each group member should write one or more clarifying questions that stem from the
―problem(s)‖ identified by the group on a sticky note (one question per note). Place the
sticky notes on the Clarifying Questions chart paper.
8. As a group, review the questions and group similar questions together if possible.
Develop a title for each group such as: Questions about Achievement; Questions about
Relationships among Variables; etc.
9. Reach consensus on the clarifying questions that seem most appropriate to move the
inquiry deeper. Record these questions on a new piece of chart paper. Leave room
between questions on the chart paper, or put each question on a separate page.
10. Under each question, identify the evidence (data elements) that needs to be collected to
address each of the clarifying questions. If possible, note where each piece of data can
be found and how it can be collected.
11. Share the clarifying questions and additional data elements needed with the whole
group. The District Data Team will record the questions and data elements on a sheet of
chart paper for the whole group to see.
12. Use template 2.5.2T: Focusing Question Investigation Template to record the key ideas
for future reference.
Note: The District Data Team may choose to close the meeting at this point, or the Team
may ask the group to help prioritize the clarifying questions that would be most useful and
meaningful to extend the inquiry process. Either way, the Team should clarify next steps for
how the inquiry process will move forward, and how the stakeholders in attendance at this
data overview may be impacted.
2.5.1T: Data Overview Brainstorming Protocol—Version 1.0 2/2

FOCUSI NG QUES TION INV ESTI GA TION
TEMPLA TE 2.5.2T
Purpose To capture the results from the delivery of your data overview Related Documents
presentation. 2–Inquiry Module
2.5.1T: Data Overview
Description Use this template as soon after the delivery of the data
Brainstorming
overview as possible to record the key clarifying questions
Protocol
and other ideas it generated. This template can then be
shared with stakeholder groups in the district as work
proceeds around data collection.
Focusing Question
Hypothesized “Problem(s)” Discovered Through a Review of High-Level Data
1.
2.
3.
4.
Clarifying Questions Related to these “Problems”

1.
2.
3.
4.
Identifying Data Elements Needed:

Begin by referencing the notes from the data overview regarding the data needed to inform
each of the clarifying questions in your inquiry. For each clarifying question, list below the data
elements the district will need in order to address the question. For each data element, list
which domain of data the element represents and whether it is currently collected and
accessible to the Team. If the Team has completed 1.5.1T: Data Inventory Template, it might
want to use it for reference, as well as 1.5.2R: ESE Data Resources. If the data needed to
continue your inquiry is not currently being collected or is not readily accessible, indicate your
plan to acquire the required data.
2.5.2T: Focusing Question Investigation Template—Version 1.0 1/3

Clarifying Question #1:________________________________________________________
Domain Ease of
Data Elements Needed Demographics, Access* Plan to Acquire Required Data
to Address the Perceptions,
1–4, or Elements
Clarifying Question Processes, or (If level of Access is 1–2 or N/A)
Student Outcomes N/A
(add more rows as

needed)
*Access refers to the degree to which the data are available to District Data Team members. Rate
Access on a scale of 1–4 (1 = hard to access; 4 = easily accessible) or N/A if the needed data element is
not currently being collected.

Domain Ease of
1–4, or Elements
(add more rows as

needed)

Domain Ease of
1–4, or Elements
(add more rows as

needed)

Domain Ease of
1–4, or Elements
(add more rows as

needed)

MODULE 3: INFORMATION
Table of Contents
Introduction—1
Preparing Data For Analysis—2
Analyzing Data—4
Analyzing Assessment Data—5
Incorporating Other Types of Data—6
Facilitating the Process—7
Module Summary—8
Information
3.1.1T: Data Collection Planning Tool Also revisit tools from Inquiry:
3.2.1T: Practice Making Valid 2.4.1T: Building Data Displays Protocol
Inferences 2.4.2R: Data Display Rubric
3.3.1T: Data Analysis Protocol 2.4.3R: Types of Data Displays
2.4.4R: More Data Display Resources
District Data Team Toolkit—Version 1.0 Module 3: Information
INTR ODUCTI ON
WHERE ARE WE NOW?

progression through the model—you are now in Module 3: Information.
Getting
Ready
Raw data alone does not support the inquiry process. Central to turning
raw data into information is the process of data analysis. The Information
module can help a District Data Team build its capacity to analyze data
by considering the appropriate use of assessment results and the
formation of valid inferences.
Through the data overview process introduced in the Inquiry module, the
District Data Team identified the data needed to move the inquiry process
forward. Next the Team must collect and organize the data in order to be
poised to analyze the data and make meaning from them.
MODULE OBJECTIVES
The Information module will help a District Data Team:
 Collect and organize data relevant to the inquiry process
 Distinguish between observations and inferences
 Make inferences from multiple sources of data

PREPARI NG DATA FOR

ANA LYSI S
At this stage of the process, the District Data Team should develop a list
of data needed to address the clarifying questions related to its focus of
inquiry. Now the Team must actually collect the data and organize these
in a meaningful way that promotes rigorous analysis. The Focusing
Questions Investigation Template (2.5.2T) and the Data Inventory
Template (1.5.1T) can be useful in taking this next step.
Activity 3.1 Data Collection
This tool will help guide the collection of specific data needed to answer
a focusing question and related clarifying questions.
(3.1.1T: Data Collection Planning Tool)
Once data are collected, the Team will want to display the data in a
meaningful way that prompts curiosity and allows viewers to make
comparisons and inferences about causality. Displays should show as
much information as possible in as small an area as possible, without any
distractions or extraneous information. The Inquiry module introduced a
variety of data displays that may be useful for the Team to revisit at this
stage in the process. In addition, the Team may want to consider the
following questions:
Do the displays highlight contrasts and differences?

Do they show multiple factors?
Is evidence from different sources integrated?
Is the data of high quality and integrity?
Is it relevant to the questions being investigated?
Are displays laid out so important comparisons are in the same
page or eye span? (It’s best not to have to turn pages to compare
data)1

REVISIT 2.4 Building a Data Display
These tools were first introduced in Module 2: Inquiry, but may be
useful at this stage of the process.
The Building Data Displays Protocol enables District Data Team
members to apply the principles of data display construction to tell a
story related to a focusing question. The Data Display Rubric provides
a framework for the Team to assess the quality of the data displays it
creates. The Types of Data Displays and More Data Display Resources
provide some ideas for different ways that data can be represented.
(2.4.1T: Building Data Displays Protocol)
(2.4.2R: Data Display Rubric)
(2.4.3R: Types of Data Displays)
(2.4.4R: More Data Display Resources)

ANA LYZING DATA
A thoughtful and rigorous analysis of data is key to a successful inquiry

process. It can be easy to unknowingly approach data with the answer
already in our minds, consciously or unconsciously seeking evidence that
supports what we already believe to be true. Approaching data with a
truly open mind takes practice and discipline.
The first step in data analysis, as described in the data overview process
in the Inquiry module, is the objective description of what the data say.
What patterns and trends are evident in the data? It is very important to
Approaching data focus on this first step before making inferences or drawing conclusions
with a truly open from the data, because clarifying questions often need to be posed and
additional data collected before valid inferences can be made. Colleagues
mind takes practice
on a District Data Team can play an important role in helping each other
and discipline.
use language that is as specific and objective as possible when
discussing information and data. For example, helping each other
distinguish between observations and inferences:
Observation: Factual interpretations and statements about quantities,

e.g., “Over half the principals report…”; the presence of specific
information and/or numerical relationships between ideas, e.g., “Over
90% of the district’s schools have teams…”; or patterns, e.g., “most
principals report that their teams are focused on…” An observation
captures an unarguable fact and may be indicated by phrases such as
I observe that…, some patterns/trends that I notice…, or I am
surprised to see...
Example: About one third of our students performed below proficient

in mathematics.
Inference: A conclusion, explanation, or conjecture that is drawn from

a data set, such as using a smaller set of data to make broader
generalizations or predictions. An inference reflects the meaning that
the observer is making from the data, and may be indicated with
phrases like I predict…, I think…, because…, or therefore…, or by
imprecise qualifiers like smarter, adequate, or poorly.
Example: About one third of our students are not on track to meet the
mathematics criteria for graduation.

Both observations and inferences play crucial roles in the data analysis
process. What is important is to distinguish between the two. The Team
should be sure to rigorously examine the data for patterns, trends, and
outliers that can be factually explained, prior to making any inferences or
conclusions about what those patterns may mean.
ANALYZING ASSESSMENT DATA

Much of the data analysis work that the District Data Team will undertake
will involve assessment data. Prior to engaging in analysis, it is important
for the Team to have a common understanding of assessment terms,
concepts, and how these data should and should not be used to form
inferences about student performance. If the Team has not done so
already, it may want to review the resources on assessment literacy in the
Getting Ready module in order to expand the Team’s capacity in this
area.
Activity 3.2 Making Valid Inferences From the Data
During this activity, the Team will view multiple data displays and check
the inferences made by another data team for validity.
This activity also appears in the ESE Data Warehouse course DW 102.
The Data Displays used are all “Pre-defined Reports” from the ESE
Data Warehouse. You may want to revisit Activity 1.5 Assessment
Literacy.
(3.2.1T: Practice Making Valid Inferences)

INCORPORATING OTHER TYPES OF DATA

To this point, making inferences has been based on only one measure,
the MCAS. Yet no single data source can provide a complete picture of
the business of teaching and learning within the district. The District Data
Team can increase the validity and credibility of its inferences if it can use
a variety of related data sources to provide more information about the
question being investigated. Because unconscious biases and
assumptions can unknowingly skew analysis of data, a group can
increase the validity and credibility of the inferences it generates by
ensuring at least two data elements yield the same or similar information.
The Team might consider the impact of simultaneously examining data

from two or more of the domains discussed in the Getting Ready module:
student outcomes; perceptions of stakeholders; demographics of
students; faculty and staff; and school and district processes.2 Similarly a
team might consider looking at data from different intervals throughout the
year.
Questions the Team might ask when triangulating across data sets
include:
What patterns or inconsistencies are evident across the different

data sets?
Do different data sets reveal the same patterns and trends? If not,
what can be learned from the differences? (For example, does the
same student score at comparable levels of proficiency on
different assessment measures?)
How has the data changed over time? (Longitudinally)
How does the data compare with data from other populations in
the district?
Activity 3.3 Data Analysis Protocol
These protocols can guide the District Data Team in the process of
analyzing data from non-traditional and/or multiple sources.
(3.3.1T: Data Analysis Protocol)

FACILITATING THE PROCESS

When designing the format for a discussion of data, the district may want
to assign a facilitator who can help the group with the following:
Ensure all Team members have an equal voice in sharing

observations of the evidence that has been gathered
Put as much data on the table as possible, from high-level to fine-

grained observations
Keep the conversation at the level of specific and objective

evidence, redirecting people if the language drifts to become more
general and/or judgmental
Beware of allowing broad generalizations based on only one

source of evidence
Challenge each other’s assumptions and generalizations by

asking “why?” and “what’s the evidence?”
Be prepared to be surprised
Think ahead about what the group might want to report out to
others in the district and how, and look for ways to generate
reports and visuals as part of the discussion process. For
example, might the group want to leave certain flip charts up for
display and public comment? Would it help to type notes directly
into a laptop so they don’t need to be rewritten later?
It is crucial for the Team to create the conditions for thoughtful

consideration of the evidence. If the Team reads the data wrong, it can
misdiagnose the appropriate course of action. As with medicine, car
repairs, or other problem-solving processes, a misdiagnosis could not
only result in wasted time and resources, it could also actually cause
damage. The more time the Team spends engaging with data, critically
looking at the data, and asking each other hard questions about the
inferences drawn from these data, the more capacity it will build and the
more confidence it can have in the subsequent conclusions and actions
taken.

MODULE SUMMARY
The Information module provides concepts and tools that enable the
District Data Team to further the inquiry process introduced in the
preceding module, Inquiry. It offers guidance for collecting data
specifically related to the focusing and clarifying questions generated in
the Inquiry module, and revisits tools from that module to guide
meaningful displays of that data.
The module helps clarify the difference between making factual

observations from data and making inferences about what the data mean.
It also provides a protocol for the District Data Team to use to engage
with the data related to its inquiry process and the focusing and clarifying
questions that are guiding it.
The District Data Team should emerge from this stage in the process with
inferences or conclusions drawn from the data analysis process, and
perhaps also with some new questions for consideration.
All of this work sets the stage for the next module, Knowledge, which will
help the Team place the information that it gathers in the context of
research and practice literature, as well as local knowledge and expertise.
This will help the Team narrow and refine its focus even further as it
moves toward identifying strategies and actions steps to address the
problems that it has identified.
References
1
Adapted from Tufte, E. (2009, November 10). Presenting Data and
Information. Boston, MA.
2
Adapted from Bernhardt, V.L. (2004). Data Analysis for Continuous
School Improvement. Larchmont: Eye on Education.

DA TA COLLE CTION P LANNING TOOL 3 . 1 . 1 T
Purpose To guide the collection of specific data needed to answer a focusing question Related Documents
and its related clarifying questions. 3–Information Module
Description Use this template to identify who will collect specific data needed for analysis.
Time 30–90 minutes
Instructions: In the table below, begin by listing the specific data elements needed in order to address each of the clarifying
questions in your inquiry process. If the Team has completed 2.5.2T: Focusing Question Investigation Template, it can simply use
the list of data documented there.
For each data element, indicate the required information. A Data Inventory (1.5.1T) can help identify the location/owner of the data.
For this stage in the process, the most important details to note are who will collect the information, by when, and in what format.
Question/Issue being addressed: ___________________________________________________________
Data Element Needed Location/Owner Who Will Collect By When? In What Format?
It for the Team? Paper, Electronic, etc.
3.1.1T: Data Collection Planning Tool—Version 1.0 1/2

Data Element Needed Location/Owner Who Will Collect By When? In What Format?
It for the Team? Paper, Electronic, etc.
3.1.1T: Data Collection Planning Tool—Version 1.0 2/2

PRACTICE MAKING VALID INFERENCES 3.2.1T
Purpose To practice making valid inferences. Related Documents
Description This activity can be used within your Data Team or with 3–Information Module
other audiences to improve data analysis skills. During
this activity, you will have the chance to view multiple
data displays and ―che ck‖ the inferences made by
another data team for validity.
3.2.1T: Practice Making Valid Inferences—Version 1.0

1
Practice Making Valid Inferences
1. Read a scenario.
2. Review the accompanying data display to observe what
the data say.
3. Consider the statements provided.
4. As a Data Team, decide which statements are
observations (factual interpretations) and which are
inferences (conclusions, explanations, or conjectures).
5. Also, note whether each statement is true, not
necessarily true, or false.
6. Possible answers are provided at the end.

2
Scenario #1
SCENARIO
The Data Team in District A wanted to examine the performance of 8th grade
students on the 2007 MCAS ELA and Mathematics tests. The Team posed this
focusing question.
FOCUSING QUESTION
How did our 8th graders, district-wide, perform on the 2007 MCAS tests?

3
ESE Data Warehouse Pre-defined Report
R-303: District Performance Distribution
4
District Performance Distribution Report (R-303)
Observation Not
Statement or True Necessarily False
Inference? True
1A. Our students are smarter in
English than they are in
mathematics.
1B. Compared to the state, our
students performed poorly in
mathematics.
1C. About one third of our students
performed below proficient in
mathematics.

5
Scenario #2
SCENARIO
After comparing the performance of the students in District A to the performance

of students statewide, the Data Team posed a clarifying question.
CLARIFYING QUESTION
How did the mathematics performance of the 8th graders in our district change
over the past three years?

6
R-305: District Performance Distribution by Year
3.2.1T: Practice
Making Valid 7
Inferences—
Version 1.0
District Distribution by Year Report (R-305)
Observation Not
Inference? True
2A. Students who were in 8th grade
in 2007 made gains from year-
to-year since 2005.
2B. 8th grade performance has
improved from year-to-year.
2C. Our year-to-year trend
performance follows the state’s
trend performance.

8
Scenario #3
SCENARIO
The District Data Team reviewed the longitudinal performance of the district’s
students and concluded that the percent of students scoring at the lowest level
decreased each year and the percent scoring at the Advanced level increased
dramatically in 2007. This was encouraging, but the Team felt that performance
could still be improved. The Team formulated the following clarifying question.
CLARIFYING QUESTION
With which specific strands and standards did the students have the most
difficulty?

9
R-306: District Standards Summary Report
3.2.1T: Practice
Making Valid
Inferences— 10
Version 1.0
District Standards Summary Report (R-306)
Observation Not
Inference? True
3A. Our students performed better
than students statewide in each
of the strands.
3B. Out of all the strands, our
students performed worst in
Measurement.
3C. Compared to student
performance statewide in the
strand Patterns, Relations,
and Algebra, our students
performed the best on the
symbols standard.
3.2.1T: Practice Making Valid Inferences—Version 1.0 11
Scenario #4
SCENARIO
The review of the District Standards Summary Report helped the Team
determine specific areas where the students were weak on the 2007 test. The
Team delegated several members to review this report for the three prior years
to see if these strands were problems for the students on those tests.
The Data Team also wanted to learn more about the performance of subgroups
on specific test items. The Team posed the following clarifying question.
CLARIFYING QUESTION
How did the ELL students in our LEP program perform across all test items as
compared to all students in the district and in the state?

12
R-302 District Item Analysis Graph

13
District Item Analysis Graph (R-302)
Observation Not
Inference? True
4A. The performance pattern of all
students in our district follows
the state more closely than the
pattern for LEP students.
4B. Item 36 is the most difficult
item.
4C. Our LEP program is not
preparing our students
adequately.

14
Possible Answers
Instructions:
Reflecting on the statements that have been made using the data reports:
• Which are Observations and which are Inferences?
• Which are True, False, or Not Necessarily True (more data needed)?
• What clarifying questions would help make better inferences?
Scenario #1
1A. Inference (NNT) – The English (reading comprehension and writing) assessments are developed to
assess completely different knowledge and skills, so a mathematics score cannot be directly compared
to an English score.
1B. Inference – (F) The statement is false because our students performed BETTER than the state at each
performance level. It is an inference because poorly is a conclusion that is not factually precise.
1C. Observation – (T) 29% of our students performed below proficient.
Scenario #2
2A. Inference – (NNT) While the data do show increases in MCAS performance from 2006 to 2007, there
could be slight differences in the student cohort due to changes in population since 2005. Additionally,
the data do not reflect individual student growth over time, only MCAS scores for a class from one year
to another. You would need more data to be sure.
2B. Observation – (T) The percent of students below proficient decreased over time. Caveat–again, these
are different groups of students.
2C. Observation – (F) For the first two years, our students showed a small decrease in the percent of
students below proficient, while the state stayed at about the same level (percent at warning actually
increased slightly). In the most recent year, there was a decrease in percent below proficient at the
state level, but a much larger decrease among the tested students in District A.
15
Possible Answers (continued)
Instructions:
Reflecting on the statements that have been made using the data reports:
• Which are Observations and which are Inferences?
• Which are True, False, or Not Necessarily True (more data needed)?
• What clarifying questions would help make better inferences?
Scenario #3
3A. Observation – (T) A larger percentage of District A students was successful in each strand than
students statewide.
3B. Observation – (T) Relative to all other strands, our students did indeed score the poorest in
Measurement.
3C. Observation – (F) They performed best in Models relative to the state (10 percentage points
difference).
Scenario #4
4A. Observation – (T) LEP student pattern is up and down and district pattern and state pattern are very
similar overall. Stress that this is probably due to the relatively small size of the population. Smaller
populations show greater variation.
4B. Inference – (NNT) A factual interpretation (observation) is that the LEP group and the State scores
are lowest for Item 36, but not for the District. It is an inference that this is the most difficult item for
these groups, as there could be other reasons why so many students scored low on it.
4C. Inference – (NNT) You can’t infer this from the data. For example, the most difficult items for the
LEP group may be those that have the most language, such as story problems. The next step is
looking at the actual items and drawing conclusions about what might have made the items difficult
for the LEP subgroup. Also, the LEP students did better than the other two populations on several
items.
16
D A T A A N A L Y S I S P R O T O C O L 3 . 3 . 1 T
Purpose To collaboratively analyze data. Related Documents

3–Information Module
Description The protocol is a collaborative process for analyzing
data that can be applied to many types of data.
This is a collaborative protocol designed to be used by groups of 3–5 people. It is necessary to

identify the questions that are being addressed and prepare (or gather) the necessary data prior
to beginning the protocol (such as those generated by 2.1.1T: Question Formulation Protocol).
Be sure all are clear on the protocol before beginning.
1. Write the question(s) being analyzed at the top of a piece of chart paper. Check to make
sure each person understands the question. (1–5 minutes)
2. Distribute copies of the data in either graphical or numerical displays to each member
of the Team. Ask each person to silently observe the data by taking notes and jotting
observations. (5 minutes)
Note: By this point, the Team may have three levels of data: high-level data that spurred
the inquiry in the first place; data used in the data overview (2.3.1) to generate clarifying
questions; and even more specific data collected subsequently to address these
clarifying questions. In some cases, the first two data sets may be fairly similar.
Engaging with all data sets simultaneously can better poise the group to see patterns,
trends, and outliers that had not previously been evident.
3. Observe: (15 minutes) Ask Team members to take turns (round-robin fashion) and
report one of their observations. Observations should be facts or evidence that can be
readily seen in the data and stated without interpretation.
Instruct participants to use a sentence starter like one of the following to keep the
observations factual:
I see…

I observe…

I notice…

Remember. Only discuss the facts at this stage of the process!

If you catch yourself using any of the following, STOP!
However… Because… Therefore…
3.3.1T Data Analysis Protocol—Version 1.0 1/2

After participants have shared their initial observations, probe for deeper analysis by
asking a combination of the following questions:
How do data sets (or populations) compare to each other?
Such as comparing one grade to another, or school vs. district vs. state
What are the commonalities among a given data set (or population)?
Such as among students who are scoring below standard, or those who are
achieving?
What patterns or similarities are evident across different data sets?
Such as comparing local formative assessment data with state assessments like
MCAS, or comparing student achievement with teacher attendance.
What inconsistencies or discrepancies (if any) are evident?
What is not represented in the data?
What questions do the data raise?
Capture the observations in list form on the chart paper as quickly as possible and
without comment. Capture questions on a separate sheet. Continue until all Team
members have reported all of their observations. (Note: During this step, it is acceptable
for Team members to make observations based on those made by others in the group.
Allow the process to proceed as long as logical and factual observations can be made.)
Note: It is often helpful to make a very distinct transition from the observation stage to
the interpretation stage, clarifying when the group can begin to allow statements that
may not be factually based.
4. Interpret: (20 minutes) Ask each Team member to review the entire list of
observations. Working together, code (or group) the observations into categories of
findings. To facilitate this process ask questions such as:
What assumptions might be underneath what we are noticing in the data?
What clues help explain why a certain population is meeting or missing targets?
What areas in the data stand out as needing further explanation?
What patterns or themes do we see in our observations?
Which of these observations are most relevant and important to our inquiry?
And finally:
Based on our observations, what do we know now?
5. Extend: (10 minutes) On a new piece of chart paper, write ― New Questions and
Conclusions.‖ Work as a group to identify new questions that this analysis has raised
and any possible conclusions that have been identified. The questions may serve as the
basis for another round of analysis, so it may be helpful to conclude by prioritizing them.
Any conclusions will become the basis for subsequent action.
This protocol is based on work presented by Nancy Love, author of ―

Using Data/Getting Results (2002),‖ who, in turn adapted it from
Bruce Wellman’s and Laura Lipton’s ― Data-Driven Dialogue (MiraVia LLC, 2004).‖ Additional questions adapted from Guide for
Standard Bearer Schools: Focusing on Causes to Improve Student Achievement (2007). Community Training and Assistance Center
(CTAC). Boston, MA.
3.3.1T Data Analysis Protocol—Version 1.0 2/2

MODULE 4: Knowledge
Table of Contents
Introduction—1
Clarifying the Problem—2
Writing a Problem Statement—2
Root Cause Analysis—4
Understanding Root Causes—4
Overview of Root Cause Protocols—6
Facilitating the Process—8
Connecting to Research and Local Knowledge—10
Connecting to Research and Local Expertise—10
Cataloguing Problems Under Investigation—12
Module Summary—13
Knowledge
4.1.1T: Writing a Problem Statement 4.2.5T: Dimensions Bulls-Eye

4.2.1T: Why, Why, Why? 4.3.1T: Problem Investigation Plan
4.2.2T: 20 Reasons 4.3.2R: Educational Research Websites
4.2.3T: Fishbone Analysis 4.4.1T: Problem Catalogue Template
4.2.4T: Graphic Representation
District Data Team Toolkit—Version 1.0 Module 4: Knowledge
INTR ODUCTI ON
WHERE ARE WE NOW?

progression through the model—you are now in Module 4: Knowledge.
Getting
Ready
In the Knowledge step of the Data-Driven Inquiry and Action Cycle, a

group engages in deeper analysis of the data and collaborates to begin
using this new perspective to inform strategic action. The tools in this
module can help a Team collaboratively refine questions that emerge
from data analysis, articulate a problem statement, connect possible
solutions with research, and lay groundwork for results-oriented action.
The most important parts about this transition from analysis to action is
taking time to make sure all members of a group are clearly in agreement
on the problem being addressed, and that an effort is made to connect
the problem to research and to other district efforts to solve the same
problem. Being purposeful during this step helps a Team avoid repeating
past mistakes and strengthens its ability to take effective action.
MODULE OBJECTIVES
The Knowledge module will help a District Data Team:
Clearly articulate a problem statement
Identify and explore root causes of the problem
Cross-reference solutions with research and local knowledge
Begin to capture information on the district’s improvement efforts

CLARI FYI NG THE PR OBLEM
In the Inquiry module, the District Data Team formulates a number of

questions for study and refines them through discussion and data
analysis. During the Information module, the Team delves more deeply
into the data and triangulates multiple data sources to get the best
understanding possible of the issues at the heart of the focusing and
clarifying questions. The Team may emerge from this analysis with a
clear set of conclusions that address the initial focusing and clarifying
questions. It may also generate a number of new questions raised by
their investigation of the data.
A problem Teams that want to explore the questions that emerge from the data
statement can help analysis may want to engage with the root cause activities outlined later
the Team focus its in this module to gain new perspectives on the factors that may explain
the patterns, trends, or aberrations evident in the data. If this process
work prior to
does not help the Team gain agreement on the problem to be addressed,
validating potential then it will likely reveal a need for more data or different questions, which
solutions with would cycle the Team back to the Inquiry stage of the process.
research and then
Teams that emerge from the data analysis in the Inquiry module with
moving on to action.
strong conclusions may be ready to move toward planning action by first
crafting a problem statement.
WRITING A PROBLEM STATEMENT

A problem statement can help the Team focus its work prior to validating
potential solutions with research and then moving on to action. A problem
statement can also help the Team communicate with stakeholders that
will need to be engaged in this work, including students, families,
teachers, and other school and district personnel.
A problem statement can help the Team articulate:
The original undesirable situation (the problem)
The desired outcome (the goal)
The underlying problem causing the original problem
The solution to that underlying cause

Writing a problem statement can serve as a self-check to see if the Team

is ready to move on. If the Team is all in agreement about the nature of
the problem being addressed and how to best approach resolving it,
writing a problem statement may be a fairly straightforward activity. Resolving problems
However, the Team may find that while working through the process of with teaching,
writing a problem statement, Team members are not in alignment or
curriculum
agreement. In fact, frequently a number of ideas about how to address
those issues are raised during the discussions. If that is the case, the implementation, or
Team will want to use one of the Root Cause Analysis activities in the resource allocation
next section to calibrate the Team and then return to 4.1.1T: Writing a all involve shifting
Problem Statement.
how adults
One element of a problem statement involves describing the nature of the throughout the
problem identified in the data analysis. Articulating the type of problem district approach and
can help the Team think wisely about the best strategies to put in place to
conduct their work.
resolve it. Problems can be broadly categorized as relating to skills,
attitudes, knowledge, and/or resources.
However, ultimately, regardless of the specific nature of the problem, it is

the adults in a district who create and maintain the learning opportunities
for the students they serve. This includes not only teachers and
principals, but all district employees, such as secretaries, facilities staff,
central office staff, district leadership, and school committee members.
Resolving problems with teaching, curriculum implementation, or
resource allocation all involve shifting how adults throughout the district
approach and conduct their work. Therefore, in articulating a problem
statement, it is important that the proposal for addressing the problem
paint a clear picture of how this will change–what the work needs to look
like and how the district will help personnel make the needed changes.
From this angle, when identifying the solution to address the underlying
problem, a district would be wise to first consider how it can reallocate
existing resources and improve existing initiatives. The inclination is often
to identify new strategies or initiatives, but the Team should first evaluate
the efficacy and impact of current initiatives before adding new ones. A
new initiative should be added only if it is unarguable that the need exists.
Activity 4.1 Writing a Problem Statement
This is a collaborative process meant to help the Team discuss a problem

in depth and write an agreed upon statement about how it will be
addressed.
(4.1.1T: Writing a Problem Statement)

ROOT CA USE ANA LYSIS
UNDERSTANDING ROOT CAUSES

Engaging in a root cause analysis can help the Team gain agreement on
the exact nature of a problem it has identified, and/or on the best steps to
take in addressing it. Before taking action, the Team needs to agree on
the problem that needs to be fixed. This is rarely an easy task.
In simplest terms, a root cause is an underlying factor or condition that

creates a problem and that, if addressed, would eliminate or dramatically
alleviate the problem. A root cause analysis can help a group with widely
varying opinions narrow the field of contributing factors until it agrees on
what one(s) will yield the biggest bang for the buck if it acts on it.
In mechanical systems, diagnosing a root cause is an essential part of the

troubleshooting process before beginning work. For example, if a
person’s car won’t start in the morning, s/he having a problem. There are
a number of potential root causes of that problem until some further
investigation is done. Among many other things, it could be possible that:
The battery cable isn’t connected

The battery is dead
The starter motor isn’t working
The car is out of gas
Investigating each of these potential root causes helps to determine what

is or is not the root of your trouble. Going through this effort is important,
because replacing a starter motor after getting a tow to a repair shop
would be an unnecessary expense if the real problem was simply that the
individual spouse or teenager brought the car home with an empty tank of
gas.
Naturally, discerning root causes for problems in education is not nearly

as straightforward as this example suggests, and root causes are not
always known. For example, a student’s low academic achievement could
be the result of:
Something the student is or is not doing

Something the teacher is or is not doing

Something the teacher’s support network, e.g., principal, coach,

district central office, district processes, is or is not doing
Something the student’s support network, e.g., family, friends,
community, is or is not doing
Something the physical learning environment is or is not providing
A combination of the above
Researchers engage in rigorous discourse to discern the most significant

factors behind student achievement and learning. Practitioners must do
their best to tap the best available knowledge when trying to determine
In order to change
the root causes of problems in their own district. student outcomes, a
district needs to
Discerning the root cause of educational problems is a difficult task for change the actions
another reason. People tend to have strong beliefs about problems in
of its personnel in
schools and how they should be solved. These beliefs are influenced by
personal values, political issues, opinions about strategies tried in the concrete ways that
past, and many other factors. Therefore, it is important for any group lead to the desired
attempting to solve a problem to take the time to collaboratively get all outcome.
issues surrounding it out on the table for discussion.
Something to keep in mind is that in education, a problem with student

achievement likely has a cause that lies in someone’s practice. A district
cannot change the outcome (such as low test scores) without changing
someone’s behavior. Like in the example about the car, the Team will be
following a chain of cause/effect relationships to find where the work
needs to be done. But unlike replacing a part in a car, computer, or other
inanimate machine, the Team will likely end up determining that the root
cause lies in processes or techniques that people in the system are using,
but that, despite everyone’s best efforts, just aren’t getting the job done
the way it needs to be done. While in some cases it may be that a
process isn’t being implemented correctly, in other cases it may be that
the right strategy hasn’t been applied in the first place. In order to change
student outcomes, a district needs to change the actions of its personnel
in concrete ways that lead to the desired outcome.
The collaborative tools shared here are meant to help the Data Team
understand and agree on the issues that are most responsible for the
problems it has have identified, in order to begin planning well-considered
and researched strategies and engage people in the process of changing
practice. These activities are not intended to be used to place blame on
anyone in the system, but rather to understand where the most energy
and attention should be placed in order to get different results.

It should be noted that root cause activities are useful for analyzing the
factors that contribute to success, as well as those that contribute to a
problem. For example, if an initiative produced very strong results,
engaging in these activities could help the Team capture lessons to scale
up in other areas of the district’s work.
OVERVIEW OF ROOT CAUSE PROTOCOLS

Below are five possible approaches for engaging in a root cause
discussion, listed in order from simpler to more complex activities. Each
has its own strengths and challenges when working with different groups.
Read through them all and consider trying each of them out within the
District Data Team to get an idea of how they work and how well they
would engage other audiences. Any of them can be used with any sized
group, but some general suggestions for use are below.
In choosing the approach that is best for the situation, the Team will want
to consider the complexity of the problem and the depth of additional
analysis needed in order to gain agreement on the root cause. It will also
want to ensure it has the time and facilitation skills required to conduct the
activity successfully.
Why, Why, Why?—This protocol is a relatively quick and informal

technique for identifying root causes of problems. It can be used
by individuals or groups of 3–8 people, and requires little
facilitation. This technique is especially useful to start a discussion
and determine if there is disagreement among the participants. A
more formal process should be used for in-depth discussion.
20 Reasons—This protocol works well with a large or small
group. It enables the group to brainstorm many issues related to a
particular problem and then carefully consider whether they are
actual causes or simply excuses.
Fishbone Analysis—This very formal protocol works well with
groups of about three to five participants. The protocol provides
the chance for the group to consider and discuss many possible
explanations for a problem and enables participants to categorize
causes.
Graphic Representation—This protocol enables a group to
discuss, analyze, and display relationships among contributing
factors. In essence, the group creates a visual representation of
the problem or situation, including all the factors that influence it
and possible solutions that have come to light during the analysis.
This highly collaborative technique works best with groups of up to
five people and requires a somewhat skilled facilitator.

Dimensions Bulls-Eye—This protocol, derived from the

Department’s Performance Improvement Mapping (PIM) process,
is a lengthy but effective way to brainstorm possible root causes,
sort them into one of three dimensions of district improvement,
and prioritize key root causes for action. This process works best
with a group 7–10 people and requires strong facilitation.
Each of these activities should steer participants toward evaluating the

extent to which existing systems and structures are functioning as
intended and genuinely impacting teaching and learning. The end result
of any of these activities should be that the group has a clear and uniform
idea of the problem, its potential root causes, and how the Team will
proceed to take action.
Regardless of the method used to surface root causes, it is important for

the Team to identify evidence that verifies its ideas. If the Team is not
careful, it can unwittingly reinforce false perceptions and negative
stereotypes. Thus, Team members should constantly ask each other
―How do you know?” When available information and data have been
consulted, the Team should discuss whether significant evidence exists to
confirm the Team’s hypothesis about the causes. Without this self-check
against valid evidence, the root causes that the Team identifies to target
for action may not be deemed credible by stakeholders.
After completing any of the Root Cause Analysis Protocols, the Team
should return to the Writing a Problem Statement worksheet and prepare
a newly aligned view of the problem and potential solution.
Activity 4.2 Root Cause Analysis Protocols
These activities can facilitate the Team’s discussion of root causes. The
protocols can be used in many different situations to explore problems in
a collaborative way. The Team should select the approach that seems
best for its particular group or situation, or create its own using these as
templates.
(4.2.1T: Why, Why, Why?)
(4.2.2T: 20 Reasons)
(4.2.3T: Fishbone Analysis)
(4.2.4T: Graphic Representation)
(4.2.5T: Dimensions Bulls-Eye)

FACILITATING THE PROCESS
The Team should think strategically about which groups to involve in the
process of root cause analysis. While the District Data Team on its own
could likely generate valuable insight on a problem, it is often best to
engage those closest to the problem in the identification of the root
causes that, if addressed, would improve the situation. As well-
intentioned as the Team may be, it may miss valuable information by not
going closer to the source.
For example, a district might consider having teams from a number of

different schools engage in the protocols, then notice the patterns that
emerge, and use that information to decide how to best leverage district
resources. Whether school or district personnel are conducting the
activity, the focus should always remain on the teaching and learning and
those factors that most directly impact it.
Each of these root cause protocols is based on the premise that adult
behavior and district processes impact student learning outcomes. If we
If we believe that believe that all students can learn, and they aren’t, then we need to look
all students can at what we can do differently. While some root causes may indeed be out
of the district’s hands, such as student mobility or the effects of poverty,
learn, and they
the District Data Team needs to look very closely at how the district
aren’t, then we need conducts the business of educating students and what aspects of this
to look at what we work may or may not be contributing to the problem at hand. When done
can do differently. well, engaging in a root cause activity can promote honest and
sometimes difficult conversations about how personnel in all corners of
the district conduct their work, including the members of the Team itself.
However, participants frequently disagree about root cause explanations
for the original problem, the sequence of causes and effects, or the
relative importance of various possible causes during the brainstorming
phase of this activity. The group may even come up with explanations
that are directly contradictory to one another. This has some important
implications for facilitating the process of discussing root causes.
Any time a protocol requires brainstorming, keep the group

focused on first listing as many ideas as possible. Do not allow
debate, discussion or even comments like great idea at that time.
Once all the ideas are generated, the group can then ask for
clarifications, probe for greater understanding, and move past less
important disagreements to focus on more significant points
The Team should encourage and record dissenting views rather
than immediately dismiss any that might arise. The subsequent
discussion and reflection will sometimes reveal which perspective

is more likely to be true. If this does not happen, the Team can
consult research and local knowledge for more insight
Many of the ideas generated in the activities should be regarded
as biases, opinion, or conjecture until proven otherwise with data
or research. The Team must objectively look at the assumptions it
holds and check them against research, data, and expert opinion.
Just because everyone in the group happens to agree does not
mean any given potential root cause is right
In fact, unlike cars that won’t start, we often don’t really know the
right answer. The Team has to pick one potential root cause that,
based on data analysis, research, and local knowledge, seems
like it may make the most impact, and then try to resolve it. By
monitoring progress, evaluating results, and continuing the inquiry
process, the Team can model the truly adaptive nature of
education where educators learn the work by doing it, and develop
the answers together along the way
When designing the format for a discussion of root causes, a district may
want to assign a facilitator who can help the group with these key points,
as well help the Team:
Ensure all Team members have an equal voice in sharing

observations of the evidence that has been gathered
Put as much data on the table as possible, from high-level to fine-
grained observations
Challenge each other’s assumptions and generalizations by
asking why and what’s the evidence?
Be prepared to be surprised
The Team might also consider engaging stakeholders directly in these

discussions, as a way to gain new perspective on the topic as well as
gain buy-in on the strategies that emerge from this work. This could be
especially important if the Team is small and/or is not very representative
of the stakeholders impacted by the problem, such as if it only consists of
district-level personnel.

CONNE CTIN G TO RESEAR CH

AND LOCA L KNOW LEDGE
Once the Team has clearly defined the problem and everyone has agreed
to a general strategy to alleviate it, the Team might feel ready to move
straight to building an action plan with specific goals, timelines, and data
collection points. But, before moving on, it is important to begin making
connections to research and local knowledge, looking outward for
information that might be helpful in shaping the Team’s work.
CONNECTING TO RESEARCH AND LOCAL

EXPERTISE
By this point, the Team has clarified the problem and has articulated:
The original undesirable situation (the problem)

The desired outcome (the goal)
The underlying problem causing the original problem
The solution to that underlying cause
Up to this point, the Team has worked on its own—or perhaps with some
input from stakeholders—to identify the underlying problem and a
proposed solution. Taking time to consult local experts, research
literature, and others outside the District Data Team who have gone down
the same path, can increase the effectiveness of the plan for action, as
well as increase its credibility and validity.
The attached Problem Investigation Plan guides the Team to articulate

the information it wants to gather prior to going out and getting it. Taking
this step can help the Data Team maintain focus prior to diving in to what
could be an overwhelming amount of research and information.
When consulting research, the District Data Team should be mindful that
the Internet makes it much easier to connect to a wide range of scholarly
research—however, not all research is good research. The District Data
Team has a responsibility to ensure that the research it uses is credible,
and as such should look for research from credible independent sources.
4.3.2R: Educational Research Websites includes a list of useful websites

that can connect districts to other websites with searchable educational
research reports. This list, while by no means meant to be

comprehensive, can serve as a starting place for gathering credible

researched reports related to the problem the Data Team is addressing
and interventions the Team is considering. In addition, Team members
may want to consider tapping academic and/or research institutions
directly, whether by linking to their own undergraduate and/or graduate
institutions, or by connecting to those located in or near their district.
When identifying local knowledge and expertise that can further clarify the
problem and aid the development of an effective action plan, the Team
may want to consider:
Who has credible perspectives and expertise related to this

problem?
Who has knowledge of relevant content, systems, and history?
Who has a solid understanding of the experiences of the
stakeholders impacted by this problem? Who among the
stakeholder group(s) itself may have valuable insight?
Who does the Team not usually hear from, and whose expertise
could be valuable at this point?
Who can the Team trust to share knowledge and lessons learned
that may deviate from what is commonly held as truth?
Activity 4.3 Problem Investigation
This activity helps the Team identify assumptions that need to be checked
and questions that need to be answered about a problem or potential
intervention. Several websites are provided to help connect the Team to
related research.
(4.3.1T: Problem Investigation Plan)

(4.3.2R: Educational Research Websites)

CATALOGUING PROBLEMS UNDER

INVESTIGATION
The District Data Team can facilitate the endurance of a data-driven
decision making initiative by beginning to capture the work and evidence
from improvement efforts. This will allow the district to begin creating a
database (either electronic or in file cabinets) of questions and problems
people have addressed, strategies they have used, and the results of
those efforts. In addition, having a systematic way of organizing the
results of various inquiry processes will make it much easier to make
connections to research in professional journals, books, and web
resources. Using categories in a database will allow the Team to create a
searchable catalogue of improvement efforts over time, removing the
need to rely only on the institutional knowledge of district personnel.
The next activity is designed to give the District Data Team a systematic
way to capture problems. The Team should begin by documenting
information relevant to the inquiry process in which it is engaged. Over
time the Team can collect information from other teams, as well as from
its own subsequent inquiry processes. The way the Team elects to
capture and share this evidence and knowledge is highly unique to its
local systems, personnel, time, and resources. As the Team engages with
the template, it will likely want to refine the categories and format to suit
local needs and initiatives. However, beyond determining the exact
headers on a template, the Team also needs to make a long-term plan for
collecting, storing, and using this information.
Activity 4.4 Problem Catalog Template
This template contains a model for a catalogue of problems being

addressed by groups throughout a district.
(4.4.1T: Problem Catalog Template)

MODULE SUMMARY
The Knowledge module is intended to strengthen the transition that a

Data Team makes between initial data analysis and subsequent action. It
is meant to help a group take a broad look at what it now knows, after
having engaged in data analysis, and compare it to what else is known,
prior to thinking about what do to next. Spending time and purposeful
effort in the Knowledge step of the inquiry process can strengthen the
Team’s understanding of the problem before it enters the Action step.
Essential steps in the process of turning information to knowledge are

formally stating the problem being worked on and beginning to investigate
strategies for improvement by consulting research and local knowledge
and expertise. In order to accomplish these two items, it is frequently
necessary to spend some time calibrating the entire Team around what
the real problem is (or might be) by exploring perceived underlying root
causes. These can be challenging conversations at times, so it is best to
conduct them using a protocol designed to facilitate safe discussions
about root causes. The Team should consider the value of engaging
stakeholders in the knowledge-building phase, for example by including
stakeholders in discussions of root causes.
The District Data Team should emerge from this stage in the process with
a clearly articulated problem statement that outlines the original problem,
the suspected cause, the goal for improvement, and a proposal for
moving forward.
The next module, Action, guides the Team in articulating a logic model
and crafting or revising a plan to take action on the identified problem.

WRITI NG A PR OB LEM STATEMENT 4 . 1 . 1 T
Purpose To formalize a problem in order to focus action. Related Documents

1 4–Knowledge Module
Description This framework helps a Team to discuss a problem in
depth and write an agreed-upon statement about how
it will be addressed. The process forces the Team to
think purposefully about who is affected by the
problem, possible causes, and potential solutions.
Time About 1 hour.
`
Directions: As a Team, work through the boxes from top to bottom to craft a problem
statement. The next page contains a completed sample of the Writing a Problem Statement
worksheet. A blank copy of the worksheet appears on the last page.
Restate the initial problem that launched this inquiry process, or

Original problem or focusing
rewrite the focusing question or one of the clarifying question as a
question
statement.
Stakeholders who are most Who is most directly impacted by this problem? Alternately, who
affected by the problem would benefit the most if this problem were resolved?
For example, skills, attitudes, knowledge, resources, or something

Type of problem
else.
Based on the data analysis and/or the root cause analysis, what
Suspected cause of the does the Team think is the most significant cause(s) contributing to
problem this problem? What, if addressed, would make the greatest impact
on resolving the problem? (Include specific evidence).
Goal for improvement and The wishes, dreams, and general vision describing the target. The
long-term impact Team will write a clearer, measureable goal statement in Module 5.
High-level strategy that represents promising practices drawn from

Proposal for addressing the research, local knowledge, and local expertise. (Note sources if
problem possible). This will become the basis for subsequent action
planning.
Tie the above statements into 3–5 coherent sentences that could be
Final problem statement
easily understood by a wide range of stakeholders.
1
Adapted from Sagor, R. (2000). Guiding School Improvement Through Action Research. Association for
Supervision and Curriculum Development, Alexandria, VA.
4.1.1T: Writing a Problem Statement—Version 1.0 1/3

Sample of Completed Writing a Problem Statement Worksheet

Students are not reading at grade level by grade 3.
question
Stakeholders who are most Third grade students at our school.

affected by the problem
Resources: Without good information about where our kids are

Type of problem starting, we have no way of knowing if our goals may be
unattainable for some of them.
Teachers don’t get sufficient training and support in our reading

Suspected cause of the program.
problem Students’ reading levels are not measured accurately in grades
K–3.
Goal for improvement and We want all our third graders to read at grade level or above.
long-term impact
Proposal for addressing the Start a teacher mentoring program in reading.

problem Implement more rigorous reading assessments in grades K–3.
Many third grade students at our school

do not read at grade level.
We believe that this is a result of teachers not having sufficient
training in our reading program and not accurately measuring
students’ reading levels in grades K–3.
We want all third graders at our school to read at grade level or
above.
We will start a teacher mentoring program focused on reading
and implement more rigorous reading assessments in the
primary grades.

Writing a Problem Statement Worksheet

question
Stakeholders who are most

affected by the problem
Type of problem
Suspected cause of the

problem
Goal for improvement and

long-term impact
Proposal for addressing the

problem

WHY, WHY, WHY? 4 . 2 . 1 T
Purpose Determine a root cause for a problem. Related Documents

4–Knowledge Module
Description A Team brainstorms answers to ― Why?‖ a problem 4.2.2T: 20 Reasons
might be happening in order to arrive at an agreed 4.2.3T: Fishbone Analysis
upon root cause. 4.2.4T: Graphic
Time < 30 minutes. Representation
4.2.5T: Dimensions Bulls-Eye
Directions
Why, Why, Why? is a relatively quick, informal way to identify root causes of problems. Start by
writing the problem being addressed and then ask the group to give a reason for ― Why this
might be happening?‖ Record the answer after the first ― Because‖ and then ask the question
again in reference to the first ―Because.‖ Repeat the process three to five times, asking ―Why?‖
for the previous ― Because‖ until the group feels that it has arrived at the root cause of the
problem. If after three to five questions and answers, the group does not agree that it has found
a root cause, consider using another root cause protocol in the Toolkit.
4.2.1T: Why, Why, Why?—Version 1.0 1/2

Problem/Barrier/Issue
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
______________________________________________________________________
Why?
Because: ______________________________________________________________
______________________________________________________________________
______________________________________________________________________
Why?
Because: ______________________________________________________________
______________________________________________________________________
______________________________________________________________________
Why?
Because: ______________________________________________________________
______________________________________________________________________
______________________________________________________________________
4.2.1T: Why, Why, Why?—Version 1.0 2/2

2 0 R E A S O N S 4 . 2 . 2 T

Description A Team or large group brainstorms 20 reasons 4.2.1T: Why, Why, Why?
why a problem might be occurring in order to come 4.2.3T: Fishbone Analysis
to agreement about what the real cause of the 4.2.4T: Graphic
problem might be. Representation
Time 45 minutes to an hour. 4.2.5T: Dimensions Bulls-Eye
Directions: Use a computer and projector to display the 20 Reasons worksheet on the last
page, or use chart paper to recreate the simple list.
1. Begin by writing the problem in the box at the top of the page.
2. Ask the group to give possible reasons for why the problem may be occurring. It may be
helpful to use a round-robin response order to get people started, but try to allow the
team to call out reasons as they come to mind. Record them all until you have reached a
full list of 20 reasons.
3. Allow the group to review the list silently for a few moments.
4. Ask each member to identify what s/he thinks might be the root cause of the problem.
Place a checkmark next to the statement as s/he speaks and encourage him/her to
explain his/her reasoning before moving on to the next person.
5. Continue to facilitate the discussion until the group feels that it has identified a potential
root cause.
Key Points
 It should be emphasized that this is a brainstorming activity and all responses are
welcome and valid.
 You may find the last several reasons are more difficult to come up with, but frequently
the effort is worth it, as the root cause will likely appear near the bottom of the list.
 Many problems do in fact have more than one root cause. It is fine to identify more than
one root cause, but do push the group, through reflection and discussion, to narrow the
list to no more than three root causes.
Additional Information
Participants frequently disagree about explanations for the original problem, the sequence of
causes and effects, or the relative importance of various possible causes during the
brainstorming phase of this activity. Rather than allowing debate during the brainstorming of the
list, keep the group focused on listing possible reasons first. When the group reflects individually
to identify possible root causes, it can move past less important disagreements to focus on the
root of the problem.
The group may come up with explanations that are directly contradictory to one another. If this
occurs, record them all rather than immediately dismissing any. The reflection and subsequent
discussion will sometimes reveal which are more likely to be true.
4.2.2T: 20 Reasons—Version 1.0 1/3

Example of Partially Completed 20 Reasons Worksheet
Problem: Our ELL population struggles to meet proficiency on the ELA section of MCAS.
Root
# Possible Explanation
Cause?
1 ELL students have a wide variety of needs and abilities that are difficult to meet.
2 Programs we have for ELLs are not being implemented effectively in every school.
Many teachers have not received enough PD and support to help them work with
3
ELLs effectively.
4 Many ELL students also have learning disabilities.
5 The number of ELL students in our schools is increasing faster than we thought.
The MCAS requires high levels of comprehension of vocabulary that may be

6
unfamiliar.
8 .
10
11
12
13
14
15
16
17
18
19
20

20 Reasons Worksheet
Problem:
Root
# Possible Explanation
Cause?
10
11
12
13
14
15
16
17
18
19
20

F I S H B O N E A N A L Y S I S 4 . 2 . 3 T

Description A Team works through a formal fishbone diagram to 4.2.1T: Why, Why, Why?
brainstorm and individually rule out possible causes in 4.2.2T: 20 Reasons
order to arrive at an agreed upon potential root cause. 4.2.4T: Graphic
Time 1 hour. Representation
4.2.5T: Dimensions Bulls-
Eye
Directions: Follow these steps to complete the fishbone diagram.

1. Write your problem in the box at the ―
head‖ of the fish.
2. Identify major categories and write them in the boxes. (The diagram has four ― ribs‖ and
boxes, but you may have fewer or more than that. The first time you use this tool, try to
use four.)
3. For each major category, brainstorm possible causes. Write them next to the appropriate
―rib‖ of the fish.
4. Analyze each possible cause identified to determine whether it is a root cause by asking:
Would the problem have occurred if the cause had not been present?
Would the problem reoccur if the cause was corrected?
If the answer to both of these questions is no, you have found a likely root cause.
5. Place checkmarks next to ideas that are not root causes.
6. Circle root causes.
Key Point
 The categories used most often when addressing problems in student achievement are
Students, Families, Processes, Curriculum, and Teachers, but categories may vary
depending on the problem.
Additional Information
During the brainstorming, participants may come up with possible causes that do not fit easily
into one of the previously identified categories. This can indicate a need to identify a new
category or broaden an existing category. Do not discard an idea solely because it does not fit
into a previously identified category. The purpose of the major categories is to provide a
structure to guide the brainstorming. These categories should be used to inspire, rather than
restrict, participants’ thinking.
In the early stages of the process, participants often use the activity as an opportunity to vent
frustrations and criticisms. This can be acceptable in the beginning, but be sure to steer them in
a more constructive direction as the activity progresses.
4.2.3T: Fishbone Analysis—Version 1.0 1/3

Example of Completed Fishbone Diagram
Families Students
Do not speak Parents do not They don’t try Students don’t

English at home help students at hard enough like school
home
Do not behave Students miss

Poverty Parents do well enough in school too much
not accept class
help
Parents do not come Students don’t

to conferences or participate enough Our ELL population
ask for help
struggles to pass
the ELA section of
Kids are allowed
Inconsistent
Teachers feel MCAS.
to fall through overwhelmed
the cracks implementation of
programs
Class sizes are
Not enough ELL
too large
We have not changed specialists
Not enough materials
with the change in
We treat many population Some teachers
students the same do not have Not enough
even though their Every school seems experience with professional
needs are different to be doing ELL students development
something different
Processes Teachers

Fishbone Diagram Worksheet
Problem:

G R A P H I C R E P R E S E N T A T I O N 4 . 2 . 4 T

Description A group works together to design a graphic 4.2.1T: Why, Why, Why?
representation of a problem and the processes that 4.2.2T: 20 Reasons
surround it to identify areas of strength and weakness 4.2.3T: Fishbone Analysis
to address, including potential root causes. This 4.2.5T: Dimensions Bulls-
technique enables a group to discuss, analyze, and Eye
display relationships among factors, so members can
simultaneously see the big picture and the fine details
it’s comprised of. This highly collaborative technique
works best with groups of up to five people.
Time 90 minutes to 2 hours.
Directions: Use the following steps as guidelines in developing and reflecting on a

graphic representation about a problem or situation. Bear in mind that the steps are
intended as a reminder to help you through the process. In practice, the process is less
linear than this sequence of steps implies. To complete this activity, you will need chart
paper, markers, and sticky notes.
1. Identify a problem and write it on chart paper.
2. Working independently, brainstorm factors that influence the situation and write them on
sticky notes. (Allow approximately five minutes).
3. Compare and discuss the factors suggested by different members of the team, adding,
discarding, and revising factors as needed.
Look for duplicates. A factor that was suggested by multiple members is likely to be
relevant and important. Select one and discard the others.
Look for similarities and consider combining ideas that are similar, but not exactly the
same.
Refine ideas that are imprecise.
Consider whether each factor is too specific or too general.
Debate and decide whether each factor is relevant.
4. Arrange the sticky notes on the chart paper in a pattern that indicates how the factors
are related.
Cluster related ideas.
Discuss their relationships.
Does one have a strong impact on another?
Which is the cause and which is the effect?
Do two or more factors combine to affect another factor or group of factors?
Are two or more factors influenced by another factor or group of factors?
Arrange the sticky notes to show the relationships.
4.2.4T: Graphic Representation—Version 1.0 1/3

5. Experiment with different arrangements for the sticky notes. Once the team has reached
consensus on an arrangement for all the factors, draw circles and arrows on the chart
paper to clarify the relationships.
6. Study the entire graphic representation with a critical eye, asking questions such as:
Are the relationships between variables shown correctly?
Are there other variables or issues that should be added?
Revise and refine the overall arrangement based on the answers.
7. Focus on each relationship individually, asking such questions as:

Do we know for certain that these factors are related in this particular way?
What evidence do we have that enables us to make this conclusion?
8. If there are relationships that you believe are valid and important, but you cannot be
certain that they are, list them on another piece of chart paper and note how you could
investigate them further.
9. Study the graphic representation and identify factors on which to focus solutions by
asking such questions as:
Which factors seem to be at the root of the problem?
Would a significant change in one or two factors solve the problem?
Adapted from How To Conduct Collaborative Action Research, by Richard Sagor, 1992, ASCD,
Alexandria, VA.

Graphic Representation Example
Many of our students score poorly on constructed

response (CR) questions in mathematics.
Mathematics CR questions
“Problem of are used on
the Day” used tests and

homework
in every
classroom assignments
Students Students
solve the solve the
problems problems
Teachers do Most students Some students

1 or 2 Students
think aloud to complete the don’t complete
are asked to
the written
explain demonstrate CR with
the solution written explanations
thinking Students
orally demonstrate explanations
the solution
on blackboard Teachers
score
Our Rubric
for Students
Students
Constructed solve the
solve the
Responses problem
problem
correctly incorrectly
Students Poor or no Students Poor or no

explain their written explain their written
thinking in response thinking in response
a paragraph a paragraph
Full credit Partial No credit

credit

D I M E N S I O N S B U L L S - E Y E 4 . 2 . 5 T
Purpose Determine possible root causes for a problem. Related Documents

Description This protocol will help a Team brainstorm possible 4.2.1T: Why, Why, Why?
root causes, sort them into one of three dimensions, 4.2.2T: 20 Reasons
and prioritize key root causes for action. 4.2.3T: Fishbone Analysis
Time 1–2 hours. 4.2.4T: Graphic
Representation
Note: This activity is an adaptation Step Four of the Performance Improvement Mapping (PIM)
process—Identify the most significant causes of the weaknesses in students' knowledge and
skills—available at http://www.doe.mass.edu/sda/regional/pim/.
Directions:
1. Write the inference or conclusion from your data analysis (3.3.1T) where all can see,
e.g., flip chart or projected by an LCD.
2. Brainstorm all the possible underlying causes that might have contributed to this
outcome. For each potential root cause, write a short summary on a piece of paper and
tape it on the wall where everyone can see and read it.
Note:
The group may want to give individuals silent think/work time before
brainstorming as a group.
Make sure that the written causes are specific enough to be interpreted after the
discussion is over. For example, a cause written as curriculum does not describe
what really is lacking.
3. Once the brainstorm is complete, consolidate any duplicate or very similar ideas.
However, avoid consolidating causes in ways that make them too broad and vague.
4. Review all the causes and note any that are outside of the direct control of the district,
e.g., those dealing with student behavior, families, or the community. For each of these
causes, discuss the following:
Is this potential root cause important enough for the district to focus time and
energy on as part of an action planning process?
If so, can this cause be stated in terms of something over which the district has
control?
After discussing each of these causes, the Team has two options:
Rewrite the cause in terms of actions the district could take, such as securing
resources, modifying processes, and/or shifting actions of district personnel. (See
examples below).
4.2.5T: Dimensions Bulls-Eye—Version 1.0 1/6

Put the cause aside to be dealt with at another time (Remove it from the wall and
record it on a ―waiting room‖ list. The Team may want to discuss this list at a later
time with district leadership, the School Committee, Union leadership, and/or
community partners).
Examples of rewritten causes:
Problem Brainstormed Root Cause Rewritten and Refined Cause
Students do not know how to Students’ families don’t The district lacks supplementary
read grade level text fluently and read to them at home. reading interventions for students
with comprehension (7.10) who do not make adequate progress
through regular classroom instruction.
Students do not know how to Special education students Special education teachers lack a
estimate and compute with refuse to memorize the variety of strategies for teaching
fractions, including simplification rules for computations computational skills involving
of fractions (8.N.10) involving fractions. fractions.
5. Once all causes have been written in terms that represent things over which the district
has control, sort them into one of three dimensions by moving the papers on the wall. It
may help to have a separate flip chart or wall space designated for each realm.
Core realm: Contains factors that most directly affect student outcomes. These
tend to be classroom-level factors.
Enabling realm: Contains conditions that must be in place in order to make the
core elements successful in affecting student outcomes. These tend to be a mix
of school- and district-level factors.
Supporting realm: Contains conditions that are helpful toward making the core
elements successful in affecting student outcomes. These tend to be a mix of
district- and community-level factors.
Note that the amount of control that teachers and the school have is greatest at the center.
Conversely, district control is greatest in the enabling and supporting realms. The district has
the unique perspective, responsibility, and authority to act at the enabling and supporting levels
in order to make systemic improvements that affect student learning and achievement.
Dimensions of district improvement:
Realm Definition Sphere of Amount of School Amount of
Implementation and Teacher Control District Control
Core Factors that most directly Classroom School and teachers District has
affect student outcomes have a great deal of responsibility, but
control less direct control
Enabling Conditions necessary in School School has some District has
order to make activities in control significant control
core realm successful and leverage
Supporting Conditions that are helpful District/ School has little District has some
in making activities in core Community control control and
realm successful leverage

6. Once all potential root causes are sorted, review the enabling and supporting
dimensions. If there are very few causes in either of these categories, ask the Team to
consider what other factors might affect the problem the Team is investigating and add
any new ideas to the list.
Stop: If the data analysis in 3.3.1T yielded multiple conclusions, repeat this process for the
other conclusions (problems) before proceeding further.
7. Once the Team has generated and sorted potential root causes for each of the
conclusions, look across all the causes and ask:
Are there any issues that arose in one brainstorm that are similar to those in
others, suggesting they affect multiple areas within the district?
Do these causes primarily affect a subgroup of students, teachers, or other
stakeholders, or do they affect a much wider segment of the population we
serve?
Consolidate the issues that affect multiple areas or stakeholders. Record these in
worksheet 4A: Far-Reaching Causes, and record the remainder in 4B: Problem-Specific
Causes.
It would be impractical to address all of the causes identified. Therefore, narrow the list of
causes to identify those which can be addressed most productively by the district.
8. Rate each cause based on the impact it is likely to have on student learning and
achievement, and on the amount of control the district has over it. Causes that rate high
on the amount of impact and the amount of district control should become the focus of
subsequent action planning.

Dimensions of District Improvement
Core realm: Contains factors that most directly affect student outcomes: instruction,
curriculum, and assessments.
Enabling realm: Contains conditions that must be in place in order to make the core
elements successful in affecting student outcomes.
Supporting realm: Contains conditions that are helpful toward making the core
elements successful.
SUPPORTING ENABLING REALM

REALM
Such as: Such as:
Parents Materials CORE REALM
Governance Schedules Instruction
Personnel Professional Curriculum
Funding development Assessments
Community Organizational
resources structures
Student School leadership
readiness to Supervision
learn Staff accountability
for performance
Adapted from: Conley, David T. (1997). Roadmap to restructuring:

Charting the course of change in American education. Eugene, OR:
Clearinghouse on Educational Policy and Management

4A: Far-Reaching Causes
Record the potential root causes that apply to a number of identified problems, function areas, or groups of stakeholders.
Guiding Questions:
1. Which potential root causes have the greatest impact on the work of the district?
2. Which causes does the district have the most immediate control over?
3. What evidence does the Team have to verify its theories about why this problem exists?
Impact on
student District’s
Potential Root Cause achievement control Evidence
1 = minimal 1 = very little
2 = some 2 = some
3 = substantial 3 = a lot

Worksheet 4B: Problem-Specific Causes
Use this worksheet to record the potential root causes that apply to only one identified problem, function area, or group of
stakeholders. Be sure to indicate the target for each potential root cause.
Guiding Questions:
1. Which of these potential root causes have the greatest impact on the work of the district?
2. Which causes does the district have the most immediate control over?
3. What evidence does the Team have to verify its theories about why this problem exists?
Impact on District’s
student control
Problem, function area, achievement 1 = very
Potential Root Cause Evidence
or stakeholder group 1 = minimal little
2 = some 2 = some
3 = substantial 3 = a lot

P R O B L E M I N V E S T I G A T I O N P L A N 4 . 3 . 1 T
Purpose To structure a research effort aligned with a problem. Related Documents

Description This document serves as a way for a Data Team to 4.3.2R: Educational
identify questions that need to be answered about a Research
problem being addressed in the Data-Driven Inquiry Websites
and Action Cycle.
Time 30 minutes.
Directions:
1. Restate the underlying problem and proposed solution articulated in the Problem
Statement.
2. As a group, brainstorm questions about the problem or proposed solution that should be
checked before moving forward. It can be useful to note the underlying assumptions the
group has, e.g., that a certain factor is the most significant root cause, or that a certain
solution will have the greatest impact, and translate those into a question for
investigation. If the list is long, the Team may want to prioritize them.
3. For each question, complete the information below until all are captured and a clear plan
to investigate each is identified. Copy the table as many times as necessary to
document how the Team will address each question it has about the problem or solution.
It is not necessary to consult both research and local expertise for each question.
Example:
Problem or solution under investigation: Teachers don’t get sufficient training and support in
our reading program, so we are going to start a teacher mentoring program.
What other districts similar to ours have

Question
implemented mentoring programs focused on
we have:
reading, and how well did they work? Lead Investigator:
Research Local university teacher training department District teacher
sources to Regional Laboratory Reference Desk developer
consult:
MA ESE Literacy Office District literacy director
Local expertise Collaborative of districts
to consult: Principal and coach at a high-performing school
in our district
Date for Three weeks from now
completion:
4.3.1T: Problem Investigation Plan 1/2

Problem or solution under investigation: ________________________________________
Question
we have: Lead Investigator:
Research
sources to
consult:
Local expertise
to consult:
Date for
completion:
Question
Research
sources to
consult:
Local expertise
to consult:
Date for
completion:
Question
Research
sources to
consult:
Local expertise
to consult:
Date for
completion:
Question
Research
sources to
consult:
Local expertise
to consult:
Date for
completion:
4.3.1T: Problem Investigation Plan 2/2

EDUCATI ONA L RESEAR CH WEBSITES 4 . 3 . 2 R
Purpose To connect districts to websites with searchable Related Documents

educational research reports. 4–Knowledge Module
4.3.1T: Problem
Description This list can serve as a starting place for gathering
Investigation Plan
researched reports related to the problem the Team is
addressing and interventions the Team is considering.
Time N/A.

ERIC—Education Resources Information Center; a
1 http://www.eric.ed.gov/ federal site for collected educational resources,
including research.
What Works Clearinghouse—A website operated by
the Institute for Education Sciences to provide "a
2 http://ies.ed.gov/ncee/wwc/
central and trusted source of scientific evidence for
what works in education.”
IES REL Network—Institute for Education Sciences
3 http://ies.ed.gov/pubsearch/ search engine for publications, including research
from 10 Regional Education Laboratories.
The Regional Educational Laboratory Northeast
and Islands (REL-NEI) is part of the Regional
http://www.relnei.org/referencedesk.2009-12- Educational Laboratory Program. The REL-NEI
4
31.php Reference Desk is a free service that provides quick-
turnaround responses to education-related research
questions, offering a quick scan of existing research.
U. of Texas at Austin: Data Use Website—Dept. of
Educational Administration, College of Education;
5 http://edadmin.edb.utexas.edu/datause/index.htm
includes publications; site developed by Chief Data
Champion Jeffrey Wayman.
SEDL—(formerly the Southwest Educational
Development Laboratory); a private, nonprofit
6 http://www.sedl.org/
education research, development, and dissemination
(RD&D) corporation based in Austin, Texas.
RTI Action Network—A program of the National
7 http://www.rtinetwork.org/
Center for Learning Disabilities.
The IDEA Partnership—Reflects the collaborative
work of more than 55 national organizations, technical
assistance providers, and organizations and agencies
8 http://www.ideapartnership.org/journals.cfm at state and local levels. Click on "MANY VOICES" to
find hundreds of articles and citations from web-based
journals and other periodicals; they are building a
larger online library to open in March 2010.
Promising Practices Network—RAND corporation's
website, whose stated purpose is "providing quality
evidence-based information about what works to
9 http://www.promisingpractices.net
improve the lives of children, youth, and families." All
of the information on the site has been screened for
scientific rigor, relevance, and clarity.
4.3.2R: Educational Research Websites 1/1

PROB LE M CA TA LOGUE TE MPLA TE 4.4.1T
To give the District Data Team a systematic way to capture Related Documents
Purpose 4–Knowledge Module
problems.
This template contains a model for a catalogue of problems being

Description
addressed by groups throughout a district.
Time Ongoing.
4.4.1T: Problem Catalogue Template 1

Problem Catalogue
Problem Academic
Problem Full Problem Statement Subject Team Date Date Results Available
ID Content
Keywords (or summary) Area(s) Investigating Begun Completed (location)
Number Standard(s)
0 K–5, ELA, • Many third grade students at ELA N/A Reading 13-Jan-09 Reading
Mentoring, our school do not read at grade Intrervention Intervention
Training, level.
Assessment • We believe that this is a result
of teachers not having sufficient
training in our reading program
and not accurately measuring
students’ reading levels in grades
K–3.
• We want all third graders at our
school to read at grade level or
above.
• We will start a teacher
mentoring program focused on
reading and implement more
rigorous reading assessments in
the primary grades.
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
Collected Research and Practice Literature: Articles and Sources
Location or
Associated
Associated Problem Person with Date Added
Problem ID Title of Article, Study, or Other Item Source: Publisher or Website
Keywords Copies to this List
Number(s)
Available?
0 K–5, ELA, [Article Title 1] [Website link] Online 2/8/2010

Mentoring,Training,
Assessment
0 K–5, ELA, [Article Title 2] [Publisher name] Central comptuer 1/13/2010
Mentoring,Training, network drive
Assessment
0
0
Collected Research and Practice Literature: Articles and Sources
Location or
Associated
Associated Problem Person with Date Added
Problem ID Title of Article, Study, or Other Item Source: Publisher or Website
Keywords Copies to this List
Number(s)
Available?
0
0
MODULE 5: ACTION
D i s t r i c t D a t a T e a m T o o l k i t
Table of Contents
Introduction—1
Crafting a Logic Model—2
Why Develop a Logic Model?—2
Components of a Logic Model—3
Building a Logic Model—4
Articulating Meaningful Measures—8
Putting It All Together—11
Taking Action—12
When Are Action Plans Necessary?—12
Module Summary—14
Action
5.1.1T: Logic Model Template 5.2.2R: Elements of a Well-Written

5.1.2T: Logic Model Checklist Measure
5.2.1T: Crafting Meaningful Measures 5.3.1T: Action Plan Template
Checklist 5.3.2T: Action Plan Checklist
District Data Team Toolkit—Version 1.0 Module 5: Action
INTR ODUCTI ON
WHERE ARE WE NOW?

progression through the model—you are now in Module 5: Action.
Getting
Ready
If a District Data Team has progressed through this Toolkit in sequence, it

began by forming a focusing question and presented a data overview
related to this question (Inquiry). The Team then gathered additional data
to answer clarifying questions (Information) and identified an area to be
acted upon (Knowledge). Then it consulted the research literature and
local practice to identify one or more strategies to address the identified
problems. This module will help the Team craft a logic model that
illustrates how and why the strategies will work. It will also help the Team
create action plans, if needed, to implement the strategies.
MODULE OBJECTIVES
The Action module will help a District Data Team:
 Craft a logic model or theory of action to guide subsequent action

and evaluation
 Articulate meaningful measures of implementation and change
 Develop action plans, if necessary, to implement new strategies or

to implement existing strategies more effectively

CRA FTING A LOGI C MODEL
WHY DEVELOP A LOGIC MODEL?

Now that the District Data Team has selected the strategy or strategies it
will employ to address the identified problems, it might be tempting to
Internal stakeholders move directly to the implementation phase. However, whether the Team
will want to know identified new strategies or proposed to leverage existing ones, it must be
what their roles and able to justify to internal and external stakeholders how and why the
responsibilities will allocation of these resources will have the intended impact on the skills,
knowledge, and behavior of the targeted adults and students.
be in implementing
these strategies. Internal stakeholders, i.e., the district leadership team, central office staff,
External stakeholders principals, and teachers, will want to know what their roles and
responsibilities will be in implementing these strategies. External
must be convinced
stakeholders, i.e., the school committee, parents and families, state
that the proposed education agencies, and the public, must be convinced that the proposed
changes are changes are compelling, will result in rapid improvement, and will use
compelling. resources effectively.
A logic model articulates to these internal and external stakeholders, in

plain language, the connections among the selected strategies, needed
resources, intended outcomes, and desired impact on goals at the district,
school, or classroom levels.
Compelling logic models:
 Show cause and effect. The model succinctly communicates that

given x strategies and resources, we expect y and z to occur as a
result.
 Are bold yet achievable. Taken as a whole, the strategies are
likely to lead to ―
improved results for students, long-term gains in
school and school system capacity, and increased productivity
and effectiveness‖ in the targeted areas.1
 Give implementation and outcome measures. The model
describes the changes that the strategies, when effectively
implemented, will have on the skills, knowledge, and behavior of
adults and students in the system.

COMPONENTS OF A LOGIC MODEL

A typical logic model has the following components:
INPUTS OUTPUTS OUTCOMES IMPACT
Problem Measures of Measures of

Strategies Resources Goals
Statement(s) Implementation Change
 Inputs consist of:
A problem statement(s) that sets the context.
Strategies that outline a course of action and answer the

question: what are the specific means, methods, or
approaches we will use to solve the problem(s) we identified?
The identification of key resources needed to support
successful implementation of the strategies.
 Outputs, or measures of implementation, communicate if the
adults involved have implemented key action steps.
 Outcomes, or measures of change, communicate if the adults
and students involved have acquired and are applying the skills,
knowledge, and expertise needed to achieve the goals of the
initiative.
 Impact, or goals, articulates the goal for improvement and long
term impact, long-term, sustainable effects the district expects to
see as a result of the model.
Each component is explicitly tied to the others. However, it is not

necessary to have a one-to-one relationship between items in one column
and items in the next. For example, a district may develop a logic model
with 3 strategies, 12 essential resources, 5 key measures of
implementation, and 3 key measures of change, in order to reach 1 goal.
The resulting model gives the reader a clear sense of how the district
perceives the problems it identified, the strategies selected to address
those problems, the supports needed to implement those strategies
successfully, and the outputs and outcomes that should occur if the
strategies are implemented effectively.

BUILDING A LOGIC MODEL

In completing the logic model, the Team does not proceed from left to
right, but does a form of backward mapping. The Team begins by naming
the problem and proposed solutions, articulates the impact, names the
outcomes indicating progress toward that goal, and then returns to the
inputs to complete the model. The numbered columns below show the
progression:
1 4 5 6 3 2

Adapted from: W.K. Kellogg Foundation (2004). Logic Model Development Guide. Battle Creek, MI: Author.2
1. The Team begins by providing a problem statement (or

statements)—a brief description of the problem it wants to address,
including naming the nature of the problem, the stakeholders who are
most affected by the problem, and the suspected cause of the
problem, based on rigorous analysis. This statement should be based
on rigorous data analysis and/or root cause analysis. It should
represent what the Team thinks is the most significant cause(s)
contributing to the problem. What, if addressed, would make the
greatest impact on resolving the problem? It may be helpful to
reference specific evidence.
If the Team has completed 4.1.1T: Writing a Problem Statement, then

its work for this and other sections of the logic model will already be
complete.
2. Next the Team articulates the overall goals the initiative is intended to
accomplish or, stated another way, the impact the initiative will have
on the culture of the district, its schools, or its classrooms. This
section describes the wishes, dreams, and general vision describing
the target, the sustained effects, or consequences the district expects
to see over a multi-year period. The Team should seek the support of
the superintendent in developing appropriate goal statements
because they should align with the district’s strategic goals. (These
can also be drawn from 4.1.1T).

3. Having identified the impact of the initiative, the district articulates the
outcomes, or measures of change (or outcomes) that indicate
progress is being made toward those goals. These measures provide
evidence of shifts in the skills, knowledge, and behavior of the adults
and students targeted by the strategies. They articulate in specific and
measurable in terms what will change, for whom, by how much, and
by when.
 For adults, these measures demonstrate how teachers,

principals, and/or district personnel will approach their work
differently as a result of the actions taken.
 For students, these measures demonstrate the extent to which
the shifts in adult practice are having an impact on their
learning and achievement.
These measures often describe things that can be hard to quantify

easily and effectively. Yet well-defined measures can provide a
powerful focus for all involved in the effort. As the maxim goes, what
gets measured gets done.
A diverse array of
The Team should articulate both short- and long-term outcome measures will
measures that will demonstrate progress toward the goal, rather than
provide a richer
rely solely on data collected once the strategy is fully implemented. A
diverse array of measures will provide a richer picture of the effect of picture of the effect
the strategy than measures that capture similar evidence. The Team of the strategy than
may want to consider collecting outcome data related to the four measures that capture
domains introduced in Module 1: Getting Ready—demographics,
similar evidence.
perceptions, and school (or district) processes, as well as student
outcomes.3
The section Articulating Meaningful Measures later in this module

provides more guidance for creating and refining these measures.
4. Once the Team has clearly articulated its outcomes, it then goes back
to the second column and lists its strategies—the specific means,
methods, or approaches to solving the identified problem(s). These
should represent promising practices drawn from research, local
knowledge, and local expertise. Sources should be noted if possible.
(These can also be drawn from 4.1.1T if the Team has completed it).
Strategies may include:
 Policies, such as student discipline and suspension policies, or

those related to developing and retaining an effective,
academically capable, diverse, and culturally competent
educator workforce.

 Structures, such as time in the schedule for common planning

or technology and other infrastructure.
 Systems, such as the district's method of screening the
reading skills of entering students, as well as the many central
office processes that support school operations.
 Curricula, such as pacing guides, textbooks, and
supplementary materials.
 Instructional strategies, such as the use of graphic organizers
to organize details in a story; grouping strategies; and the use
of formative assessments to gauge student performance on a
specific skill.
Strategies may range from relatively large initiatives, such as aligning

professional development to the district’s improvement plan, to small
actions, like the use of particular teaching techniques in the
classroom. It is important to consider what degree of specificity is
needed to address the identified problem, given its size and scope. If
the proposed strategy (or strategies) is either too broad or too narrow
in relation to the problem it is addressing, it is unlikely to be useful in
generating the desired impact. It will also be harder to convince
internal and external stakeholders how the strategy will address the
issues identified.
As stated earlier, a single strategy is unlikely to achieve the district’s

goals or withstand external scrutiny. Typically a combination of
strategies implemented in coordination is required to address the
problem statement.
In deciding on the strategies to reach the desired goal, first consider

what strategies are already in place, and consider the value of
continuing or refining these before creating any new strategies. The
Inventory of District and School Initiatives (Activity 2.2.1) may be
useful for this step.
5.
What the Team identifies for resources answers the question: What
supports are available to the district or our schools to implement our
strategies? While the most obvious resource is funding, a district’s
primary resource is its personnel. The Team should consider: How is
the time of teachers, administrators, and support staff throughout the
district being used, and how might it be used differently to address the
strategy or strategies?

Other resources include technology, materials, systems,

infrastructure, polices, political support, and partnerships with
organizations like the Department itself.
In identifying these resources, the Team is making a commitment to

allocate them, protect them, and prioritize their use in service of these
strategies.
6. Measures of implementation (or outputs) communicate how the

reader of the plan will know the strategy or its key action steps are
carried out and if so, to what extent. These measures generally apply
only to the adults in the system, and not to the students. They answer
the questions what will take place, with or for whom, to what extent,
and by when? These measures tend to be more concrete, discrete,
and more easily quantified than the measures of change.
The Logic Model Template provides a framework for documenting the

Team’s work. The Logic Model Checklist can assist the Team in reflecting
on the strength of its logic model and the degree to which it will help drive
the work forward.
The next section, Articulating Meaningful Measures, provides more

guidance for refining these measures.
Activity 5.1 Crafting a Logic Model
These tools will guide the District Data Team in crafting and refining a
logic model to guide implementation and evaluation of its strategies.
(5.1.1T: Logic Model Template)

(5.1.2T: Logic Model Checklist)

ARTICULATING MEANINGFUL MEASURES

It is critical that the District Data Team clearly and meaningfully articulate
measures of implementation and measures of change, as they will serve
a number of functions. For example, these measures–also known as
indicators or benchmarks–can be used to:
 Guide and inform the work and subsequent evaluation
 Provide a range of evidence for analysis and triangulation with

other data sources
 Inform mid-course corrections, if needed
 Support the ability of the Team to communicate successes and

needs to stakeholders on an ongoing basis.
In crafting its measures of implementation and change, the District Data

Team is essentially deciding what will be evaluated. Since the measures
that are articulated will be used to monitor progress, it is important to
articulate only those measures whose evidence will be collected and
analyzed.1 The Team should not identify a measure if the evidence it
Well-defined would generate does not relate directly to an evaluation question.
measures can
provide a powerful In deciding what to evaluate, and therefore what to measure, the Team
might consider:
focus for all involved
in the effort.  What are the questions internal and external stakeholders
(including the Team itself) want to have answered?
 How will the information be used, and by whom?
By reflecting on these questions, a Team can narrow the focus and scope
to those measures that will best document and communicate progress
toward outcomes and impact.
As mentioned earlier in this module, measures of change in particular
often describe things that can be hard to quantify easily and effectively.
However, well-defined measures can provide a powerful focus for all
involved in the effort.
1
See Module 6: Results for more information on the evaluation process.

Measures of Implementation… Measures of Change…

(Outputs) (Outcomes)
Answer: What will take place, for Answer: What will change, for whom,
whom, to what extent and by when? by how much, and by when?
Focus primarily on adult actions Address both the adults and the
students involved in the strategy
Focus on the short term Focus on the intermediate and long
term
Focus on the fairly technical aspects Focus on elements that are less
of the work (such as ensuring all tangible, such as the acquisition of
teachers receive a particular training, skills and knowledge, or the shifting of
or that a certain number of students habits and beliefs
are enrolled in a program) For adults, these measures
demonstrate how teachers, principals,
and/or district personnel will approach
their work differently as a result of the
actions taken (such as teachers
implementing new techniques from a
training)
For students, these measures
demonstrate the extent to which the
shifts in adult practice are having an
impact on their learning and
achievement. (such as students
shifting their aspirations)
Are used to monitor if things got Are used to monitor if actions are
done having the intended effects
Communicate whether Communicate whether implementation
implementation of the strategy or is expanding capacity and resulting in
strategies is taking place as desired positive outcomes
Help determine whether particular
strategies or initiatives should be
continued, expanded, or discontinued
On a larger scale, these measures can
help prove whether the district’s theory
of improvement, as articulated in its
logic model, is sound and effective
Internal stakeholders are the primary External stakeholders are the primary
audience, e.g., the Team itself and audience, e.g., other district personnel,
those directly involved with the as well as the wider community
action plan
The Team should articulate short-, intermediate- and long-term measures

(sometimes known as formative and summative measures) that span a
progression of time and that will demonstrate progress toward the goal.
Similarly, the Team should embed into its work the means to collect,
monitor, and evaluate the evidence generated by these measures, rather

than waiting until implementation is complete to begin that process. In

specifying these measures, the Team is making a commitment to engage
with the evidence in an ongoing way to see what can be learned and to
make changes to the operating plan in mid-course, if necessary.
For a useful mnemonic device to follow in developing indicators, consider

SMART:
 Specific: What are the specific criteria against which the outcome
will be judged?
 Measurable: What will be the method or tool used to measure
progress?
 Action-oriented: Is there consensus among stakeholders that this
is a worthy outcome?
 Realistic: Is this outcome sufficiently bold, yet still achievable
given available resources, knowledge, and time?
 Timed: Does the indicator specify by when the outcome will be
achieved?
A diverse array of
measures will Evidence for these measures, like other data, comes in a variety of forms.
The Team might consider articulating data that represent the four
provide a richer
domains of data outlined in Module 1: Getting Ready—demographics,
picture of the effect perceptions, and school or district processes, as well as student
of the strategy than outcomes. Another way to think of it is to consider data that can be
measures that capture counted (such as assessment data), seen (such as classroom
observation data), and heard (such as stakeholder feedback).
the same or similar
evidence, and will A diverse array of measures will provide a richer picture of the effect of
facilitate self- the strategy than measures that capture the same or similar evidence,
and will facilitate self-monitoring and measurement of success in each
monitoring and
area. Sample measures include:
measurement of
success in each area. Achievement, assessment, improvement, and percent
completion data
Stakeholder perceptions, such as a survey of teachers,
administrators, school committee, students, and/or families
Self-assessments, such as the Common Planning Time self-
assessment or Essential Conditions rubric2
Observation data, such as Learning Walkthrough evidence
2
For more information on these and other district support resources, visit http://www.doe.mass.edu/sda/ucd/
or email districtassist@doe.mass.edu.

Organizational data
External review data (such as MASBO or PQA)
For further guidance on crafting meaningful measures, see the attached

checklist and resources.
Sample measures include:
 Measure of Implementation: By May of 2010, 50 percent of

middle school mathematics teachers will have received
professional development in developing algebraic thinking. By the
start of SY 2010–2011, 95 percent will have been trained.
 Measure of Change (for adults): Within six months of receiving
training, 75 percent of mathematics teachers will be observed
implementing the material introduced in the training, as evidenced
by data collected through Learning Walkthroughs and feedback
forms completed by instructional coaches.
 Measure of Change (for students): Across the district’s
elementary schools, the number of students reaching grade level
benchmarks for demonstrating algebraic thinking, as measured by
a district-wide common assessment, will increase by 20 percent
between November 2010 and February 2011.
Activity 5.2 Elements of a Well-Written Measure
These resources can help the Team articulate meaningful measures.

The first document is a checklist to guide the creation or revision of a
measure. The second resource provides an example of how a measure
can be made stronger and more informative, and provides sample
scenarios to give the Team practice articulating measures.
(5.2.1T: Crafting Meaningful Measures Checklist)

(5.2.2R: Elements of a Well-Written Measure)
PUTTING IT ALL TOGETHER

Once the Team articulates the six components of a logic model, the
resulting document, when read from left to right, represents the district’s
theory of change—a rationale for what the district believes is the best way
to fulfill its mission, vision, and goals. While completing the model can be
a long and sometimes difficult process, the result will be increased clarity
and agreement on where to focus the time and energy of all involved. A
clear logic model can also serve as the basis for evaluation of the
district’s efforts, which is discussed in Module 6: Results.

TA KING A CTI ON
In developing its logic model, the District Data Team identified a strategy
(or, more likely, a set of strategies) that will be used to address the
problems it identified. Now the Team is ready to begin taking action.
In some cases, the logic model alone will be sufficient to guide the work
and the Team will not need any additional detail. In other cases, an action
plan may be useful or even necessary.
WHEN ARE ACTION PLANS NECESSARY?

An action plan is a strategic series of steps designed to ensure that an
identified strategy is implemented. For each step, the plan outlines the
necessary resources and measures of implementation, as well as an
owner who will ensure it is accomplished and a deadline by which the
step is expected to be completed.
An action plan can be a valuable resource to:
Guide the implementation of certain strategies

Provide a basis for monitoring the progress of those strategies
However, action plans are not always necessary. In some cases, a strong
logic model with clear strategies and measures may be all the Team
needs to guide implementation and monitoring of the work.
In considering whether or not to craft an action plan, the Team might ask:
Do action plans already exist for the strategies the Team has
identified?
Would an action plan significantly enhance the district’s ability to
delegate steps and/or monitor their completion?
Generally speaking, action plans are only required for new strategies or
strategies that require changes to function as intended. For example, a
district initiating professional development in sheltering instruction for
English language learners may need to specify the action steps required
to implement the training to make sure it occurs. On the other hand, if
walkthroughs were selected as a strategy and the district already uses

walkthroughs effectively to observe classrooms, there is no need to

develop an action plan for that strategy.
Also, action plans guide strategy implementation. However, they do not

give any indication of the effects of those actions, and therefore are not
adequate for monitoring outcomes. This is done by an evaluation plan,
which is discussed in Module 6: Results.
If the District Data Team determines that an action plan would be a useful
tool for the situation it is addressing, it will want to reference the attached
documents that describe the components of an action plan and guide the
Team in building one.
Activity 5.3 Crafting an Action Plan
These tools will guide a District Data Team in crafting and refining an
action plan to guide its work.
(5.3.1T: Action Plan Template)

(5.3.2T: Action Plan Checklist)

MODULE SUMMARY
Module 5: Action builds the capacity of the District Data Team to craft a
logic model that provides a rationale for how the strategies it selected to
address the identified problems will work. The logic model is the district’s
rationale and justification for addressing the problems the District Data
Team helped identify. It gives an argument for why existing strategies,
given some adjustments, and possibly some new strategies as well, will
achieve superior results than in the past. Although the Team may not be
the primary author or owner of the district’s logic model or action plans, it
will certainly play an important support role in their creation.
This module also guides the Team on when and how to develop action
plans to support the implementation of these strategies.
However, the logic model is only a theory of improvement. Until it is put to

use—and rigorously monitored—the Team may find that certain
components need refinement in order to achieve the intended results.
Perhaps a strategy was not fully implemented according to the timelines
set out in its action plan because it lacked a key resource. Perhaps the
evaluation data yielded little change in the skills, knowledge, and
behaviors of students or adults. The next module, Results, guides the
Team in creating an evaluation plan that will prepare it to analyze whether
or not the strategies have had the desired impact.
References
1
Race to the Top Executive Summary, p. 2. U.S. Department of
Education. Washington D.C. November 2009
2
W.K. Kellogg Foundation (2004). Logic Model Development Guide.
Battle Creek, MI: Author
3
Bernhardt, V. L. (2004). Data Analysis for Continuous School
Improvement. Larchmont: Eye on Education.

LOGIC MODE L TE MP LATE 5 . 1 . 1 T
Purpose Articulate to internal and external stakeholders–in plain language–the connection between Related Documents
the problem statements identified and the strategies identified to address them. 5–Action Module
5.1.2T: Logic Model Checklist
Description Use the template below, the guidance in the text, and the checklist to craft a logic model
that shows the links between the team’s ideas about improvement, strategies, resources,
outputs, outcomes, and the intended impact on district and/or school practices.
Time Varies. 2–3 hours to start, but may take many meetings to refine.
Directions:
Follow the numbering sequence below, respond to the guiding questions in each column. For example, begin in the first
column by naming the problem and proposed solutions; then jump to the last column and name the end goal (the desired
impact) before identifying the measures of change that will demonstrate improved outcomes for adults and students.
It is not necessary to have a one-to-one relationship between items in one column and items in the next. For example, a
district may develop a logic model with 3 strategies, 12 resources, 5 key measures of implementation, and 3 key measures of
change, in order to reach 1 goal.
See the text in Module 5 for additional guidance on completing the logic model.
In crafting the logic model, consider:

Does the model give a credible chain of events, from strategies through outcomes? Does the model succinctly communicate,
Given x resources and strategies, we expect y and z to occur as a result?
Are strategies and outcomes articulated in the logic model compelling? Taken as a whole, are the strategies likely to lead to
improved results for students, long-term gains in school and school system capacity, and increased productivity and
effectiveness in the targeted areas?
Does the model give measures of implementation as well as outcomes?
Use 5.1.2T: Logic Model Checklist to refine the completed model.
5.2.1T and 5.2.2R provide more detailed guidance for crafting and refining meaningful measures of implementation and change.
5.1.1T: Logic Model Template—Version 1.0 1/2

District/School Name:
Today’s Date: Date Approved by Superintendent:
1 4 5 6 3 2
Problem Goals
Measures of
Statements &
Strategies Resources Implementation Measures of Change (Outcomes) (Desired
Proposed
(Outputs) Impact)
Solutions
What are the specific What supports are What intermediate and longer-term results do we expect
How will we know What are the
means, methods, or available to the to achieve as measured by changes in skills,
What is our proposal whether the strategies sustained effects or
approaches we will district or our knowledge, and behavior?
for addressing the we described were consequences we
use to solve the schools to
problem? implemented by the Students expect to see over a
problem(s) we implement our Adults
adults? multi-year period?
identified? strategies?
5.1.1T: Logic Model Template—Version 1.0 2/2

LOGIC MODE L CHE CKLIS T 5 . 1 . 2 T
Purpose To help a Team refine or revise a logic model that will articulate its theory of action for Related Documents
addressing an identified problem. 5–Action Module
5.1.1T: Logic Model Template
Description This checklist can guide a District Data Team in designing or refining a logic model to
guide its action and evaluation plans.
Time 1–2 hours.
District/School Name: _____________________________ ____________ Date: ___________
Problem Statements & Proposed Solutions Y/N

Does the problem statement indicate the people who are affected?
Does the problem statement identify the underlying cause of the problem?
Does the problem statement propose how to address the problem?
Strategies Y/N
Are strategies described at the district level and at the school or classroom levels, as needed, to address the problems that were
identified?
Taken as a whole, are the strategies articulated in the logic model sufficiently bold and ambitious?
Taken as a whole, are the strategies articulated in the logic model sufficiently attainable?
Taken as a whole, can the strategies withstand scrutiny?
Taken as a whole, do the strategies represent a justifiable use of resources in light of current strategies and initiatives in the district?
Are existing strategies described in ways that indicate how they will achieve different results than in the past?
Do the descriptions of the strategies communicate to internal and external stakeholders how the strategy will address the issues
identified?
Resources Y/N
Do the resources identify the full range of supports needed by the district or its schools to implement the strategies?
5.1.2T: Logic Model Checklist—Version 1.0 1/2

Measures of Implementation (Outputs) Y/N
Are outputs described in terms of adult actions and behavior?
Do outputs document technical steps for which the actions are clear and the results easily quantified?
Are outputs specific in terms of what will take place, with/for whom, to what extent, and by when?
Measures of Change (Outcomes) Y/N
Do outcomes describe the intermediate (formative) and longer-term (summative) results the district expects to achieve?
Do outcomes provide evidence of shifts in the skills, knowledge, and behavior of the adults and students targeted by the strategies?
Do outcomes adequately describe how teachers, principals, and/or district personnel will approach their work differently as a result of the
strategies?
Are outcomes specific in terms of what will change, for whom, by how much, and by when?
Goals (Desired Impact) Y/N

Does the goal statement describe the sustained effects or consequences the district expects to see over a multi-year period?
Does the goal describe how the systems, processes, practices, and structures described elsewhere in the logic model will transform the
culture of the district and what it believes to be true about good teaching and learning?
(Overall) Y/N
Does the model articulate to internal and external stakeholders, in plain language, the connection between strategies, outcomes, and the
desired impact of goals at the district, school, or classroom levels?
Does the model give an idea as to the roles and responsibilities of internal stakeholders, i.e., the district leadership team, central office
staff, principals, and teachers, in implementing the strategies?
Does the model serve as convincing evidence to external stakeholders, i.e., the school committee, parents and families, state education
agencies, and the public, that the proposed changes are compelling, will result in rapid improvement, and will use resources effectively?
Does the model give a credible chain of events, from strategies through outcomes?
Taken as a whole, are the strategies articulated by the model likely to lead to improved results for students, long-term gains in school
and school system capacity, and increased productivity and effectiveness in the targeted areas?
Does the model outline both implementation measures (outputs) and measures of change (outcomes)?
Does the model reflect input from the superintendent so it aligns with the district’s strategic goals for systemic improvement?
5.1.2T: Logic Model Checklist—Version 1.0 2/2

CRAFTING MEANINGFUL MEASURES CHECKLIST 5.2.1T
Purpose To guide the creation of useful measures of implementation Related Documents

(outputs) and measures of change (outcomes). 5–Action Module
5.2.2R: Elements of a
Description This checklist can guide a District Data Team in crafting and
Well-Written
refining measures that can be useful in monitoring the
Measure
implementation of a logic model, action plan, or evaluation plan,
and determining progress toward desired goals.
Time 30–90 minutes
Note:
This activity may be conducted with the District Data Team, or with other stakeholders
involved in the improvement effort being discussed.
This activity is most useful if the Team has drafted a logic model, action plan, or evaluation
plan prior to beginning.
Measures are also known as indicators or benchmarks, and can be short-term (formative) or
long-term (summative).
Directions:
Use this checklist to reflect on and refine the measures you have crafted to demonstrate
implementation and change.
First respond to the questions separately for each individual measure you have crafted, then
reflect on all the measures collectively in relation to the identified strategy (or strategies) or
action steps.
Examples:
Unrefined measure: Increase the percentage of students who pass the MCAS tests on the first
attempt.
Refined measure: To increase the percentage of students who pass the MCAS ELA test on the first
attempt by 4 percentage points each year from 2010 to 2014.
5.2.1T: Crafting Meaningful Measures Checklist—Version 1.0 1/2

District/School Name: ____________________ Date: _____ __
Foundational Questions Y/N Y/N Y/N
1. Does the measure respond to the questions:
What will change?
For whom?
By how much? (or how many or what percent—be specific)
By when?
2. Does the measure articulate what evidence is needed to demonstrate progress?
The specific criteria against which the measure will be evaluated?
3. Does the measure paint a clear picture of what, precisely, it will look like to achieve
full implementation or the full degree of change desired?
4. Is there consensus among stakeholders that this is a worthy outcome?
5. Is there consensus among stakeholders that this measure is needed to evaluate the
success of the strategy or action step?
Refining Questions Y/N Y/N Y/N
6. Is this measure sufficiently bold, yet still achievable given available resources,
knowledge, and time?
7. Does the measure suggest or explain the connection between the data and the
strategy or action(s) that contribute to success in the targeted areas?
8. Are the data generated by the measure adequate to evaluate the effectiveness of
the strategy or action step(s) they relate to?
9. Is the description of the measure specific and measurable enough to allow for clear
communication with stakeholders, as well as for monitoring and evaluation?
10. Is the timeline realistic given current capacity?
11. Has the Team identified a means to embed collection and analysis of the measure
into the action plan?
Looking at the Measures Collectively Y/N Y/N Y/N
12. Will the measures demonstrate the success of a given strategy or action step(s)?
13. Does the achievement of all the benchmarks for a given action step show that the
strategy was successfully implemented? If not, what other benchmarks need to be
included?
14. Do the measures indicate substantial and clear progress toward the desired goal at
intervals when monitoring and evaluation will occur? e.g., short and long-term
measures to demonstrate results are evident at 3 months, 6 months, etc.
15. Is there a sufficient balance of quantitative and qualitative indicators to measure
results?
16. Is there a balance of data from a variety of domains (demographics, perceptions, and
school or district processes, as well as student outcomes)?
17. Is there a sufficient range of measures to promote self-monitoring and measurement
of success for each strategy?
5.2.1T: Crafting Meaningful Measures Checklist—Version 1.0 2/2

E LE M E NTS OF A W E L L- W R I T TE N M E A S UR E 5 . 2 . 2 R
Purpose To learn how to articulate clear measures of Related Documents

implementation (output) and change (outcomes). 5–Action Module
5.2.1T: Crafting Meaningful
Description This activity provides an example of how a typical
Measures Checklist
measure is refined to be more specific and useful.
This can serve as a guide for districts as they craft
measures of implementation and change related to
their strategies and goals.
Time 30 minutes.
Directions:
1. Open and study the text below: Elements of a Well-Written Measure.
2. Using this document as a guide, each Data Team member should create a well-written
measure from each of the following scenarios by:
Identifying each of the four elements of a well-written measure
Writing a clear statement
3. As a Data Team, reach consensus on the most appropriate measure for each scenario.
Possible ―answers‖ are included at the end of this document, which you may want to
consult AFTER your discussion!
5.2.2R: Elements of a Well-Written Measure—Version 1.0 1/5

Elements of a Well-Written Measure
Typical Goal or Desired Impact

 Increase the achievement of all students.
Typical Measure
 Increase the percentage of students who pass the MCAS tests on the first attempt.
A clearer and more useful measure would address the following questions:
1. What will change?
The percentage of students who pass the MCAS ELA test.
2. For what population?

Students who are taking the MCAS for the first time.
3. By how much?
What percentage passed in 2005–06? (57% in ELA; 58% in mathematics)
What percentage passed in 2006–07? (73% in ELA; 57% in mathematics)
What percentage must pass to meet NCLB requirements? (100%)
When do the NCLB requirements need to be met? (2014)
What incremental increase must be sustained to meet the NCLB requirements?
(ELA: 100% (in 2014) – 73% (in 2006–07) = 27 percentage points difference;
2014–2008 = 7 tests remaining to show progress; 27/7 = 3.9 percentage
points/year needed to reach target if linear improvement is assumed.
4. By when?
Annual improvement target?
By 2014?
After a certain number of years to allow the impact of an intervention to be felt?
The resulting measure (compare to the typical measure above):

 To increase the percentage of students who pass the MCAS ELA test on the first
attempt by 4 percentage points each year from 2010 to 2014.

Scenarios
Scenario 1: The Scenic Cove School District needs to determine if the new grade 3–6 English
curriculum purchased in 2009 and used with the English language learners with special needs is
effective in reducing the performance gap between special education and general education
ELL students as measured by the MCAS English test from 10 percentage points to 0 by the
2012 test.
Elements:
What will change?
For whom?
By how much?
By when?
Statement:
Scenario 2: In response to chronically low mathematics performance across the district, the
superintendent reallocated resources from the professional development budget to the salary
account to enable hiring mathematics coaches to provide embedded professional development
for the middle school mathematics teachers in one of the two district middle schools. The
superintendent hoped to see a significant increase in the percent of proficient students (at least
a 10 percentage point increase) in the middle school students whose teachers had participated
in the embedded professional development provided by mathematics coaches within three
years.
Elements:
What will change?
For whom?
By how much?
By when?
Statement:
Scenario 3: The superintendent of the Scenic Cove School District reviewed cohort graduation
data (same students grade 9 to grade 12) for the class of 2009 and was shocked to see that
only 80% of the cohort graduated on time, while the state average was 95%. The
superintendent instructed the assistant superintendent to work with the District Data Team to
develop a measurable improvement target for the class of 2012, in order to bring the district in
line with the current state graduation rate.
Elements:
What will change?

For whom?
By how much?
By when?
Statement:
Scenario 4: The district leadership team, with support from the District Data Team, reviewed
MCAS performance in ELA at each grade level over the past three years. The data revealed
that the percentage of proficient female students was consistently five percentage points or
more higher than the percentage of proficient male students at each grade level. The District
Data Team was asked to create a measurable improvement target for male students to
eliminate this gap in performance by the 2012 test administration date.
Elements:
What will change?
For whom?
By how much?
By when?
Statement:
Scenario 5: During the district principals’ meeting, the principals of the four district high schools
noted that the data displays the District Data Team provided clearly showed a positive
relationship between high absence and low performance on the grade 10 MCAS ELA test. On
the 2009 MCAS Mathematics test, 30% of the grade 10 students had been absent for 10 or
more days prior to test administration. Of these, 90% scored at the failing level. The principals
worked together, with support from the District Data Team, to craft an improvement goal for
attendance in their schools that would have no student with 10 or more absences prior to the
2011 MCAS test administration date.
Elements:
What will change?
For whom?
By how much?
By when?
Statement:
Scenario 6: While investigating the cohort graduation rate, the assistant superintendent noticed
that students who were retained in grade 9 generally didn’t graduate with their class. Five
percent of the students in the class of 2009 cohort had been retained in grade 9 and only 10%
of these students graduated with their class. To develop an action plan to address this problem,
the assistant superintendent must craft a measurable improvement target for grade 9 retention
for the class of 2013.
Elements:
What will change?
For whom?
By how much?
By when?
Statement:

Possible Answers to Scenarios
Scenario 1
Elements:
What will change? Gap between special education and general education
For whom? English language learners with special needs
By how much? 10 percentage points
By when? 2012
Statement: To decrease the gap between special education and general education ELL student
performance on the MCAS English test from 10 percentage points on the 2009 tests to 0 on
the 2012 test.
Scenario 2
Elements:
What will change? Percent of students proficient in MCAS Mathematics
For whom? Middle school students
By how much? 10 percentage points
By when? Within three years
Statement: To increase within three years the percentage of middle school mathematics students in the
target school who score at the proficient level or above on the MCAS mathematics test.
Scenario 3:
Elements:
What will change? Graduation rate
For whom? Class of 2012
By how much? From 80% to 95%
By when? 2012
Statement: To increase the cohort graduation rate from 80% to 95% for the class of 2012.
Scenario 4:
Elements:
What will change? The gap between proficient male and female students
For whom? Male students at all grade levels
By how much? To equal the percentage of proficient female students
By when? 2012 MCAS test administration date
Statement: To increase the percentage of proficient male students at each grade level to equal the
percentage of proficient female students by the 2012 MCAS test administration date.
Scenario 5:
Elements:
What will change? The percentage of students with 10 or more days of absence
For whom? Grade 10 students
By how much? No student with 10 or more absences
By when? 2011 MCAS test administration date
Statement: To decrease to 0 the percentage of grade 10 students with 10 or more days of absence prior
to the 2011 MCAS test administration date.
Scenario 6:
Elements:
What will change? Grade 9 retention rate
For whom? Class of 2013
By how much? By 5 percentage points
By when? 2013
Statement: To decrease the grade 9 retention rate for the Class of 2013 by 5 percentage points.
ACTI ON PLAN TE MPLATE 5 . 3 . 1 T
Purpose To document the series of steps needed to ensure that the strategies identified by the Related Documents
District Data Team to address the identified problem areas are implemented as intended. 5–Action Module
5.3.2T: Action Plan Checklist
Description Develop an action plan (if needed) to implement new strategies or to implement existing
strategies more effectively.
Time 30–45 minutes for each strategy requiring an action plan.
Note: As discussed in the text of Module 5: Action, it is not always necessary to develop an action plan. In considering whether or
not to craft an action plan, a district might ask:
 Do action plans already exist for the strategies we have identified?
 Would an action plan significantly enhance the district’s ability to delegate steps and/or monitor their completion?
Generally speaking, action plans are only required for new strategies or strategies that require changes to function as intended.
Components of an Action Plan
In relationship to the logic model, an action plan has the following components:

Measures of
Problem Statement(s) Strategies Resources Measures of Change Goals
Implementation
Strategy:
Action Steps Resources Measures of Implementation Owner Deadline
5.3.1T: Action Plan Template—Version 1.0 1/4

Essentially, action plans drill down to provide more detail linking the strategies and resources to the measures of implementation and
ultimate goals. If the Team has already developed a logic model to articulate the district’s theory of action in supporting improvement,
then the strategies, resources, and many of the implementation measures will have already been identified. If it does not have a logic
model, then the steps below can help the Team create a plan.
Building an Action Plan
Begin by asking:
If the district intends to leverage an existing strategy to address the problems it identified, does the strategy need to be
refined, adjusted, or improved prior to its implementation?
If the district identified a new strategy to address the problems it identified, does the new strategy require multiple
steps to be implemented?
If the response to either of the above questions is “NO”, then the strategy does not require an action plan. The District Data
Team should focus its time and energy on developing action plans for the other strategies it identified, or proceed to Module 6:
Results to develop an evaluation plan.
If the response to either of the above questions is “YES”, use the template below to craft a plan that will guide implementation of
the activities:
Begin by articulating the overarching strategy that the action plan will support.
Respond to the guiding questions in each column, proceeding from left to right. For example, begin in the first column by
naming the specific action steps. For each action step, indicate the necessary and available resources, the measures of
implementation (which can be drawn from the logic model, if it exists), the owner, and the deadline.
Each action step should have corresponding information in each of the other columns. For every action step, there should be
resources, at least one measure of implementation, an owner, and a deadline for completion.
Note:
If the Team has crafted a logic model (5.1.1T and 5.1.2T), it should have that available for reference, as the strategies,
resources, and many of the implementation measures will have already been identified.
5.3.2T: Action Plan Checklist provides additional guidance on refining the completed plan.

Directions for Completing an Action Plan
1. Indicate the strategy—the specific means, method, or approach to solving the identified problem(s). If the Team has
completed a logic model, these are outlined in the second column. If the Team has not completed a logic model, it will need to
determine the best approach to addressing the identified problem. Since the strategy is the driver of the action plan, it is
essential that the Team think carefully about this component.
2. List the action steps that describe the major steps that must be taken to implement the strategy. They should be listed in
order of completion, note which need to be completed before others, and which need to be sufficiently described so others
can understand them and carry them out.
In articulating action steps, a district should stay focused on the big picture, naming only the most significant, far-reaching
steps that need to be monitored as evidence of progress toward the goal. However, the owner of a specific action step may
wish to add detail to the action plan to guide his or her particular work, e.g., if he or she has to manage a team of people to
get the work done, or has many details to track.
3. Indicate the resources needed to implement one or more action steps, if needed. In some cases, key resources may be
lacking or not yet allocated to the project, in which case one activity would be to secure those resources. For example, ESE
offers a wide range of technical assistance, but accessing that assistance for a particular project would require someone from
the district approaching the appropriate office to see what is available. As with the strategy, if the Team has completed a logic
model, it will have already identified this information.
4. Indicate the measures of implementation that tell when the action step or strategy is fully realized or carried out. For
example, the measure of implementation of a professional development strategy might be that a certain percentage of
teachers receive training over a specified period. Again, if the Team has completed a logic model, it will have already
determined what will serve as evidence that key action steps have been implemented, and can use those measures as a
starting point. However, since the action plan by definition provides more detail, the Team may want to add additional
measures of implementation that capture a finer grain of detail.
5. Indicate the owner—the individual most closely responsible and accountable for a given action step. It is essential that this be
a specific person and that they have the resources, capacity, authority, and support required for completing the step.
6. Give the deadline by when the action step will be completed. Completion of the last step signifies the date by which the
strategy is expected to be fully operational and by which measures will be available for analysis.

District/School Name: Today’s Date:
Strategy:
Measures of
Action Steps Resources Owner Deadline
Implementation (Outputs)
What steps must be taken to What specific supports are How will readers of the plan know Who is most closely responsible and By when will the step
implement our strategy? needed to implement this action the action step or strategy is fully accountable for taking each action step? be completed?
step? realized or carried out?

ACTI ON PLAN CHECKLIST 5 . 3 . 2 T
Purpose To craft an action plan to support the implementation of strategies identified in the previous Related Documents
module, Knowledge. 5–Action Module
5.3.1T: Action Plan Template
Description This checklist can guide a District Data Team in designing or refining an action plan to guide
implementation of a strategy.
Time 1–2 hours, depending on the number of action plans to be developed.
Directions:
Before creating the action plan, ask:

If the district intends to leverage an existing strategy to address the problems it identified, does the strategy need to be
refined, adjusted, or improved prior to its implementation?
If the district identified a new strategy to address the problems it identified, does the new strategy require multiple
steps to be implemented?
If the response to either of the above questions is “NO,” then the strategy does not require an action plan. The District Data
Team should focus its time and energy on developing action plans for the other strategies it identified, or proceed to Module 6:
Results to develop an evaluation plan.
If the response to either of the above questions is “YES,” use the template below to craft a plan that will guide implementation of
the activities.
Note: If the Team has crafted a logic model (5.1.1T and 5.1.2T), it should have that available for reference, as strategies, resources,
and many of the implementation measures will have already been identified.
5.3.2T: Action Plan Checklist—Version 1.0 1/2

Strategy Y/N
If this is an existing strategy, is it described in a way that indicates how it will achieve different results than in the past?
Does the description of the strategy communicate to internal and external stakeholders how it will address the issues identified?
Action Steps/Activities Y/N
Are just the key, far-reaching action steps that need to be monitored as evidence of progress toward the goal identified?
Are action steps listed in order of completion?
Is it implicit which action steps need to be completed before others?
Are action steps sufficiently described so that others can understand them and carry them out?
Resources Y/N
If necessary, are any special supports (technology, materials, funding, infrastructure, people, polices, political support, etc.) required
to implement the strategy indicated?
Measures of Implementation (Outputs) Y/N
Are outputs described in terms of adult actions and behavior?
Do outputs document technical steps for which the actions are clear and the results easily quantified?
Are outputs specific in terms of what will take place, with/for whom, to what extent, and by when?
Owner Y/N
Is each action step ascribed to an individual person (not a group, title, or team)?
Does each person assigned an action step have the resources, capacity, authority, and support needed to complete the step?
Deadline Y/N
Does the deadline indicate by when the step will be completed?
5.3.2T: Action Plan Checklist—Version 1.0 2/2

MODULE 6: RESULTS
D i s t r i c t D a t a T e a m T o o l k i t
Table of Contents
Introduction—1
Planning for Evaluation—2

Why Develop an Evaluation Plan?—2
Deciding What to Evaluate—3
Monitoring Progress—4
Building an Evaluation Plan—6
Conducting an Evaluation—8
Analyzing the Evidence—8
Communicating Results—10
Evaluation as Continuous Improvement—11
Module Summary—12
Results
6.1.1T: Evaluation Plan Template 6.2.2T: Building Data Walls

6.2.1T: Communication Organizer 6.2.3T: Evaluation Report Template
District Data Team Toolkit—Version 1.0 Module 6: Results
INTR ODUCTI ON
WHERE ARE WE NOW?

progression through the model—you are now in Module 6: Results.
Getting
Ready
Once a District Data Team, or any other district team, has begun
implementing a logic model or improvement plan, it will want to monitor its
progress toward the desired goal. The Results module can help a Team
build capacity to evaluate the desired outcomes, monitor its progress on a
given strategy (or strategies), and communicate those results to various
stakeholders.
MODULE OBJECTIVES
The Results module will help a District Data Team:
 Decide what to evaluate

 Develop an evaluation plan
 Analyze evaluation data
 Identify and develop a communication strategy
 Continue the process of inquiry

PLA NNI NG FOR EVA LUATI ON
WHY DEVELOP AN EVALUATION PLAN?

Taking action means nothing if the District Data Team does not take time
to reflect on the impact of the work. From the outset, a district should
have a plan to analyze and evaluate the evidence generated by
implementation of the designated strategies.
The evaluation plan helps the Team answer the questions:
To what extent have new skills, knowledge, and expertise been

acquired by the targeted adults and/or students?
To what extent are these new skills, knowledge, and expertise
being put to use effectively by the targeted adults and/or
students?
To what extent have the adults and/or students shifted their habits
or beliefs in a way that impacts learning?
Since what gets measured gets done, it is wise not to wait until the end of
implementation to draft an evaluation plan. Instead, the Team should
begin thinking very early in the implementation process—if not before
implementation even begins—about the most important evidence to
measure.
These measures of implementation and change will be used to indicate if

the strategy has had the desired impact, and therefore they should be
articulated clearly and meaningfully. Module 5: Action provides guidance
on crafting useful measures. In short, the measures should be bold yet
achievable, and help a Team answer the questions:
What did the district attempt to change and why?

What did the district do to try to make the change?
What results were achieved?
What effect(s) did the action steps have on the habits, beliefs, and
ways of working of those involved?
What will happen next?

An evaluation plan can guide the overall evaluation process, where the
Team reflects on, and reports publicly, the extent to which new skills,
knowledge, and expertise have been acquired by the targeted adults
and/or students, and the extent to which they are having an impact on
student achievement and organizational culture.
It outlines the specific measures that will be monitored, the evidence to be

collected, and the date(s) for analysis, and can be a useful tool for
keeping a team organized and focused on the desired change the district
is trying to bring about.
DECIDING WHAT TO EVALUATE

In crafting its measures of implementation and change, the District Data
Team is essentially deciding what will be evaluated. As discussed in
Module 5: Action, since the measures that are articulated will be used to
monitor progress, it is important to articulate only those measures whose
evidence will be collected and analyzed. The Team should not identify a
measure if the evidence it would generate does not relate directly to an
evaluation question.
In deciding what to evaluate, the Team might consider these questions:
 What are the questions internal and external stakeholders

want to have answered?
What questions are stakeholders (including the Team
itself) asking about the work of the district? What
improvements are they seeking?
What responses will stakeholders want in relation to the
original focusing questions and data that drove the inquiry
process?
 How will the information be used, and by whom? Consider the
ways the data might be used both internally and externally.
Will the data be used to inform organizational learning and
continued improvement of practice?
Will the data be used for compliance with external entities?
A combination of these?
By considering these questions, the Team can narrow the focus and
scope of its evaluation to those measures that best document and
communicate progress toward outcomes and impact.

If the Team is unsure how the data will be used or by whom, they may not
be worth collecting. If the Team has identified a substantial amount of
If the District Data data to collect, it should seek guidance from district leadership in setting
Team has crafted a priorities appropriately. For example, while it is wise to consider
logic model, then responding to the questions stakeholders ask, the Team might not want
to get distracted by answering all of their questions, but rather focus on
much of its work in
those that will most further the district’s vision, mission, and strategic
deciding what to plan.
evaluate is already
If the District Data Team has crafted a logic model (5.1.1T and 5.1.2T),
done, as the logic
then much of its work in deciding what to evaluate is already done, as the
model articulates the logic model articulates the essential outcomes, or measures of change, in
essential outcomes, adult and student practice that the district will look for as it implements its
or measures of strategy or strategies.
change.
If the Team does not have a logic model to work from, it will need to think
strategically about what to evaluate, when, and why. The Team should
consult with district and school leadership in order to select the areas that
will provide the most useful information for the Team, district leadership,
and other stakeholders.
MONITORING PROGRESS
As soon as the first action step is underway, the Team (or some other
entity) can begin monitoring the progress of the district’s work. It will likely
monitor the implementation of the actions related to the district’s
strategies. Likewise, the Team may also begin monitoring those
strategies for efficacy and impact. However, the two forms of monitoring
should not be confused:
When we monitor implementation, we analyze measures that help

answer the question, did the work get done?
When we monitor change, we analyze measures that help answer

the question, is our work having the intended effects?
Monitoring implementation (or outputs) helps the Team note the

progress of the action steps adults are taking to implement the strategy. It
helps answer the question, what took place for whom, when, and to what
extent? If the Team has decided to use an action plan to implement one
or more strategies, then it can use the dates in the “deadline” column to
monitor implementation. The Team can note whether or not steps were
fully implemented by the desired date and modify implementation as
needed to meet the targets.

Monitoring change (or outcomes), by contrast, is when the district

evaluates whether its initiatives are having the intended effects on student
and adult performance. It helps the Team answer the question what
changed, for whom, by how much, and when? Or, more succinctly, we’ve
taken action—so how did we do?
This work of monitoring both implementation and change can begin once
the strategies are in motion, and may in fact coincide with one another,
making it all the more important to distinguish between the two.
Likewise, in the midst of the process, a district should be mindful not to

equate output data (such as the percent of teachers trained) with
outcome data (such as how this training has changed instructional
practice and improved student performance).
The following chart summarizes key differences between the two

processes.
Monitoring Implementation… Monitoring Change…

Based on data generated by the Based on data generated by the
measures of implementation in measures of change in the logic
the logic model model
Did the work get done? Is our work having the intended
effects?
What took place, for whom, to What changed, for whom, by how
what extent and when? much, and when?
Focused on the short term Focused on the intermediate and
long term
Internal stakeholders are the External stakeholders are the
primary audience, e.g., the Team primary audience, e.g., other
itself and those directly involved district personnel, as well as the
with the action plan wider community

BUILDING AN EVALUATION PLAN

While a logic model outlines a theory of action, and an action plan may
provide more detail on the steps taken to implement the strategy, an
evaluation plan guides the process of evaluating results. It organizes the
information that will form the basis for the evaluation process and the
discussions about whether or not the desired changes took place.
If the Team developed a logic model in Module 5: Action, then the

relationship of the evaluation plan to the model is straightforward.

(Optional) (Optional) (Essential) (Essential)
Goal:
Data to be Source or Date for Person

Measure
Collected Location Analysis Responsible
1. To build an evaluation plan, first identify the measures which would

demonstrate progress toward the goal. The most important measures
to focus on are those related to the acquisition of skills and
knowledge or shifts in habits and values by the adults and students
involved in the initiative—in other words, the measures of change or
outcomes. However, depending on the situation, it may also be
useful to analyze evidence related to the measures of
implementation, and potentially even the allocation of resources.
(Note the essential and optional labels in the diagram above). This is
discussed further in the next section, Conducting the Evaluation.
If the Team has completed a logic model, it will already have
articulated these measures for both adults and students, for both the
short- and long-term. For help on developing good measures of
change, see the section on Crafting Meaningful Measures in Module
5: Action.

2. Specify the data to be collected—the actual evidence the district will

collect to document these changes. If the Team has completed a
data inventory (1.5.1T), it may be a useful reminder of the range of
evidence the district currently collects.
3. Give the source or location where this evidence can be found when
the Team is ready to analyze it. The data inventory can be useful for
this section as well.
4. Specify the date for analysis by which the Team will engage
thoughtfully with the information provided by each. If the measures
are written well, these dates will already be implied.
5. Indicate the person responsible for ensuring that the evidence for a
given measure will be available for analysis by the Team at the
designated time.
Activity 6.1 provides a template for an evaluation plan. Again, if the Team
has crafted a logic model, it already has the majority of elements needed.
If the Team has not yet done this, the evaluation plan template will guide
it to think about the necessary information.
Activity 6.1 Crafting an Evaluation Plan
This template can guide a district in articulating a plan for monitoring

implementation and outcomes of its improvement efforts.
(6.1.1T: Evaluation Plan Template)

CONDUCTI NG AN EVA LUATI ON
ANALYZING THE EVIDENCE

If the measures are specified concretely enough, then the monitoring
dates will be implied.
Once the evaluation plan is in place, the team is ready to begin collecting
and analyzing relevant data.1 Because a strategic plan or logic model is
based on a causal chain of events, each of which is dependent on the
next, the focus areas may need to be examined chronologically: one
cannot look for impact without outcomes; one cannot look for outcomes
without outputs; one cannot look for outputs without inputs.
What is most important to evaluate are the outcomes and impact—the

measureable changes in the ways adults and students approach their
What is most respective work related to teaching and learning, and the relationship of
important to evaluate those changes to the district’s vision for improvement. In many ways the
evaluation process mirrors the process in Module 3: Information,
are the outcomes and
reflecting the cyclical nature of the inquiry process.
impact…the Team
will want to evaluate As mentioned earlier, the District Data Team will want to begin evaluating
the measures of change soon after the work to implement the strategy
the outcomes of a
has begun, and at the same time it monitors the implementation of the
strategy once it has steps themselves for early indicators of the impact of implementation.
been fully
For example, if the strategy involves training teachers in a specific
implemented. technique, early monitoring of implementation might communicate the
percent of teachers trained, while early monitoring of change might
investigate the impact of the training on the practice of those teachers,
and might note differences between the practice of teachers who have
received the training and those who have not.
Naturally, the Team will want to evaluate the outcomes of a strategy once
it has been fully implemented. If it has engaged in analyzing outcome
data along the way, this final evaluation will not take as much additional
effort as it would if a district has not been evaluating the early evidence of
actual change in practice.
As mentioned earlier, the Team will want to begin by evaluating the

measures of change it has identified, first for students, and then for the
1
The Data Analysis Protocol (3.1.1T) in Module 3 can help the Team shape its evaluation of outcome
data.

adults. If this outcome data does not demonstrate any movement toward
the desired goal, the Team should first consider:
Is the best evidence possible being used to demonstrate the

desired changes? What other evidence might be available to show
whether the strategy is having the desired impact?
Is more time or a greater degree of implementation needed before
the action steps will begin to yield the desired changes?
If the Team feels it has identified the best measures and has allowed
enough time for change to occur, and those changes are not evident, then
it will need to review the elements put in place to achieve those results.
Do modifications need to be made to the action steps in order to

have greater impact?
Are adequate and appropriate resources being used in service of
this strategy and related action steps?
Did the Team implement the best strategies possible for the given
problem and its context?
Did the Team diagnose the problem correctly in the first place?
It is important to consider these questions in order, and one at a time. If
the Team gets the information it needs by considering the first question, it
does not need to spend time on the remaining questions.
In essence these questions mirror the elements of the logic model
outlined in Module 5: Action. They guide the Team to reflect on each
component of the logic model, starting at the end and working backwards
to the Team’s definition of the problem itself.

Statements Implementation Change
For example, if a District Data Team has evaluated all aspects of its logic
model and has deemed that, in fact, it did not identify the real problem, it
may want to visit or revisit the guidance in earlier modules to see if it was
asking the right questions in the first place. The District Data Team Self-
Assessment (0.1.1T) can help a Team understand its strengths and
weaknesses in regard to inquiry, action, and results, and can direct it to
tools and resources in other modules in this Toolkit that could be useful at
this point of the process.

COMMUNI CATING RE SULTS
Interim evaluation data should be communicated to stakeholders to

demonstrate initial successes, as well as any early lessons learned and
adjustments made. This is especially important given that the goal is to
shift ways that adults and students in the district think about and
approach their work. Indications of progress can serve as an incentive for
the individuals involved to continue the work. The results of the evaluation
plan can also contribute to the knowledge base in the district about what
works, what did not work, and why, given a particular district, school, or
classroom context.2
The results of any Team’s efforts are important to a number of audiences
for a variety of reasons. For example, if a district did an exceptional job of
raising the percentage of English language learners scoring Proficient and
Advanced on the MCAS by ten percent or more for several years, many
people would need to know for many different reasons. To name a few:
The data will justify to external stakeholders that the district’s logic
model was a sound approach to improving the performance of
adults and students in the organization.
The District Data Team and other internal stakeholders would
want to identify the key actions and processes contributing to
results in order to embed successful practices across the district,
and to inform the formation of new initiatives and inquiries.
The schools’ faculty will build on this record of success to
collaboratively approach new challenges.
Activity 6.2 Communicating Results

Use the Communication Organizer to think about the specific
audiences that need to know about your results and the messages
each audience needs to receive.
Use the report template for guidance on creating a written summary of
the work, if appropriate for the context.
Use Building Data Walls to consolidate in one display a range of information
that summarizes the district’s focus on inquiry and data-driven action.
(6.2.1T: Communication Organizer)
(6.2.2T: Building Data Walls)
(6.2.3T: Evaluation Report Template)
2
Module 1: Getting Ready provides guidance for managing the change effort and addressing some of
the concerns that stakeholders may raise during the process.

EVALUATION AS
CONTI NUOUS IMPR OVEMENT
Engaging in inquiry is an ongoing process, and ultimately an embedded

way of approaching one’s work. Purposefully taking time to reflect on the
steps taken and not taken and the intermediate and longer-term results
will determine whether the identified strategies are transforming the
culture of the district and improving student achievement. If at any time a
district finds it is off target, it will want to reflect back on earlier steps in
the process and adjust as necessary.
For example, if the District Data Team did not see the desired results, it
may need to modify its evaluation plan. It may find that it does not have
enough quality data, or that data are not presented in a way that
highlights key trends or outliers. The Team may want to reflect on
whether it identified the best action steps and resources for the strategy,
of if the strategies and action steps it decided on in the first place were
the most appropriate. Continuing to reflect back on the process, the Team
may even find that the initial focusing question used to frame the inquiry
process might not have been the best one, and it may want to re-engage
in inquiry with a different question in mind.
By intentionally engaging in ongoing collaborative reflection, the Team

can build its capacity to ask good questions, use data to plan effective
action, and ultimately see greater results in the work of teaching and
learning in the district.

MODULE SUMMARY
Evaluation is an essential, yet often overlooked aspect of the

improvement process. Rigorous and timely evaluation holds people
accountable for results, builds the confidence of internal and external
stakeholders in supporting the district’s theory of improvement, and
contributes to improved outcomes for students.
This module guides a District Data Team in the creation and

implementation of an evaluation plan, simultaneously linking the Team
with resources in all of the previous modules in the Toolkit. The module
re-introduces the District Data Team Self-Assessment as a way to reflect
on the Team’s capacity for inquiry and data use, and a means to identify
the best parts of the Toolkit with which to engage.
Regardless of the success of its strategy, logic model, or implementation

plan, a District Data Team is likely to emerge from this stage in the
process with new questions to investigate through a process of inquiry.
These questions may be based on successes that the Team wishes to
disseminate, or on failures to hit the mark that require further attention to
resolve. Either way, they can lead the Team to deepen its investigation
into its own practice and that of the stakeholders in the district, further
embedding a culture of data-driven inquiry, action, and learning that is
ultimately the business of education.
References
W.K. Kellogg Foundation (2004). Logic Model Development Guide. Battle
Creek, MI: Author

EVALUATION PLAN TEMPLATE 6 . 1 . 1 T
Purpose The evaluation plan provides structure and guidance for the Team as it considers the impact of a Related Documents
certain strategy (or strategies) on the identified problem. 6–Results Module
Description The District Data Team completes the evaluation plan before implementation of a strategy begins,
or very soon after, in order to articulate the data it will collect, when, and from where, in order to
evaluate if the strategy and action steps are having the desired effect on student and adult
outcomes.
Time 1–2 hours to create. Ongoing time to gather and analyze data.
Notes:
If the Team has crafted a logic model (5.1.1T and 5.1.2T), it should have that available for reference, as the strategy and
related measures of change (outcomes) will have already been identified.
If the Team has completed a data inventory (1.5.1T), it can reference that for information on available data, their location, and
the ease with which the Team can access the data.
Directions:
1. Begin by articulating the goal or desired impact that the Team is trying to achieve.
2. Respond to the guiding questions in each column, proceeding from left to right.
Begin in the first column by naming the specific measures that will serve as indicators that the strategy is working. It is
essential that this includes any measures of change (outcomes), but in some cases this may also include measures of
implementation (outputs) and even resources.
Then for each measure indicate the specific evidence that will be collected, its source or location, the date for analysis
by the Team, and the person responsible.
Each measure of change should have corresponding information in each of the other columns.
3. Use the plan to ensure the Team has the right data at the right time and is prepared to analyze it for results.
See the text in Module 6 for additional guidance on completing the evaluation plan.
See Module 3: Information for guidance on analyzing data.
6.1.1T: Evaluation Plan Template—Version 1.0 1/2

Goal (Desired Impact):
Specific Date for

Measure Evidence Source or Location Analysis by Person Responsible
to be Collected Team
6.1.1T: Evaluation Plan Template—Version 1.0 2/2

COMMUNI CATION OR GA NIZER 6 . 2 . 1 T
Purpose Communication planning. Related Documents

6–Results Module
Description Team members identify the specific findings that they need to share from the evaluation, as 6.2.2T: Building Data Walls
well as the audiences with whom they need to communicate. 6.2.3T: Evaluation Report
Time 30 minutes–1 hour. Template
What needs to be Why?

How? By whom?
communicated? To whom? (What do we (How do we hope By When?
(Communication (Person
(Finding or (Audience) want them to they use this (Deadline)
Tools/Venues) Responsible)
Message) know?) information?)
 School Board  Annual report
 District Leadership  Quarterly report
Team  District newsletter
 District Data Team
 Data wall displays
 School
Improvement Team  Website
 School Faculty  Email to relevant
 Parents audience
 Students  Presentation
 Other  Other
_____________ ____________
 District Data Team  Data wall displays
 School
 Website
Improvement Team
 Other  Other
_____________ ____________
6.2.1T: Communication Plan Template 1/2

What needs to be Why?
How? By whom?
communicated? To whom? (What do we (How do we hope By When?
(Communication (Person
(Finding or (Audience) want them to they use this (Deadline)
Tools/Venues) Responsible)
Message) know?) information?)
 School
 Website
Improvement Team
 Other  Other
_____________ ____________
 School
 Website
Improvement Team
 Other  Other
_____________ ____________
 School
 Website
Improvement Team
 Other  Other
_____________ ____________
6.2.1T: Communication Plan Template 2/2

BUILDI NG DA TA WA LLS 6 . 2 . 2 T
Purpose To guide the creation of meaningful data walls. Related Documents

6–Results Module
Description These resources guide a Team through the creation 6.2.1T: Communication
of two types of data walls. Organizer
Time About 45 minutes. 6.2.3T: Evaluation Report
Template
Overview
One effective strategy for communicating results is the use of data walls. A data wall is a visual
representation of data related to a specific question or problem. It is comprised primarily of
numbers, charts, and diagrams, using only enough text to annotate the data and articulate the
inferences and conclusions that the Team has agreed on. A data wall may also capture any
questions that have emerged as a result of the inquiry process. Ultimately, a data wall should be
dynamic, interactive, and evolve over time as new data are added and new conclusions drawn.
An interactive data wall contains data that will be updated and manipulated often over a period
of time, making the data wall a ― living‖ display. For example, a district might create a wall to
track the percent of students across the district that are meeting district-wide or statewide
standards. As new assessment data comes out, the Team could engage stakeholders in
updating the wall and noticing shifts in outcomes. Such a data wall might be used with
stakeholders most closely involved with implementing an action plan and monitoring its progress.
Consolidating in one display a range of information that summarizes the district‘s focus for
inquiry and data-driven action can serve several purposes, including:
Providing a visual reminder of the district‘s strategies and the corresponding

implementation, which could lead to continued motivation to stay focused on the challenge
Visually demonstrating how different, and potentially seemingly disparate, data sources
informed the district‘s understanding of the problem
Disseminating a common message to different stakeholders
Engaging stakeholders in conversations about district priorities and goals for long-term
impact
Demonstrating how engaging with data can be an integral part of continually improving a
district‘s work, and not merely a compliance exercise done for the benefit of others
Data walls can be portable (such as on a tri-fold science fair display board or rolling bulletin
board) or may be a permanent installation on a wall. It is increasingly common to find data walls
in the lobbies and offices of schools, often displaying data related to student assessment
scores. The District Data Team might consider the value of displaying data walls in central office
spaces as well, including areas devoted to functions such as finance, operations, and human
resources. The Team may also consider having the same display in all buildings district-wide.
6.2.2T: Building Data Walls—Version 1.0 1/3

Guidelines for Creating and Using an Interactive Data Wall
An interactive data wall contains data that will be updated and manipulated often over a period
of time, making the data wall a ―
living‖ display. For example, a district might create a data wall to
track the percent of students across the district who are meeting district-wide or statewide
standards in the area of literacy. This may be related to strategies and interventions that the
district has implemented to reach a district goal of students meeting literacy benchmarks by a
certain time or grade level.
1. Find a data set that is closely related to the focus of inquiry. Display the data in a space
where they will be easily accessible and easily viewed. After initial data is posted, and
once action is taken and similar data are later collected, the data on the interactive data
wall can be manipulated. A district‘s effort around progress monitoring students at the
end of each marking period in third grade relative to reading achievement is one
example of a dynamic data set that could be updated frequently. The periodic monitoring
of student achievement would inform the district as to whether or not literacy strategies
employed were resulting in an increase in student reading ability and if the district were
meeting its goal related to the number of students proficient in that grade.
2. Identify the individual unit that will be monitored. In schools, this is the student. However,
a district-wide data wall might use the classroom, grade level, or school as the unit to
monitor. Use small magnets, post-its, or another material to create a marker for each
unit, e.g., each classroom, grade level, or school.
a. It may be best to label each unit in a way that maintains privacy, while also
allowing the identification of specific units, e.g., specific classrooms.
b. The Team might consider using additional coding of the markers to add a third or
fourth dimension of data. For example, green markers could represent classrooms
with teachers in their 1st or 2nd year, yellow could represent those in their 3rd–6th
year, and orange could represent those with 7 or more years of experience.

3. Create a grid (data wall) with a data set on each axis and sort all ‗units‘ accordingly, for
example:
Any School District

Grade Three Reading Proficiency by Classroom*
Second Marking Period
0–25% 26–50% 51–75% 76–100%
School
A
School
B
School
C
School
D
*Depending on district size, the unit of analysis may be by classroom or by grade level.
In this grid, an audience can view sets of classroom level data across a district in relation to how
many students were scoring at benchmark in each class. Subsequent administrations of a
district-wide literacy assessment could then be administered and classrooms redistributed in
relation to the number of students achieving benchmark. The movements of data points on the
data wall provide stakeholders an opportunity to connect with data and see a picture of both
aggregate and disaggregate results due to the nature of the display.
4. As data are updated, the Team can track movement of individual units from one
administration to the next.
Note:
a. Prior to manipulating or updating data, document the existing data wall in some form,
such as with a picture, so the Team can reflect back on changes over time.
b. Ideally, when engaging with the data, those closest to the data should be the ones to
manipulate and update the data wall.

E V A L U A T I O N R E P O R T T E M P L A T E 6 . 2 . 3 T
Purpose To report on the results of action taken by the Related Documents

District Data Team. 6–Results Module
6.2.1T: Communication
Description The District Data Team can use this outline to report
Organizer
on its evaluation of the impact of the strategies in
6.2.2T: Building Data Walls
furthering the district’s goals.
Time 4–6 hours
This is an outline of the major components of an evaluation report. The associated activities or
resources in the Toolkit are indicated in parentheses. If the Team has completed any of the
activities listed, it will want to have those available for reference as much of the information in
the report derives directly from them.
I. Overview: A summary that describes the problem being addressed by the action plan.
A. Original focusing question (and potentially also the related clarifying questions)
(See 2.1.1T)
B. Summary of initial findings from the original data displays and data overview
(See 2.5.1T and 2.5.2T)
C. Description of the suspected cause of the problem (See 4.1.1T)
D. Goal or desired impact (See 4.1.1T and 5.1.1T)
II. Implementation Description: What the district did to address the problem.
A. Brief narrative (1–2 paragraphs) identifying the strategy and major steps taken to
address the problem (See 4.1.1T, 5.1.1T, and 5.3.1T)
B. Description of key resources that contributed to the effort. (See 5.1.1T and
5.3.1T)
III. Evaluation Results: What the effect was of implementing the strategy.
A. Data displays depicting the measures of implementation (outputs) and the
measures of change (outcomes), highlighting the acquisition of skills, knowledge,
and expertise, as well as shifts in habits and beliefs related to teaching and
learning. (See 2.4.1T through 2.4.4R)
B. Short narratives to describe findings from analysis of this data (See 3.3.1T)
IV. Recommendations and Next Steps: How the district will apply what it learned.
A. Identification of new focusing questions (See 2.1.1T and 3.3.1T)
B. Identification of immediate next steps to re-enter the Data-Driven Inquiry and
Action Cycle
6.2.3T: Evaluation Report Template—Version 1.0 1/5

Section I: Overview
Original
Focusing
Question
Summary of
Initial Findings
Suspected Cause
of the Problem
Goal (or Desired

Impact)

Section II: Implementation Description
Description of
Strategies and
Major Actions
Taken
Description of
Key Resources

Section III: Results
Use this section to summarize your results with data displays and written descriptions of your findings.
Attach pages as necessary.

Section IV: Recommendations and Next Steps
New Focusing
Questions
Such as: New team formulation, creation of new data displays and data overviews,
audiences for communication…
Next Steps

District Data Toolkit

Uploaded by

Copyright:

Available Formats

District Data Toolkit

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

District Data Toolkit

Uploaded by

Copyright:

Available Formats

District Data Team Toolkit

Helping districts establish, grow, and

MODULE 0: INTRODUCTION TO THE TOOLKIT

District Data Team Toolkit

What is the Toolkit?—1

How Can the Toolkit Help Our District?—2

What’s in the Toolkit?—3

How Should the Toolkit Be Used?—3

How Do I Navigate the Toolkit?—4

Where Should Our District Begin?—4

Tools and Resources for

0.1.1R: Objectives for All Modules

WHAT IS THE TOOLKIT?

Craft questions about accountability, equity, and continuous improvement

Massachusetts Department of Elementary and Secondary Education 1

HOW CAN THE TOOLKIT HELP OUR DISTRICT?

Organize a District Data Team to facilitate productive use of data

2 Massachusetts Department of Elementary and Secondary Education

WHAT’S IN THE TOOLKIT?

1. Getting Ready module provides guidance on forming a District Data Team

2. Inquiry module launches a process of inquiry by identifying high-level

3. Information module guides the process of analyzing the data identified in

4. Knowledge module helps place information (analyzed quality data) into

5. Action module provides frameworks for putting new knowledge to work by

HOW SHOULD THE TOOLKIT BE USED?

Massachusetts Department of Elementary and Secondary Education 3

HOW DO I NAVIGATE THE TOOLKIT?

Resources provide further reading, documentation, or guidance to help

Activity 0.1 Overview of Toolkit Contents

(0.1.1R: Objectives for All Modules)

(0.1.2R: Tools and Resources for All Modules)

WHERE SHOULD OUR DISTRICT BEGIN?

Activity 0.2 District Data Team Self-Assessment

4 Massachusetts Department of Elementary and Secondary Education

including a number of personnel from:

Public Consulting Group (PCG);

MA ESE Offices of Urban District Assistance, School Improvement Grants

Management, Information Technology, Data Collection, and the Center for

Curriculum and Instruction; and

Brockton, Chelsea, Fitchburg, Holyoke, and Lynn Public School Districts.

Massachusetts Department of Elementary and Secondary Education 5

Purpose To provide an overview of the objectives for all Related Documents

Module 1 (Getting Ready) will help a district:

0.1.1R: Objectives for All Modules—Version 1.0 1/1

Purpose To provide an overview of the tools and resources Related Documents

Module 1: Getting Ready (Organize for Success)

Module 2: Inquiry (Ask the Right Questions)

Module 3: Information (Collect and Analyze Data)

0.1.2R: Tools and Resources for All Modules—Version 1.0 1/2

4.2.1T: Why, Why, Why?

4.2.3T: Fishbone Analysis

4.2.4T: Graphic Representation

Module 5: Action (Apply Meaning)

5.1.2T: Logic Model Checklist

5.2.1T: Crafting Meaningful Measures Checklist

Module 6: Results (Look for Improvement)

6.2.1T: Communication Organizer

0.1.2R: Tools and Resources for All Modules—Version 1.0

MA ESE District Data Team Toolkit