METHODS AND PROCESSES FOR

CONDUCTING DATA QUALITY

ASSESSMENTS

DECEMBER 2014
This publication was produced for review by the United States Agency for International
Development. The forms and the content of this report were created by Patricia Vondal,
and Management Systems International with the exception of the USAID DQA Checklist.

Methods and Processes for Conducting DQA’s

METHODS AND PROCESSES FOR
CONDUCTING DATA QUALITY
ASSESSMENTS (DQA)

Contracted under AID-440-C-14-00003

Vietnam Evaluation, Monitoring, and Survey Services

DISCLAIMER
The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency
for International Development or the United States Government.

Methods and Processes for Conducting DQA’s

CONTENTS
Introduction 1

1. Data Quality Assessment (DQA) Checklist 3

2. DQA Process 7

3. DQA Questions for Meetings with Prime Implementing Partners (IPs) 15

4. DQA Questions for Site Visits with IP Partners 18

5. Data Verification Worksheet 20

6. Summary of Data Quality Issues per Indicator 21

Annex A. Lessons Learned 22

Annex B. Training Slides 25

Annex C. Instructions for Small Group Exercise during Workshop 39

Annex D. Template for the DQA Report 41

Methods and Processes for Conducting DQA’s

INTRODUCTION
This report provides guidance on the methods and processes for conducting Data Quality Assessments
(DQA) based USG Regulations and USAID policy (ADS 203.) The materials and methods have been
used by other USAID Missions and were recently updated and improved based on the consideration of
experience from DQA exercises with USAID Missions in Kenya, Ethiopia and Pakistan.

The report provides a comprehensive package of everything needed to conduct a USAID DQAs exercise,
including a presentation of USAID’s five data quality standards, and how the application of each of those
standards can affect the quality of data collected for performance indicators used for management
decision-making and learning. The report also contains guidance on the step-by-step process of
conducting a DQA and the forms that support the assessment.

1. DATA QUALITY ASSESSMENT (DQA) CHECKLIST

This Checklist is recommended by USAID for use in conducting an assessment of indicators selected for
the assessment. MSI made some modifications to help clarify certain aspects of the form and to support a
final judgment of the quality of the data for that indicator. It walks DQA Team members through an
assessment based on the five quality standards, and provides space for documenting actions required to
strengthen the quality of the data.

2. DQA PROCESS
This section of the report provides clear guidance on recommended methods and process to complete the
DQA. It is divided by three phases of the process: preparation, collecting information, and completing
the DQA and the DQA report. Each phase provides specific steps that can be followed and includes the
forms that must be filled out during each stage of the DQA.

3. DQA QUESTIONS FOR MEETINGS WITH PRIME

IMPLEMENTING PARTNERS (IPS)
This is a questionnaire that can be used during the initial DQA meeting with the Prime IP that collects and
aggregates data related to the indicators selected for the assessment. This questionnaire is designed to
capture information on the IP’s overall performance monitoring system including data flow,
documentation, data base management, and procedures used to maintain data integrity. The questions
provide important information to examine standards related to validity, reliability, integrity and
timeliness.

4. DQA QUESTIONS FOR SITE VISITS WITH IP PARTNERS

This is a questionnaire that is used for each of the IP’s partners that report and transmit indicator data to
the prime IP. Responses to the questions help verify the information provided by the prime IP and also
help pinpoint gaps and problems in the overall data collection/data management system that can adversely
affect quality.

Methods and Processes for Conducting DQA’s 1

5. DATA VERIFICATION WORKSHEET
This worksheet was designed to provide an efficient process for verifying data from a subset of indicators.
Some of the data for this worksheet will be obtained during the meeting with the prime IP, and the
remainder of the data is obtained during the meeting with IP partners. It allows one to isolate
inconsistencies in data reported by the IP partner to the prime, and from the prime to USAID.

6. SUMMARY OF DATA QUALITY ISSUES PER INDICATOR

This form provides an overall summary of both data quality issues and action items to strengthen data
quality for each of the five data quality standards. It provides a section for the outcome of the data
verification process for those indicators selected for verification. Summaries are meant to be brief. The
completed DQA Checklists and Data Verification Worksheets provide the information required for this
summary. The form can be used in the DQA report. Completed checklists and worksheets are for back-
up information that can also be used for auditors.

ANNEX A. LESSONS LEARNED

This Annex compiles lessons learned from training USAID staff and from DQA processes undertaken in
Kenya, Ethiopia and Afghanistan. These lessons were also used to inform the methods and data
collection forms included in this report.

ANNEX B. TRAINING SLIDES

These slides are for use in a workshop setting. They cover the purpose and requirements of a DQA, the
five data quality standards with examples, and guidance for the DQA process. They are designed to
prepare USAID technical staff to conduct the DQA. Most of the slides contain notes at the bottom for the
trainer’s use.

ANNEX C. INSTRUCTIONS FOR SMALL GROUP EXERCISE

DURING WORKSHOP
Meant for the workshop trainer, this exercise is structured to engage workshop participants in beginning
the assessment of indicators associated with their DO portfolio. The exercise concludes with a participant
discussion on issues related to the DQA process.

ANNEX D. TEMPLATE FOR THE DQA REPORT

This template provides a suggested format and organization for compiling a report based on the
completion of the DQA process. There is current report template specifically recommended for use by
USAID.

Methods and Processes for Conducting DQA’s 2

1. DATA QUALITY ASSESSMENT (DQA) CHECKLIST
This Data Quality Assessment (DQA) Checklist is provided as a recommended tool that an operating unit
(OU) may use to complete its DQAs. If the OU has used a different tool for conducting DQAs in the past,
they are free to continue the use of that tool instead. The checklist below is intended to assist in assessing
each of the five aspects of data quality and provide a convenient manner in which to document findings.

USAID Mission or Operating Unit Name:

Title of Performance Indicator:

[Indicator should be copied directly from the Performance Indicator Reference Sheet]
Linkage to Foreign Assistance Standardized Program Structure, if applicable (i.e. Program Area, Element, etc.):

Result This Indicator Measures [For USAID only] (i.e., Specify the Development Objective, Intermediate
Result, or Project Purpose, etc.):

Data Source(s):
[Information can be copied directly from the Performance Indicator Reference Sheet]
Partner or Contractor Who Provided the Data:
[It is recommended that this checklist is completed for each partner that contributes data to an indicator– it should state
in the contract or grant that it is the prime’s responsibility to ensure the data quality of sub-contractors or sub grantees.]

Period for Which the Data Are Being Reported:

Is This Indicator a Standard or Custom Indicator? Standard Foreign Assistance Indicator

Custom (created by the OU; not standard)
Data Quality Assessment methodology:
[Describe here or attach to this checklist the methods and procedures for assessing the quality of the indicator data. E.g.
Reviewing data collection procedures and documentation, interviewing those responsible for data analysis, checking a sample of
the data for errors, etc.]

Date(s) of Assessment:
Assessment Team Members:

USAID Mission/OU Verification of DQA

Team Leader Officer approval

Methods and Processes for Conducting DQA’s 3

 Y N COMMENTS
ES intended
VALIDITY – Data should clearly and adequately represent the O result.

1 Does the information collected measure what it is

supposed to measure? (E.g. A valid measure of overall
nutrition is healthy variation in diet; Age is not a valid
measure of overall health.)

2 Do results collected fall within a plausible

range?

3 Is there reasonable assurance that the data collection

methods being used do not produce systematically
biased data (e.g. consistently over- or under-
counting)?

4 Are sound research methods being used to

collect the data?
RELIABILITY – Data should reflect stable and consistent data collection processes and analysis
methods over time.

1 When the same data collection method is used to

measure/observe the same thing multiple times, is the
same result produced each time? (E.g. A ruler used
over and over always indicates the same length for an
inch.)

2 Are data collection and analysis methods documented

in writing and being used to ensure the same
procedures are followed each time?

TIMELINESS – Data should be available at a useful frequency, should be current, and should be
timely enough to influence management decision making.

1 Are data available frequently enough to inform

program management decisions?
2 Are the data reported the most current
practically available?

3 Are the data reported as soon as possible after

collection?
PRECISION – Data have a sufficient level of detail to permit management decision making; e.g. the
margin of error is less than the anticipated change.
1 Is the margin of error less than the expected change
being measured? (E.g. If a change of only 2% is
expected and the margin of error in a survey used to
collect the data is +/- 5%, then the tool is not precise
enough to detect the change.)

2 Has the margin of error been reported along with

the data? (Only applicable to results obtained
through statistical samples.)

Methods and Processes for Conducting DQA’s 4

 3 Is the data collection method/tool being used to
collect the data fine-tuned or exact enough to
register the expected change? (E.g. A yardstick
may not be a precise enough tool to measure a
change of a few millimeters.)

INTEGRITY – Data collected should have safeguards to minimize the risk of transcription error or data
manipulation.

1 Are procedures or safeguards in place to minimize

data transcription errors?

2 Is there independence in key data collection,

management, and assessment procedures?

3 Are mechanisms in place to prevent unauthorized

changes to the data

SUMMARY
Based on the assessment relative to the five standards, what is the overall conclusion regarding the quality of the
data?

Significance of limitations (if any):

Actions needed to address limitations prior to the next DQA (given level of USG control over data):

IF NO DATA ARE AVAILABLE FOR THE COMMENTS

INDICATOR
If no recent relevant data are available for this
indicator, why not?
What concrete actions are now being taken to collect and
report these data as soon as possible?

When will data be reported?

Methods and Processes for Conducting DQA’s 5

RECOMMENDATIONS FOR CONDUCTING DATA QUALITY
ASSESSMENTS
1. Data Quality (DQ) assessor should make sure that they understand the precise definition of the
indicator by checking the Performance Indicator Reference Sheet. Please address any issues of
ambiguity before the DQA is conducted.

2. DQ assessor should have a copy of the methodology for data collection in hand before assessing the
indicator. For USAID Missions, this information should be in the PMP’s Performance Indicator
Reference Sheets for each indicator. Each indicator should have a written description of how the
data being assessed are supposed to be collected.

3. Each implementing partner should have a copy of the method of data collection in their files and
documented evidence that they are collecting the data according to the methodology.

4. DQ assessor should record the names and titles of all individuals involved in the assessment.

5. Does the implementing partner have documented evidence that they have verified the data that
has been reported? Partners should be able to provided USAID with documents (process/person
conducting the verification/field visit dates/persons met/activities visited, etc.) which
demonstrates that they have verified the data that was reported. Note: Verification by the
partners should be an ongoing process.

6. The DQ assessor should be able to review the implementing partner files/records against the
methodology for data collection laid out in the PMP (for USAID Missions only). Any data quality
concerns should be documented.

7. The DQ should include a summary of significant limitations found. A plan of action,

including timelines and responsibilities, for addressing the limitations should be made.

Methods and Processes for Conducting DQA’s 6

2. DQA PROCESS
This section provides the steps suggested in preparing and conducting a high quality DQA for indicators
selected by the USAID Mission. It is based on recent DQA experience from USAID Missions in Kenya,
Pakistan and Ethiopia.

The section is divided into the following sections:

I. Preparation for the DQA

1. USAID DQA Team Composition

2. Key Materials Required for the DQA

3. Step 1: Indicator Review Session for the DQA Team

4. Step 2: Logistics – Arranging Meetings with the Prime IP and with IP Partners

II. Conducting the Assessment

5. Step 3: Meeting with the Prime IP and Phase I of Data Verification

6. Step 4: Site visits with Key IP Partners and Phase II of Data Verification

III. Summarizing the Findings

7. Step 5: Completing the Data Quality Assessment

Methods and Processes for Conducting DQA’s 7

 I. PREPARATION
1. USAID DQA TEAM COMPOSITION
A minimum of two USAID staff should attend the meetings with Implementing Partners (IPs) so that
one person can ask the questions and the other can record responses. Ideally the DQA team should
include the COR or AOR, the activity manager or any other lead technical person that works most
closely with the IPs that contribute data for the indicators selected for assessment. If the DO Team
includes an M&E Specialist, that person should also be represented on the DQA Team. If not, you
may want to include a manager from the Program Office. Ultimately, the size and composition of
the DQA team is up to USAID and must be based on the size of the Mission staff.

Roles: The senior manager should lead the DQA effort and one member of the team should be
selected to record responses during meetings with IPs and during site visits to their partners. Other
DQA team members, such as a DO Team M&E Specialist, can ask specialized questions about the
IP’s M&E system.

2. KEY MATERIALS TO HAVE IN ADVANCE OF THE DQA PROCESS

 DQA Checklist
 PIRS for each indicator selected for DQA
 The last quarterly report submitted by IPs that contribute data for those indicators
 DQA Questions for Meetings with IPs form
 DQA Questions for Site Visits with IP Partners form
 Data Verification Worksheet

If possible, arrange for the Team’s designated note taker a laptop or iPad to bring to meetings with
IPs and site visits. This process is more efficient if the information and responses to questions are
typed directly onto soft copies of the worksheets and forms during the meetings.

3. STEP ONE: HOLD AN INDICATOR REVIEW SESSION WITH USAID

MANAGERS SELECTED TO PARTICIPATE IN THE DQA

Purpose: The session will refresh the memory of the managers who attended the DQA workshop,
help them form a DQA Team, and guide them through completing the first concrete step of the data
quality assessment for the indicators selected. The review will ensure that the newly formed DQA
teams know the strengths and weaknesses of the indicator data they receive from their IPs, and will
highlight missing information and issues for discussion during the meetings with their Prime IP and
Partners.

Process: To begin the session, a facilitator/trainer should provide the participants with a brief review
of USAID’s five data quality criteria, the forms and documents to be used, and the process for
conducting the DQA. These topics will have been covered in the DQA workshop, but a brief refresher
will assist the participants to begin the job.

During the Indicator Review Session, DQA teams should complete a DQA Checklist for each
indicator selected for the data quality assessment, filling in as much information as possible. The
Team’s designated note taker should, ideally, fill in the information directly onto the soft copy of the
worksheet so that the Checklist can be pre-populated prior to meetings with the Prime IPs and IP
partners.

Methods and Processes for Conducting DQA’s 8

 The worksheet provides an assessment check off list (Yes/No) for all variables listed under each of
the five data quality criteria accompanied by any comments the reviewer think important. The
comment column can also be used to highlight missing information. Tools useful for conducting this
preliminary assessment include:

 The PIRS for the indicator, especially if it has been updated by the IP, will help your review.
This is an important tool to use as one of the PIRS sections covers data quality issues related to
the indicator, and what the IP intends to do to address any issues identified.

 The M&E section of the IP’s latest quarterly report reporting the last quarter’s value for the
indicators selected for the DQA process. The M&E section may also provide additional
information that can be used for your assessment.

Again, to maximize efficiency of the process, the DQA team should try to complete as much of the
assessment as possible prior to IP meetings for each of the indicators using one Checklist per
indicator.

If more than one prime IP reports data for a given indicator selected for the DQA, a separate
Worksheet must be filled out for each IP that contributes data for that indicator.
At the end of the session, the DQA team members select a set of the indicators they have assessed
that will undergo a data verification process. Criteria for selecting a subset of indicators for data
verification might include:

 Importance of the indicator’s data for program management and decision-making.

 Multiple Prime IPs contribute data for the indicator and it is not certain that each IP is
collecting/analyzing the data the same way.
 One prime IP contributes data for the indicator, but the IP relies on many partners for actually
collecting and transmitting data to them for consolidation. This is another scenario whereby
USAID is not sure that all of those partners are not collecting and analyzing the data the same
way before sending it to the prime IP.

Begin filling out the Data Verification Worksheet for each Prime Implementing Partner using the
aggregated data for the quarter that was submitted to you in the IP’s last quarterly report.

Missing information

There will be some questions on the worksheet that cannot be assessed based on the team’s review.
Missing information should be obtained through meetings with the prime implementing partner
responsible for providing data on that indicator. Develop specific questions for these meetings based
on the missing information that you can use during those meetings.

4. STEP TWO – LOGISTICS: ARRANGING MEETINGS WITH IPS AND SUB-

CONTRACTORS
Arranging Meetings with the Prime IP

Methods and Processes for Conducting DQA’s 9

 Arrange a two hour meeting at the IP’s office. The length of the meeting depends on the number of
indicators selected for the DQA that each IP reports on. There could be three or four indicators to
review during the meeting.

Inform the IP about the purpose of the meeting, and that for the purposes of the DQA, the COP, M&E
Manager, Database Manager, and lead technical members of the USAID project/activity should be
present. Persons representing each of these roles will each have something to contribute regarding
data quality issues.

Inform the IP that you have specific questions that you will ask during the meeting. You do not want
them to give you a presentation. It is your decision as to whether or not you electronically send a
copy of the worksheets and forms that the Team will use during the meeting.

If more than one IP contributes data to the indicator selected for a DQA, arrange a separate meeting
with each of those IPs.

Arranging Site Visits with IP Partners

Site visits with the IP’s key partners are important because they generate/collect, document, aggregate
and transmit indicator data to the prime IP.

Ask the IP COP to arrange site visits with each of their partners on behalf of your DQA team. Site
visits generally last no more than one hour. It is often important for someone from the IP to introduce
the DQA team members to the partner. But be very clear with the IP that they cannot participate in
site visits with the DQA team after providing introductions.

Provide guidance to the IP on the types of personnel that should be present for the site visits so that
they can inform their partner/s. Site visits to the offices of partners in government may take more
time to arrange.

 The manager of the office, clinic, facility or organization (NGO, government office, agricultural
coop, private business, health clinic, etc.)
 Individual/s responsible for collecting and/or recording and documenting the data to send to the
Prime IP.
 Who should attend the site visit meeting will likely vary depending on the project, the indicator,
and the type of organization or facility you visit.

II. CONDUCTING THE ASSESSMENT

5. STEP 3 – HOLD MEETINGS WITH THE PRIME IMPLEMENTING
PARTNER
Purpose: The purpose of this meeting is to:

 Understand the IP’s overall data systems and processes which affect the quality of the data
received by USAID.
 Fill out information gaps on the DQA Checklist from the DQA Team’s initial assessment during
the Indicator Review Session.

Methods and Processes for Conducting DQA’s 10

  Begin the data verification process for those specific indicators selected by the DQA team during
the Indicator Review Session.
 Increase IP understanding of USAID’s DQA criteria and requirements, and subsequently to
strengthen the quality of their overall data system and processes.

Materials:

 The DQA Checklist that you began filling out as part of your assessment prior to the meeting.
Please be sure to bring a DQA Checklist for each indicator to be assessed during the meeting.
 The PIRS for the indicator that you used to fill out the Worksheet
 DQA Questions for Meetings with Implementing Partners form
 Data Verification Worksheet
 Copies of the above forms for each DQA Team Member and for the Prime IP COP

Meeting Process:

1. Begin the meeting with introductions and then present the purpose of the meeting. It is important
that IP staff attending the meeting understand the purpose and importance of conducting DQAs as per
USAID regulations. They should know that undergoing a DQA will help them to strengthen the
quality and integrity of the data they collect and report to USAID.

2. After the introductions and explanation of meeting purpose have been concluded, begin asking
questions using the form titled DQA Questions for Meetings with Implementing Partners. There
is space to document IP responses, your notes, and to record any additional issues. Ask them to
provide copies of PIRS and other documentation as highlighted next to certain questions on this form.

Make sure the responding partner answers only one question at a time. There is a tendency for IPs to
want to make a presentation but this will not get the DQA team the specific information it needs.

Responses to questions from this form will provide the DQA team with an overview of the partner’s
entire M&E system as a whole including M&E staff members and their roles, data flow, DQAs they
conduct to assess the quality of data submitted by other partners, their own data quality and integrity
procedures they use, and so on. All of these issues have a bearing on the quality of the data you
receive in IP quarterly reports. As a result of asking these questions, you may be able to highlight
certain problems affecting the quality of the data that should be corrected or modified by the IP.

3. After completing the coverage of questions listed on this form, the USAID DQA Team Leader
can then turn to issues on the DQA Checklist per indicator for specific information that the team
could not address on their own before coming to the partner meeting. Focus your questions on those
particular issues for each question. Make sure all your questions are answered.

4. Begin the Data Verification Process (Phase I) for those indicators that were selected by the
DQA team for verification during the Indicator Review Meeting. Ask the IP M&E Manager and/or
Database Manager to bring up the pages from their M&E database. Record the number (value) of
these selected indicators for the last quarter on the Data Verification Worksheet. If there are
multiple partners reporting data to the IP for the indicator, the number should represent the
aggregation of numbers reported from all those partners.
Methods and Processes for Conducting DQA’s 11
 5. Document and compare the value of the indicator reported in the IP’s latest quarterly report with
the value for the indicator in the IP’s database. If there is a discrepancy between the two numbers (or
however the value is recorded, e.g., %, Yes/No, etc.), record this information on the Data
Verification Worksheet.

6. If the prime IP receives indicator data from multiple partners, have the IP data manager bring up
the monthly data on the same indicators that were transmitted to the IP from those offices,
facilities/organizations that you have arranged to visit. You will need to see and then record the data
from those IP partners for each month in the last quarter reported to USAID on the Data Verification
Worksheet. During the site visits to these partners, you will compare the value in the IP database for
the last quarter with the value recorded and transmitted to the IP for those months.

6. STEP FOUR: SITE VISIT MEETINGS WITH

OFFICES/FACILITIES/ORGANIZATIONS
Site visits are only necessary if IP has several subcontractors, grantees, facility managers, etc. that
collect and submit data to the IP for aggregation. In some cases, those partners are required to
perform computations with the data before submitting it to the primary IP.

Purpose: The purpose of site visit meetings is to gain an understanding of how the indicator data is
collected, recorded and submitted to the primary implementing partner, as well as the partner’s
understanding of what the indicator definition is and whether they are collecting data using methods
conveyed by the primary IP. Data viability and integrity issues will be examined related to storage,
data access and data accuracy.

Materials:

 DQA Questions for Meetings with Implementing Partners form

 DQA Questions for Site Visits with IP Partners form
 Data Verification Worksheet

Meeting Process: Begin with introductions and the purpose of the meeting.

1. Begin asking the questions listed on the form DQA Questions for Site Visits. There is ample
room to record notes, responses, etc. Most of these questions are closely related to the ones asked
during the partner meetings and will serve as a confirmation of the information provided by the Prime
Implementing Partner. Note that for many of these questions you should ask staff at the site location
to share supporting documentation, copies, etc.

2. Ask the Director or Manager of the partner organization to show you all data collection forms and
where the data are stored. At the same time, ask the individual most closely associated with the data
to walk you through how the data are collected, recorded, transmitted and ask the Director or
Manager present to walk you through how the data are collected and recorded

Methods and Processes for Conducting DQA’s 12

 3. For those indicators that are to be verified, ask to be shown the data reported to the prime partner
for each of the three months of the last quarter that they reported. Record these data on the Data
Verification Worksheet.

III.SUMMARIZING THE FINDINGS

7. STEP FIVE: COMPLETING THE DATA QUALITY ASSESSMENT
Now that data collection process is finished, selected members of the DQA Team must process the
data to form judgments on the quality of data for each indicator they assessed. There are several tasks
involved.

Data Verification – Verifying Numbers:

 For each indicator selected for verification, compare the value of the indicator reported in the last
quarterly report submitted by the Prime IP (aggregated from all partners reporting to the prime)
with the value shown to the DQA Team from the IP’s database during the Partner Meeting. Is it
the value the same? Or is there a discrepancy between the two values? Record this on the Data
Verification Worksheet if this has not already been done. If the values recorded from these two
sources differ, resolving this discrepancy is one action item.

 For each of these same indicators, compare the data you obtained during the site visits for each of
the three months from the last quarter with the data provided to you during the Partner Meeting
for these same facilities/organizations. Are they the same values for each of these three months?
Does data from each month in the last quarter per partner total to the same value the partner
recorded? If not, resolving these discrepancies is another action item.

Completing the DQA Checklist

 Use the information recorded from the prime implementing partner meetings and site visits to
complete the DQA Checklist and to judge the quality of the data for each of the indicators
according to the five USAID Data Quality Standards.

 On the top of page 3 of the DQA Checklist, summarize your overall conclusions on data quality
for the indicator. Include in the summary your findings on data verification for those indicators
that went through this additional process.

 On the bottom of page 3, record the action to be taken for each data quality issue you identified
for that particular indicator, the specific actions that should be taken to rectify these issues, the
person responsible in your office for ensuring that the issues are addressed and completed by the
Prime IP, and the timeline by which this action should be completed.

 On page one of the DQA Checklist, provide your overall rating for this indicator: acceptable,
acceptable if corrections are made, or not acceptable.

Methods and Processes for Conducting DQA’s 13

 Summary of Data Quality Issues per Indicator

 Using the form Summary of Data Quality Issues per Indicator, briefly summarize the DQA
findings for each of the five standards and for the actions required to strengthen data quality. If
the indicator was selected for verification, summarize those findings and actions as well.
 The information should be transferred and summarized from the DQA Checklist.
 If more than one IP reports on this indicator, include the name of each partner that contributes
data on this form.
 Using the completed DQA Checklists for the indicator from each of the partners, summarize the
findings and actions required across all the IPs reporting data, noting the partner name associated
with specific issues under any of the five data quality standards. This allows for more rapid
follow-up.

Items for the DQA Report to be provided by the DQA Team for Each Indicator:

 The completed DQA Checklist Worksheet, signed by your team leader

 The completed DQA Questions for Meetings with IPs
 The completed DQA Questions for Site Visits
 The completed Data Verification Worksheet
 Summary of Data Quality Issues per Indicator

The amended PIRS based on the DQA issues based on the completed DQA Checklist

Methods and Processes for Conducting DQA’s 14

3. DQA QUESTIONS FOR MEETINGS WITH PRIME IMPLEMENTING PARTNERS
(IPS)
Note: These questions concern your partner’s overall Data Collection/Data Quality Control systems, procedures and processes. These questions
are not meant to cover each of the selected indicators. Some of the information from this form can be used to complete the DQA Checklist
Worksheet.

Project/Activity Name:

Partner Name: Date: Reviewer:

QUESTIONS Yes/No NOTES Documentation to collect

for questions indicated
or N/A

1. What is your staffing for this activity’s

M&E/data management?
If more than one person, what are their
roles?

3. Do you have an updated PIRS for each Document/s needed

indicator we are discussing today?
4. Do you have documentation for your
staff and for each reporting partner on
each indicator’s definition, data source,
details of data collection, how data should
be disaggregated and how the value
should be calculated
5. Does each reporting partner have Document/s needed
written guidelines from you on: a) what
they are supposed to report on, b)
specific formats to use for reporting, c)
when reports should be transmitted to
you, and d)to whom reports should be

Methods and Processes for Conducting DQA’s 15

submitted?
6. Have reporting partners been trained
to understand this information? Do they
understand the indicator definition and
what they are supposed to document?

7. Are all partners reporting data on the

same indicator/s using the same data
collection and documentation methods
per your guidance?
8. What data quality control procedures Document/s needed
and processes are documented and
followed for the project as a whole?

9. Have reporting partners been trained

on data quality control procedures?
10. Who has responsibility for viewing the
quality of data from sub-reporting levels
(service points, sub-grantees, etc.)?
11. What do you do when data
discrepancies have been uncovered in
reports from partners? Is this
documented?
12. What are your procedures for
correcting errors in the data?
13. What are the data flow procedures Document/s needed
for transmitting data collected per
indicator from reporting partners to your
office?
14. Are there problems getting complete
data sets from your reporting partners?

Methods and Processes for Conducting DQA’s 16

15. Are their problems getting data from
your reporting partners on time?
16. If you have problems with getting
complete data sets and/or data sets
coming in late, what are you doing (or
what have you done) to address this? Is it
documented?
17. Do you have a documented process Document/s needed
for aggregating data contributions across
partners or grantees that collect data for
this indicator?
18. What are your data storage and data Document/s needed
access procedures to protect the integrity
of the data? What procedures do you
follow to reduce errors?
Record any summary comments and issues for feedback to your IP on their overall M&E systems, processes and procedures for follow-
up discussion.

Methods and Processes for Conducting DQA’s 17

4. DQA QUESTIONS FOR SITE VISITS WITH IP PARTNERS
Name of Activity

Partner Name: Date: Reviewer:

Reports Data to (Prime IP name):

QUESTIONS Yes/ Responses/Notes Ask to see

No documentatio
n where
or indicated
N/A

1. Has (name of prime IP) provided you with information on the Ask to See
definition of each indicator you collect data for that you send to
them? Is it understandable?
2. Have they also provided you with details on how to record (or
collect if applicable) the data for each indicator?

3. Is there any written guidance about collecting or recording Ask to see

information on these indicators that you can share with us?
4. Is the written guidance understandable/clear?

5. Can you please show us the tool you use to record the Ask to see
information?
4. Did (insert name of the IP) provide any training on this guidance,
use of the tools and how to document the information for this
(these) indicators?

5. Do you have any problems in filling out all the required

information for this (these) indicators? If yes, why?

Methods and Processes for Conducting DQA’s 18

6. Are you able to provide all the data to (IP) on time as
scheduled? If no, what are issues?
7. ASK ONLY IF TRAINING PROVIDED: Was the training useful
for understanding how to record (collect) the data for this (these)
indicators?

9. If you asked for additional training or assistance, what were the

issues you had?

10. How often do you submit data to (IP name)? Monthly?

11. How do you give the data to (IP name) each month?
a) Enter data into the database linked to IP?
b) Someone from (IP name) comes to pick up the forms you filled
out?
12. ONLY ASK THIS QUESTION IF DATA IS SUBMITTED ON
PAPER FORMS: Before you submit the data to (IP name) each
month (or however often); do you first sum up the information
each month for each indicator? If not, who does that?
13. Has (IP name) provided you with any written guidance or Ask to see
training to reduce errors in recording the data (or for data entry
of information from written form into database if they have one)?
14. Are the data protected so that no one can alter the
information that was recorded unless there is an error?

15. If you or (IP name) finds an error, are there written procedures Copy Need
you follow for correction of the data?

Methods and Processes for Conducting DQA’s 19

5. DATA VERIFICATION WORKSHEET1
NAME of PRIME IMPLEMENTING PARTNER: DATE:
Name of Aggregate Quarterly Discrepancy: Contrasting Monthly Values Month Month Month Aggregate
Indicator Value for Value plus/ minus for the Indicator Received 1 2 3 Value for
Quarter Documented OR Per Partner in Prime IP Quarter
from In IP No Database with from
Last Quarterly Database discrepancy: Monthly Values Partner
Report 0 Documented in Partner Records
Records
1. Number of NGOs assisted by USG
Partner 1:
Monthly Values from Partner in 3 2 2 7
IP database
Values Documented in Partner 3 2 2 7
Records
Discrepancy, plus or minus 0 0 0 0
Partner 2:
Monthly Values from Partner in 3 3 4 10
IP database
Values Documented in Partner 2 3 2 7
Records
Discrepancy, plus or minus +1 0 +2 +3

Other data verification issues to examine may include: 1) data entry error (the source can be at IP or one of their partners, but one needs to look across
documents to check this); and 2) how partners use data collection, calculation and/or recording methods to determine if mistakes exist.

1
Note: this is just one way of verifying data. Using this method one can check for discrepancies between recorded and reported partner data, and for discrepancies between what is in the IP
database and what was reported to USAID in a quarterly report.
Methods and Processes for Conducting DQA’s 20
6. SUMMARY OF DATA QUALITY ISSUES PER INDICATOR
The table is used to summarize data quality issues for a particular indicator. If more than one prime IP (activity)
contributes data to measure the indicator the form asks USAID to list each of these organizations. The identified
issues should be summarized across all the IPs that contributes data. This is important because it highlights any issues
uncovered by the DQA process that indicate differences in data collection methods among primary IPs. In these cases,
IPs should be brought together to address and find common solutions in order to increase the quality of data for the
indicator.

The summary of issues is drawn from the DQA Checklist. You may need to refer back to any other documentation
collected or examined, or to responses from IPs and IP partners from questions asked during those meetings.

Summary of Data Quality Issues Per Indicator

Title of Indicator:
Was Indicator Selected for Verification of Data? Yes ____ No _____
List of projects/activities that contribute data to this indicator:
Summary Rating of Indicator Quality: Acceptable Acceptable with Improvements ___X__ Not
Acceptable ______

USAID Data Summary of Identified Issues Action Items to Strengthen Quality

Quality
Standards
Validity:

Reliability:

Precision:

Timeliness:

Integrity:

Data
Verification
Outcome (if
selected for
verification)
Action Items Completed: Date ___________ Signature of DQA Leader _______________________

Methods and Processes for Conducting DQA’s 21

ANNEX A. LESSONS LEARNED
LESSONS LEARNED FROM DQA TRAINING AND THE PROCESS OF
CONDUCTING AND FINALIZING A DQA
GETTING BUY-IN FROM SENIOR STAFF
Brief senior management on the process of conducting a DQA and the time it takes to go through this multi-stage
process. Senior management should be helped to understand that their staff have to be accorded sufficient time in
their schedule in order to participate in data quality assessments. Stress the value of quality data for program
management and decision-making. It is critical that the quality of performance monitoring data can be trusted.

TRAINING AND PREPARATION OF USAID STAFF

The training is delivered through two mechanisms: 1) A one day workshop on DQA and Conducting DQAs and 2)
Indicator Review Session. Both are equally important and neither should be given short shrift.

Staff who will be participating in data quality assessments MUST attend both the workshop and the indicator review
session. Individuals who do not first attend the DQA Workshop should not be allowed to participate in the Indicator
Review Session. If they are allowed to attend the session without first attending the workshop, experience shows that
these individuals are too disruptive during the review session at the expense of other participants. They will have too
many questions and very little understanding of what they are to do. This holds up the session and is annoying to all
other participants.

The one day workshop includes important preparation for USAID staff to begin data quality assessment of the
indicators they are responsible for. Do not try to make the workshop a ¾ day event. Because there is a tendency for
people to arrive late and to come back from breaks and lunch late despite your best efforts, there will not be enough
time if you end the workshop at 3:00 or 4:00 PM. This will come at the expense of the exercise at the end where
participants actually begin assessing one of the indicators using the indicator PIRS and quarterly reports from the IPs
that report data for the indicator. An alternative is to split the Workshop into two ½ days. Day 1 for the purpose of
DQAs and going over each of the five DQ standards. Day 2 for the actual process of conducting a DQA. During Day
2 each form should be gone through carefully.

The portion of the Workshop focusing on the DQA Process includes mention of steps to finalize the DQA. It must be
impressed on workshop participants that their job does not stop with the collection of data. The DQA Checklist and
Data Verification Worksheet must be completed, a Summary of Data Quality Issues per Indicator must be filled out,
and the PIRS for the indicator must be updated for inclusion into a DQA Report.

The Indicator Review Session provides a deeper understanding of the forms DQA members will fill out and an
opportunity to get clarification and ask questions about the forms and the DQA process they will follow. This should
be a half day to ¾ day event. It is a very important part of staff preparation, and will make the whole process more
efficient and effective.

Have on hand at the beginning of the Review Session the PIRS for each indicator and the last quarterly report from
each IP that submits data for the indicators that will be assessed DQA team members to use. They must fill out the
DQA checklist for the indicator during this session using these sources. It’s a mistake to wait to fill it out during the
meeting with their IP. Have team members prepare questions to ask their IP to obtain information that cannot be
filled out on the Checklist during the Review Session. The team members can split up the indicators selected for the
DQA between them. They should also record the value of the indicator data from the IP’s last quarterly report on the
Data Verification Sheet.

Roles and Responsibilities – Assign roles and responsibilities for each person that will be part of the DQA Team
during the Review Session. This should include a lead person for asking questions and an individual who records
Methods and Processes for Conducting DQA’s 22
responses and other information. The recorder should arrange to download each of the forms on a laptop or iPad so
that time is saved at the end from transferring hand written notes onto forms. This is the time to download the four
forms:

DQA Checklist

Questions for Meetings with IPs

Questions for Site Visits with IP Partners

Data Verification Worksheet

Scheduling Meetings - Before the Review Session ends, each team must decide who will contact the IPs for Meetings
and to let those IPs know who should attend, the two hour length of the meeting. They should also request the IP to
schedule site visits to (pre-selected) IP partners.

Tracking Time - One of the DQA team members should be selected in advance to keep track of the time during the
meeting with the IP. DQA members should be clear that there are several parts of the meeting: the introduction,
asking questions from the questions for meetings with IPs, asking additional questions to fill in information on the
DQA checklist, and lastly, data verification. If the indicator being assessed has been selected for data verification, be
sure to accord sufficient time at the end of the meeting to begin the verification process.

MEETINGS WITH IMPLEMENTING PARTNERS

Teams must make sure that they have hard copies of the four forms in addition to the soft copy on the note taker’s
laptop or iPhone. One set of the forms should be given to the IP for distribution to staff participating in the meeting.
An alternative that promotes greater efficiency is to send these forms electronically to the IP before the meeting is
held. Let them know in advance what indicators will be assessed and that you will also want to see their database (or
records if not a database.) This will help IPs prepare appropriately.

Do not allow an IP spokesperson to give a presentation on what they do to ensure data quality. It is likely they will
not know all the factors that promote data quality or USAID’s five data quality standards. Inform them that you have
specific questions that you want to address, one at a time.

SITE VISITS WITH IP PARTNERS

As a matter of courtesy, this meeting should not take more than one hour. If staff from the prime IP come with DQA
team members to provide introductions, it is very important that they do not stay and participate in this meeting.
Partners will not feel free to respond, or to respond as openly as they would otherwise to the “Questions for Site Visits
with IP Partners” if IP staff members remain for this meeting. DQA team members must ask to actually see the forms
to be walked through the processes partners use to collect and document data that are transmitted to the prime IP.

COMPLETING THE DQA PROCESS

DO Team leaders and other senior staff should be informed or reminded that DQA participants must be accorded time
to complete the assessment following the collection of data at meetings with the Prime IP and IP partners. Their work
at this point will form the basis for a final judgment on the quality of data for those indicators selected, and in turn, the
specification of concrete actions that must be taken to increase the quality of data related to any problems that were
found. Remind DQA team members what they must do to complete the DQA process.

COMMON SOURCES OF DATA QUALITY PROBLEMS

While this is not meant to be an inclusive list, the issues below are among the top sources of data quality problems.
They will affect the ability to verify data.

Methods and Processes for Conducting DQA’s 23

  Poorly defined indicator definition, type of data to be collected, data collection method, and data calculation
method (if required). Leads to confusion and erroneous interpretations of what to collect and how.
 Multiple prime IPs collecting and reporting data on the same indicator may be using different definitions, data
collection and calculation methods. If this is the case, you cannot role up the data for the indicator across the
IPs that report. If not addressed, you will not have valid data for the indicator.
 Inadequate training and training follow-up from prime IPs to their partners who are responsible for collecting
data for the indicator. This leads to poor understanding, confusion, and to mistakes in what data to collect,
the data collection and recording method. If an IP partner experiences a lot of staff turnover, training may
have to be repeated.
 Inadequate documentation of indicator definition, type of data to be collected, data collection and calculation
methods and required documentation provided to the prime IP’s implementing partners.
 Failure of prime IPs to conduct data checks on quality of data submitted by partners on a rotating basis. If
done on a regular basis, problems can be addressed and rectified.
 Failure to document procedures for correcting data when mistakes are identified. These must be clarified to
the prime IP’s partners, and the IP ensure that corrections are made.
 Failure to conduct site visits related to data quality issues. Site visits are conducted related to implementation
issues, technical problems, etc. But they are less often conducted by qualified staff from the prime IP to
assess data quality issues and to provide guidance when needed.

Methods and Processes for Conducting DQA’s 24

ANNEX B. TRAINING SLIDES

USAID’s Data #?
%?
Quality
Standards

& Conducting
a DQA

DQA Workshop – Table of Contents

I. What is a DQA? Why Does USAID Require Them?

II. Selecting Indicators for the DQA

III. DQA Five Quality Standards

IV. Conducting the DQA:

Stage I. Preparation

- Small Group Exercise

Stage II Conducting the Assessment

Stage III. Summarizing Findings

2

I. What is DQA and What is the Purpose?

USAID ADS 203.3.11.2 The purpose of a DQA is to ensure

that the USAID Mission/Office and DO Team are aware of
the:
1. Strengths and weaknesses of the data, as determined
by applying the five data quality standards, and

2) Extent to which the data integrity can be trusted to

influence management decisions

Methods and Processes for Conducting DQA’s 25

 Consider Trade-Offs

Conducting a rigorous DQA for each indicator requires a

significant allocation of staff time and USAID resources

Which indicators scheduled for assessment are most critical

for management decision-making?

Judge those most critically, and take the extra step to verify
the data

Government Performance and Results

Modernization Act (GPRAMA) requires ….
…that a data quality assessment must occur for indicators,
which are reported externally (standard), at some time within
the 3 years before submission.
ADS says…..
USAID Missions/Offices may choose to conduct DQAs more
frequently if needed.

No requirement to conduct DQAs for data that are not

reported to USAID/Washington (from custom indicators)
5

II. Selecting Indicators for a DQA

• standard indicators reported to USAID/Washington must
be assessed for data quality at some time within a 3 year
period

• new indicators within six months of establishing baseline

data

• custom indicators developed specifically for Mission if

technical or other USAID staff have questions about the
quality of that data

Methods and Processes for Conducting DQA’s 26

 II. Five Criteria USAID Applies to
Assessing Quality of Data

Reliability

Integrity

Precision

Timeliness

First: To Assess the Quality of Data for an

Indicator
USAID uses its five data quality standards to examine:

• Quality of the indicator:

indicator: direct measure, clear about what
data should be collected?
• Data Collection Instruments
• Data Collection Methods
• Database Management
• Actual Data Collected

USAID’s Data Quality Standards

 Validity
 Integrity
 Precision
 Reliability
 Timeliness

Methods and Processes for Conducting DQA’s 27

 Validity

The data should clearly and accurately represent

the intended result
Validity of data is compromised if:

1. The indicator is not a direct measure of the

indented result

2. Measurement errors are made

Validity

Directness of Indicator - Which indicator is

most valid?
Result: Expanded citizens’ knowledge of
their rights
1. Number of citizens in target population who
participate in civic education programs
2. % of target population who are able to identify 3 or
more key civil rights in a survey
3. Number of civil rights cases brought to court by
targeted community organizations

Validity

Measurement Errors- Two Types Related

to Surveys

A. Errors as a result of poor data collection

processes

Examples: Inadequate preparation of data collectors,

leading questions contained in the survey instrument,
data collected at the wrong time of the year to be
relevant

12

Methods and Processes for Conducting DQA’s 28

 Validity

Measurement Error

B. Sampling/ Selection (or Representativeness)

Data are representative if they accurately reflect the
population they are intended to describe

Result: Improved Performance of Learners (national)

Indicator: Primary School Completion Rate
Data: Data is derived from 100 schools in the capital.

13

USAID’s Data Quality Standards

 Validity
 Integrity
 Precision
 Reliability
 Timeliness

14

Integrity

Data collected should have

safeguards to minimize the risk of
transcription error or data
manipulation
i l ti

15

Methods and Processes for Conducting DQA’s 29

 USAID’s Data Quality Standards

 Validity
 Integrity
 Precision
 Reliability
 Timeliness

16

Precision

Data should have a sufficient level of

detail to permit management decision-
making

Three examples:
1 Data
1. D t on th b off policy
the number li reforms
f passedd
each year as opposed to data on progress made
toward the reform or the content of the reform

2. Data where the margin of error is less than the

anticipated change

17

3. Is an international index precise enough to

measure the intended result of your program?

• Corruption Perception Index (Transparency International)

• Human Rights Index (UN Human Rights Commission)
• Ease of Doing Business Index (World Bank)
• Government Effectiveness Index (World Bank)

Ask Yourself First: What variables are being measured

inside the index?

18

Methods and Processes for Conducting DQA’s 30

 USAID’s Data Quality Standards

 Validity
 Integrity
 Precision
 Reliability
 Timeliness

19

Reliability

Data should reflect stable and consistent

data collection processes and analysis
methods over time.

20

Reliable Data For Comparing Performance

…..From Quarter to Quarter and Year to Year, Depends on

consistent use of the same:

• Data Collection Instrument or Method

• Data Calculation/Analysis Method
• Data Source……

Each time data are collected

Methods and Processes for Conducting DQA’s 31

 USAID’s Data Quality Standards

 Validity
 Integrity
 Precision
 Reliability
 Timeliness

22

Timeliness

Data for Indicators Used for Performance

Monitoring Should:

• be available at a useful frequency,

• b
be current, and
d
• be timely enough to influence management
decision-making.

23

III. Conducting the DQA

• Stage I: Preparation Work by the DQA Team

• Stage 2: Collect Information and Conduct Data

Verification (for selected indicators)

• Stage 3: Completing the DQA - Summarize

information, provide judgment on assessment results,
document next steps to strengthen data quality

24

Methods and Processes for Conducting DQA’s 32

 Conducting a DQA – STAGE 1: Preparation
1. Attend the DQA Workshop

2. Hold an Indicator Review Meeting for DQA Teams

-Fill out the DQA Checklist for each indicator selected
for assessment.
- Highlight missing information to ask about during
meeting with your IP

3. Arrange meetings with your Prime IP, and ask them to

arrange meetings on your behalf with their partners
that provide data
25

Tools for Conducting DQAs

• The DQA Assessment Checklist
• PIRS for each Indicator
• Last Quarterly Report submitted by the IP
• Data Verification Worksheet
• DQA Questions for Meetings with IPs
• DQA Questions for Site Visits to IP Partners
• Summary Sheet of Data Quality and Verification Issues

26

DQA Checklist

• Helps DQA team members assess the quality of the

indicator and data collected by walking through each one
of the five criteria used by USAID

• Provides a method for highlighting strengths and

weaknesses and recommended next steps for improving
the quality of the data.

• Asks the DQA Team to judge the quality of the data based
on the completed worksheet
27

Methods and Processes for Conducting DQA’s 33

 DQA Checklist, continued.
• The Checklist should be filled out for each IP that
contributes data for a given indicator

• This allows one to pinpoint where data quality errors are

introduced when more than one IP reports on a given
indicator
• Gaps in information required can be filled by use of the
indicator PIRS, meetings with your IP, and with your IPs
partners and subcontractors

28

Small Group Exercise to Begin the DQA

1. Examine the indicator provided to your group
2. Begin filling out the DQA Checklist as much as you
can
3. Examine the indicator’s PIRS and using that
information, try to complete the checklist
4. Are there still gaps in information?
5. If yes, circle these and prepare a set of questions
for use in the meeting with your IP

29

Stage 2 – Conducting the DQA in the field

Tools Required for the Meeting with your IP

– Questions for Meetings with IPs about their data collection
procedures and processes
– DQA Checklist
– List of Questions for the IP to fill in gaps on the DQA Checklist
– Data Quality Verification Worksheet

– IF POSSIBLE, a laptop or iPad for the designated Team note taker

to enter information on worksheets and questionnaires on line

30

Methods and Processes for Conducting DQA’s 34

 Step 1: Meetings with your IPs
Following introductions and purpose of the meeting, begin
asking questions on the form “DQA Questions for Meetings
with Your IPs”

This will provide you with an overview of key elements of

their data collection and data management system and
procedures used for ensuring quality & integrity of the
data

These questions are best answered by the IP’s M&E

Manager and Database Manager (if both positions exist)
31

Step 2: Filling in Gaps in the DQA Checklist

• Begin asking questions you developed to fill in information
gaps in the DQA Checklist before coming to the meetings

• Some of the information required will come from IP

responses to the “DQA Questions for IP Meetings” form

• Some of the information gaps can be filled or validated by

site visit meetings with the IP’s partners

32

Step 3 – Begin first phase of Data Verification

Materials needed:
IP’s last quarterly report submitted
Data Verification Worksheet
1. Examine the data for the indicator for each of the three
months in the last quarter and the value for the quarter in
the IP’s database
2. Compare the value for the quarter with the value for the
indicator documented in the IP’s quarterly report to see if they
match
3. Record these figures on the Data Verification Worksheet
33

Methods and Processes for Conducting DQA’s 35

 Data Verification Continued
• Ask the IP to show you the data reported for the indicator
from each partner for each of the three months

• Use the Data Verification Worksheet to record this

information for the indicator along with the name of each of
the reporting partners

• This information will be used to compare with data in

partner records during site visits

34

Step 4: Site Visit with IP Partners

• Following Introductions and purpose of the

meeting, begin asking questions from the form
“Questions for Site Visits with IP Partners”

• Ask to see tools and record books for collecting

and/or calculating data, and how the data are then
documented and transferred to the prime IP

35

Step 5: Second phase of Data Verification

• Ask the partner to show you the data recorded that they
submitted to the prime IP for each month in the last
quarter that was reported on

• Record the data for each month on the Data Verification

Worksheet from this partner

36

Methods and Processes for Conducting DQA’s 36

 Stage 3: Summarizing Findings (1) Data
Verification
Compare the data that you recorded on the Data Verification
Worksheet from the Prime IP with the data you recorded on
this Worksheet from the IP partners.

Does it match up? Continue the process comparing the data

you documented from each IP partner you visited with the
data from the Prime IP.

Based on this assessment is the value for the quarter

accurate? Is it the same value reported to USAID?

37

(2) Completing the DQA Checklist

1. Using data from:
• Questions for IPs, Questions for IP partners
• Completed
Completed Data Verification Worksheet
• Any documentation examined during meetings

2. Record actions required based on each data quality

issue
3. Make an overall judgment on quality of the data for the
indicator

38

(3) Summary of Data Quality Issues Per Indicator

• For each indicator, briefly summarize the data quality
issues from the DQA Checklist form for each of the five
standards and the action items identified to strengthen
data quality
• If the indicator was selected for data verification,
summarize the result of the verification and action items
required
• If more than one prime IP reports on this indicator,
summarize the issues for each standard across IPs, noting
which IPs are the source of data quality problems

39

Methods and Processes for Conducting DQA’s 37

 LAST STEPS!
1) Amend the indicator PIRS based on the DQA
– A copy should be added
to the Mission PMP

2) Compile the DQA report

– There is no standard
format, but do include:
• List of indicators assessed
• Methodology used
• Summary of Data Quality Issues
• Back up documentation for Annex
• Amended PIRS forms

4
0

Methods and Processes for Conducting DQA’s 38

ANNEX C. INSTRUCTIONS FOR SMALL GROUP
EXERCISE DURING WORKSHOP
In this exercise, workshop participants will actually begin the work they will be doing to assess the
quality of the indicators that have been selected for an assessment. The exercise helps them to better
understand what they are expected to do once they begin the actual assessment process.

Small Group Composition - Participants from the same Development Objective (DO) Office should
partner for the work. The size of each small group will depend on the total number of workshop
participants. The ideal is 2 to 3 individuals.

Instructions - Give each small group one of the indicators selected. The best way to arrange this is to
give participants an indicator to work on that is associated with an activity they manage. They will be
provided with ½ hour to fill out as much information on the DQA Checklist as they can, and to begin
entering the initial data required for the Data Verification Worksheet.

Each group should be provided with:

 One (or maybe two depending on the complexity of the indicator) indicator
 PIRS for the indicator
 The latest quarterly report submitted to USAID from the IP that contributes data for this indicator.
If there are multiple prime IPs that contribute data to the indicator, choose one.
 Data Quality Assessment Checklist (handed out during the workshop earlier on)
 Data Verification Worksheet (handed out during the workshop earlier on)

Process - Let the participants know in advance that they will not be able to provide answers and
comments for each of the USAID data quality standards based on the PIRS and the quarterly report.
Instruct them to circle or highlight those areas on the checklist they cannot answer. The rest of the
information needed to complete the DQA Checklist will come from meetings with the prime IP and
meetings with IP partners. The groups can write up questions to address these gaps in the information if
helpful.

Stress that it is more efficient for them to begin filling out the DQA Checklist before their meeting with
the Prime IP. The actual meeting with the IP can focus on those checklist items that the small group was
not able to answer. Other efficiencies are gained by assigning one note taker who has soft copies of the
DQA Checklist and the Data Verification Worksheet loaded onto a laptop or an iPad.

Report Out: Depending on the number of small groups at the workshop and the time left for the
workshop, select two (or more) small groups to report out on the results of their initial assessment. Each
report out should be no more than 10 minutes.

Report Out Directions for Small Groups:

 Record on a flip chart the name of the indicator and the result it is intended to measure
 Briefly inform the other participants how far they were able to get in filling out the DQA
Checklist and Data Verification Worksheet. What parts were most difficult to fill out and why?
Because of lack of information? Because the PIRS and the last Quarterly Report didn’t provide
useable information? Or because they didn’t themselves understand some of the criteria used to
judge one or more of USAID’s data quality standards

Methods and Processes for Conducting DQA’s 39

  After all presentations are over, pick out several common themes that cut across groups for a
larger group discussion on what the work will entail. Provide more explanation for those criteria
and standards that aren’t well understood.

Inform them that they will complete this process for each indicator they will be verifying during a
scheduled Indicator Review Session. At that time, they will also learn about how to make logistical
arrangements, the process for handling meetings with the prime IP and with the IP’s partners. They will
also be able to ask more detailed questions.

Methods and Processes for Conducting DQA’s 40

ANNEX D. TEMPLATE FOR THE DQA REPORT
CONTENTS

ACRONYMS

INTRODUCTION
Technical offices are required to conduct data quality assessments on all reportable indicators at
least once in three years to ensure that they are aware of the strengths and weaknesses of the
data, as determined by applying the five data quality standards… and are aware of the extent to
which the data integrity can be trusted to influence management decisions” (ADS 203.3.5.2).
The introduction of new indicators must be assessed within six months after baseline data are
collected.

This report includes the results of a DQA process conducted by USAID/Hanoi from X to Y.

List of Indicators Assessed per Development Objective (indicating which one were subject to
data verification)

METHODOLOGY
Include a brief summary of the three stage approach
Stage 1. Preparation
Stage 2. Conducting the DQA
 Meetings with IPs
 Site Visits to IP Partners
 Verifying Data for Selected Indicators
Stage 3. Finalizing the DQA

OVERARCHING FINDINGS
a. Data Quality
b. Partner Data Collection Systems and Processes

RECOMMENDATIONS FOR USAID/HANOI TO STRENGTHEN

DATA QUALITY

SUMMARY OF INDICATOR DATA QUALITY ISSUES PER

INDICATOR
Insert completed summary sheets for each indicator

Methods and Processes for Conducting DQA’s 41

ANNEX A. DATA QUALITY ASSESSMENT CHECKLISTS BY
INDICATOR

ANNEX B. AMENDED PIRS FORMS

ANNEX C: DQA FORMS UTILIZED FOR THIS ASSESSMENT

Include: DQA Checklist, Questions for Meetings with IPs, Questions for Site Visits with IP
Partners, Data Verification Worksheet, Summary of Indicator Data Quality Issues

Methods and Processes for Conducting DQA’s 42