The Immunization Data Quality Self-Assessment (DQS) Tool: WHO/IVB/05.04 Original: English Distr.: General

WHO/IVB/05.
04
ORIGINAL: ENGLISH
DISTR.: GENERAL
The immunization
data quality self-assessment
(DQS) tool
The Department of Immunization, Vaccines and Biologicals
thanks the donors whose unspecified financial support
has made the production of this publication possible.
This publication was produced by the

Vaccine Assessment and Monitoring team
of the Department of Immunization, Vaccines and Biologicals
Ordering code: WHO/IVB/05.04

Printed: March 2005
This publication is available on the Internet at:

www.who.int/vaccines-documents/
Copies may be requested from:

World Health Organization
Department of Immunization, Vaccines and Biologicals
CH-1211 Geneva 27, Switzerland
• Fax: + 41 22 791 4227 • Email: vaccines@who.int •
© World Health Organization 2005
All rights reserved. Publications of the World Health Organization can be obtained from Marketing
and Dissemination, World Health Organization, 20 Avenue Appia, 1211 Geneva 27, Switzerland
(tel: +41 22 791 2476; fax: +41 22 791 4857; email: bookorders@who.int). Requests for permission to
reproduce or translate WHO publications – whether for sale or for noncommercial distribution – should
be addressed to Marketing and Dissemination, at the above address (fax: +41 22 791 4806;
email: permissions@who.int).
The designations employed and the presentation of the material in this publication do not imply the
expression of any opinion whatsoever on the part of the World Health Organization concerning the legal
status of any country, territory, city or area or of its authorities, or concerning the delimitation of its
frontiers or boundaries. Dotted lines on maps represent approximate border lines for which there may
not yet be full agreement.
The mention of specific companies or of certain manufacturers’ products does not imply that they are
endorsed or recommended by the World Health Organization in preference to others of a similar nature
that are not mentioned. Errors and omissions excepted, the names of proprietary products are distin-
guished by initial capital letters.
All reasonable precautions have been taken by WHO to verify the information contained in this publica-
tion. However, the published material is being distributed without warranty of any kind,
either express or implied. The responsibility for the interpretation and use of the material lies with the
reader. In no event shall the World Health Organization be liable for damages arising from its use.
Printed by the WHO Document Production Services, Geneva, Switzerland
ii
Contents
Acknowledgments ............................................................................................................ v
Abbreviations .................................................................................................................. vii
Executive summary ......................................................................................................... ix
A. Introduction .............................................................................................................. 1
B. Immunization data quality self-assessment toolbox ........................................ 2
1. DQS options: overview ................................................................................... 2
2. Data accuracy .................................................................................................... 3
3. Completeness/timeliness of reporting ......................................................... 18
4. Assess the quality of the monitoring system ............................................... 20
5. Assessing the quality of immunization card recording
(health unit level) ............................................................................................ 22
6. Monitoring of wastage ................................................................................... 23
7. Monitoring of immunization safety .............................................................. 25
8. Denominators of immunization coverage ................................................... 26
C. Where to conduct a DQS? ................................................................................... 27
D. Present the DQS findings ..................................................................................... 30
1. Present the DQS results ................................................................................ 30
2. Using Excel to enter and represent the data ............................................... 33
E. Conduct a DQS workshop ................................................................................... 34
Some proposed workshop principles ..................................................................... 35
F. Integrate DQS results into the routine activities ........................................... 37
Annex A: Sample chart for monitoring doses administered
and drop-outs in children less than one year of age ......................... 39
Annex B: Example of a completeness/timeliness reporting table .................... 41
Annex C: Standard questions to assess the quality of the
monitoring system ................................................................................... 43
Annex D: Child immunization card exercise (example for 20 infants) ........... 57
Annex E: Sampling of health units ......................................................................... 59
Annex F: Data quality self-assessment workshop schedule .............................. 62
iii
iv
Acknowledgements
Special thanks is given to Abdallah Bchir, Craig Burgess, Jan Grevendonk,

François-Xavier Hanon, Stephen Hadler and Ezzedine Mohsni for assisting with
the technical content of the publication.
The data quality self-assessment (DQS) has been developed subsequently to the
immunization data quality audit procedure (WHO/V&B/03.19), which was designed
for use for the Global Alliance for Vaccines and Immunization (GAVI). The DQS
has been tested in a number of countries (Nepal, Morocco and Togo) in which local
support and feedback was extremely useful and appreciated. Respective WHO
regional and country offices and ministries of health (immunization divisions) of
these countries are deeply acknowledged. Financial support from GAVI has
contributed to the design and testing of the DQS.
v
vi
Abbreviations
AD auto-disable (syringe)
AEFI adverse events following immunization
BCG bacille Calmette-Guérin (existing TB vaccine)
DTP diphtheria–tetanus–pertussis vaccine
DQS data quality self-assessment
HU health unit
MOH ministry of health
NGO nongovernmental organization
NID national immunization day
OPV oral polio vaccine
QI quality index
QQ questions on quality
RED Reaching Every District
SE standard error
TT tetanus toxoid
UNICEF United Nations Children’s Fund
VVM vaccine vial monitor
VPD vaccine-preventable disease
vii
viii
Executive summary
What is the DQS? The DQS is a flexible toolbox of methods to evaluate different
aspects of the immunization monitoring system at district and health unit (HU) levels.
Immunization “monitoring” refers to the regular ongoing measurement of the level
of achievement in vaccination coverage and other immunization system indicators
(e.g. safety, vaccine management). Monitoring is closely linked with reporting because
it involves data collection and processing.
Target audience. This document is to be used primarily by staff who will adapt the
toolbox for a specific area (usually staff at national and regional levels). The adapted
tool should then be used by staff collecting and using immunization data at the national,
provincial or district levels.
Uses of the DQS. The DQS aims to assist countries in diagnosing problems and to
provide orientation to improve district monitoring, as highlighted in the Reaching
Every District (RED) approach.
The DQS aims to determine:

• the accuracy of reported numbers of immunizations, and
• the quality of the immunization monitoring system.
The assessment includes a review of data accuracy at different levels and a

self-designed questionnaire reviewing monitoring quality issues (e.g. availability of
vaccination cards, use of tally sheets, directly-observed recording and reporting
practices). These are then analysed, strengths and weaknesses identified,
conclusions reached and practical recommendations made. These recommendations
aim to improve the use of accurate, timely and complete data for action at all levels.
How is a DQS performed? One approach is to hold an initial national participatory

DQS workshop involving key people from the national and district levels to review
country monitoring practices and design a self-assessment. This workshop is
immediately followed by a practical assessment in a number of districts and
health units to provide a self-diagnosis of the monitoring system of the country.
Other approaches can be developed and self-assessments can be designed and
conducted without this preliminary workshop.
ix
The final goal of the DQS is to integrate into routine practice the tool options that
are most relevant for a country so that constant attention is given to improve
monitoring practices and management of immunization activities.
How to use this document? A number of options for evaluating monitoring processes
are presented in this document. They should be explored, selected and refined
according to specific needs. The DQS does not aim to be standardized across
countries. The same flexibility is required when selecting where to conduct the DQS
in a country.
x
A. Introduction
The data quality self-assessment (DQS) consists of a flexible toolbox, designed for
staff at the national, provincial or district levels to evaluate different aspects of the
immunization monitoring system at district and health unit (HU) level in order to
determine the accuracy of reported numbers of immunizations and the quality of
the immunization monitoring system.
In this manual, monitoring refers to the measurement of the level of achievement in

vaccination coverage and other system indicators (e.g. safety, vaccine management,
etc). Monitoring is linked closely with reporting because it involves data collection
and processing.
The options described in the toolbox (Section B) should be explored, selected and
refined according to specific needs. The tool does not aim to be standardized across
countries. The same flexibility should be applied for the selection of DQS sites,
which is discussed in Section C.
The DQS aims to diagnose problems and provide orientation to improve district
monitoring and use of data for action, as highlighted in the Reaching Every District
(RED) approach.1 Basic knowledge of Excel is helpful when entering and analysing
collected data but the self-assessment can be conducted without computerized support.
To date, two Excel workbooks are available for different components of the toolbox
(Section D).
The approach described here to introduce the DQS concept in one country is through
a national participatory workshop (see Section E) involving key people from the
national and district levels. This workshop is immediately followed by an assessment
in a number of districts and HUs that provides a self-diagnosis on the monitoring
system of the country. Other approaches can be developed and self-assessments can
be conducted à la carte.
The final goal of this assessment tool is to integrate the options that are most relevant for one
country into routine practice (Section F) so that constant attention can be given to improve
monitoring practices and management of immunization activities.
1
Increasing immunization coverage at the health facility level. Geneva, WHO, 2002 (WHO/V&B/
02.27). RED is a global strategy aimed at increasing coverage and decreasing drop-out rates. It is a five-
part strategy: reaching the underserved, providing supportive supervision, increasing use of data for
action, increasing micro-planning capacity at district levels and using local populations in planning
immunization sessions.
WHO/IVB/05.04 1
B. Immunization data quality
self-assessment toolbox
1. DQS options: overview
The DQS toolbox proposes several options to assess different aspects of the
monitoring system at different levels.
Table 1. Description of the main areas a DQS can assess
Option District Health Unit Main measures

Assess reporting accuracy X X Accuracy ratio
Assess recording accuracy X Accuracy ratio
(sample in the community)
Assess completeness/ X X Completeness of district reporting (%)
timeliness of reporting Timeliness of district reporting (%)
District report availability at national level (%)
Completeness of HU reporting (%)
Timeliness of HU reporting (%)
HU report availability at district level (%)
Assess the quality of the X X Quality index (QI) scores
monitoring system
Assess the quality of X Integrated in the QI
immunization card recording
Estimate vaccine wastage X X Unopened vial wastage at district store level
Opened vial wastage at HU level
2 The immunization data quality self-assessment (DQS) tool

2. Data accuracy
2.1. Assess reporting accuracy

The principle is to verify the reported information on coverage data, that is, to compare
the data available from one level (a form, report, chart, etc.) against the same
information that has been collated or reported at a more central level. “More central”
should be understood as higher in the data flow: it could be in the same facility
(e.g. tally sheets against registers in the same HU) or between two different facilities
(e.g. registers at the HU against monthly reports found at the district level).
A description of a typical data flow follows (para 2.1.1).
This exercise is critical because it provides an opportunity to evaluate coverage data

accuracy and correct it. But also, by looking at data and the associated work, it is an
appropriate gateway to stimulate discussion on the use of the tools and the meaning
of the data; it also motivates staff concerned with data entry and use.
2.1.1. Description of the administrative immunization-reporting

system flow
A typical reporting flow of immunization coverage data is shown in Figure 1.
In some countries there may be, in addition to the district level, other intermediate
levels between district and national, such as the province, governorate, region, zone,
or state as well as intermediate levels between HU and district (subdistrict, etc).
Figure 1: Reporting flow of immunization coverage data
Health Unit District National
WHO/UNICEF
HU HU joint reporting
report report form
District National
HU tabulations/ tabulations/ tabulations
monitoring chart monitoring chart
Tally District District

sheets report reports
Child
register
Community
(vaccination card)
WHO/IVB/05.04 3
The flow of information begins at the HU level. An HU is defined as the administrative
level where the vaccinations are first recorded; it might include private health facilities,
facilities of nongovernmental organizations (NGOs), hospitals, or a simple health
post. Typically, when a health worker administers a dose of vaccine, the date of
vaccination is immediately recorded on the child’s individual vaccination card and on
the immunization register and the dose is tallied on an appropriate sheet allowing for
the easy re-counting of all doses provided. The individual vaccination card is either
kept in the HU or (preferably) stays with the child’s caretaker (in the community)
while the register and the tally sheets are archived in the HU.
HUs usually report to a district health office on a regular basis (monthly or quarterly).
The HU report includes the number of doses of every antigen given during the
reporting period. To prepare the report, an HU officer obtains the number of doses
administered from the tally sheets. Alternatively he/she uses the child registers to
count the doses administered and put the added figure in the report.
The HUs should keep a copy of all reports sent to the district. The HUs should
display the cumulative number of doses administered in a graph on display to monitor
the progress towards coverage targets (Annex A, the monitoring chart).
At the district office level, administrative personnel receive the reports, log the date
they are received (e.g. on a completeness and timeliness chart – see Annex B), and
follow up on late reports. They then aggregate the information from all the HUs
they oversee and send a periodic district report to the national level (or to the next
intermediate level - if one exists). Tabulations (number of doses reported by each
HU) are made (computerized or not) to allow for the calculation of the district totals.
Copies of the reports sent to the national level are kept in the district office.
At the national level (national headquarters of the national immunization services/

programme), tabulations collating the district report information are made.
Subsequently, the country sends the national data to the international community as
an official report to WHO and UNICEF (available under the immunization coverage
link found in each www.who.int country profile pages).
Important note: In parallel with the upward flow of information, data should be analysed at each
level and fed back to appropriate levels so that the information is used for direct action.
It is important to note that the availability of all the forms is subject to many factors,
including the national policy in use. It is recommended that reports and registers
should be kept for a minimal period of three years after the end of the calendar year
they have been used.

2.1.2. Selecting the information to be assessed
To check that reported immunization coverage data are precise and accurate, a number
of verification processes can be undertaken and virtually all possible sources of
information (those described in Figure 1) could be retrieved and verified, i.e. compared
with another source. Therefore, in order to save time and resources it is important to
determine:
• which level (or levels) need to be checked against other level(s);
• which antigen (may include any antigen: infant, maternal vaccination or any
other supplement e.g. vitamin A) will be verified;
• which documents (form/report) need to be retrieved for each level and where
on the form/report the information should be looked for;
• which time period the verification will cover. This provides a good idea of a
system. A full year is preferable i.e. the whole previous year. However,
local factors will influence this decision, such as a change in the reporting
system, time available, availability of forms, etc.
In addition, an agreement needs to be made in the case of missing information:

should one document not be available, it can be considered either as zero information
(0 dose verified) or as unavailable information (NA). In the former case,
the information to which it is compared is kept. In the latter, the information,
to which it is compared is disregarded. An alternative in the case of missing
information is to check for the same information in another document (e.g. in case of
an HU report missing at the district level, replace it by the HU report available at
the HU level).
Note: The levels selected below include the district and HU levels only, but the
same principles apply should one or several intermediate levels exist.
2.1.3. Verifying coverage data sent by the health unit level

HU coverage data on the number of immunizations provided to the community is
sent to the district on the HU monthly or quarterly reports. The data is potentially
verifiable from the following sources:
• immunizations recorded in an immunization register;
• immunizations tallied on a form;
• monitoring charts describing the progress of the coverage of the HU throughout
the year;
• meeting reports, feedback or feed-forward forms describing achievements.
The assessor will decide which source will be used to verify the information contained
in the HU reports. The HU monthly or quarterly report can be retrieved at the HU
or district level.
Accuracy of the HU sources can also be checked and bring useful information on
the correct use of one or the other tool. For example, the verification of tally sheets
against registers could lead to the finding that a higher number of tallied vaccinations
are due to the poor recording in the registers.
WHO/IVB/05.04 5
Important note: Full understanding of the correct and recommended recording and reporting
procedures is required when selecting the sources that will be verified. Recommendations do
vary from country to country and this influences the interpretation of results.
Example: In Zanzibar, according to the national policy, immunized children who do not belong
in the target area of one health unit should be tallied (on a tally sheet), but not recorded on the
health unit immunization register. Hence the comparison of re-counted immunizations in the
register and in the tally sheet for the same time period might bring out discrepancies attributable
to a correct practice (according to the national policy) and not due to poor recording.
2.1.4. Verifying the coverage data sent by the district level

There are two possibilities to verify the information that is collated by the district
and sent to the more central level: (a) the information coming from HUs collated at
the district level, and (b) the information sent by the district to the more central
level. For the latter, the information reported to the more central level needs to be
available.
a) A monthly or quarterly district report sent to the more central level

(coverage data on the number of immunizations provided in all HUs of the
district) is potentially verifiable from the following sources:
• all HUs (or subdistrict) monthly or quarterly reports (physical copy) that
are sent to the district;
• tabulations (computerized or not) compiling the HU reports (or subdistrict
reports);
• monitoring charts describing the progress of the coverage of the district
throughout the year;
• meeting reports or feedback or feed-forward forms describing the
achievements.
There needs to be a decision on which source will be used to verify the

information contained in the district reports. The district monthly or quarterly
report can be retrieved at the district or national level.
b) The district reports, district summary data or district tabulations may also be
compared against district data available at higher levels. The sources at
national level include the most recent national tabulations or the district reports
(physical copy) found at national level.

2.2. Verifying in the community the accuracy of the recorded information
available in a health unit
The only verifiable recorded information on individual vaccinations is the coverage
information recorded on an immunization register. The principle is to check for
discrepancies between infants or mothers vaccinated according to the register and
those according to the child vaccination card (or mother vaccination card).
The exercise is not only useful in detecting overreporting or underreporting but also
allows examination of the correct recording of immunization cards. It can also assess
the proper use of the immunization register and allow an estimation of valid doses
(i.e. doses given at the right time and with a proper interval).
In situations where the child was indeed vaccinated but the date put on the register
was systematically wrong (for example because the health worker puts the date of
planned vaccination instead of the actual date of vaccination), the exercise can provide
an estimation of timely doses, i.e. given in the recommended time schedule, according
to the information retrieved from the card.
The two following options can be proposed:

a) If the suspected problem is overreporting in the register, a sample of infants
or mothers should be taken from the immunization register in the HU.
Then the assessor can search for the children/mothers in the community to
verify the information recorded (antigen, date of vaccination, etc).
b) If the suspected problem is underreporting in the register, a sample of children
or mothers should be taken from the community. The assessor takes the
available information (antigen and date of vaccination) from the immunization
cards if the childen or mother and verifies it later in the HU register.
Card retention in the community may be a problem and the assessors need to agree
on what to do in case of missing cards. It is recommended that the history of
vaccination by parents’ recall is used if a card is not available.
Similarly, the assessors need to think about which strategy to adopt if a child in the
community cannot be retrieved – option a. Reasons may indeed include overreporting
but also family move, temporary absence, etc. It is recommended to make every
attempt (including contacting neighbours, administrative entity, etc.) to verify whether
children recorded on a register exist.
In option b, the assessors should make sure that the vaccinations that are verified
from immunization cards in the community have been provided by the selected HU(s)
and not by other units so that they can potentially be retrieved in the registers.
Experience has shown that verification at the community level is a time-consuming

exercise and a cheaper alternative can be to take infants coming to the HU. With this
method, a balance is found between the number of children/mothers to be verified
and the logistic and time constraints.
WHO/IVB/05.04 7
Selection of children/mothers in a register (option a)
A minimum of 5–10 children/mothers should be selected per HU. According to
time and logistics, they can be selected from the register:
• from the same locality (to limit transportation costs) if the address is mentioned
in the register;
• by retrieving x of the most recently immunized infants/mothers in the register
(the most recent will be less likely to have moved from the area);
• by choosing randomly within a time period;
• or a combination of the above options.
Selection of children in the community (option b)

A minimum of 5–10 children/mothers should be selected per HU. According to
time and logistics, they can be taken from the same locality (to limit transportation
costs) or from different areas among the population covered by an HU.
Once a village/area has been selected, it is recommended that the strategy developed
in the immunization coverage cluster survey reference manual: Immunization
coverage cluster survey reference manual (in print) is used to randomly retrieve the
defined number of children/mothers. The age of the children to be retrieved should
be in the range of the children recorded in the register. For example, if the HU
registers from the last three years are available, children 0–36 months could be
retrieved in the community. However, it is recommended that children 0–12 months
are assessed (although taking one birth cohort only will take more time than several
birth cohorts) in order to determine the current recording practice.
2.3 The measure (accuracy ratio)

2.3.1 Definition
The main quantitative measure of data accuracy is the ratio between the number of
vaccinations verified or re-counted from a source at one level (numerator), compared
to the number of vaccinations reported by that level to more central levels
(denominator). This ratio gives the proportion of reported numbers that could be
verified. It is expressed as a percentage. The antigen, the source of information and
the time period will need to be defined.

Examples of accuracy ratios:
• Verifying coverage data sent by the HU level:
No. of re-counted DTP3 (0–11 months) in the HU register during given time period x 100
No. of DTP3 (0–11 months) reported in the HU reports found at the district level during same time period
• Verifying the coverage data sent by the district level:
No. of TT2+ reported in all HUs of the district (as in the HU reports found at the district level) in year Z x 100
No. of TT2+ reported by the district in the same time period
• Verifying in the community the recorded information available in an HU:
No. of vitamin A doses recorded on immunization cards of children in the community x 100
No. of vitamin A doses recorded on the registers for the same children in the HU
Each time, the verified information (from the “lower” level in the data flow) is on
the numerator and the reported information (retrieved from the “higher” level in the
data flow) is on the denominator, so that:
• a percentage < 100% shows that not all reported information could be verified;
• a percentage > 100% shows that more information was retrieved than was
reported.
It is theoretically possible to develop several accuracy ratios, basically for each level
and source assessed against another one. The assessment should focus on accuracy
ratios that are most relevant in order to avoid confusion with a high number of
different accuracy ratios.
2.3.2 Interpretation
Possible reasons for low verification: accuracy ratio < 100%

Overreporting
• Intentional
− Often linked with pressure from a higher level
• Non intentional
− Inclusion of vaccination conducted outside target group
− Reporting of doses used instead of immunizations
− No use of standard tools to adequately report the daily number of immunizations
performed
− Transcription or calculation error
Loss of verifiable information
WHO/IVB/05.04 9
Possible reasons for very high verification: accuracy ratio > 100%
Underreporting
• Reports not complete at the time of forwarding
• No use of standard tools to adequately report the daily number of immunizations
performed
• Transcription or calculation error
Loss of information
These lists are not exhaustive.
2.3.3 Aggregating the accuracy ratios

The exercise of extrapolating values (e.g. HU values) to a level (e.g. to the district
level) to obtain a valid estimate for that level is only correct either when all facilities
of this level (all HUs of the district) have been assessed or when the selection of
facilities (e.g. selection of 3 HUs) has been unbiased, i.e. randomly conducted. If this
is not the case, it may be preferable not to aggregate the accuracy ratios and interpret
them according to the local situation. Section C describes the site-selection options.
2.3.3.1 To aggregate the same accuracy ratios (same level)

If the assessment has been conducted in a number of districts and health units, it is
possible to aggregate the same accuracy ratios in order to obtain a national ratio
(when the district or provincial ratios are aggregated), or a district figure (when HU
ratios are aggregated). The principle is to weight each district/HU according to its
importance – in terms of total target population – which can also be estimated by the
number of vaccinations reported during a year.
In the following example, an accuracy assessment has been conducted in two

randomly selected HUs in each of three randomly selected districts of province A.
The vaccinations (one determined antigen) re-counted in the registers of six HUs
from the three districts were compared to the reports sent by the respective HUs for
the same time period. Table 2 also shows the total number of vaccinations that were
reported by each district during the same time period. Table 2 also shows the total
number of vaccinations that were reported by each distribut during the same period.
We would like to get an estimate of the HU registers/HU reports accuracy-ratio for

Province A (see Table 2).

Table 2: Findings of accuracy assessment (HU registers/HU reports)
in six HUs, province A
District HU Re-counted number Reported number HU accuracy Target Weight of

(target of vaccinations in of vaccinations ratio population the HU in the
population) the HU register by the HU of the HU district sample
1 1 315 336 94% 447 58.2%
(5000) 2 280 275 102% 321 41.8%
2 3 125 154 81% 151 43%
(3000) 4 120 139 86% 200 57%
3 5 60 54 111% 81 48.5%
(1000) 6 78 79 99% 86 51.5%
First, we will obtain an accuracy ratio for each district, giving each HU its respective
weight. The weight of each HU corresponds to the proportion of HU population
out of the total sample:
e.g. for District 1, the weight of HU1:
= 447 / (447+321)
= 58.2%
Then the District 1 accuracy ratio is:
(315 x 58.2%) + (280 x 41.8%) = 0.97 or 97%

(336 ) (275 )
Then, to obtain a provincial estimate, the weight of each district will be taken into
account. Each district accuracy ratio should be multiplied by the proportion of the
district in the province in a similar calculation:
(5000 x 97%) + (3000 x 84%) + (1000 x 105%) = 0.94 or 94%

(9000 ) (9000 ) (9000 )
which is the provincial HU registers/HU reports accuracy ratio estimate. One should
not simply take an average of the three accuracy ratios of the three districts to obtain
a provincial accuracy ratio such as: 97 + 84 + 105 = 0.95 or 95%. This is because
3
the weight of each district should be taken into account.
WHO/IVB/05.04 11
2.3.3.2 To aggregate accuracy ratios from different levels
One can also combine the accuracy ratios of two different levels to provide an overall
accuracy figure. The basic principle is to multiply the ratios. Procedures to obtain
an estimate for one level should have already been conducted as described in 2.3.3.1.
In the following example, the two accuracy ratios for the same antigen and time
period:
register HU / reports HU found at district, i.e. regHU
repHU
and
copies of all district reports found at the district/district data found in the
national tabulation, i.e. repDIS
tabDIS
are multiplied to provide one accuracy measure:
(regHU) x (repDIS)
(repHU) (tabDIS)
For example: (94%) x (97%) = 0.91 or 91%
2.3.4. Calculating confidence intervals around an accuracy estimate

• If facilities assessed have been randomly selected, the aggregated measure of
the sample can be inferred to the whole area with confidence intervals around
the estimate. In the case of a 95% confidence interval, one can say that there is
a 95% chance that the interval will include the population parameter.
Confidence intervals are constructed using the standard error (SE),
characterizing the variability of the sample statistic. The basic formula proposed
here for the calculation of upper and lower bounds for 95% confidence intervals
around an accuracy ratio is:
P + 1.96 x SE(p)
Where
SE(p) = P x (1-P)
N
P is the accuracy ratio
N is the total sample size (reported values)

Table 3: Collection sheet for accuracy of health unit record versus reporta
Indicate the antigen: ________________________________ Indicate the year: ________________
WHO/IVB/05.04
Vaccinations – number retrieved: Total Total
(selected months)b
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Register
(tally the re-counted
immunizations)
Register (total) a a’
Month complete
(Yes/No)
Tally sheet b b’
Month complete
(Yes/No)
HU reports
(HU level) c c’
HU tabulation d d’
HU report e e’
(district level)
District
tabulation f f’
a
If information not available put NA. Accuracy ratios:
b
For the column “Total (selected months)” it is optional to circle the months to be considered. a/b: % a’/b’: %
b/e: % b’/e’: %
b/f: % b’/f’: %
13
14
Table 4: Collection sheet for accuracy of district tabulation. Example for three selected months and two antigensa
Region: _________________________________________ District: ________________________
Monthly reports Year X Antigen 1 Total Antigen 2 Total
Health unit name Oct Nov Dec Oct Nov Dec
HU1
HU2
HU3
HU4
HU5
HU6
HU7
HU8
HU9
HU10
HU11
HU12
TOTAL (calculated sum of HUs) a1 a2
Total as sent by the district b1 b2
Date: ______/______/_______
a
Put NA if not available.
Accuracy ratios:
The immunization data quality self-assessment (DQS) tool

a1/b1: %
a2/b2: %
Table 5: Collection sheet for accuracy of district reporta
Indicate the antigen: ________________________________
WHO/IVB/05.04
Indicate the year: __________________________________
Vaccinations – number retrieved Total
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Reports a
(district level)
Tabulation b
(district level)
Reports c
(national level)
Tabulation d
(national level)
a
If report not available put NA
Accuracy ratios:
c/d: %
b/d: %
15
16
Table 6: Data collection form for antigen X: from the register to the community
District: __________________________________________ Name of HU ____________________________ Selected Village: _______________________________________
In the HU In the community
Serial Register Address Name Date Date Card Date Antigen X BCG Verified Remarksa
No. No. of child of birth antigen X possession antigen X vaccination scar
vaccination (card) history
(register)
D M Y D M Y Yes No D M Y Yes No Yes No Y N
1
2
3
4
5
6
7
8
9
10
Name of the interviewers ________________________ Team _____________________________ Date of interview ____________________________________________________
a
The “Remarks” column can be used to record whether the vaccination is timely, whether the dose is valid, etc.

2.4. Designing the assessment forms for data accuracy
Once the team has decided which antigen (including vitamin A) to verify and the
time period that will be verified, ad hoc forms should be designed allowing for easy
data collection on site. The following four tables (Tables 3, 4, 5 and 6) present standard
collection sheets that can be locally adapted.
Table 3 corresponds to paragraph 2.1.3: verifying coverage data sent by the

HU level. As it is designed, the assessor can tally in the row “Register (tally)” the
number of re-counted immunizations and put the re-counted monthly figure in the
column“Register (total)”. Tally sheets can also be verified. The assessor may indicate
whether the re-count is based on complete available information or not (e.g. if one
register or some tally sheets were lost), and may then base the ratio re-counted :
reported on the full amount of months (“Total” column ) or on those selected months
[column “Total (selected months)” ] where they are sure the information was fully
available.
The information can be verified against the HU reports available at the HU,
aggregated data tables (if the HU had to aggregate the information) in HU tabulations,
HU reports available at the district level, or in tabulations (aggregated data tables)
of the district.
A number of accuracy ratio options are presented in Table 3 but it should be decided
which are going to be the most relevant.
Table 4 corresponds to (a) under paragraph 2.1.4, verifying the coverage data sent
by the district. It assesses whether the monthly totals of the HUs (as found in the
HU reports) correspond to the figure aggregated and sent by the district to the
higher level. It can also be used for assessing the availability of HU monthly reports
at district level. It should be adapted for the number of HUs in the district, number
of months for which the HU reports are checked, etc.
Table 5 corresponds to (b) under paragraph 2.1.4, verifying the coverage data sent
by the district. It compares all sources of information for the reported district figure,
either at district or national level. Again, the appropriate accuracy ratio(s) should be
chosen.
Table 6 corresponds to option (a) under paragraph 2.2, verifying recorded

information available in an HU. It aims to compare immunizations recorded in a
register to those recorded on the immunization cards (community level). The example
here aims to retrieve 10 children from an HU register for a specific antigen.
WHO/IVB/05.04 17
3. Completeness/timeliness of reporting
Each district should be monitoring the completeness and timeliness of units reporting,
as a quantitative core measure of the quality of the reporting system.
Completeness of HU reporting is defined as a percentage with the number of reports

received in the numerator and the number of reports expected during a period of
time as a denominator. This definition does not include the quality of reporting, i.e.
whether a given report is complete (all fields filled in). This is addressed in the quality
of the monitoring system.
Timeliness of HU reporting is defined as a percentage with the number of reports

that were received on time (by the deadline set by the EPI office) as the numerator
and the number of reports expected during a period of time as denominator.
During a DQS, a number of approaches can be adopted:

• These two measures are available at a more central level for a given district,
and the assessment comprises verification of the measures provided by the
district;
• These two measures are not available at a more central level for a given district,
and the assessment involves getting the information available at the district.
In both situations, findings should be discussed in terms of causes, actions that have
been taken to correct the problem, and solutions if the problem persists.
3.1 Verification of completeness and timeliness figures

• Reported completeness of HU reporting can be verified by re-counting the
number of HU reports available at the district level for a given period. This is
referred to as an indicator of the availability of reports, defined as the
proportion of reports physically available (retrievable) at the time of the
assessment for a given time period divided by the total number of reports
expected to be available. Note that here, the non-available reports are excluded
from the denominator but this could be discussed.
• Reported timeliness of HU reporting can be verified by looking at the date of
sending/reception written or stamped on the reports. According to the national
policy, this date can be +5 days or 1 week, etc. after the end of the reporting
period. This is defined as the proportion of reports physically available
(retrievable) with a date stamped on time for a given time period divided by
the total number of reports available. Again note that the non-available reports
are excluded from the denominator but this could be discussed. The notion
“on time” should also be defined, depending on whether the assessor wants to
be strict on a given timeline, or whether he wants to allow for some flexibility
(deadline + x days). This verification obviously depends on the local policy to
write the reception and sending dates on the report itself.
If the amount of time needed to verify each report of all HUs for one year is too
high, an alternative is to choose randomly a number of months for which the
information will be collected.

3.2 Obtaining the completeness and timeliness of health unit reporting for a
district
The ideal situation is to obtain from a district a completeness/timeliness table
(see Annex B) which should immediately provide the indicators for a given period of
time.
If this is not available, the procedures described under paragraph 3.1 can also be
done as a proxy for getting HU reporting completeness and timeliness figures.
However, non-available HU reports may indeed not have been obtained from the
lower level or may have been lost or destroyed (physically) by the district.
It is of particular interest for the national level to go into this option of the DQS;
usually, the national level has an idea of the district to national completeness and
timeliness but hardly knows the situation at the lower level (i.e. completeness and
timeliness of HU to district level).
WHO/IVB/05.04 19
4. Assess the quality of the monitoring system
4.1 Overview
The assessment of the quality of the immunization monitoring system is based on
questions or observations or exercises that can be posed or made or assessed at
each visited level (district, sub-district, HU, etc). Each question should have a
“yes”, “no” or “NA” (not applicable) response so that they can be given a score
according to the Yes or No response. A list of proposed questions for each level is
presented in Annex C. These questions/observations/tasks can be grouped into the
different assessed components of the monitoring system. Table 7 proposes a number
of components of the monitoring system based on the usual steps of the collection
and use of data.
Table 7: Proposed components of the monitoring system
District level HU level

Recording Recording
Paper-form practices
Computer practices
Archiving Archiving
Paper-form practices
Computer practices
Reporting Reporting
Demographic information Demographic information
Core output / analyses Core output/ analyses
Evidence of using data for action Evidence of using data for action
The questions should be selected and revised according to each country situation.
The grouping into components is also adaptable: these components can be used as
well as other ones to be defined (e.g. availability of forms etc).
District data of the current and previous year should be analysed to identify and
quantify causes of poor data quality and to find solutions. This will help to refine the
qualitative questions that will be asked during the DQS process.
4.2 The measure – quality index

The quality index (QI) is a quantitative measure of the quality of each component
of the monitoring system. In calculating QI scores, one to three points are given
for each question answered or observation made or task performed correctly.
Scores are calculated for each of the identified components, with the number of
points corresponding to correct answers as the numerator and the number of possible
scores as the denominator. A “no” scores 0, a “yes” scores from 1 to 3 according to
its importance, and an “NA” is not recorded in the denominator. The overall QI is
the proportion generated as the sum of all numerators and all denominators.
For each component and each level of the monitoring system, i.e. at district and
HU, average scores can be obtained and standardized as a percentage or on a scale
from 0 to 10.

The QI is the proportion of: scores for all questions answered “yes”
sum of maximum scores that could be obtained
How to determine the QI
Example of questions on quality (QQ) for HU level reporting component:

Q1: Have all the HU reports of the last year been sent on time to the district?
Q2: Are the HU reports correctly filled out?
Q3: Is the procedure for dealing with late reports known by the HU officer?
Q4: Is HU officer aware of the necessary form to complete if there is a report of a severe adverse
event following immunization (AEFI)?
Q5: Were all the monthly reports from the HU signed by the HU officer for the current year?
Possible score Actual score
Q1: 3 Yes 3
Q2: 1 No 0
Q3 2 No 0
Q4 a
2 NA –
Q5 2 Yes 2
TOTAL 8 5
QI = 5 / 8 = 62.5%
a
If the HU officer was not trained in AEFI and did not receive the ad hoc forms, Q4 is NA and the
possible score for Q4 is removed from the denominator.
The decision about which weight to assign to a question can be determined by asking
each participant in the questionnaire to score the question, then divide the sum of the
scores by the number of people in the team and choose the next round number to
determine the weight (Table 8 below). The weights for each question should be
agreed upon before the assessment.
Table 8: Method of assigning weights for each qualitative question in the DQS
(scores in the table are examples, allowed range in the Excel tool is 1–3)
Score Staff 1 Score Staff 2 Score Staff 3 Average score Weight

a b c (a+b+c) / 3 (rounding the
decimal)
Q1 3 2 2 2.3 2
Q2 1 1 2 1.3 1
Q3 3 3 2 2.6 3
Etc…
Once the QQs have been selected, a form should be printed in hardcopy for the field
assessment.
WHO/IVB/05.04 21
5. Assessing the quality of immunization card recording
(health unit level)
The assessment of the quality of the immunization card recording can be done during
an immunization session: assessors ask the mother/father for filled cards after her/
his child has been immunized and check whether the vaccination(s) were correctly
provided and recorded. This is suggested in countries where the proportion of non-
valid doses has been shown to be high (from coverage survey data).
If it is not possible to attend an immunization session it is also possible to conduct a

“child immunization-card exercise”, to simulate an actual immunization session. The
child immunization card exercise requires advance preparation before arriving at
the HU (Annex D).
To conduct the exercise, ask the vaccinator to complete a health card for a child who
is supposedly brought to the HU on the day of the assessment. Then ask the vaccinator
to determine the next return date. This will assess the vaccinator’s abilities to determine
what vaccines are needed for a child and to correctly complete the vaccination card.
Annex D describes an example of exercise done for 20 children.
The observation and the exercise can be integrated into the quality index score and
one should determine which score to give in case of successful and unsuccessful
answers from the health worker (see QQs in Annex C).

6. Monitoring of wastage
6.1 Overview
Two options can be explored during a DQS at HU or district level:
• The first option is to go through the documents that provide information on
vaccine wastage and determine whether the wastage calculations and
monitoring are understood and done correctly. This can be assessed specifically
or through QQs.
Information about the number of used doses can usually be found in the
following documents:
− the HU/district vaccine ledger, describing all vaccine movements
(shipments/deliveries and despatches), with the balance;
− the HU/district monthly reports, where these contain information on the
number of vials used at the HU or in the district (sum of the HUs).
− stock receipts, invoices, etc.
Questions can include enquiries about the availability of these documents,

whether they have been correctly and completely filled in (e.g. recorded in the
ledger, or reported in the monthly report); whether the wastage rate was
monitored; and whether specific actions were undertaken. Annex C includes a
list of proposed questions.
• The second option is to review the documents, allowing for wastage calculation
for a specific time period, and determine the vaccine wastage for the setting.
This second option allows you to obtain a figure for the HU or the district,
discuss it, and promote monitoring of wastage based on real calculations.
6.2. Definitions
2
Unopened vial vaccine wastage can be calculated at the store level (district).
At district level, the wastage of unopened vials falls mainly into the following
categories:
• vaccines discarded due to vaccine vial monitor (VVM) indication,
• heat exposure,
• vaccines frozen,
• breakage,
• theft,
• vaccines discarded due to expiry dates,
• missing inventory.
2
Monitoring vaccine wastage at country level: Guidelines for programme managers. Geneva, WHO,
2003 (WHO/V&B/03.18).
WHO/IVB/05.04 23
The last item corresponds to “unexplained number of doses not matching an inventory
count” when one is conducted.
For example: on 1 August, according to the vaccine stocks ledger, the DTP balance
is 3000 doses but the physical inventory of the refrigerator contents of that day
records 2940 doses; the balance should therefore be adjusted to 2940 in the book,
with a note that 60 doses are “missing”. The 60 doses fall into the category of
unopened-vial wastage.
The unopened vial wastage is calculated as the proportion of unopened doses wasted
(numerator) out of the number of doses handled by the store (denominator), where:
Number of = Number of doses in + Number of doses

doses handled stock at the beginning received during
of the period the period
The information is usually available in the vaccine ledger of the district.
Additionally, the total vaccine wastage occurring in one district can be calculated
from all figures coming from all HUs vaccinating in the district in addition to the
unopened vial wastage at the district store. This calculation needs information from
all HUs.
At HU level, the global vaccine wastage is calculated, corresponding to the wastage

of opened (administered wastage) and unopened vials. Opened vials are those
opened for vaccination, with a proportion of the doses being administered and the
remaining being wasted.
The global vaccine wastage rate (%) = 100 - vaccine usage rate, where:
vaccine usage number of doses administered
rate (%) = number of doses issued
and:
number of number of number of doses number of
doses issued = doses in stock at + received during - doses in stock
issued the beginning of the period at the end of
the period the period
Interpretation
Whatever the figure found at any level, it is crucial to try to identify and discuss the main causes of
wastage. The importance of monitoring wastage should always be stressed. The level of
immunization coverage should also be taken into account in the interpretation: classically at
higher coverage levels, including more difficult-to-reach children (e.g. through outreach sessions),
the wastage rate is likely to increase, and a higher wastage figure may be more acceptable.

7. Monitoring of immunization safety
Immunization safety should be monitored by every district. Indicators should be

defined and the information properly collected in order to be able to plan and take
corrective actions.
Standard indicators for the monitoring of safety for a district include:

• proportion of HUs with AD syringes out of stock during the month;
• proportion of HUs with ratio: number of immunizations (injectable) < 1;
AD syringe use
• AEFI rate in the district (no. AEFI reported / target population of the district)
given a time period;
• proportion of HUs with the ratio (Syringes used / Safety boxes used) < 100;
• proportion of HUs with incinerators / safe waste disposal;
• proportion of HUs reporting at least one needle-stick injury.
(This list is only indicative: it is not exhaustive and does not consist of a minimal set
of necessary information.)
Options during a DQS may include the following.
1. The verification that indicators have been effectively defined and that they
are well monitored. This can be done through QQs (see examples in Annex C,
QQ 6, 21, 42 and 50 for the district level).
2. The verification of the quantitative data collected which allowed for the
calculated indicator. This can be based on information available at the district
or in selected HUs. A procedure similar to the verification of coverage data
can be undertaken according to the selected indicator.
WHO/IVB/05.04 25
8. Denominators of immunization coverage
The different population groups targeted for routine immunization services are usually:
• infants (i.e. 0–11 months of age) for primary vaccinations, and
• pregnant women for TT vaccination.
Falsely high or low estimates of population numbers can introduce large inaccuracies
in coverage estimates. District and locality denominators are often officially provided
by a more central level, based on national statistics and census projection, but they
may be inaccurate. It is therefore of great importance for the more peripheral levels
(district and HU) to take into consideration local information to estimate and use a
number for their populations that is as precise as possible. This can be done with the
use of birth registries, local household census, etc.
In any case, the denominators should include the entire population living in the
catchment area of the HU or the district, even in the case of moving populations,
populations not registered, contraindications, etc.
Similarly, the denominators should not be determined in order to meet a local

target. They should include the whole population of the area regardless of a target
(i.e. a proportion of the total) set locally.
The DQS is assessing the denominator issue through the QQs in order to explore
the understanding and practice (Annex C, QQs 23–31 for the district level, and
QQs 29–32 for the HU level). These questions can of course be adapted and revised.

C. Where to conduct
a DQS?
Ideally a DQS should be introduced in a country and conducted on an ongoing basis.

In a district though, the number of sites to be visited is constrained by the desired
precision and logistic issues, such as time available, the number of team members,
distances, ease of travel, the ability to obtain security clearance in difficult areas, and
the availability of vehicles, drivers and accommodation.
A choice of sites to be visited must be made. However, the greater the sample size,
the more precise the results will be.
To provide a reasonable idea of the situation in one district, visits are recommended
to at least three HUs per district, with a maximum of six HUs. Visits to more HUs
are not are not recommended because this is not likely to provide additional
information and will wastes resources. Figure 2 shows that the maximal reduction in
the standard error (SE) – hence a higher precision, see para. 2.3.3.1 – is obtained
when the sample size is increased up to six HUs; then, with higher sample sizes, the
decrease of the SE is marginal. The DQS does not provide elements for sample size
calculation as it is felt that the discussions behind all obtained figures is more important
than the figures themselves. Common sense and logistic practicalities should dictate
the number of visited places.
WHO/IVB/05.04 27
Figure 2: Standard error reduction (in %) according to the sample size
(i.e. number of sampled health units)*
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
0 5 10 15 20 25
* Example calculating standard errors for a 1000 sample size per health unit with 50% accuracy ratio.
Alternatively, all districts/HUs can be assessed over a period of time by the higher
level, for instance, as part of supervisory visits.
The selection of the visited districts or HUs can be done according to the following
four options:
1) Representative selection, based on random sampling: This approach is based
on the assumption that the selection of sites should be representative of the
entire system if the recommendations are to be relevant to the whole system.
This option has the advantage of providing estimates which can be applied to
the district. It avoids any temptation to conduct the assessment in areas supposed
to be very strong or weak. Annex D provides guidelines on how to proceed
with random sampling.
2) Representative selection within defined strata: A stratum refers to a
subpopulation of an entity. It may be defined in many ways, e.g. according to
the importance of constitutional units (i.e. number of immunizations provided
by an HU), its type [e.g. hospital/HU or urban/rural], its location.
This option has the advantage of providing estimates which can be extrapolated
to subpopulations. It is useful to differentiate problems and actions which can
be different from one HU type to another.

3) Convenient selection. If a DQS is conducted in districts and HUs where data
quality is suspected to be poor, it examines “the worst case scenario” within
the district/country. This alternative method may be preferred as it has the
greatest potential impact. In this approach, sites can be selected in cooperation
with local staff who are aware of areas with potential problems.
Potential problems in data quality, such as those outlined below, can be identified
from district/HU data to orientate the selection:
− inconsistencies in the reports from the district;
− negative drop-out;
− coverage >100%;
− inconsistent coverage between antigens given during the same visit
(e.g. big differences between OPV3 and DTP3);
− poor completeness (missing reports);
− poorly completed reports, including inaccuracies in sums or calculations ;
− volatile trends (i.e. big changes from year to year);
− poor timeliness of reporting;
− discrepancies between survey and administrative coverage results;
− poor card retention in coverage surveys;
− incoherence between coverage and disease surveillance data;
− incoherence between the vaccine wastage rate and district strategies;
− proportion of the population vaccinated through outreach;
− incoherence between the vaccine utilization or wastage rate and coverage
figures (e.g. number of immunizations higher than vials open);
− suspicion of overreporting (achievements systematically too close to target,
discrepant achievements according to the antigen, etc).
Poor data quality HUs/districts also include those where the turnover of health
staff is high or where key posts are vacant. Supervision reports also indicate
good or poor recording, reporting and monitoring practices.
4) A combination of the above. This approach combines the advantages of the
problem-oriented approach and the fact that a selection bias for any “preference”
can be avoided.
The conclusions drawn from the sample will need to take the sampling strategy into
consideration. If the sample is not representative, then the results cannot be
generalized; they can only be extrapolated to the structures which were sampled.
Findings obtained from one district cannot be extrapolated to other districts. However
it is likely that common problems and difficulties are shared within a number of
areas. Results could be disseminated through feedback reports and meetings so that
solutions can be shared.
WHO/IVB/05.04 29
D. Present the
DQS findings
The data quality assessment provides a certain amount of information on the status
of records and practices related to the reporting system.
All options in the DQS toolbox provide quantitative measures which can be followed
easily over time and used to compare different areas. The use of the tool will be
particularly interesting when several districts can be compared or a district can be
compared to itself over time.
Assessment findings should be presented and discussed to the level that was assessed
but also to the national level so that lessons can be drawn and solutions proposed for
the whole country.
1. Present the DQS results
1.1 Accuracy
The raw figures and accuracy ratios can be presented in tables such as Table 9.
Table 9: Presentation of figures and accuracy rations
Accuracy of district tabulation, DTP3, year 2003

(October–December), district X
• Total of HU reports • 3465

• District tabulation • 3545
• Accuracy ratio • 98%
DTP3 re-counting at HU level
(register against HU reports, year 2003)
Based on 4 HUs, district X
• Verified • 6848
• Reported • 10 845
• Accuracy ratio • 63%
Graphic presentations can be helpful to present and discuss the findings. An example
is the use of a bar chart (Figure 3).

Figure 3: Example presentation showing the proportion of re-counted measles
immunizations that were reported by 6 HUs in 2003, District X.
100%
80%
60%
40%
20%
0%
HU1 HU2 HU3 HU4 HU5 HU6
% verified (re-counted on the registers)
Reported (on the HU reports found at district level)
The accuracy can also be presented in terms of “accurate months” defined as months
for which the verified information was perfectly accurate (100% match).3
For example, district W, 2003, measles vaccinations:

• HU A: 12 months verified, 11 months accurate;
• HU B: 12 months verified, 2 months accurate;
• HU C: 6 months verified, 6 months accurate;
• Aggregated figure for district W: 18 / 30 = 60%.
3
This can be defined and some flexibility may be allowed, e.g. 95%; 90%...
WHO/IVB/05.04 31
1.2 Quality index scores
The measures can also be presented using bar charts, in percentages or using the raw
numbers. The following representation (Figure 4), called a radar graph, provides a
way to compare all components: average scores are presented in this example on a
scale from 0 to 10. It is easily produced in Excel.
Figure 4: Example presentation showing the quality indices for five

components of a monitoring system, on a 0–10 scale
Recording
10.00
8.00
6.00
4.00 Archiving and

Data use reporting
2.00
Core outputs Demographics
These presentations need to be complemented by ad hoc discussions on each topic.

The DQS findings are only important if strengths and weaknesses that were identified
can be discussed at each level. The main intention is to present appropriate and
realistic recommendations for improving the system. The major challenge is to ensure
that the assessment is useful to the district concerned and that the recommendations
are implemented.
Every presentation should be followed up by an action plan – drafted at the time of the meeting –
outlining roles and responsibilities.

2. Using Excel to enter and represent the data
To date, two simple Excel workbooks are available to assist with data entry and
analysis:
• One is on the QQs, which aggregates the quality indices of selected HUs for
the district level. Automatic charts are presented using the radar graph option
described above.
• One is on the calculation of wastage, which aggregates the vaccine wastage
rates of selected HUs for the district level.
Instructions on how to use the workbooks are detailed on the respective “Read me”
worksheets of the two workbooks.
WHO/IVB/05.04 33
E. Conduct a
DQS workshop
The suggested approach is to introduce the DQS in a country through a national

workshop, followed immediately by an assessment in a number of districts and HUs.
The workshop aims:

• to sensitize key health personnel on the importance of accurately monitoring
immunization activities and data quality;
• to train key health personnel from the national and district levels in data quality
self-assessment methodology (theoretical and practical);
• to enable the participants to conduct a data quality assessment in a number of
districts and facilities of the country immediately after the workshop;
• to make appropriate recommendations to adapt the DQS to the country-specific
context as a sustainable self-assessment tool.
A typical workshop schedule is proposed in Annex F. During phase 1, the participants

revise and discuss available monitoring tools, then design and test their own assessment
tool. During this phase participants thoroughly review the options they have selected
from the DQS toolbox. The test makes sure that all components of the assessment
are understood and allows last minute refinements.
Phase 2 consists of conducting the assessment itself, in the areas and facilities chosen
by the participants, using the forms they will have designed (field work). After the
assessment, the participants convene again to share the results, perform a global
analysis, and make overall recommendations.
The suggested timeframe in Annex F is four days for phase 1, then three days for
data collection and two days for data analysis and feedback, but this should be adapted
to the time available, logistics and the number of participants. If people cannot take
the above suggested time off, a DQS workshop can be organised in six days –
specifically two days theory, two days field work, one and a half days data analysis
and a half day providing recommendations and debriefing. Careful planning is essential
to success. For this reason, the facilitators should be in the country for three days
prior to the workshop for coordinated preparation.

Some proposed workshop principles
• Target audience: The number of participants should ideally fall between

15 and 25, with a ratio of 1 facilitator per 6 participants. Facilitators should
receive a list of the names and titles of these individuals at least one week prior
to training so they are familiar with their target audience and relevant skill
base, including language skills. Staff should include a balanced representation
from national (30%), provincial (40%) and district (30%) levels to permit:
a) national level understanding of the principles of the DQS;
b) training of certain key individuals who train others in the DQS in the future;
c) district feedback into the processes.
• Although monitoring data is not a new concept, the practical aspects of applying
the DQS are new and sometimes difficult to understand if only using theoretical
concepts in a classroom-lecture style. Field work helps test the DQS but, just
as importantly, it gives the opportunity for participants from provincial and
national levels to witness the ground realities of immunization monitoring
systems. The in-class sessions themselves are most successful when participants
are actively encouraged to participate through a range of adult learning
techniques, such as simulations, practical exercises, games, illustrated lectures,
role plays, small group competitions and prizes.
• A good ice-breaking exercise consists of the monitoring-card game (day 1).
It consists of a series of 50 questions on monitoring systems which are asked
of the participants who should be split into groups. (An Excel workbook
presents these questions on cards for participants to randomly choose.) If a
group answers correctly, it is allowed to move (throwing a dice) on a
50-square game board and the participants can gently compete. The card
questions provide an excellent overview of the available tools and best practices,
and engender a spirit of camaraderie in the workshop.
• Two approaches can be envisaged during phase 1 of the workshop: (1) the
“start-from-scratch” approach, with an entirely self-devised assessment,
and (2) a “menu” approach with participants provided with a range of possible
qualitative questions and forms that are locally relevant and presented as a
menu of options from which to choose. In the menu approach, the questions
can be simultaneously pre-assigned to their proper categories (recording,
reporting, demographics, use of data, availability of forms, etc.) and structured
by subgroup so participants can effectively choose how to prioritize questions
and design their national questionnaire. For the accuracy component, the sources
of data and levels of analysis could also be presented as a series of options from
which the participants choose, with the possibility of revising or adaptating
them after field work. The first approach provides better ownership of the
process but is more time consuming and necessitates more intensive guidance
throughout the workshop. The second approach necessitates careful planning
and excellent understanding of the local situation prior to the workshop.
• If participants are able to use Excel, computers provide an opportunity to
learn how to create small databases, analyse data and create ways of displaying
data. This saves time when transferring data for presentations, and Microsoft
PowerPoint presentations can also enhance the efficiency of the workshop.
WHO/IVB/05.04 35
• Preparation: All facilitators should arrive in country early to permit three full
days of preparatory work before the workshop. This would permit two field
days for travel to several health centres in several districts to be able to give a
realistic overview of data flow from community up to national level, followed
by one day for revising the menu of qualitative and accuracy questions.
Promoting local facilitators is critical in encouraging ownership and
sustainability of the process. Facilitators should receive adequate briefing one
month in advance and should be allocated tasks so they can begin their individual
preparations. Where possible, presentations and session plans from prior
workshops should be shared widely.
• Because the field visits require good coordination, it is important that focal
point(s) for logistics in-classroom are appointed. The facilitators should know
who the focal points are and should have good channels of communication
with these individuals. At least a week prior to the workshop, the organizers
should prepare a detailed list of supplies needed; this should include an ample
quantity of office items such as markers, flipcharts, scissors, staplers, as well
as access to other necessary equipment (computers, printer, photocopier). Field
travel should be carefully coordinated focal point to assure adequate cars for
transport. The lead facilitator should be aware of the budget allocation in order
to address any possible budget constraints. The workshop can be integrated
with other health-sector monitoring issues as usually the same staff members
are busy with a variety of health data. Hence this would be a good opportunity
to explore whether DQS principles can be used for other health indicators.
After data collection on site, each team presents its findings and recommendations,
emphasizes the most important or urgent points;,suggests persons/parties who should
be responsible for follow-up action, and draws up a timetable of corresponding
activities.
In presenting its findings, the team should review the terms of reference, explain the
methodology used, summarize observations (supplemented by supportive objective
information), provide recommendations and acknowledge the contribution of
everyone who has helped to make the review a success. Any visual aids used during
the presentation should be shared for use during future meetings/training sessions.
Involvement of local partners and academic institutions. In order to build the capacity for a
country to perform data quality self-assessments and sustain the tool implementation, hence
maintain a high standard of monitoring practices, it is important to involve local partners and
academic institutions in a DQS workshop and its follow-up. This could also be a gateway for an
eventual extension of the tool to other health indicators.

F. Integrate DQS results into
the routine activities
The key to DQS success depends on the success of the workshop and first assessment.
A well-focused assessment should result in the following.
• A documented monitoring system: The tool should help managers to estimate
whether the information collected is reliable (accurate), and whether the
information is properly used (the monitoring system is of good quality).
• Identified weaknesses and strengths of the system: Major problems should
be localized.
• Recommendations for improvement of the performance of the system:
Monitoring immunization services is meaningful only if the information that is
produced can be used and leads to action. In particular, DQS results should be
used to:
− adjust district microplans accordingly;
− review the effectiveness of applied strategies;
− change priorities in the plan;
− guide future supervisory visits to focus on the issues found in the DQS.
• Suggestions on specific activities that the country concerned should

introduce for improved reporting of data.
• Suggestions on how to improve the assessment itself.
• A follow-up plan: A follow-up plan should be drafted, and include the creation
of a team or focal point.
Finally, recommendations should include ways to use and promote the DQS options
that the workshop participants developed. The goal of the DQS is to integrate the
tool into routine practice (sustainable self-assessment) which should be facilitated
by the fact that the tool is self-designed.
Options include the following.

• Integrate the DQS options into supportive supervisory visits/feedback
practices. An ad hoc supervisory checklist can be built upon the QQs. A
supervisory visit can also include the calculation of an accuracy ratio or wastage
rate.
• Make a plan to repeat the assessment in the same districts or extend it to new
districts.
WHO/IVB/05.04 37
• Integrate the DQS concept into the national training schedule.
• Form a core team or designate a focal point to be responsible for follow-up of
DQS findings, help incorporate these into supportive supervisory visits, and
involve local partners and academic institutions.
• Include key quantitative DQS measures as core indicators at the district level.
• Integrate DQS measures and tools into Reach Every District (RED) workshops
and microplanning activities.
• Develop district-level DQS guidelines or a workbook (or integrate into existing
district material).

Annex A:
Sample chart for monitoring doses administered
and drop-outs in children less than one year of age
WHO/IVB/05.04 39
40
Sample chart for monitoring doses administered and drop-outs in children less than one year of age
Fictia District, Peri-urban area Year: 2001
150 150
138 138
125 125
113 113
100 100
88 88
75 75
63 63
50 50
38 38
25 25
13 13
0 0
Jan Feb Total Mar Total Apr Total May Total Jun Total Jul Total Aug Total Sep Total Oct Total Nov Total Dec Total
DTP1 10 12 22 7 29 12 41 14 55 15 70 14 84 7 91
DTP3 8 9 17 8 25 10 35 11 46 12 58 12 70 7 77
DO# 2 5 4 6 9 12 14 14
DO% 20 23 14 15 16 17 17 15
DO = drop-out

Annex B:
Example of a completeness/timeliness
reporting table
WHO/IVB/05.04 41
42
Example of a completeness/timeliness reporting table
Insert the date the HU reports were received at the district office. If a report is received after the deadline, enter the date in red.
Display the table in the district office.
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Total Total
completeness a timeliness a
HU1
HU2
HU3
HU4
HU5
HU6
HU7
HU8
HU9
HU10
Total received this month (No.)
Total received this month (%)
Cumulative completeness b (%)
Total on time this month (No.)
Total on time this month (%)
Cumulative timeliness c (%)
Key
a
Total completeness or timeliness: refers to the reporting completeness of the selected HU. Can be filled in at the end of the year, or could be updated each month giving the
HU completeness and timeliness at each moment of the year. This may be easier in a computerized worksheet.

b
Cumulative completeness: reports received up to that month divided by reports expected up to that month.
c
Cumulative timeliness: reports received on time up to that month divided by reports expected up to that month.
Annex C:
Standard questions to assess
the quality of the monitoring system
WHO/IVB/05.04 43
44
Standard questions to assess the quality of the monitoring system
QUESTIONS ON QUALITY (QQ), DISTRICT LEVEL
Question Comment Weight

(from
1 to 3)
Recording component (district)

1 Are vaccine receipts and issues recorded in a vaccine ledger? To assess appropriate record-keeping of vaccine receipts and issues.
2 Does the district manager know the annual vaccine requirements for the district?
3 Is the lot number/batch number and expiry date of the vaccine recorded?
4 Is the current ledger up to date for a given vaccine (or vitamin A)?
5 Is there a log of syringe supply and delivery to HU up to date? Is the stock available identical to the quantity recorded in the register (count).
6 Are district staff aware of standard operating procedures to record a severe adverse
event following immunization (AEFI)?
7 Do the district’s reports (found at district level) have at least one date stamped or written Determine what proportion of correct HU reports you would need to answer “Yes”.
on them? Define for which period (e.g. for the previous year)
Define the date significance:
the date the report was signed
the date of receipt at district level (stamped or written on it by the
district office)
8 Are the district reports (that are sent to more central levels) completely and correctly To select a number of fields to be checked in all district reports and check whether
filled in? these have been correctly filled in.

Standard questions to assess the quality of the monitoring system (cont’d...)
WHO/IVB/05.04
(from
1 to 3)
Archiving component (district)

9 Is there a separate file or sub-file for each HU and are the reports inside filed by date? Storage should facilitate retrieval and monitoring (and be well organized).
10 Have all HU data from the previous month been processed?
11 Are supervisory reports available?
12 Are copies of the last feedback to the health facilities easily available?
13 Can copies of all district reports (that were sent to more central levels) be found?
Computerized archiving (district)
14 If the district is computerized is the last date of backup within one week? Check diskette for last saved date; look at the file creation date.
(look at the date the file was created on the diskette)
15 Can the official immunization tabulations for the previous year be reproduced from an To check official immunization tabulations = final summary of previous year data.
archived electronic file?
16 If more than one computer has immunization data, is there either a functioning network or
a written, well-organized method of data transfer? (If yes, read it.)
17 Is the date of printing /production on every tabulation/chart produced or, if the data is
archived, is there a date showing when the archived file was created?
45
46

(from
1 to 3)
Reporting component (district)

18 Have the district reports of the last year month been sent on time?
19 Is the procedure for dealing with late reports known and applied?
20 Did all the monthly (quarterly) reports from the HUs use the same form/format for the
current year?
21 Is there a system for investigation of individual reports of adverse events following Serious AEFIs should be rapidly reported and investigated.
immunization (AEFI) from the district to the higher level functioning/operational? Investigators should be looking for any evidence of programmatic error that
must be rapidly corrected and/or rumours that cause problems.
22 Did all the visited HUs report adequate supply of administrative forms tally sheets/reporting
forms/health cards?

WHO/IVB/05.04
(from
1 to 3)
Demographic information component (district)

23 Is the district denominator for immunization of infants and pregnant women (and school Known: the interviewed senior staff member should be able to tell (without looking)
children, if applicable) known? approximately how many infants the district contained and how the figure was
calculated (if relevant).
24 Is there a district map of the catchment area showing HUs and providing immunization Ideally, the map should include denominator, target, type of strategy.
strategies (fixed, outreach, mobile)?
25 Is the proportion of infants per strategy-type known for the district? Usually fixed – outreach – mobile team, etc. This should be used in a
district microplan.
26 Has the same denominator for child immunization been used on different tabulations, Indicate for which year.
reports, charts, tables, etc?
27 Are the denominators used in the current year different from the denominators used in Should be different from previous year.
the previous year?
28 For the previous year, is the district denominator value (for child immunizations) found at
the district the same as that used at national level?
29 Is the denominator established independently? The denominator should be established independently from locally set-up targets.
30 Are the denominators of each HU available for the previous year? Answer “Yes” if available. Totals should add up to the district total.
31 For the previous year, has only one denominator value (check at least total population) Check with various initiatives (e.g. polio, nutrition, malaria) whether the
been seen in all health projects/programmes? denominator is consistent at district level.
47
48

(from
1 to 3)
Core outputs/analyses component (district)
32 Is there a target number of children that the district strives to vaccinate during a calendar
year or reporting period?
33 Is there a chart or table of immunization coverage by report period for the current year Is it on display?
(monitoring chart)? Is it UP TO DATE?
Does it cover all antigens?
34 Is the completeness of the immunization reporting from HU recorded and monitored at District staff should be able to describe what percentage of HU reports was
district level? received on time, received but not on time, and not received at all during the
previous year or the last months.
35 Does the district record and monitor timeliness for HU immunization reporting? District staff should be able to say (based on printed information) what percentage
of HU reports was received on time during the previous year or the last months.
36 Is the drop-out rate monitored? Discuss the importance and reasons for drop-outs. Are there managerial practices
that could be changed to reduce the drop-out rate?
37 Is there monitoring of HU/district vaccine wastage? Discuss the importance and reasons for wastage.
38 Is there a graph by month of the incidence of vaccine-preventable diseases (VPDs) – How do these data correspond to coverage data (i.e. more cases in areas with
broken down by VPD? poor coverage). When was the last VPD outbreak? Was it investigated?
Why did it occur?
39 Is an up-to-date chart/table of the completeness of the current year’s immunization data Completeness = reports received or not received from the HUs. (Here the
available? score 0/1 is only for completeness.)
40 Is the HU performance monitored at the district level? Monitoring of HUs: graph/figures showing how all HUs are performing during
the current year.
41 Are supervision activities monitored? A written schedule of supervision that includes visiting every HU within a specific
period of time.

42 Has the district selected an indicator for the monitoring of immunization safety?
WHO/IVB/05.04
(from
1 to 3)
Evidence of using data for action (district)
43 Is there an analysis of HU data performed regularly with HU staff? Analysis can be done within supervisory visits, meetings at district level, etc.
Explore the quality of analysis as well as the exhaustiveness of the HUs said to
be analysed: none of them should be left out.
44 Do you send regular monthly written feedback to the HUs?
45 Are areas of low access identified and evidence of action taken to deal with it? Discuss the importance and reasons for low access. How do the three strategies
(fixed site, outreach and mobile teams) relate to the issue of access in the district?
46 Have reasons for any high drop-out been identified, and are there plans/actions Are there managerial practices that could be changed to reduce the drop-
to deal with it? out rate?
47 Is there monitoring of HU vaccine stock-outs? The manager should be able to say (based on written information) whether any
(A stock-out is an interruption in vaccine supply [for any vaccine].) HU has encountered a vaccine stock-out. If no vaccine stock-out is reported,
ensure that the monitoring is possible and is being implemented.
Staff should be monitoring the level of reserve stocks and taking action if stock
goes below a specified reserve level.
48 Are there problems with completeness and timeliness of reports? Are the late or incomplete reports usually from the same HUs. What was done to
follow them up? What other actions were taken to encourage/induce timely
reporting.
49 Are the recommendations made for the last three supervisory visits followed up in
subsequent visits?
50 Has the monitoring of the selected immunization safety indicator been adequate during
the last 12 months?
51 Are surveillance and coverage data compared to look for inconsistencies and then
49
followed up to understand why?
50
Standard questions to assess the quality of the monitoring system
QUESTIONS ON QUALITY (QQs), HEALTH UNIT LEVEL
Question This question should be designed to find out: Weight

(from
1 to 3)
Recording component (HU)
1 Are there tally sheets for infant vaccinations on the desk (or easily available) and do they The main concern is evidence of use of availability (official form) and tally sheet.
have entries for the last immunization day?
2 Are registers used for recording individual information about child immunizations? Each HU should have a book or register where each immunization
history can be registered and traced back.
3 Can a child’s vaccination history be easily and rapidly retrieved in the registers? A new dose should not be entered as a complete new entry but entered in the
location where previous doses have been entered. Score 0 if the register is
used as a new entry for any immunization.
4 Are registers (or pre-printed forms) used for recording individual information about There may be registers or health cards if cards kept in HU.
women’s TT immunizations?
5 Observe at least five vaccinations:
Were all vaccinations well registered on the child health card/tally sheet/register?
6 Are individual immunization records used, updated and given to the child’s caretaker Blank cards should be available in the HU. Immunization cards are often
at the time of the immunization visit? integrated in “Road to Health” or other health cards.
7 Are vaccine receipts recorded in a vaccine ledger? Check against available stock (count doses in the refrigerator).
8 Ask the child’s caretaker: Do you know the expected date of receiving vaccine? Find out whether the expected dates are known.
9 Is the ledger up to date for all vaccines and/or a selected vaccine? Up to date = all receipts and issues recorded immediately.
Check against stock (in therefrigerator). Compare the date of last entry and the

date of the last immunization session.
WHO/IVB/05.04
(from
1 to 3)
10 Is the receipt of a selected vaccine in the ledger complete for the entire year?
11 Is there a log (vaccine ledger/stock card) for receipt/issuing of syringes supplied Can perform a stock check.
(AD/non-AD reconstitution syringes)?
12 Does the HU record vaccine batch-number and expiry date?
13 Are all individual recording forms available for the entire previous year? Individual recording form = tally sheet or register.
14 Did every person doing the child immunization card exercise get a perfect score for: Need to define how the scoring will be if a perfect score is not obtained.
DTP1
DTP3
measles?
15 Is the cold chain temperature monitoring chart completed daily? Check the chart and compare the latest reported temperature with the actual
temperature in the refrigerator.
51
52

(from
1 to 3)
Reporting component (HU)
16 Have all reports for the previous year been signed by the officer-in-charge or Score for example:
officer authorized to submit the HU report? If >50% reports are signed score “Yes”.
If <100% discuss with the HU why some have not been signed.
17 Does each report from the previous year have at least one date stamped or written on If >50% reports are signed score “Yes”.
it by the HU – either as “signed date” or “compiled date”? If <100% discuss with the HU why some have no data stamped or written in.
This can be answered at district level.
18 Are the HU reports correctly filled in? Select a number of fields to be checked in all HU reports and check whether
these have been filled in correctly.
19 Are health staff aware of standard operating procedures and the necessary forms to Ask health staff what is supposed to be done if a child becomes severely ill or
complete if there is a report of a severe AEFI? dies after a vaccination. Ask to see any forms that are to be used.
20 Are the HU reports completely filled in? Select a number of fields to be checked in all HU reports from the previous year
and check whether these have been filled in.

WHO/IVB/05.04
(from
1 to 3)
Archiving component (HU)
21 Can copies of all previous reports from this HU be found in the HU? For current and previous year.
22 Is there one location where the previous immunization reports and recording forms
are stored?
23 Are the reports of the HU organized in a file by date? The main concern is that the reports are easily retrievable.
24 Are HU reports available for the entire year?
25 Are the child registers available for all periods of the previous year?
26 Can all tally sheets covering the previous year be found?
27 Are registers for TT vaccinations to pregnant women available for the entire
previous year?
28 Is the latest feedback on data from district easily available?
53
54

(from
1 to 3)
Demographic information component (HU)
29 Does the HU have data on the number of infants born in its catchment area? The number of births should be different from the previous year. Discuss if there
is a difference with the denominator available at more central level. Discuss ways
to collect denominator information from community (e.g. birth register), data from
national immunization days (NIDs), or other sources. Discuss if the target was
set up by the district or HU level.
30 Does the HU have a target set on the number of children that should be vaccinated Discuss how realistic the value is.
during the calendar year?
31 Does the HU have a system that allows the collection of information on new births This may include community health workers, traditional birth attendants, outreach
in the community? clinics, etc. A system means (a) organized way to collect the information in every
village/community and (b) a written track available at the HU.
32 Does the HU have a target by type of strategy (fixed/outreach/mobile) with a map
showing the catchment area by strategy including the outreach villages?

WHO/IVB/05.04
(from
1 to 3)
Core outputs/analysis (HU)
33 Does the HU have a (target) number of children that it strives to vaccinate during a
calendar year or a reporting period?
34 Is there a mechanism in place to track defaulters? Can be an appropriate use of a correctly filled register, tickler file, etc.
When was the last time a child was followed up?
35 Does the HU have achievements split by type of strategy – fixed/outreach/mobile? It is important is to know the proportion of numbers actually reached by each
strategy.
36 Does the HU have an up-to-date chart or table (preferably on display) showing the Monitoring coverage chart – must be UP TO DATE.
number of vaccinations by report period for the current year?
37 Is there a monthly chart/graph of VPD cases (broken down by VPD)? How do these data correspond to coverage data (i.e. more cases in areas with
poor coverage). When was the last VPD outbreak? Was it investigated?
Why did it occur?
38 Does the HU monitor drop-out rate? Preferably on display with the same monitoring chart as the coverage one,
but score 1 if the health worker can tell you the drop-out rate for his HU.
Discuss the importance and reasons for drop-outs.
39 Does the HU monitor vaccine wastage? Discuss the reasons for wastage and any ways it might be reduced.
Discuss whether the health worker knows how much the vaccine wastage is
and how it can be calculated.
55
56

(from
1 to 3)
Evidence of using data for action component (HU)
40 Is there a mechanism in place to track defaulters or vaccine doses that are due? Check how the HU can know when a child should return for a vaccine dose
(e.g. DTP, Hib, HepB, polio, measles or yellow fever vaccine dose).
41 Is there a map showing the catchment area, including the outreach villages? This ideally shows strategy type.
42 Are areas of low access identified and is there evidence of actions taken to deal with this? If there is low access (evidenced by low BCG or DTP1 coverage), how does it
relate to the effectiveness of the three strategies (fixed site, outreach and/or mobile
teams).
43 Have reasons for any high drop-out been identified; are there plans/actions to deal with Are there any managerial practices that can be changed?
this?
44 Have actions been taken on the last feedback from the district?
45 Is there interaction with the community regarding immunization? Ask for information on Are health staff actively involved in any community committees or meetings on
“what” and “when”? health, investigations of outbreaks or any rumours of AEFIs, etc?

Annex D:
Child immunization card exercise
(example for 20 infants)
A set of 20 cards should be prepared according to the histories described below

(child 1–20). Each card represents a child arriving at the HU on the day of the
evaluation. The health worker examines a card, determines what vaccinations should
be given, and makes the appropriate marks/recordings on the provided sample copy
of the register/tally sheet. The vaccinator can also write down the date that the child
should return for his/her next vaccination. The dates should correspond to the dates
of the HU’s planned vaccination sessions. If “OPV0” has to be included, discuss this
preliminary and fill in the child immunization cards accordingly.
Example child A – due for DTP2, OPV2.

BCG, DTP1, OPV1 given.
Child old enough for OPV2/DTP2 vaccination.
Example Child B – not due for any vaccination.

BCG, DTP1, OPV1, DTP2, OPV2, DTP3, OPV3 given.
Child too young to receive measles vaccination yet.
Child History
Child 1 due for DTP3, OPV3
BCG, DTP1, OPV1, DTP2, OPV2 given
Child old enough for OPV3/DTP3 vaccination
Child 2 due for BCG
Child born two days ago
BCG, DTP1, OPV1 given
Child 4 not due for any vaccination
BCG, DTP1, OPV1 given on schedule,
OPV2/DTP2 given only two weeks ago
Child old enough for OPV3/DTP3
Child 5 due for BCG, DTP1, OPV1
No vaccinations given
Child 6 due for measles
BCG, DTP1, OPV1, DTP2, OPV2, DTP3, OPV3 given
Child old enough for measles vaccination
No vaccinations given
WHO/IVB/05.04 57
Child History
BCG given at birth
BCG, DTP1, OPV1 given on schedule, DP2/OPV2 given just
two weeks ago
BCG, DTP1, OPV1, DTP2, OPV2
Child 11 due for BCG
No vaccinations
Child born 2 weeks ago
No vaccinations
BCG at birth
BCG given late
Child 16 due for DTP3, OPV3, measles
BCG late, DTP1, OPV1 late, DTP2, OPV2 late
BCG, DTP1, OPV1, DTP2, OPV2, DTP3, OPV3 given
Child 18 due for DTP3, OPV3, measles
BCG, DTP1, OPV1 given late
BCG, DTP1, OPV1 given
Relevant totals (for children 1–20) are:

BCG = 6; DTP1 = 6; DTP2 = 4; DTP3 = 5, measles = 5.

Annex E:
Sampling of health units
A classical procedure is described below. It allows a random selection of HUs with

a probability proportional to the size (estimated by the number of immunizations
given during the previous year), using systematic sampling. The tool used to assist in
the randomization is a random-number table.
1) Obtain the list of all HUs providing immunization services. This list is then
the sampling frame from which the sample is to be selected.
2) A sampling interval is then determined. The sampling interval is a number
used to systematically select HUs from the sampling frame. To determine the
sampling interval, take the total (all HUs) cumulative number of vaccinations
(in this example DTP3) divided by the number of HUs you want to sample
(say 6 in the following example).
In practice, make a table listing all the HUs in the district, and make a cumulative
total of their DTP3 vaccinations.
List of all health units with their respective DTP3 vaccination numbers,
and cumulative DTP3 totals
HU name DTP3 Cumulative DTP3

Bennet 85 245 85 245
Dundee 45 124 130 369
(This is 85 245+45 124)
Jamestown 36 875 167 244
(This is 130 369+36 875)
Nyeri 96 185 263 429
Pokot 76 359 339 788
Rossem 77 125 416 913
Travert 22 654 439 567
Unison 57 692 497 259
Waverly 57 265 554 524
Natoye 22 115 576 639
Erpent 10 847 587 486
Tuki 5 000 592 486
District total 592 486
WHO/IVB/05.04 59
If there were a total of 592 486 doses of DTP3 given among 12 HUs available for
sampling, the sampling interval would be: 592 486 / 6 = 98 748 (which has five
digits). (Six is the number of HUs to be sampled.)
To select the first HU, firstly you choose a random number between one and the
sampling interval.
Step 1: Choose a direction (right, left, up or down) in which you will read the
numbers from the table.
Step 2: Select a starting point: close your eyes, and touch the random number table
with a pointed object. Open your eyes. The digit closest to the point where
you touched the table is the starting point. Check that the starting point
will give a number which is going to be less than or equal to the sampling
interval. If not, start again before going on to Step 3.
Step 3: Read the number of digits required (determined by the sampling interval)
in the direction chosen in step 1. Because each individual digit in the table is
random, the sequence(s) of digits can be used across spaces between the
five-digit numbers. The number you end up with is your random number.
For example, let us say you decided to read numbers to the right, and you
identified your starting point as the number 3 in row 01, column 8 (see the
table of random numbers in this Annex). If the sampling interval had four
digits, then your random number would be “3861”. The numbers “6” and
“1” come from row 01, column 9.
NOTE: Remember that the random number selected must be equal to or
smaller than the sampling interval. If it is not, then another random number
must be selected. You can decide (before selecting your starting point) a
direction to go to choose it (right, left, up or down from the first selected
digit).
In our example, column 3, row 07 of the random number table gave the
number 92780. The first selected HU will be Dundee, as it is the first HU
where the cumulative population listed for that HU will equal or exceed
the random number.
Step 4: Identify the second HU by adding the sampling interval to the random
number. The cumulative population listed for that HU will equal or exceed
the number you calculate. Repeat for subsequent HUs. In our example,
these will be:
92 780 + 98 748 = 191 528 Nyeri selected
191 528 + 98 748 = 290 276 Pokot selected
290 276 + 98 748 = 389 023 Rossem selected
389 023 + 98 748 = 487 771 Unison selected
487 771 + 98 748 = 586 519 Erpent selected

Random numbers table
Column
Row 0 1 2 3 4 5 6 7 8 9
01 88008 13730 06504 37113 62248 04709 17481 77450 46438 61538
02 01309 13263 70850 11487 68136 06265 36402 06164 35106 77350
03 45896 59490 98462 11032 78613 78744 13478 72648 98769 28262
04 50107 24914 99266 23640 76977 31340 43878 23128 03536 01590
05 71163 52034 03287 86680 68794 94323 95879 75529 27370 68228
06 76445 87636 23392 01883 27880 09235 55886 37532 46542 01416
07 84130 99937 86667 92780 69283 73995 00941 65606 28855 86125
08 00642 10003 08917 74937 57338 62498 08681 28890 60738 81521
09 64478 94624 82914 00608 43587 95212 92406 63366 06609 77263
10 02379 83441 90151 14081 28858 68580 66009 17687 49511 37211
11 32525 44670 57715 38888 28199 80522 06532 48322 57247 46333
12 01976 16524 32784 48037 78933 50031 64123 83437 09474 73179
13 67952 41501 45383 78897 86627 07376 07061 40959 84155 88644
14 38473 83533 39754 90640 98083 39201 94259 87599 50787 75352
15 91079 93691 11606 49357 55363 98324 30250 20794 83946 08887
16 72830 10186 08121 28055 95788 03739 65182 68713 63290 57801
17 40947 75518 59323 64104 24926 85715 67332 49282 66781 92989
18 44088 70765 40826 74118 62567 75996 68126 88239 57143 06455
19 19154 29851 16968 66744 77786 82301 99585 23995 15725 64404
20 13206 90988 34929 14992 07902 23622 11858 84718 22186 35386
21 24102 13822 56106 13672 31473 75329 45731 47361 47713 99678
22 59863 62284 24742 21956 95299 24066 60121 78636 61805 39904
23 57389 70298 05173 48492 68455 77552 87048 16953 45811 22267
24 63741 76077 44579 66289 88263 54780 76661 90479 79388 15317
25 17417 56413 35733 27600 06266 76218 42258 35198 26953 08714
26 85797 58089 91501 34154 96277 83412 70244 58791 64774 75699
27 65145 97885 44847 37158 54385 38978 20127 40639 80977 73093
28 24436 65453 37073 81946 36871 97212 59592 85998 34897 97593
29 20891 03289 98203 05888 49306 88383 56912 12792 04498 20095
30 81253 41034 09730 53271 92515 08932 25983 69674 72824 04456
31 64337 64052 30113 05069 54535 01881 16357 72140 00903 45029
32 35929 76261 43784 19406 26714 96021 33162 30303 81940 91598
33 34525 54453 43516 48537 60593 11822 89695 80143 80351 33822
34 27506 45413 42176 94190 29987 90828 72361 29342 72406 44942
35 92413 00212 35474 22456 76958 85857 85692 75341 32682 00546
36 76304 57063 70591 06343 38828 15904 79837 46307 40836 69182
37 17680 92757 40299 98105 67139 01436 68094 78222 61283 40512
38 43281 36931 26091 42028 62718 38898 64356 19740 77068 78392
39 30647 40659 23679 04204 67628 81109 73155 68299 62768 58409
40 26840 42152 80242 57640 19189 47061 44640 52069 98038 49113
WHO/IVB/05.04 61
Annex F:
Data quality self-assessment
workshop schedule
Location: _______________________________________________
Dates: __________________________________________________
Day 1
Session 1: Introduction and overview

09.00 Opening of workshop
MOH–WHO representatives (national–regional)
09.10 Introductions
09.20 Review of workshop objectives and DQS overview
10.00 Global/regional perspectives – 3 options proposed:
What is data used for at regional level?
The GAVI alliance and the need for improving data quality
The RED approach
10.45 Coffee
Session 2: Monitoring tools
11.00 Brainstorming on monitoring tools (card game)
13.00 Lunch
14.00 Review of country monitoring tools
15.00 Coffee
15.15 Designing QQs (group work)
16.15 Group feedback

Day 2:
Session 3: Finalizing the questions and weighting for the DQS

9.00 QQs revision and weighting
10.30 Coffee
10.45 Plenary session & group presentations and feedback
Session 4: Supportive supervision
11.45 Introduction to supportive supervision
12.00 Organizing a supervision of monitoring practices (group work)
13.00 Lunch
Session 5: Site selection and other options
14.00 Sampling strategies
15.00 Coffee
15.15 DQS options:
Levels to be assessed
Site selection
Wastage – completeness
Day 3:
Session 6: Accuracy of reported data

09.00 Data accuracy measurement at health facility and district levels
09.30 Group work preparation: collection and analysis at health facility,
district and community levels
10.30 Coffee
10.45 Plenary session and feedback
13.00 Lunch
Session 7: Guide for the assessment
14.00 Elaboration of the assessment guide – brainstorming session
15.00 Coffee
15.15 Survey guide continued
Optional: data collation and analysis
16.15 Practical arrangements for the pre-test
WHO/IVB/05.04 63
Day 4:
AM: Pretest
Session 9: Finalization of the tool
14.30 Feedback
15.30 Revisions of the QQs and the forms
16.30 Assessment in practice
Days 5, 6, 7:
Data collection in the field
Day 8:
09.00 Feedback from the assessment

Data collection, field difficulties and suggestions for improving the
assessment
10.30 Coffee
11.00 Data analysis (group work)
13.00 Lunch
14.00 Data analysis and recommendations. Development of practical district
materials – or how to integrate DQS into routine practice
15.00 Coffee
15.30 Data analysis and recommendations (continued)
Day 9:
09.00 Group presentations

10.30 Coffee
11.00 Recommendations (plenary)
13.00 Lunch
14.00 Data analysis and reporting
15.00 Coffee
15.30 Formal presentation of results
16.30 Closure

The Immunization Data Quality Self-Assessment (DQS) Tool: WHO/IVB/05.04 Original: English Distr.: General

Uploaded by

Copyright:

Available Formats

The Immunization Data Quality Self-Assessment (DQS) Tool: WHO/IVB/05.04 Original: English Distr.: General

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Immunization Data Quality Self-Assessment (DQS) Tool: WHO/IVB/05.04 Original: English Distr.: General

Uploaded by

Copyright:

Available Formats

WHO/IVB/05.

This publication was produced by the

Ordering code: WHO/IVB/05.04

This publication is available on the Internet at:

Copies may be requested from:

© World Health Organization 2005

Printed by the WHO Document Production Services, Geneva, Switzerland

Special thanks is given to Abdallah Bchir, Craig Burgess, Jan Grevendonk,

The DQS aims to determine:

The assessment includes a review of data accuracy at different levels and a

How is a DQS performed? One approach is to hold an initial national participatory

In this manual, monitoring refers to the measurement of the level of achievement in

1. DQS options: overview

Table 1. Description of the main areas a DQS can assess

Option District Health Unit Main measures

2 The immunization data quality self-assessment (DQS) tool

2.1. Assess reporting accuracy

This exercise is critical because it provides an opportunity to evaluate coverage data

2.1.1. Description of the administrative immunization-reporting

Figure 1: Reporting flow of immunization coverage data

Health Unit District National

Tally District District

At the national level (national headquarters of the national immunization services/

4 The immunization data quality self-assessment (DQS) tool

In addition, an agreement needs to be made in the case of missing information:

2.1.3. Verifying coverage data sent by the health unit level

2.1.4. Verifying the coverage data sent by the district level

a) A monthly or quarterly district report sent to the more central level

There needs to be a decision on which source will be used to verify the

6 The immunization data quality self-assessment (DQS) tool

The two following options can be proposed:

Experience has shown that verification at the community level is a time-consuming

Selection of children in the community (option b)

2.3 The measure (accuracy ratio)

8 The immunization data quality self-assessment (DQS) tool

Possible reasons for low verification: accuracy ratio < 100%

These lists are not exhaustive.

2.3.3 Aggregating the accuracy ratios

2.3.3.1 To aggregate the same accuracy ratios (same level)

In the following example, an accuracy assessment has been conducted in two

We would like to get an estimate of the HU registers/HU reports accuracy-ratio for

10 The immunization data quality self-assessment (DQS) tool

District HU Re-counted number Reported number HU accuracy Target Weight of

e.g. for District 1, the weight of HU1:

Then the District 1 accuracy ratio is:

(315 x 58.2%) + (280 x 41.8%) = 0.97 or 97%

(5000 x 97%) + (3000 x 84%) + (1000 x 105%) = 0.94 or 94%

are multiplied to provide one accuracy measure:

For example: (94%) x (97%) = 0.91 or 91%

2.3.4. Calculating confidence intervals around an accuracy estimate

12 The immunization data quality self-assessment (DQS) tool

Indicate the antigen: ________________________________ Indicate the year: ________________

Region: _________________________________________ District: ________________________

Monthly reports Year X Antigen 1 Total Antigen 2 Total

Health unit name Oct Nov Dec Oct Nov Dec

Total as sent by the district b1 b2

The immunization data quality self-assessment (DQS) tool

Indicate the antigen: ________________________________

Vaccinations – number retrieved Total

Indicate the antigen: ________________ Indicate the year:

Region: _________________ District:

District: ____ Name of HU Selected Village: _

Name of the interviewers __________ Team _ Date of interview ______________________________________