Pnadg566 Mini Survey

USAID Program Design and Evaluation
Methodology Report
Conducting Mini Surveys in

Developing Countries
Krishna Kumar
Senior Social Scientist
Center for Development Information and Evaluation
U.S. Agency for International Development
Month Year1990, revised July 2006

December BureauBureau
for Policy
for Policy
and Program
and Program
Coordination
Coordination PN-ADG-566
XX-XXX-XXX
The findings and conclusions in this report are those of the author and do not necessarily
represent the views of USAID.
This paper is available from USAID’s Development Experience Clearinghouse (DEC). To order
or download, go to dec.usaid.gov and enter the document identification number (see front cover)
in the search box. The DEC may also be contacted at 8403 Colesville Rd., Ste. 210, Silver Spring, MD
20910; tel 301-562-0641; fax 301-588-7787; email docorder@dec.usaid.gov.
Editorial, design, and production assistance was provided by IBI–International Business Initiatives,
Arlington, VA, under contract no. HFM-C-00-01-00143-00. For more information, contact IBI’s
Publications and Graphics Support Project at 703-525-2224 or pgsp@ibi-usa.
2 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONTENTS
PREFACE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
DESCRIPTION OF MINI SURVEYS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Advantages and Limitations of Mini Surveys . . . . . . . . . . . . . . . . . . . . . . 2
When Are Mini Surveys Appropriate? . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
PLANNING A MINI SURVEY: SEVEN STEPS . . . . . . . . . . . . . . . . . . . . . . . . 3
PREPARING MINI SURVEY QUESTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Wording and Length of Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Open-Ended and Closed Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Making Questions Specific . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Wording Questions to Aid Recall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Avoiding Loaded Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Wording Sensitive Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
DESIGNING THE QUESTIONNAIRE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Putting Questions in a Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Length and Format of Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Pretesting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
SELECTING RESPONDENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Probability Versus Informal Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Probability Sampling Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Informal Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Sample Size for Mini Surveys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
ADVICE FOR INTERVIEWERS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Initial Contact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Using the Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Techniques for Getting Responses to Questions . . . . . . . . . . . . . . . . . . 23
Recording and Editing the Interview. . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
ANALYZING AND PRESENTING THE SURVEY DATA . . . . . . . . . . . . . . . 25
Coding Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Statistical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Presenting Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES i

PREFACE
This monograph was originally published in 1990 in the Development
Information and Evaluation Series on program design and evaluation
methodology. Because of continuing high demand, it has been re-
issued. This edition contains a new section, researched by Development
Associates, on software that can be used for mini surveys.
I hope that the monograph will continue to be used by development aid
practitioners, particularly those engaged in conducting evaluations and
field assessments.
I wish to acknowledge again the invaluable advice, comments, and sug-
gestions in writing the monograph that I received from Kurt Finsterburch
of the University of Maryland, Michael Hendricks of Hendricks &
Associates, Charles R. Perry of the U.S. Bureau of the Census, and former
colleagues John Eriksson and the late Annette Binnendijk.
I am also grateful to Janet Kerley for her contributions to this version
and to Hilary Russell of International Business Initiatives for her editing.
Krishna Kumar
Senior Social Scientist
USAID
ii USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

DESCRIPTION OF MINI SURVEYS multitude of variables. Such surveys, undoubt-
edly costly and time-consuming, require the
[T]he extensive questionnaire survey with the 30
support of an efficient organization. However,
pages of questionnaire (multidisciplinary, each
discipline with its own questions), which if asked mini surveys can be done on a smaller scale;
are never coded, or if coded never punched, or if they concentrate on a few variables and use a
punched, never processed, and if processed and small sample.
printed out, never examined, or if examined, never
Mini surveys focus on a narrowly defined issue,
analyzed or written up, or if analyzed and writ-
ten up, never read, or if read, never understood question, or problem. The kinds of questions
or remembered, or if understood or remembered, they address include the following:
never used to change action. Rural surveys must be
• What proportion of targeted farmers are using
one of the most inefficient industries in the world.
the recommended technical package?
Robert Chambers, 1981
• How do project participants evaluate the
A survey of 20 respondents is better than no survey. services provided by a microenterprise develop-
For example, in the absence of a survey one only ment project?
has hunches about who is for or against vari-
ous policy alternatives facing a decision maker. • Are most farmers willing to pay user fees to
The sample of 20 at least gives a rough idea of access necessary health facilities?
what people are thinking and is better than one’s
hunches. The number of questions is deliberately kept
small—usually only between 15 and 30. In this
Kurt Finsterbusch, 1976
respect, mini surveys differ significantly from
Surveys are the most widely used method of traditional household or agricultural surveys,
data collection. In popular belief, they are whose questionnaires may run into several
synonymous with social research. This is not pages. Mini survey questionnaires are designed
without justification, since analysts usually rely to be completed within half an hour, at most.
on surveys in one form or another when doing To save time and resources, sample size is also
basic and applied research in social, economic, kept small, usually 25–70 cases. This is the most
political, and cultural subjects. important characteristic of mini surveys, one
that distinguishes them from large socioeco-
A survey requires direct collection of informa- nomic surveys.
tion from individuals. The basic element is a
structured questionnaire. In the developing In addition, mini surveys largely (though not
world, most surveys are administered in person, exclusively) use closed questions. Such questions
by telephone, or through the mail. Web-based list major response categories, and respondents
surveys are less frequently used. Respondents identify one category or more that they consider
are carefully selected, generally—although not appropriate. The essential idea is to quantify
always—on the basis of probability sampling. responses so statistical analysis can be done rap-
Responses gathered from questionnaires are idly. In this respect, mini surveys differ from key
coded and statistically analyzed to yield findings informant interviews or informal surveys that
and conclusions. use open-ended questions.
In popular perception, surveys are large investi- For mini surveys, the use of probability sam-
gations that involve hundreds (even thousands) pling is preferred. Each unit in the population
of respondents, and they generate data on a has an equal chance of being selected, and the
CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 1

sample is representative. Though this kind of being gathered. Interview and coding errors
representative sampling is efficient and ensures are reduced because samples are small and
unbiased findings, it may not be feasible questions are few. The overall quality of data
because of time and resource constraints. When thus tends to be satisfactory.
this is the case, informal sampling procedures
• Mini surveys are generally managed at rela-
are acceptable. However, because informal sam-
tively low cost. Manpower requirements are
pling is based on convenience and individual
minimized because of small sample sizes and
judgment, results can be biased.
relatively brief questionnaires. A mini survey
interviewer does not require much outside
ADVANTAGES AND LIMITATIONS
help; he or she can manage with two or three
OF MINI SURVEYS
full- or part-time assistants.
Mini surveys have several advantages:
Notwithstanding, mini surveys have limitations
• Unlike other rapid, low-cost data collection
that should be carefully weighed.
methods, mini surveys generate qualitative
data from sources such as key informant inter- • Compared to large surveys, it is harder to
views, rapid rural appraisal, or focus group dis- generalize from mini survey findings. This is
cussions. Such data permit analysts to say, for especially true of mini surveys that did not use
example, that 50 percent of women farmers probability sampling. When this is so, analysts
surveyed indicated that the technical assistance cannot be sure that the sample is representative
provided by the project was valuable, or that and cannot compute the sampling error. Even
40 percent reported that their incomes had experienced researchers can make mistakes
increased because of their participation in the when they rely on informal sampling.1
project. However, qualitative data-collection • In many instances, small sample sizes do
methods do not generate quantitative data. not permit elaborate statistical analysis. For
• Because mini surveys can be completed within example, if only 8 out of 50 farmers in the
three to seven weeks, they are the only prac- sample are women, the analyst cannot make a
tical alternative when quantitative data are comparative analysis of the behavior of male
needed but there is insufficient time to mount and female farmers.
a comprehensive survey. If an evaluation team • Credibility is always a problem. Many policy-
has only about four weeks in the field to assess makers and decisionmakers consider findings
the impact of a microenterprise project, it from mini surveys unreliable—not without
cannot launch a comprehensive survey of local justification—because of their small sample
entrepreneurs targeted by the project. How- size. The remark is often heard, “So, you are
ever, the team can easily design and implement generalizing about the whole project on the
a mini survey that can produce reasonably basis of 35 respondents!”
credible data for the evaluation.
1
• The number of nonsampling errors tends to This occurred when interviewers questioned available
heads of households for a survey of rural households in
be low in mini surveys. Each requires only a Lesotho. Many men in these villages worked in the South
few interviewers, who tend to be better trained African mines, and their households enjoyed higher incomes
than other villagers. But because only available heads of
and supervised than interviewers used for large households were interviewed, those working in South Africa
surveys. Mini survey interviewers also tend to were underrepresented in the sample. Thus, the findings of
the otherwise carefully planned survey were undoubtedly
have a better grasp of the small volume of data inaccurate.

WHEN ARE MINI SURVEYS
APPROPRIATE? Box 1. Three Examples of Appropriate Mini Surveys
Mini surveys are not substitutes for large, care-
fully designed, and efficiently implemented • A mini survey was conducted to test the effectiveness of agricultural
extension in a South Asian country. A sample of 60 farmers was ran-
surveys that study complex social and economic
domly selected in three villages; 30 were contact farmers (who were
subjects. When rigorous and reliable data from receiving extension advice) and 30 were noncontact farmers. The find-
heterogeneous populations are needed for major ings did not show any significant differences in adoption behavior: more
policy or program initiatives, large sample sur- or less the same percentage in each group adopted the recommended
veys may be indispensable. variety of wheat. The explanation was that the government had been
promoting the new variety for the past six years. Most farmers had
Mini surveys are appropriate when limited become fully aware of its advantages and limitations and did not require
time and resources do not permit or justify the much extension advice.
launching of a large sample survey. Their data • A private voluntary organization undertook a mini survey in an African
are often satisfactory in project and program country to learn whether hospital clients would be willing to pay to use
settings. Mini surveys may be extremely useful the facility. A sample of 60 respondents representing six villages were
when conducting feasibility studies, preparing asked to identify factors they would consider in choosing between a
paying and a nonpaying hospital. The survey revealed that fees were not
project papers, assessing beneficiary responses,
a significant determinant. Instead, income, the quality of the facility, and
and preparing final and impact evaluations. its proximity were the factors that influenced their choice.
Within such contexts, analysts are more inter-
• In a Southeast Asian country, an evaluation team reviewed the impact
ested in broad patterns, trends, and tendencies
of a large education project on the quality of primary schooling. Though
than in precise measurements. For example, the team had access to the project over seven years, they sought the
in evaluating an agricultural project for small views of experts and decisionmakers. They therefore designed and
farmers, it is often immaterial if the beneficiary conducted a survey of 35 educationists, experts, and administrators
approval rating is 60 or 63 percent; the differ- familiar with the local educational system. Each was given a structured
ence of 3 percent will hardly affect the conclu- questionnaire that took about 15 minutes to complete. The survey was
completed within 10 days. The findings confirmed the documentary
sions and recommendations of the evaluation
evidence that the project contributed to improving the quality of
team. primary schooling.
Mini surveys can be used to develop questions,
hypotheses, and propositions for further testing.
They can be a prelude to more comprehensive, PLANNING A MINI SURVEY:
large-scale surveys, and their information may SEVEN STEPS
help to sharpen study questions, design relevant 1. Formulate study objectives: The first step in
questionnaires, and develop sampling strategies. planning a mini survey is to formulate precisely
Mini surveys are also useful when quantita- the mini survey’s objectives. This is done by list-
tive data are needed to supplement qualitative ing study questions, which, in most cases, are
information. For example, a mini survey could stated in the scope of work prepared by the con-
be used to measure perceptions of beneficiary cerned USAID office. The person designing the
farmers when a USAID manager wants further survey should discuss with the office’s staff any
confirmation of conclusions emerging from key questions that are unclear or unspecific. This
informant interviews about agricultural inputs sharpens the focus of the survey and helps avoid
supplied by private traders. Box 1 contains any possible misunderstanding. A few precisely
other examples of appropriate situations for formulated study questions determine what is
conducting mini surveys. and is not to be covered by the survey. During

the planning stage, there is always a temptation guidelines for conducting interviews need be
to seek more information than can be realisti- developed at the outset that cover such topics
cally used by managers and policymakers. The as initial contact, questioning methods, and
focus on study questions curbs this temptation. recording and editing interviews. The question-
naire should also be pretested and revised in
2. Review the literature: The next step is to con-
light of findings.
duct a review of existing information, including
project or program records and documents, 5. Choose the type of sampling: Survey design-
published and unpublished studies, and statisti- ers then need to choose between probability
cal data available through public and private sampling and informal sampling. They must
agencies. An effort should also be made to also decide on the specific sampling technique.
review earlier surveys on similar and related top- Though probability sampling is preferred,
ics. Such a review will provide valuable informa- it may not be feasible because of time and
tion on substantive issues and generate a list of resource constraints. If so, the limitations of
questions that can be used in planning the mini informal sampling should be recognized and
survey. Moreover, the literature review can alert precisely stated in the report.
survey designers to potential conceptual, meth-
6. Choose the mode of contact: Survey designers
odological, and logistical problems, and may
need to determine how the questionnaire will
even reveal data or information that make a new
be administered, though the only viable method
survey unnecessary. Many agencies and organi-
in developing countries may be the individual
zations gather data with scant attention to each
interview. Mail, the simplest and least expensive
other’s efforts: it is not uncommon to find sev-
means of contacting respondents, is not practi-
eral surveys being conducted by different agen-
cal. One reason is that literacy rates are usually
cies on the same topic. As a result, there is often
very low, especially among populations that are
too much information, rather than a shortage of
the subject of development interventions. A
it, in many developing countries.
second reason is that response rates for mailed
3. Prepare survey questions: Interview questions questionnaires tend to be poor—even among
should now be prepared, keeping in mind the literate people—especially if there is little or
study objectives. This task is not simple. It no incentive to complete and mail them. In
requires careful reflection because the wording, the developing world, telephone interviews and
length, and open or closed nature of a question web-based surveys are also out of the question,
can all significantly affect the responses given. primarily because most people do not have
The recall period is also important. It cannot be access to telephones and computers.
assumed that respondents will easily remember
and report relevant details, however important 7. Analyze the data: The final stage is cod-
those details may seem to survey designers. In ing and analyzing the questionnaire data. The
addition, care must be taken in wording ques- analysis in mini surveys is invariably limited to
tions that may be sensitive in a given social and simple frequencies, percentages, rates, or—at
cultural milieu. most—simple correlations.
4. Design the questionnaire: The next logical step

is design a carefully crafted short questionnaire PREPARING MINI SURVEY
that can be easily administered to respondents. QUESTIONS
All questions should be arranged in a logical Two priests were debating whether it was right to
sequence to facilitate the interview. Practical smoke during prayers. Both marshaled all kinds of

arguments without coming to an agreement. They text. This question would be better rephrased
decided to consult their superiors and meet the next as “Should technical assistance provided by
day. the project be continued after external funding
When they met, the prosmoking priest said: “My ends—that is, should technical assistance be
superior told me that it was all right to smoke.” sustained?”
“How could it be?” replied the antismoking priest,
Often words have multiple meanings. For exam-
“My superior was emphatic that it was wrong.
ple, any, anyone, anybody, or anything may mean
What did you ask him?”
‘every’, ‘some’, or only ‘one’; fair may mean
“I asked him if it was alright to smoke while pray- ‘average’, ‘pretty good’, ‘not so good’, ‘not bad’,
ing” came the reply.
‘favorable’, ‘just’, ‘open’, ‘according to the rules’,
“That explains it,” said the antismoking priest. “I or ‘plain’; and you can be singular or plural.
asked whether it was all right to pray while smok- Those who design surveys must be extremely
ing.” careful with and limit the use of such words
A survey tale with multiple meanings.
As this tale suggests, preparing good questions Interviewers should use standard language, but
requires more than good language skills. It norms of spoken (rather than written) language
requires a robust common sense and the abil- are appropriate, since questions are read to
ity to empathize, both with the subject and the respondents. Better results may be achieved by
social and economic milieu in which the survey violating the rules of written language; commas,
is conducted. Familiarity with the literature on colons, and other punctuation marks should be
designing survey questions is essential. General avoided if they break the flow of ideas.
guidelines for drafting appropriate questions for Questions should be kept short and succinct. A
mini surveys follow. lengthy question can confuse respondents and
cause them to miss its essential point. Indeed,
WORDING AND LENGTH OF
the reliability of responses declines as the length
QUESTIONS
of a question increases. This is particularly true
Words used in phrasing survey questions should
when questions address opinions, judgments, or
be simple, widely understood, and have precise
attitudes. However, when respondents are asked
meanings. Slang and colloquialisms should be
to recall events that happened long ago, length-
scrupulously avoided. Many respondents may
ier questions may be helpful. Such questions
not understand them, and this will cause both
not only provide memory cues and aid recall,
embarrassment and errors. For the same reason,
but the time they take to read permits reflection
technical terms should not be used unless most
and improves the accuracy of responses.
respondents are technical experts.
If the word that best describes a relevant behav- OPEN-ENDED AND CLOSED
ior or concept is not understood by respon- QUESTIONS
dents, the ideal course is to begin with an expla- The choice of open-ended or closed questions
nation of the word before using the word itself. requires careful attention. Open-ended ques-
For example, the question “Should the technical tions enable respondents to tell their stories in
assistance provided by the project be sustained their own words. The interviewer reads a ques-
over time?” may confuse those not familiar with tion and tries to record the answer verbatim.
the word “sustained” in the development con- Closed questions list major response categories,

emerge. For example, when responding to an
Box 2. Examples of Open-Ended open-ended question about credit, a respondent
and Closed Questions may say that he prefers commercial banks, but
borrows from a rich uncle when interest rates
During the last year, did you borrow money are high. Such a statement will not be produced
for your business? ❑ Yes ❑ No by a closed question.
If the answer is yes, then
But open-ended questions have several limita-
Open-Ended Question tions. The data they generate are difficult to
Please tell me the source or sources from code, making this task more costly and time
whom you borrowed money consuming. Their data are also more prone to
errors, since coders have to interpret answers
____________________________________
and then classify them in appropriate categories.
____________________________________
_________________________________________________________________________________ Two coders may code answers to one question
________________________________________________________________________________ differently, increasing the likelihood of bias.
Open-ended questions also require more time:
Closed Question
respondents take more time to think and verbal-
If the answer is yes, then ize their responses.
Who among the following loaned money One persistent problem with open-ended ques-
to you?
tions is that interviewers are inclined to edit
1. Relatives and friends ❑ answers, omitting portions that do not make
sense, condensing them to fit the space provided
2. Money lender/trader ❑
in the questionnaire, or even elaborating points
3. Cooperatives ❑ considered to be unclear. Such editing results in
4. Commercial banks ❑ inaccuracies and distortions. Interviewers need
considerable discipline and skill to take verba-
5. Others ❑
tim notes and systematically question respon-
dents whose answers are inadequate.
In sharp contrast, closed questions are easy to

ask, still easier to record, and do not require
and respondents simply identify one or more highly skilled interviewers. Many respondents
they consider the most appropriate (box 2). also find them less taxing than open-ended
questions because closed questions do not
Open-ended questions encourage spontaneity require as much recall of detail or contempla-
and freedom; respondents can use their own tion. Moreover, the response categories of closed
language, concepts, and analytical categories. questions aid recall. For example, the listing
Free expression makes an interview interesting, of consumption items in expenditure surveys
and it often generates insightful quotes and rich helps respondents to recall items the household
vignettes for survey reports. might have purchased within a specified time.
The coding of closed questions is also simpler,
Open-ended questions enable respondents to less time consuming, and less likely to contain
provide additional details or to qualify their coding errors. And because the same response
statements, and unanticipated findings may categories are used, the data are comparable.

But closed questions lack all the advantages of may be inaccurate. If, for example, a question
open-ended questions. They inhibit spontaneity is asked about the sources of prenatal care, all
and force respondents to choose from response important local sources available—private pre-
categories with which they may not even agree. natal clinic, public prenatal clinic, doctor, tra-
Because closed questions do not provide ample ditional midwife, and so on—should be listed.
time for reflection and recall, responses may be This is necessary because respondents tend to
superficial; standardization may be achieved at choose from given categories; they do not sug-
the cost of oversimplification. gest new ones unless they know the subject and
feel strongly about it.
Most questions in a mini survey should be
closed, and there should be only limited use of When there are more than five possible respons-
open-ended questions. If most questions cannot es, it is usually preferable to list them on cards.
be answered using a closed format, other rapid However, this is not practical when illiteracy is
data-collection methods should be substituted, widespread. If it is, the interviewer should slow-
such as key informant interviews, group meet- ly read each category and seek the appropriate
ings, or focus group discussions. response—for example, yes, no, not applicable,
In mini surveys, open-ended questions should or don’t know. The question “How do you get
be restricted to three situations: to market?” might have more than 10 response
categories (walk, bicycle, truck, cart, train,
1. When the issue is why and how: If a survey on bus, car, taxi, horse, mule, other). Interviewers
credit wants to know the reasons for respon- should slowly read each category and ask for a
dents’ preferences for various credit sources, a yes or no response. However, the best option
closed-question format may be unduly restric- may be to avoid more than four or five response
tive. categories and provide an “anything else” cate-
2. When response categories are large or unknown: gory. This procedure cannot be used when ques-
If recommendations for improving the tions require rating an event, behavior, or opin-
operation of a health project are sought, an ion. For example, in an evaluation of a health
open-ended format will be more informative. program, respondents may be asked to rate, on
It is difficult to determine in advance the a scale of 1 to 7, the quality of care provided.
specific recommendations that respondents
may offer. Indeed, a closed-question format
MAKING QUESTIONS SPECIFIC
Questions should be as specific as possible.
may cause some respondents to agree with
recommendations they had not previously Respondents understand and respond better to
considered. specific questions, and they tend to interpret
general questions differently, depending on their
3. When respondents may have no information or backgrounds and experience. Take the ques-
opinion on an item: In these situations, the tion: “What has been the impact of structural
closed-question format may produce inau- adjustment programs?” Economists are likely
thentic answers and inaccurate data. to answer by referring to inflation, balance of
payments, economic growth, and gross national
Responding to Closed Questions income. Social scientists are inclined to focus
Response categories for closed questions need on living conditions, economic inequalities, and
to be exhaustive. If the full range of possible the availability of social services. And politi-
answers are not included, the resulting data cal scientists may refer to institution building,

social unrest, and political instability. Though pened long ago. Recall of multiple events poses
responding to the same question, respondents additional problems because respondents often
are addressing different things. But when the confuse them.
question is restricted to a specific sector (for
Despite these problems, many surveys contain
example, “How have structural adjustment pro-
questions that require vivid recall of events and
grams affected the balance of payments situa-
behaviors. A draft household survey question-
tion, inflation, and economic growth?”), respon-
naire prepared by the World Bank to study the
dents focus on the same issues and the resulting
social dimensions of structural adjustment in
data will be comparable.
Africa provides an interesting example. One sec-
Specific questions will also assist recall. If tion listed dozens of consumption items—rang-
separate questions are asked about the social, ing from newspapers to razors, soap to taxi
economic, and political effects of structural fares, and cooking oil to coconuts—and asked
adjustment, respondents are likely to mention a common set of questions about each of the
items that they might have otherwise forgot- items (box 3).
ten. Listing specific items will help refresh their
Unless respondents kept elaborate records of
memories.
their expenditures or were gifted with excep-
tional memory, most were unlikely to respond
Avoiding Double-Barreled Questions
Sometimes interviewers combine two or more
issues in one question. Consider the question: Box 3. Questions about
“Do you think that the government should Consumption Items
provide credit to farmers at affordable rates and
The following questions were asked for each
assist them in getting the improved variety of
item listed in the World Bank’s draft question-
maize seed at subsidized prices?” Respondents naire.
may be confused when more than one ques-
tion is being explicitly stated or implied. In 1. Was anything spent on ____ during the last
12 months?
addition, they may agree with one part of the
question but not with the other. In the example, 2. Do you buy ____ once a week or more?
the respondent who does not favor the gov-
3. How many times was ____ bought in the
ernment providing credit but wants seeds at last 12 months?
subsidized rates will not know how to answer.
Often respondents answer only the first part of 4. How much was spent on ____ the last 12
months altogether?
the question and ignore the rest. Interviewers
should be careful to provide separate questions 5. How much was spent on ____ in the last 3
for each issue being examined. months?
6. Were there months when you did not
WORDING QUESTIONS TO purchase this item?
AID RECALL
Interviewers must pay particular attention to 7. In how many months did you not buy this
item?
questions that require recall. Lapses of memory
are more common than many realize. People 8. How much was spent on ____ since my
tend to forget behaviors or events, especially last visit?
those that seem trivial to them or that hap-

accurately. How many of us can remember the earlier or focus on more recent events, omitting
amount we spent on razors or cooking oil dur- others that should have been included.
ing the past 12 months?
AVOIDING LOADED QUESTIONS
Three general strategies can be followed for Loaded questions are those likely to push
questions that require recall: respondents to answer in a certain way. Such
1 Narrow the reference period. As much as pos- questions undermine the validity of survey
sible, questions should focus on the recent data. Expressions such as “Don’t you agree?” or
past. For example, an interviewer is more “Wouldn’t you say?” push respondents to give
likely to obtain an accurate answer to the ques- affirmative answers. Because of the outright
tion, “How many times did you see an exten- suggestion, respondents may feel obliged to
sion agent last month?” than to the question, agree, even if they have reservations about the
“How many times did you see an extension statement. In societies where it is considered
agent last year?” Longer reference periods can impolite to disagree, especially with outsiders,
be used when the question is about a topic of loaded questions can be particularly damaging.
great importance to the respondent, such as For example, the question, “Wouldn’t you say
the purchase of a major piece of agricultural that the agricultural extension program has ben-
equipment or construction of a house. But the efited farmers?” is likely to evoke a more posi-
guiding principle is that it is better to refer to tive response than the question, “What was the
days and weeks than to months and years. impact of the agricultural extension program on
farmers?”
2 Ask for average—not specific—time spans: the
prevalent norm, not a particular incident or Using emotionally charged words, clichés, and
behavior. For example, in expenditure surveys, appeals to self-respect can also influence the
interviewers should ask, “How much meat do respondent. Consider the wording of two load-
you buy every week?” rather than “How much ed questions for a survey of smallholder farmers
meat did you buy last year?” Respondents in a developing country:
seem to give relatively accurate information • “The insurgents who are fighting the govern-
about the average or norm. In many instances, ment believe that farmers should own the land
questions concerning a short reference period they cultivate. Do you agree with them?”
can address both the norm and actual behav-
ior. • “The government of this country believes that
farmers should own the land they cultivate.
3 Use landmark or important events as a reference Do you agree with its position?”
to refresh the respondent’s memory. Instead of
asking, “Have you seen an extension worker The second question will evoke more positive
during the past six months?” interviewers response than the first, and the mere mention
might ask, “Have you seen the extension of “insurgents” will hamper an objective frame
worker since the new year?” Making the refer- of reference.
ence period more concrete will refresh the
More subtly loaded questions evoke the name of
respondent’s memory.
an authority. These questions attribute a state-
However, questions asked about events several ment to an expert, respected leader, or estab-
years in the past tend to be confusing. Respon- lished institution, then ask the respondent to
dents may then include events that happened agree or disagree.

Box 4 presents some examples of loaded ques- maize seed being vigorously promoted by the
tions directed at entrepreneurs being served by a government, and some educated mothers may
microenterprise project. Such questions encour- be reluctant to admit that they visit traditional
age responses that show the project in favorable healers when their children are sick. In addition,
light. With a few possible exceptions when data people do not like to divulge information that
are being gathered for sensitive topics, loaded they believe will damage the image of an organi-
questions should be avoided. Intelligent respon- zation or an individual. Interviewers have to be
dents easily see through such questions, which extremely careful in wording sensitive questions
will further undermine the credibility of the to obtain accurate answers.
survey.
Five strategies can be employed when sensitive
WORDING SENSITIVE QUESTIONS questions must be asked:
In project and program settings, interviewers • Questions can convey the impression that the
may have to ask sensitive questions—about behavior or incident in question is not unusu-
people, organizations, or the respondents them- al. A question about visits to traditional healers
selves. However, some respondents who do could begin with the following: “Experience
not like to answer such questions directly will has shown that even scientists, doctors, and
either evade them or give inaccurate answers. highly educated people consult traditional
For example, some farmers may not like to dis- healers in the wake of family illness. Was there
close that they are not using the new variety of any time during the past six months when you
went to see one?” By stating that educated and
respected people also visit traditional healers,
Box 4. Examples of Loaded
the interviewer minimizes the implicit threat
Questions for an Impact Evaluation
of the question.
1. Was it not an excellent idea to locate the
• A question can make the assumption that
microenterprise project in this district?
the particular behavior or event happened or
2. Experience shows that such projects happens. For example, an interviewer can ask,
contribute to the growth of small business “How many times did you visit the traditional
enterprises. Wouldn’t you say that this
healer during the past six months?” This ap-
project had similar effects in this district?
proach is often used in surveys. Because the
3. Do you disagree with the prevalent view assumption may be resented by respondents,
that the project has helped many small en- a provision should be made for a “none”
trepreneurs by providing subsidized credit?
category. A more important problem is that
4. Several experts believe that small busi- the question may lead to overreporting of the
nesses in the area would have faced many behavior or event; for example, even mothers
technical and financial problems in the ab- who had not visited traditional healers may say
sence of such projects. Wouldn’t you agree
that they did.
with this assessment?
5. Are you in favor of continuing the project • The name of an authority whom respondents
so that both credit and technical assistance are likely to trust can be cited in a sensitive
are provided to local businesses, thereby question, as in the following example: “Now
improving the economy? the Ministry of Industry concedes that there
have been serious problems with the current
technical assistance program, particularly

with its extension activity. Did you face any PUTTING QUESTIONS IN A SEQUENCE
problem in obtaining technical assistance from
the project?” By mentioning the Ministry of The First Question
Industry, the interviewer seeks to reassure the The first question should be simple and non-
respondent that the problems are recognized threatening, but also important. It should
at the highest levels and talking about them is stimulate the respondent’s interest in the survey,
acceptable. since boring or complex questions asked at the
outset will adversely affect the respondent’s will-
• Interviewers can minimize the sensitivity of ingness to cooperate.
what is being asked with a phrase such as “Did
you happen to...?” The tone as well as the An interviewer might consider starting with an
wording of such questions is important. open-ended question on an important issue.
These tend to more interesting to respondents
• A question can advance reasons for a respon- and allow them free expression. However, such
dent’s behavior. One example: “Many things questions are helpful only if respondents are
that are beyond one’s control can come in the well-informed and articulate; they may make a
way of making regular payments to a lend- respondent who is neither feel threatened.
ing institution. Was there any time during
the past 12 months when you were not able
Demographic Questions
to make your payments to the agricultural
At the beginning of the questionnaire, many
credit bank?”
interviewers routinely include demographic
questions about a respondent’s age, employ-
DESIGNING THE ment, marital status, and even religion. For
QUESTIONNAIRE mini surveys, such questions are generally super-
The same care and thought that are given to the fluous: demographic variables are rarely used in
wording of individual questions must be given to analyzing their data. Demographic questions
the construction of the total questionnaire. The thus not only take time from the interview but
tasks of both interviewers and respondents should they intimidate respondents who do not like
be made as easy and enjoyable as possible. The to divulge such information. As a general rule,
respondent, after all, is doing you a favor; and a demographic questions should be avoided,
well-designed questionnaire makes the interview- except when required for analytical purposes.
er’s job easier and improves the quality of data If required, they should come at the end of the
obtained. questionnaire.
Seymour Sudman and
Norman M. Bradburn, 1988 Funnel and Inverted Funnel Sequences
A funnel sequence in a questionnaire moves
After a set of appropriate questions is crafted, from general to increasingly specific questions.
the next task is to present them in a short ques- Funnel sequences are especially useful when
tionnaire. This task is not as simple as it first interviewers want to learn quickly about a
appears. It involves arranging the questions in a respondent’s perspective or frame of reference.
logical sequence, developing a suitable physical
format for the questionnaire, and pretesting the In a funnel sequence concerning the privatiza-
questionnaire to identify and resolve problems. tion of agricultural input supply, the first and
Each task requires careful reflection and analyti- most general question might seek opinions
cal skills. about privatization in the agricultural sector

(box 5). The next question would be slightly subject and cannot give thoughtful answers to
restrictive; it would concern privatization of general questions. The advantage is that respon-
the type of parastatal that supplies agricultural dents can think through a topic before verbal-
inputs to farmers. The third question would izing their responses. Both funnel and inverted
have an even narrower focus: perhaps progress funnel sequences can be used in mini survey
toward privatization that had been made by a questionnaires.
specific parastatal. The final question would
be the most specific of all, seeking to discover Chronological Order
how satisfied respondents were with the pace of Interviewers should ask questions that address
privatization for a parastatal. historical events in chronological or reverse
The inverted sequence reverses the funnel chronological order. For example, questions
sequence: the interviewer asks a specific ques- about respondents’ experience with technical
tion first, followed by increasingly general ques- assistance provided by a microenterprise project
tions. This is useful if an interviewer believes may begin with the most recent experience and
that respondents have not considered the work backward to earlier periods, or vice versa.
Chronological order is helpful in aiding recall;
it forces respondents to describe the sequence in
the time period under consideration.
Box 5. Example of a Funnel Sequence on Privatization
of Agricultural Input Supply Changing Topics
Often, a mini survey covers more than one
1. How is the privatization program going in the agricultural sector? topic. For example, a survey designed to exam-
__________________________________________________________ ine the impact of an international training pro-
__________________________________________________________ gram is likely to include questions on the selec-
__________________________________________________________ tion of trainees, overseas training experience,
_____________________ reentry, placement of trainees, and the contribu-
2. Do you think that the government has made progress in transferring tion of graduate trainees to institution building.
ownership of agricultural input supply parastatals to the private sector? The simple rule is that all relevant questions on
a topic should be grouped together. For exam-
❑ Yes ❑ No ❑ Don’t know ple, the questionnaire will group all questions
3. What progress has been made in privatizing the fertilizer corporation on the selection process or the reentry of train-
that markets fertilizers to farmers? ees in one place in logical order. A short, tran-
__________________________________________________________ sitional sentence can help interviewers switch
__________________________________________________________ topics. One example is, “So far, we talked about
__________________________________________________________ the working of the child survival program. We
_____________________ will now ask questions about its impact.” This
4. Are you satisfied with the progress of privatization in the fertilizer helps to lead respondents to the next topic.
corporation?
LENGTH AND FORMAT OF
__________________________________________________________ QUESTIONNAIRES
__________________________________________________________
Mini survey questionnaires must be short and
__________________________________________________________
_____________________ succinct. They should contain between 15 and
30 questions and take no more than 30 minutes
to complete. Instructions on making introduc-

tions and asking questions should be included, the effort produces a better translation and
along with instructions on recording answers fewer errors.
and, when appropriate, recording the nonverbal
Time permitting, the draft translation can be
behavior of respondents.
given to another local expert to translate back
Guidelines for the physical format of the ques- into English. The comparison of the two ver-
tionnaire are relatively simple: sions will help identify possible errors, which
can then be corrected by the translator.
• A booklet format is preferable, since loose
sheets can be easily lost or misplaced. PRETESTING
• The questionnaire cover page should provide The manager of a mini survey should carefully
space for the name of the interviewer; the pretest the draft questionnaire by conducting
name and address of the respondent; and the between 5 and 10 interviews. The number will
time, date, and place of the interview. depend on the complexity of the questionnaire
and its target population. Those interviewed for
• Each page should be numbered, and each pretesting must have backgrounds and experi-
question should be numbered on the left ence similar to intended respondents. For exam-
margin. ple, if the questionnaire is designed for entre-
• Plenty of space to record answers must be preneurs receiving technical assistance from a
provided. Economizing on paper is not pro- project, only this type of entrepreneur should be
ductive. Leaving only two or three lines for included in pretesting. When a survey is likely
recording responses to open-ended questions to cover many categories of respondents, at least
forces the interviewer to condense responses, one respondent from each category should be
which will undermine the validity of the data. included.
Questionnaires can be created on a personal In pretesting and assessing individual questions,

computer. It is more economical to photocopy particular attention should be paid to the fol-
than to print mini survey questionnaires because lowing issues:
of the relatively small size of the sample. • Is the meaning of the question clear to respon-
dents? Because a question has been carefully
Translation prepared does not mean that it will be cor-
Often, questionnaires prepared in English must rectly interpreted. Despite an interviewer’s
be translated into a local language. Because best efforts, misinterpretations occur because
errors in translation can distort the meaning of of conceptual and linguistic barriers between
questions and result in inaccurate data, trans- interviewers and respondents. Intelligibility
lations should be done by persons fluent in can be easily determined by reviewing answers
English and the local language who have strong and asking respondents how they interpreted a
backgrounds in survey research. given question.
The survey manager must brief the translator • Do respondents have difficulty in answering the
in considerable detail about the survey’s overall question? Because the meaning of a question is
objectives. This means explaining each question, clear does not ensure it can be easily answered.
its rationale, and the type of information it is In their enthusiasm for obtaining informa-
supposed to generate. The time spent on such tion, many interviewers overlook the problem
explanations is more than rewarded because involved in accurately answering a question.

For example, an expenditure survey asked probability sampling and informal sampling
heads of households to state how many yards procedures are used. The description offered of
of cloth their families had purchased during these tools is both brief and elementary. Readers
the past 12 months. Obviously, only a few who do not have a background in sampling
could answer this question, and their replies theory are strongly advised to consult experts
were suspect. If respondents do not answer a before finalizing their sampling strategies.
question or take considerable time to answer,
the question should be reconsidered. The essential concept underlying sampling
is that large groups of people, organizations,
• Are response categories appropriate? Often, inter- households, or other units can be accurately
viewers find that some are superfluous or that examined by carefully scrutinizing a small num-
additional categories are needed. ber of the group. A formula is used to draw
inferences from the sample for the whole popu-
• Is there an acceptable variety of responses to the
question? Interviewers should suspect the use- lation. The small group is called a sample, and
fulness of a question when all respondents give the large group is called a population or uni-
the same answer. For example, if all respon- verse. Thus, for example, all small entrepreneurs
dents say they have benefited from the inter- who have received technical assistance from a
vention, the question should be reconsidered microenterprise project are called the population
and possibly revised. One option is to use four or universe; those selected for interviews for the
response categories (very much, a fair amount, survey are called the sample.
little, or not at all) to obtain more precise
answers. PROBABILITY VERSUS INFORMAL
SAMPLING
During pretesting, the questionnaire needs to be In probability sampling, each unit in the popu-
assessed in its entirety, paying particular atten- lation has an equal chance of being selected
tion to the following: for the sample. The selection of units for the
• Does it read smoothly? The flow is important sample is carried out by chance procedures, and
because the questionnaire will be read by the with known probabilities for selection. Informal
interviewer, not the respondent, in most cases. sampling, on the other hand, uses convenience
or common sense rather than mathematical rea-
• How much time does it take to administer? soning. For example, an interviewer selects 30
Pretesting helps discover the answer to this farmers who are available for interviews or visits
question. If it takes more than 30 minutes, the 20 health centers regarded as “typical” by the
questionnaire must be shortened. survey designer or other experts.
• Does it retain the attention of respondents? If One widely held misconception about prob-
respondents look bored or indifferent, the ability sampling is that it requires large samples.
questionnaire should be revised by adding or In fact, statistically valid generalizations can be
deleting questions and improving the lan- made with a relatively small sample. A simple
guage. Further training of interviewers may example will illustrate. Suppose key informant
also be needed. interviews indicate that 60 percent of mothers
have acquired the treated bednets that are being
SELECTING RESPONDENTS promoted by the government. The concerned
The selection of respondents is a critical issue USAID official wants to find out if this is true,
for mini surveys. To make this selection, both but will be satisfied if the survey demonstrates

with 90 percent confidence that not less than Data generated by probability sampling are thus
50 percent of mothers have these bednets. In more credible than data derived from informal
this case, a sample of only 39 women users will sampling. Survey findings are more trustworthy
provide the needed evidence. if respondents are selected randomly, rather than
on the basis of personal convenience or judg-
A second faulty assumption about probability ment. However, those in the development field
sampling is that the size of the sample depends often find that time, logistics, and resource con-
on the size of the population, and that larger straints make probability sampling impossible.
samples are required for larger populations. At In such cases, extreme care should be taken to
best, this is only partially correct. Sampling make the sample as representative as possible,
error is determined by several factors, including and the limitations of the sampling method
sample fractions (the proportion of the sample used should be clearly stated in the report.
to the population). But an increase in the sam-
ple size only marginally contributes to a reduc- PROBABILITY SAMPLING METHODS
tion in the sampling error. This means that Four methods of probability sampling can be
the sample sizes needed to study large or small used. The choice should be dictated by the
populations are almost the same. For example, nature of the inquiry; the availability of a list of
the sample size needed to estimate the birth rate population units; time and resource constraints;
in the small country of Lesotho is the same as and, above all, the expertise of the person man-
that needed to estimate the birth rate in China. aging the survey.
As a general rule, probability sampling should
be used for mini surveys, since it minimiz- Simple Random Sampling
es—though does not absolutely prevent—the In simple random sampling, each unit of the
risk of biased selection. If informal sampling is population—whether households, people, orga-
used, certain kinds of people are less likely to nizations, or another grouping—has an equal
be selected. Those excluded for rural household chance of being selected. This type of sample is
surveys, for example, include households that easy to design, and it is quite adequate when the
are inaccessible or remote, those whose mem- population is relatively small.
bers are seasonal migrants or belong to ethnic A simple random sample can be drawn by lot-
minorities, and those with a single member. In tery. Tags bearing names or identification num-
addition, households with high social or politi- bers of all the units in the population are put
cal status may be underrepresented if enumera- into a bowl and thoroughly mixed. A predeter-
tors are intimidated by them. mined number of tags is then randomly drawn.
Although seemingly simple, the lottery method
In addition, probability sampling permits esti-
is cumbersome and time-consuming. Its preci-
mations of the sampling error—or the prob-
sion rests on the assumption that the tags have
ability of error in estimates for a given sample.
been thoroughly mixed.
For example, if probability sampling is used to
estimate the percentage of women who own A better technique is to number all units, then
treated bednets, it can be said with confidence use random numbers to select the sample. Most
that there is only a 5 or 10 percent probability statistical calculators have random-number gen-
that the sampling error will exceed 10 percent erators. If the list runs to a three-digit number,
of the estimate. This estimation is not possible then every unit is given a three-digit number
in informal sampling. (for example, the number 5 is listed as 005),

and then three random digits are run off. If sta- Systematic selection may made from a written
tistical calculators and computer software that list—such a list of farmers who received exten-
can generate random numbers are not avail- sion advice or a list of mothers who obtained
able, a table of random numbers can be used. treated bednets from a project. Systematic selec-
The population unit with selected numbers is tion may also be made from a proxy list, such as
included in the sample. If a random number rows of houses on a street or individual medical
repeats or exceeds the highest number assigned, records in a file.
it is ignored. The process continues until the Systematic sampling is undoubtedly more
desired sample size is reached. convenient than simple random sampling. It
One common problem is that accurate and up- is much easier to take the twelfth name from
to-date lists of populations to be studied are not a document than to number each name and
always available. Before constructing a sample, then draw a sample. If units are listed in an
available lists should be carefully examined and order that shows a steady trend, a reduction in
every effort made to check and improve their sampling error can be achieved. For example, if
accuracy. The time and resources spent on this a list arranges farmers by the sizes of their hold-
task will be more than amply rewarded by the ings or entrepreneurs by how much technical
assistance they received, a somewhat smaller
increased reliability of findings.
sample—say 45 instead of 50—could achieve
Survey managers should not discard units that the same degree of reliability as the larger
“do not look right.” For example, an entrepre- sample.
neur who is randomly selected should not be
There is a danger, however, of hitting a cycle.
excluded because he or she is considered to be
For example, corner houses are more expensive
atypical. If individual discretion is exercised,
in some cities or towns. If the sample interval of
the simple random sample becomes a judgment
houses selected from a map is every tenth house,
sample, and this defeats its purpose. However, if
this may coincide with or oversample corner
there is clear evidence that the entire sample is houses, which are likely to be inhabited by rela-
unusual or peculiar—for example, it draws only tively affluent people. If so, an assessment of the
from one geographical area or social class—the community’s nutritional status that is based on
best course is to discard it and start afresh. data from these households may overestimate
the quantity and quality of food consumed.
Systematic Sampling Lists thus need to be carefully examined before
Systematic sampling involves selecting units a sampling method is chosen.
from a list, but on the basis of a fixed inter-
val after a random start. If a sample of 50 is Stratified Sampling
required from a population of 455 health work- In stratified sampling, the population is divided
ers, this means a sample fraction of 50/455, or into groups called strata. This requires a com-
one in nine units. In systematic sampling, a ran- plete population list, along with additional
dom number between one and nine is used to information on variables that form the basis for
select the first health worker. The fixed interval the stratification. After strata are determined,
is nine, so every ninth health worker is selected independent random samples are drawn from
thereafter. Thus, if the initial random number is each stratum. Stratification is especially appro-
six, the selected health workers will be the sixth, priate when the sample is designed to make esti-
the fifteenth, the twenty-fourth, and so on. mates or comparisons between subgroups and

the entire population. It is appropriate when a method, decisions need to made on how differ-
simple random sampling of the whole popula- ent results for each stratum will be aggregated to
tion may not include a sufficient number of arrive at an overall estimate. The simplest pro-
cases from strata that need to be investigated. cedure is to compute the response rate of each
stratum, multiply it by the number of units in
Strata must be relatively homogeneous: there
the stratum, sum the total for all strata, and
must be less variance within a stratum than
divide the sum by the population total.
between strata. For example, project farmers
could be classified on the basis of the size of
Cluster Sampling
their holdings into three strata—large, medium,
Most of the time, the populations sought do
and small holders—for a survey of adoption
not appear on lists. For example, there is rarely
rates. This classification might be based on the
a list of farmers who received technical assis-
premise that landholding size is related to adop-
tance from an agricultural extension project or a
tion rates. Farmers with larger holdings and
list of women who purchased bednets at village
assets may be in a better position than those
stores. Even if such lists were available, the pop-
with smaller holdings to take advantage of new
ulation units in question are likely to be widely
technical packages. It is also likely that farmers
dispersed, making simple random sampling
with large holdings have greater interaction with
both time-consuming and costly.
extension workers, more contacts, and more for-
mal education. If landholding size turns out to Cluster sampling often provides a practical solu-
be an invalid criterion, another—such as education. It is based on the fact that most popula-
tion, gender, or proximity to the demonstration tion units are clustered in one way or another.
center—could be used to classify the popula- For example, farmers served by extension ser-
tion. The essential point is that the stratum vices live in villages, public health professionals
should be homogeneous. work in organizations, and teachers teach in
schools. While it may be difficult to prepare a
Stratified sampling is of two types. The first
list of all farmers, health workers, or teachers,
is proportional, because strata sample sizes are
lists for sampling purposes can be prepared of
proportional to the strata population sizes. For
villages, public health organizations, or schools.
example, if the proportion of farmers with large
holdings is only 10 percent in the study popula- Cluster sampling is of two types: single-stage
tion, the size of their strata will be 10 percent of sampling, and two-stage or multiple-stage
the sample. The problem, however, is that the sampling.
numbers selected for a relatively small group do
not permit satisfactory statistical analysis. For • Single-stage sampling means the clusters
example, from a sample of 80 farmers, only 8 or are randomly selected and every population
9 are likely to be selected. If comparisons need unit in the selected cluster is included in the
to be made, more units need to be sampled sample. In the case of a survey of agricultural
from the stratum that comprises a smaller pro- extension workers, a project has 50 clusters or
portion of the population. extension units, and each cluster has five ex-
tension workers. If, to save the transportation
This is referred to as disproportionate strati- costs and time, 10 clusters are selected through
fied sampling, since different sample fractions simple random sampling and all the extension
are employed in each stratum. Since there are workers in them are interviewed, single-stage
variations in response rates among strata in this cluster sampling is being used.

• Two-stage or multiple-stage cluster sampling of 600 medical professionals who work in 100
means the sampling is done in two or more medical centers spread across the country, com-
stages. For example, in the survey of agri- pared with the cost of a cluster sample of 70 to
cultural extension workers, 10 clusters are 80 respondents drawn from 10 medical centers.
selected, then three out of five extension work- Cluster sampling also simplifies the interviewing
ers in each are selected for interviews though process and saves time.
simple random sampling.
The major drawback of cluster sampling is the
Cluster sampling has several advantages for mini likelihood of increased sampling error. Units
surveys, especially when the sample is drawn selected in clusters, rather than independently,
from a relatively large geographic area. This may not show the same variation. If cluster
kind of sampling can drastically reduce costs, sampling is used for a mini survey, a slightly
especially when a survey covers a whole prov- larger sample size will be needed than for simple
ince or country. For example, consider the cost probability samples. Usually, a 15–20 percent
of drawing a simple random sample of 60 out increase is sufficient.
Any of the four methods of probability sam-

pling can be used. The choice should be dic-
Box 6. Examples of Probability Sampling Procedures tated by the nature of the inquiry, availability
of the list of the population units, time and
• A team evaluating the overall performance of a project that provided resource constraints, and, above all, the exper-
credit and technical assistance to 327 women entrepreneurs wanted to
tise of the investigator. Box 6 provides some
know whether participants thought the microenterprise development
project was effective and how it had affected their economic condi- examples of probability sampling procedures.
tions. Available records allowed the team to prepare a complete list
of the women entrepreneurs receiving assistance. A table of random INFORMAL SAMPLING
numbers was used to construct a sample of 30 from this list.
Convenience Sampling
• A monitoring unit wanted to learn about differences in credit require- In convenience sampling, the prime consider-
ments for men and women farmers in a development project area. A
ation is accessibility. Only those easily reached
current list of 407 male and 187 female contact farmers receiving ex-
tension advice from the project was available, and a preliminary check by interviewers are included in the sample,
found no inaccuracies. The monitoring unit constructed a sample of 60 which is why many use this method. For exam-
farmers, half of whom were women. To do this, they prepared separate ple, medical researchers often depend on vol-
lists for female and male farmers and selected every 13th male and unteer subjects; marketing firms tend to rely on
every 6th female farmer. Responses from the two categories of farmers people visiting malls or shops; and educators use
were then compared.
their pupils for their surveys. For rural surveys,
• The provincial office of a national maternal health agency wanted to enumerators are often instructed to interview
gather feedback from its staff about its activities. Its 60 clinics, scattered only respondents available when they visit. This
throughout the province, employed 400 full-time and part-time work-
saves time and transportation costs.
ers. The office had neither up-to-date lists of local employees nor the
resources to support visits to a large number of clinics. The investigator Convenience sampling is prone to sampling
thus conducted the mini survey randomly: six clinics were randomly
bias. Often, certain strata, socioeconomic sub-
selected, then 10 workers were randomly selected from each for
interviews. groups, or categories of population units are
inadvertently excluded, underrepresented, or
overrepresented. For example, enumerators who

go to villages during the day will miss farm- The relative accuracy of judgment sampling
ers working in the fields; when they interview depends on three conditions:
farmers buying inputs at cooperative stores,
1. The study population is small enough to allow in-
they exclude those who get their inputs else-
formed judgments to be made about the selection
where—say, from a moneylender. Under such
of sampling units. For instance, if the number
conditions, findings can be wrong, and flaws in
of health centers runs into the hundreds, no
the samples are not recognized.
expert can establish that the samples are repre-
As a general rule, mini surveys should not use sentative.
convenience sampling. If it is unavoidable, 2. The sample size is small enough for judgment
extreme care should be exercised. The following sampling to yield results better than those yielded
steps can reduce sampling biases (box 7): by probability sampling. For example, if officials
1. Identify categories of population units likely to in three districts in a province with 20 hetero-
be eliminated or overrepresented in the sample. geneous districts are being interviewed, better
This can be based on in-depth interviews with results would be obtained if the sample is cho-
local experts, careful review of past studies, sen by an expert, rather than using the vagaries
and a continual review of cases included in the of random chance.
sample. 3. More than one expert is involved in constructing
the sample. For example, an evaluation team
2. Design the convenience sampling to reduce the
conducting a survey of health centers can ask
probability of omission, underrepresentation,
several persons to suggest suitable sites, and
or overrepresentation. For example, arrange to
visit villages when most farmers are likely to be
there.
Box 7. How One Investigator Attempted to Reduce
3. When possible, generate a more representative Sampling Bias
sample by supplementing convenience sampling
A survey was undertaken in remote villages in a South Asian country
with other forms of informal sampling. to discover prevalent attitudes toward immunization among women of
childbearing age. Because of limited time and money, convenience sam-
Judgment Sampling pling was used. Enumerators were expected to visit villages during the day
Judgment sampling uses the judgment or to interview women. The investigator soon realized that the sample was
likely to be highly biased because a substantial proportion of women were
advice of experts or the survey designer to working outside their villages. They were thus likely to be excluded from
construct samples. For example, evaluators of the survey.
extension programs often select farmers in a
The investigator tried to persuade enumerators to conduct interviews in
few “typical” villages for interviews. However,
the late evenings. However, women enumerators refused because they
this judgment may be biased: what is viewed would be required to stay overnight in unfamiliar areas. The investigator
as typical may not be, whether this term is then suggested that at least half the interviews be held in workplaces—
applied to a village, organization, or farmer. farms, shops, and small factories. However, this proved extremely difficult,
This is a serious problem when survey designers and interviewers had limited success. In the end, the investigator selected
additional respondents to make the sample more representative.
unfamiliar with the study population depend
on outside experts who may, consciously or
unconsciously, mislead them.

then include in the sample only those centers Snowball Sampling
for which there is a general agreement. Such a Drawing on the analogy of a snowball (which
course will minimize errors arising from indi- starts small but grows bigger and bigger as it
vidual biases. rolls downhill), snowball sampling begins with a
few population units but increases until it ends
up with the required sample size.
Snowball sampling is performed in several

Box 8. An Example of Successful Snowball Sampling stages. During the first, a few persons who meet
the necessary requirements for inclusion in the
A five-nation study was undertaken in Asia to assess the impact of U.S.-
funded participant-training programs on the professional advancement sample are identified and interviewed. These
of Asian social scientists. The study required a comprehensive survey of respondents are asked to suggest the names
trainees who had received their training in the United States during two of additional persons who meet the sampling
or three decades. requirements and should be interviewed. The
As the study progressed, the investigator found that records for past train- second stage involves interviewing some or all of
ees were missing in most of the countries. Even when some records were the persons identified by the first respondents.
available, addresses were out of date because the social scientists were no The process is repeated until a suitable sample
longer with institutions they joined after completing their training. size is constructed.
Because it was impossible to construct a reliable sample under these Snowball sampling is most appropriate when
conditions, the investigator used snowball sampling. Interviews were begun there is a need is to reach small, specialized
with the few social scientists whose addresses were available. During the
populations that can only be located with great
interviews, each respondent was asked to give names and addresses of
other U.S.-funded trainees. Those suggested were then interviewed and difficulty (box 8). One limitation is that respon-
asked for names of other fellow trainees. By using this method, the investi- dents are likely to suggest persons who share
gator located most of the trainees in each country. The survey succeeded similar backgrounds, lifestyles, and social and
because fellow participants had kept in touch with one another and could professional orientations. Thus, for example, if
provide names and current addresses of colleagues. large landholders are initially contacted, they are
likely to suggest the names of other large land-
holders (box 9).
Quota Sampling
Box 9. An Example of Problematic Snowball Sampling In quota sampling, the population is divided
into various strata, and a predetermined number
An interviewer who used snowball sampling to study the growth of a sci- of people, or quota, is selected for each. The dif-
entific community in a Southeast Asian country was dismayed to discover
ference between quota sampling and stratified
at the end of the fieldwork that the sample was biased and unrepresenta-
tive. The interviewer started with a few scientists who were members of probability sampling is that convenience or the
a clique, and they referred the interviewer only to other clique members. judgment of interviewers—not probability—is
As a result, members of two other prominent factions in the commu- the basis for the selection of respondents within
nity were not interviewed. Clearly, the mistake was also caused by the each stratum. Once quotas are established,
interviewer’s unfamiliarity with the local situation and the failure to cast a interviewers are free to include anyone who
wider net when fieldwork was beginning.
meets the requirements.
As with stratified sampling, quotas can be

established on the basis of age, sex, income,

education, location, combinations of these, or How many variables should be used for estab-
any other criterion perceived to be relevant. For lishing quotas? Obviously not many. The prob-
example, an informal survey of farming practic- lem with using more than three criteria is that
es in a province may assign quotas for different specifications become more complex, and inter-
ecological zones, based on the judgment that viewers find it more difficult to locate respon-
farming practices vary significantly from zone to dents with the requisite characteristics. It is
zone. And establishing quotas for various eco- not always easy to locate women entrepreneurs
nomic strata makes sense if the impact of policy participating in a microenterprise project who
reform interventions on the standard of living are engaged in food-related businesses and reside
is being assessed, since these interventions likely in a certain district. But it becomes still more
have differing effects on each stratum. difficult when the interviewers are also asked to
select three out of six of these women who are
An evaluation of a microenterprise project that under age 35.
provides technical assistance relating to handi-
crafts, garments, and food-related enterprises SAMPLE SIZE FOR MINI SURVEYS
and covers two districts could consider at least For mini surveys, sample size is primarily deter-
three different criteria—gender, geographi- mined by time and cost considerations. If time
cal area, and the nature of the business—and is limited and respondents are scattered over a
assign quotas for each. It could develop a simple wide and not easily accessible geographic area,
matrix (table 1), then try to establish quotas to a relatively small sample size—say, 20 or 30—is
make a representative sample. inevitable. The following factors need to be con-
sidered, in addition to time and cost:
Table 1. Sample Using Three Criteria for • The homogeneity of the population: If the
a Microenterprise Survey population of interest is highly heterogeneous,
District A District B a larger sample is needed than if it is homoge-
Enterprise neous. A mini survey designed to estimate the
M F M F
adoption rate of a particular input by farmers
Garments with widely dissimilar land holdings should
Handicrafts have a larger sample than a survey of farmers
Food who cultivate holdings of similar size.
Total • The number of variables to be examined simul-

taneously: If a survey also wants to find out the
differences in the adoption rates between male
If half the entrepreneurs are involved in gar-
and female farmers or literate and illiterate
ments, efforts would be made to select half
farmers, a larger sample is needed.
the respondents from this category. On the
other hand, if only 25 percent of the busi- • The degree of precision required: While the
nesses are located in district B, only a quarter of relationship between the degree of precision
respondents would be selected from district B. required and the sample size is too complex to
Although the match may not be perfect, every be detailed here, it is safe to say that the goal
effort would be made to select quotas in pro- should be a upper limit of respondents—say,
portion to their size in the population. 70—rather than a lower limit—say, 25.

ADVICE FOR INTERVIEWERS As the excerpt from the field supervisor’s
The farmer’s wife was startled by the sudden notes suggests, preparing thoughtful ques-
arrival of an impeccably dressed, elegant woman to tions and compiling them in a questionnaire
her remote hut when she was trying to start fire on is not sufficient to generate reliable, accurate
a wooden stove. Her hut was filthy, full of smoke data. Interviewers must also present themselves
caused by wet wood. She could read horror on the appropriately, establish rapport with respon-
face of the uninvited visitor, who was undoubtedly dents, ask questions in a manner that evokes
taken aback by what she saw. When the visitor, in
accurate responses, and, above all, accurately
an unfamiliar accent, said, “I have to ask a few
report answers. This section provides general
questions of you,” the farmer’s wife was speechless
for a moment and then asked her son to call his guidance on interviewing techniques.
father. She thought that the visitor had come to
investigate her husband’s drinking habits. INITIAL CONTACT
The first 30 seconds of contact are critical in
Notes of a field supervisor
any interview. During this brief period, the
interviewer and respondent form their first
impressions of each other, and this conditions
the ensuing interview. Interviewers thus need
Box 10. Some Typical Questions and Remarks Early be careful about their overall appearance: they
in an Interview should always dress simply and inconspicuously,
and they must respect local norms of dress and
Question: Why did you pick me? behavior, even when inconvenient.
Response: The purpose of this survey is to find out the views of people Interviews should be conducted at a time most
on_____. Since it is not possible for us to talk with everyone, we have
selected a few people like you to help us out. (The interviewer can add a
convenient for respondents. For example, if
sentence or two about the sampling process.) farmers are in the fields during daylight hours,
interviews should be held during the evenings,
Remark and question: I am too busy. Why don’t you leave me out?
when they are most likely to be at home. Men
Response: It will not take more than 15 to 20 minutes of your time. How- and women employed in industrial and ser-
ever, if you are very busy now, please tell me when I can visit you again. It vice sectors can be better contacted on week-
is extremely important that I have the benefit of your ideas and experi- ends, when they are likely to be relatively free.
ence on this important subject.
Interviewers should make appointments with
Remark: I really don’t know anything about this. government officials and professionals to avoid
Response: We are interested in your opinions and experiences, not in what scheduling conflicts.
information you may or may not have. I am sure you will find questions
Interviewers should begin the interview by
interesting and will be able to answer them easily. We have asked the same
questions to many people, and they did not have any problems answer- briefly explaining their backgrounds, the objec-
ing them. In a study of this type, there are no right or wrong answers to tives of the survey, and possible uses of infor-
questions. mation the respondent will provide (box 10).
Question: Who’s behind this? At this time, interviewers should also assure
respondents of the confidentiality of this infor-
Response: This study is sponsored by the ______ project (or program). Its mation. They should also indicate the time
purpose is to get ideas and information that will be helpful in improving its
required for completing the interview—no
activities.
more than 30 minutes for mini surveys. These
remarks should be brief and to the point, and

interviewers should honestly answer any ques- TECHNIQUES FOR GETTING
tions that respondents have at this stage. RESPONSES TO QUESTIONS
One of the most challenging tasks for inter-
USING THE QUESTIONNAIRE viewers is to obtain complete responses that
An interview should be made as pleasant as meet the needs of the survey. Respondents
possible. Interviewers should never give the may give incomplete or irrelevant answers for a
impression that they are administering a quiz variety of reasons. Some have difficulty under-
or conducting a cross-examination. Nor should standing the questions; others have problems
they say anything that would imply approval expressing themselves; and still others may be
or disapproval of the respondent’s answers. reluctant to reveal their thoughts or what they
Interviewers should invariably show respect know. Interviewers should deal with these situ-
to respondents and a genuine appreciation for ations tactfully, and in a way that will not bias
their views and opinions. responses.
The interviewer should read each question Survey experts recommend five techniques to
slowly. The ideal reading pace is two words per stimulate fuller, clearer responses.
second. Studies have shown that a slow, delib-
erate pace enables respondents to understand 1. Repeat the question. Repeating a question can
the question and formulate a careful reply. help when the respondent does not understand
When interviewers hurry through questions, it, misinterprets it, misses the emphasis, seems
respondents tend to follow their example. This unable to make up his or her mind, or strays
contributes to superficial, even inaccurate, from the subject. Moreover, the repetition
responses. gives the respondent more time to reflect on
the question.
Some respondents may not be sure of their
2. Pause for the answer and convey, through a nod
answers and need to be reassured. This is espe-
or an expectant look, that a fuller response is
cially true of people of low socioeconomic status
expected. A pause gives the respondent time
who have not been exposed to survey research.
to collect his or her thoughts. In many cases,
An interviewer who feels that a respondent is
however, the respondent may have nothing
diffident and doubtful about a reply should
more to add, and a pause may cause unneces-
make a neutral conversational remark to put
sary embarrassment. Interviewer need to be
them at ease, such as “We are just trying to get
sensitive to these nuances and recognize when
people’s ideas on this,” or “There are neither
the respondent should not be questioned any
right nor wrong answers to this question.”
further.
Questions should be asked in the logical order
3. Repeat the reply. Hearing the ideas repeated
in which they are presented in the question-
may stimulate the respondent to expand on
naire. Interviewers who change this sequence
the response.
may inadvertently bias the results. If a ques-
tion does not apply to the respondent, it 4. Use neutral comments or questions, such as
should be crossed out and the reasons for its “Anything else?” “Any other reason?” “Could
nonapplicability noted. This enables the coder you tell me more about your thinking on
to know that the question is not relevant and this subject?” “Why do you feel that way?”
has not inadvertently been overlooked by the “How did this occur?” and “When did it
interviewer. happen?”These comments and questions

indicate that the interviewer is carefully listen- RECORDING AND EDITING THE
ing to what the respondent is saying. INTERVIEW
Interviewers should record responses during
5. Gently ask for clarification of inconsistent, con-
the interview. If they wait until the interview
tradictory, or ambiguous answers. At the same
has ended and try to remember what respon-
time, the interviewer needs to take the blame
dents said after the fact, they can lose relevant
for any ambiguity and must not imply that
information.
the respondent is unclear or inarticulate. To
clarify a point, the following statements help: Responses should be noted down in the
“I’m sorry, but I’m not sure if I got the point. language of respondents, keeping the same
Would you please repeat it?” “I’d like to be phrases, grammatical usages, and peculiarities
sure I understood you correctly. You said that of speech. When interviewers try to summarize
you did not borrow money for your business? or paraphrase respondents’ answers, they often
Is this correct?” unknowingly create a communication gap.
Summaries can also distort the true intent of
The success of the interview depends on the
respondents (box 11).
interviewer’s ability to recognize immediately
whether a specific answer has failed to meet the But interviewers should not get so involved in
objective of the question and, if so, to find an note-taking that they forget the respondent.
appropriate alternative. One simple technique for holding the respon-
dent’s interest while taking notes is to repeat the
response. Doing this shows that the interviewer
is listening carefully, confirms what was said,
Box 11. Distorting Effects of Summarizing and and provides respondents more time to reflect
Paraphrasing Answers on their answers. Many respondents modify
Consider the difference between the verbatim reply and the summary their replies or provide more specifics when
version reported: interviewers repeat their answers.
Verbatim: Yes, indeed! I am positive about it. To speed note-taking, interviewers can construct
and use abbreviations and codes for commonly
Summarized version: Yes.
used terms. For example, they can use “R”
The summarized version lacks the intensity of the respondent’s reply. for respondent, “DK” for don’t know, “P” for
A more serious example of distortion is the following: project, “E” for evaluation, and so on. To save
Verbatim: I am very upset by the way the technical assistance program has time, articles and prepositions can be left out
worked in this district. The program has primarily benefited well-to-do and only key words noted during the interview.
entrepreneurs, rather than the poor, struggling self-employed men and During the editing process, the interviewer
women who were supposed to be targeted by the planners of this pro- can insert these words and punctuation. For
gram. Thus it has totally failed to accomplish its stated objectives. example, a standard question, “What is on your
Summarized version: Program did not accomplish its objectives. mind?” may be written as “What mind” at the
note-taking stage.
The summarized version does not capture the respondent’s feelings on
the subject; it even distorts the reply. The essential point that the respon- Interviewers should also note the nonverbal
dent forcefully made was that the project failed to benefit “poor, struggling, behaviors of respondents, when appropriate.
self-employed men and women.”
Often facial expressions reveal more than what
an informant says. For example, if a woman

respondent seems skeptical or uncomfortable be made to present the findings in nontechnical
responding to questions about the effectiveness language.
of credit delivery to women farmers, the inter-
viewer should make a note of her reaction. Such CODING DATA
notes enable a careful examination and review Coding is indispensable for quantitative analy-
of replies during data coding and analysis. sis. It involves transforming gathered data into
categories and translating these categories into
The best time to edit is immediately after the numbers. The purpose of coding is to sim-
interview. If that is not possible, editing should plify individual responses. For example, sup-
be done the same day. The purpose of editing is pose respondents’ occupations are as follows:
to ensure that all answers are correctly recorded.
farmer, barber, farmworker, blacksmith, general
In particular, interviewers should ensure that
merchant, moneylender, baker, butcher, civil
entries are legible. They should also ensure that
servant, primary teacher, mason, and midwife.
questions that were not relevant are marked,
Because the mini survey sample size is small,
replies to open-ended questions are put in
this number of occupational categories will serve
parentheses, and articles and prepositions are
little purpose. Instead, fewer categories should
added.
be developed that are geared to the purpose of
the inquiry. If the primary interest is the farm-
ANALYZING AND PRESENTING ing population, only three categories may be
THE SURVEY DATA used: farmers, farmworkers, and others. If the
The manager of a large area development project purpose is to understand microenterprise prob-
in a southern African country came to meet with lems, a different scheme is required.
the permanent secretary of agriculture. He brought
what he thought was the major achievement of his There are two methods for coding: inductive
statistical division, a few neatly typed regression and deductive. In the inductive method, data
tables. These tables presented regression coefficients are recorded in as much detail as possible—for
for variables predicting the adoption rates for example, all occupations in the community are
high-yielding varieties of maize in the region. The listed. However, the deductive method involves
permanent secretary asked a few general questions the use of a predetermined classification scheme
and dismissed the manager, promising to look at that is strictly followed by the coder. For exam-
the tables carefully. As soon as his visitor left, he
ple, if there are three categories—farmers, farm-
deposited the tables in his waste basket, breathing
workers, and others—the coder will classify all
a sigh of relief.
responses on this basis.
An incident reported to the author
Both coding methods have advantages and
This story illustrates that the analysis of survey limitations. One shortcoming of the deductive
data in project and program settings should method is that it does not allow for new ideas
be kept simple. The purpose of the mini sur- and insights. Once the data are coded, there is
vey report should be to communicate, not to no freedom to manipulate them beyond speci-
impress. Simple statistical tools that are more fied categories without recoding the question-
likely to be understood by people without sta- naires. An advantage of the deductive method
tistical expertise are thus invariably preferred to is that it saves time and effort, and it imposes
complex and sophisticated tools. If complex sta- some order on data by eliminating superfluous
tistical tools must be used, every effort should or irrelevant details. Consider the difference in

not apparent, the inductive method should
Box 12. An Illustration of a be used. The inductive method is particularly
Coding Scheme appropriate for coding responses to open-ended
questions.
a. Age
(Code actual age in years) There are four simple rules for developing a
good coding scheme:
b. Gender
Male = 1 1. Categories must be mutually exclusive; each case
Female = 2 should be classified only once.
c. Number of children
(give actual number) 2. Categories should allow all responses to be
categorized. If the marital classification is only
d. Use of immunization programs “married,” “single,” and “divorced,” a widow
Yes =1
or widower will create classification problems.
No = 2
The category “other” or “miscellaneous” would
I don’t know = 8
No answer = 9 be required to make the system inclusive.
e. Agreement with the statement about the 3. Details needed by the survey must not be lost.
need for polio vaccine. Fewer categories make data neat and manage-
Agree strongly = 1 able, but they also limit what is available.
Agree = 2
Not sure = 3 4. The coding scheme must be related to the purpose
Disagree = 4 and scope of the mini survey.
Strongly disagree = 5
In some cases, actual numbers can be pre-
sented—such as a respondent’s age, the size of a
household, or the area cultivated. However, in
coding and analysis time if four instead of fif- most cases, an arbitrary number will be given to
teen occupational categories are used. a category (box 12).
The main strength of the inductive method is Coding open-ended questions requires con-
that it permits flexibility. New categories can be siderable time and effort. The best course is to
easily developed, even after the data have been numerically code them as well as record them
coded. For example, if a new category—say, verbatim. For example, two steps can be used
civil servants—is found to be useful, it can be to code responses relating to recommendations
added without any additional effort. Among the
for improving the quality of technical assistance
shortcomings of this method is that more time
provided by a microenterprise project. First,
is needed for coding data and the investigation
all responses are recorded in a separate sheet
can become bogged down with unnecessary
to provide a comprehensive picture. Second,
details.
after carefully reviewing the responses, a set of
Ideally, both methods should be used in a mini categories is developed for recommendations
survey. When its designers know what they received; individual responses can be coded
are looking for and have a reasonable idea of accordingly. Often, the review of verbatim
response categories, the deductive approach records will provide new insights and explana-
is preferable. But if appropriate categories are tions when the report is being prepared.

STATISTICAL ANALYSIS by listing all response categories and counting
the number of observations in each of them.
Frequencies and Percentages The accepted procedure is to list the categories
The first step in any analysis of survey data is to in the left-hand column and the number of
construct a frequency distribution. This is done observations in the right-hand column.
Box 13. Use of Percentages in Presenting Data

Percentages can either be very helpful in presenting data Or we can show the index of the numbers purchasing fertilizer
clearly or very misleading, depending on the competence (and in each district, taking the number 1 as the base (100):
honesty) of the presenter. Consider the data in this table giving
the number of farmers purchasing fertilizer in each of three Index of Fertilizer Purchasers
districts in each of the three years.
District Year 1 Year 2 Year 3
Number of Farmers Purchasing Fertilizer A 100 115 129
District Total Farmers Year 1 Year 2 Year 3 B 100 162 271
A 10,000 3,612 4,170 4,670 C 100 96 92
B 14,000 765 1,241 2,073
• The first table shows that fertilizer is most popular (or more
C 38,000 21,036 20,217 19,416 available) in districts A and C; it also shows the time trend,
but it disguises the dominance of district C in terms of num-
We can present these data in terms of the percentage of farms bers of purchasers.
in a district purchasing fertilizer in each year (noting that each
percentage is independent of the others—they do not sum to • The second table shows the dominance of district C, but
100 in any direction): disguises the trends in the number of purchasers over time.
• The third table highlights the time shifts more clearly, bring-
Percentage of Farmers Purchasing Fertilizer
ing out the rapid growth in district B and decline in district
District Year 1 Year 2 Year 3 C, while losing the relative importance in absolute numbers
or relative incidence of fertilizer purchasers in the various
A 36 42 47 districts.
B 5 9 15
Each presentation has a role to play, but each offers a partial
C 55 53 51 picture, which, in isolation, can even mislead.
Or we can present the percentage distribution by district of

the farmers purchasing fertilizer in a year:
Percentage of Fertilizer Purchasers

District Year 1 Year 2 Year 3
A 14 16 18
B 3 5 8
C 83 79 74
Total 100 100 100

Table 2. Size of Farms Cultivated by
high or low margins may be preferred. Interval
Respondents points should be determined with reference to
Farm Size No. of Percentage Cumulative
the number of observations and the objective
(hectares) Households Percentage of the survey. As a general rule of thumb, the
(f) number of intervals in a mini survey should
0–2.0 35 35 35 normally not exceed six or seven because of the
small sample size.
2.1–4.0 25 25 60
4.1–6.0 20 20 80 It is always useful to list percentages as well as
frequencies. The percentage for each category
6.1–8.0 15 15 95
is obtained by dividing the number of obser-
8.1–10 5 5 100 vations (f ) in that category by the total (N),
N 100 100 then multiplying the result by 100. Cumulative
distribution is obtained by summing the per-
Nominal categories (categories that cannot be centages in each category and all the preceding
ordered on the basis of their own attributes: categories.
gender, rural-urban residence, religion, and so
Table 2 offers a simple frequency distribution
on) are usually arranged according to number
with percentages. It shows that 60 percent of
of observations in each—from the largest to
respondents cultivate four or fewer hectares of
smallest. The categories “not applicable,” “appli-
land and only 5 percent cultivate more than
cable,” or “no response” are kept at the end,
eight hectares.
whatever their sizes.
Box 13 (on page 27) illustrates how the percent-
Intervals into which data are grouped should
ages can be used to draw relevant conclusions.
be of equal size, such as $001–100, $101–200,
$201–300. However, there are cases, such as Often a change of scale is extremely helpful.
distribution of income or rent paid, when sud- Table 3 gives data about the number of credits
den jumps occur, and unequal intervals at the approved by 10 branches of a credit institution.
Table 3. Variations of Transformed Data

Branch No. of Credits Credit Issued in Percent Difference Percent of the Mean Target
Approved Excess of 100 (126-5) from the Mean (120)
1 103 3 –23.5 86
2 125 25 – 1.5 104
3 117 17 – 9.5 98
4 133 33 + 6.5 111
5 129 29 + 2.5 108
6 148 48 + 21.5 123
7 118 18 – 8.5 98
8 132 32 + 5.5 110
9 146 46 + 19.5 122
10 114 14 –12.5 95

Column 2 shows the data with 100 removed
from each value, and column 3 shows each Table 4. Membership of Farmers’ Clubs
value as a difference from the mean. Column 4 Club No. Membership
gives the percentage with reference to the target 1 25
of 120 credits. 2 30
Often, a mini survey need not go beyond 3 35
counting frequencies and percentages. Most 4 35
study questions for a mini survey can be easily 5 38
answered with these simple techniques. 6 42
7 45
The Mode, Median, and Arithmetic Mean 8 50
For mini surveys, it may be desirable to obtain 9 56
measures of the central tendency. Such measures 10 57
include the mode, the median, and arithmetic
mean.
bolized by x). It is calculated by adding all the
• The mode is the category or observation that values of the distribution and dividing the sum
appears most frequently in the distribution, or by the total number of cases. For instance, in
the one with the largest number of responses. computing the mean for the membership of
Most distributions have only one modal farmer’s clubs as shown in table 4,
category. When two categories are nearly or
equally prominent, the distributions are called
–x 25+30+35+35+38+42+45+50+56
bimodal. =
9
• The median, a measure that divides the distri- = 39.6
bution into two equal parts, is computed by
locating the middle observation. For an odd Thus the average membership of a farmers’ club
number of cases, the middle item is calculated is about 40.
by adding 1 to N (the number of cases) and
dividing the sum by 2. Range,Variance, and Standard Deviation
Though measures of central tendency give an
Table 4 gives the membership figures for 9 and
indication about the most representative value
10 farmers’ clubs in a project area, arranged
of the distribution, they do not indicate how
by size. The median is (9+1)/2 = 5. Since the
it is dispersed. For example, two regions may
number of members in this club is 38, it is the
have the same mean for the size of agricultural
median membership for the farmers’ clubs. If
holdings, but the land might be equitably dis-
there were another club with a membership
tributed in only one region. In other words, all
of 57 (making 10 cases, an even number), the
farmers in one region might have more or less
median would be a value between 38 and 42,
the same amount of land, but a small minority
or 40. The median is not affected by extreme
might own most of the land in the other region.
values. Even if the first club had only five mem-
Measures of dispersion give information about
bers, the median would remain the same.
the dispersion or variation in the values of a
The most widely used measure of the central distribution. These measures are range, variance,
tendency is arithmetic average or mean (sym- and standard deviation.

The steps involved in calculating the variance
Table 5. Calculations of Mean and
(s2) are as follows:
Variance for a Distribution
– • The arithmetic average is computed. In this
Serial No. x x–x (x–x– )2 x2
case, the mean is 49/7 = 7.
1 3 –4 16 9
• The second step is to calculate the differ-
2 3 –4 16 9 ence between the value in each category and
3 5 –2 4 25 the mean. This difference is shown in the
4 6 –1 1 36 third column.
5 7 0 0 49 • The third step involves squaring the devia-

6 9 2 4 81 tion from the mean, as shown in column 4.
7 16 9 81 256 • The squares from the deviations are

Total 49 0 122 465 summed up and divided by the number of
cases. In this case, the result is 17.4.
An alternative to making these calculations is
Range is the simplest of the three, although not to employ a simpler formula that yields a close
the most useful. Range refers to the difference approximation to the true variance. The most
between the highest and the lowest values of the common approximation formula is
distribution, and is computed by subtracting
s2 = x2
( )
2
the lowest from the highest. Because the range x
–
depends on the two extreme scores, it is an N N
unstable measure.
Column 5 in the table gives x2 (squares of the
By contrast, variance (s2) takes into consid- raw scores of observations). When the simpler
eration the values of all the items in a distri- formula is applied to the data in Table 5, we
bution. It is computed by summing up the find
squared deviation from the mean, then dividing
()
2
the sum by the total number. 465 = 49
variance = —
7 7
N –2
variance =  (x1 – x) = 17.4
N
The variance expresses the average dispersion in
squared units, not in the original units of mea-
Table 5 shows a distribution of seven cases and
surements. This problem is solved by taking the
the calculation of its mean and variance.
square root of variance, which is called the stan-
Note: mean (x) = 49 = 7 dard deviation. Thus s = 4.2 in the example.
—
7
Pearson’s Coefficient of Correlation
(x–x– )2 The maximum value of the coefficient of cor-
variance (s2) =
N relation or association (r) is 1, which can be
both positive and negative. If two variables are
= 122 = 17.4 positively associated, it means that an increase
7 in the first variable is likely to be associated with

an increase in the second. The negative correla- When the figures from table 6 are entered in
tion suggests that when the one increases, the the correlation formula, the following results:
other decreases; in other words, the association
is inverse Thus, a value of r of –.75 is the same 10(1440)–[(130) (100)]
as of +.75, as far as the strength of the associa-
tion is concerned. In everyday usage, an r of .8 [(10 x 1878)–(130)2] [(10 x 1138)–(100)2]
and above is considered a high coefficient; an r
about .5 is considered moderate; and an r of .3 = 1400 = .87
1610
and below is considered a low coefficient.
There are several formulas for computing r. The r of .87 is indeed a very strong correlation.
Probably, the simplest is the following: It suggests that general knowledge and knowl-
edge of agricultural innovations are closely relat-
r= Nxy – (x) (y)
ed. Farmers who are more knowledgeable about
x2 – (x)2 y2 – (y)2 general matters are also well informed about
agricultural innovations.
A simple example will illustrate this formula.
To compute r for grouped data, the midpoint
Suppose 10 farmers are tested on their general
for the intervals is taken and multiplied with
knowledge, then given another test relating to
the frequencies. Chi square, simple and multi-
their knowledge of recent agricultural innova-
variate regression, and analysis of variance and
tions being promoted by a project. By using the
t-tests can also be used to analyze data. Any
correlation formula, it is possible to determine
standard textbook on statistics will provide
whether the scores on these two tests are associ-
more information on these measures.
ated (table 6)
Table 6. Association (r) between General Knowledge and Agricultural Innovation Scores
Agricultural
General Knowledge
No. Innovation (x)2 (y)2 xy
Scores (x)
Scores (y)
1 20 12 400 144 240
2 18 16 324 256 288
3 16 10 256 100 160
4 15 14 225 196 210
5 14 12 196 144 168
6 12 10 144 100 120
7 12 9 144 81 108
8 10 8 100 64 80
9 8 7 64 49 56
10 5 2 25 4 10
Total 130 100 1,878 1,138 1,440

PRESENTING DATA With access to a computer, relevant graphs can
be prepared very quickly.
Constructing Tables
Survey data are invariably presented in tables. One of the simplest graphs is a pie diagram that
The following simple rules help readers readily shows the proportions of the whole in different
absorb the data presented in tables and appreci- categories. A pie diagram would help illustrate
ate, without further analysis, the most obvious the results of a survey that seeks to discover the
patterns and relationships: economic status of 90 participants in a micro-
enterprise project and, in particular, whether
• Limit the number of rows and columns.
people below the poverty line (those earning less
Numerous columns and rows confuse read-
than $1,000 a year) were adequately represented
ers. What is appropriate for the professional
The key to presenting this data in a pie diagram
journal is not necessarily so in the context of
is to draw it to scale (fig. 1). The slices should
project and program analysis.
be restricted to five or six; otherwise the dia-
• Use clear, self-explanatory column and row gram becomes too cluttered.
headings. Liberal use of differential spacing is
needed to highlight comparisons.
Figure 1. Household Incomes of 90
• Use clear and unambiguous class intervals in Participants in a Microenterprise Project
frequency distributions.
• Transform the data into percentages and

indexes, as appropriate. Use averages, standard
deviations, and other measures to summarize
the array of data.
• Present the data in logical order. A commonly

used order is from most frequent to least fre-
quent, although reversing this order may also
be appropriate.
• Provide a title for the table that summarizes its

purpose and content.
• Add a note at the end of the table if there is

additional information that can help interpret
data, such as statistical significance.
Bar graphs are also useful for presenting survey
• Summarize in the text the highlights of the data. Unlike pie diagrams, bar graphs can pro-
table and conclusions that can be drawn vide an overview of many kinds of information
from it. at a glance. Suppose there is a need to compare
the utilization of technical assistance by men
Presenting Graphics and women entrepreneurs. The sample of 80
Graphic representation of data can be very use- was equally divided between men and women.
ful in communicating findings. Graphs can help Figure 2 presents these data in the form of a
dramatize a point without deceiving the reader. bar graph.

them are SPPS (Statistical Package for the Social
Figure 2. Utilization of Technical
Assistance by Gender Sciences), SAS (Statistical Analysis System), and
Stata.
• SPSS includes capabilities for producing charts

and graphs of various types, such as scatter
plots, density charts, diagnostic and explor-
atory plots, probability plots, cross-correlation
function plots, and multiple-use and custom
charts. The analyst can create tables and maps
by using add-on modules. In addition, this
software package allows the user to export and
import data with Microsoft Excel and Word.
• SAS has add-on features that allow researchers

to conduct the analysis and create graphics to
represent results. SAS/GRAPH software cre-
ates graphs and visuals that help to summarize
Line graphs can also be used to present data. data and present findings in charts, including
They are most suitable for presenting time vertical and horizontal bar charts, and pie,
series data, and they can also be used to show donut, subgrouped pie and donut, stacked,
frequency distributions and the relationship star, and block charts. These different types of
between two variables. A line graph should have charts can be used to represent sums, averages,
a title, scale (when appropriate), and a key that frequencies, and percentages calculated from
defines lines, values, and symbols. data.
Collecting data is just one—albeit a cen- • Stata also offers data management and graph-
tral—step in the survey process. The use of ics capabilities. It can generate visuals such as
the appropriate data management, analyses, bar charts, box plots, histograms, spike plots,
and presentation tools is critical to the effec- pie charts, scatter plot matrices, dot charts, line
tive communication of results. To accomplish charts, area charts, and two-way scatter plots.
all these interrelated steps most efficiently, use All these capabilities are included in the base
a seamless process to the extent possible. This package.
applies to the software used in survey data man-
Software packages available in the public
agement, starting with data entry, verification,
domain can also be useful. One example is
preparation of datasets, analyses, and presenta-
CSPro (Census and Survey Processing System),
tion of results.
a Windows-based, public-domain package
In addition to the Microsoft Office comple- developed by the U.S. Bureau of the Census
ment of software packages, a number of options for entering, editing, tabulating, and mapping
can be considered for survey data management census and survey data. CSPro is a valuable
and presentation. There are integrated statistical tool for data management, particularly for
software packages that allow researchers to work large survey data, and can be used to gener-
with different modules for data entry, analysis, ate tables and limited graphics. It also permits
and graphical representation of results. Among data to be exported to a number of other

platforms where graphical representation of follow the chapter on findings. An even better
results can be developed. option is to put these sections in appendices. A
table of contents is always helpful, since it indi-
Other analytical tools available in the public cates the report’s coverage and guides readers
domain include the Center for Disease Control’s interested in technical aspects.
EZ-Text, a software package that helps analysts
to create, manage, and analyze semi-structured Many survey reports contain an elaborate dis-
qualitative databases. EZ-Text assists the design cussion of the underlying concepts, sampling
of data entry templates that are tailored to ques- strategies, and procedures used to design their
tionnaires. Response to open-ended questions questionnaires. What is still worse, they include
may be entered into EZ-Text, either as a verba- numerous unnecessary tables. The preferred
tim transcript (for example, from a tape record- approach is to cover each section, but as suc-
ing) or as a summary generated from an inter- cinctly as possible. The emphasis should be on
viewer’s notes. Responses can then be coded the meaning and implications of the data analy-
in an interactive process. Online codebooks sis for the projects and programs investigated,
are created, and the code is applied to specific not the methodological aspects of the research.
response passages. Searches are then conducted
to identify text passages in the data set that REFERENCES
meet the criterion or criteria defined by the Barton, A. J. 1958. “Asking the embarrass-
analysts. Data files from different interviewers ing question.” Public Opinion Quarterly (22):
or sites can also be merged into one file of com- 67–68.
bined analyses. The ability to export and import
the codebook helps coordinate the efforts of Casely, Dennis J., and Krishna Kumar. 1988.
The collection, analysis, and use of monitoring
multiple coders, who are simultaneously work-
and evaluation data. Baltimore: Johns Hopkins
ing with copies of the same database file.
University Press for the World Bank.
Preparing the Report Chambers, Robert. 1981. Rapid rural appraisal:

The items in a typical academic report—the Rationale and repertoire. Public Administration
purpose and scope; conceptual framework; and Development 1(2): 95–106.
research methodology; summary of data, find-
Finsterbusch, Kurt. 1976a. Demonstrating
ings, and their implications; conclusions; and
the value of mini surveys in social research.
appendices—should appear in a survey report, Sociological Methods and Research 5(1): 117–36.
but in a different sequence. A mini survey
report should begin with the summary. It Finsterbusch, Kurt. 1976b. Mini surveys: An
should then move directly to the findings and underemployed research tool. Social Science
their implications. This arrangement suits deci- Research 5(1): 81–93.
sionmakers because they are more likely to be
Scott, Christopher. 1987. Sampling for moni-
interested in findings and recommendations
toring and evaluation. World Bank Working
than in methodologies, sampling strategies, or Paper. Washington, D.C.: World Bank.
conceptual frameworks. Presenting findings
and implications at the end of the report is Sudman, Seymour, and Norman M. Bradburn.
not a good option. The sections on conceptual 1988. Asking questions. A Practical Guide to
framework, research methodology, and data can Questionnaire Design. San Francisco: Jossey-Bass.

The U.S. Agency for International Development (USAID) is an independent federal agency that receives overall foreign
policy guidance from the Secretary of State. For more than 40 years, USAID has been the principal U.S. agency to extend
assistance to countries recovering from disaster, trying to escape poverty, and engaging in democratic reforms. USAID
supports long-term and equitable economic growth and advances U.S. foreign policy objectives by supporting
• economic growth, agriculture, and trade
• global health
• democracy and conflict prevention
• humanitarian assistance
The Agency’s strength is its field offices located in four regions of the world:
• Sub-Saharan Africa
• Asia and the Near East
• Latin America and the Caribbean
• Europe and Eurasia

1300 Pennsylvania Avenue, NW
Washington, DC 20523
Telephone: 202-712-4810
www. usaid.gov

Pnadg566 Mini Survey

Uploaded by

Copyright:

Available Formats

Pnadg566 Mini Survey

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pnadg566 Mini Survey

Uploaded by

Copyright:

Available Formats

USAID Program Design and Evaluation

Conducting Mini Surveys in

Month Year1990, revised July 2006

2 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES i

ii USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 1

2 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 3

4. Design the questionnaire: The next logical step

4 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 5

In sharp contrast, closed questions are easy to

6 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 7

8 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 9

10 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 11

12 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

Questionnaires can be created on a personal In pretesting and assessing individual questions,

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 13

14 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 15

16 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 17

Any of the four methods of probability sam-

18 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 19

Snowball sampling is performed in several

As with stratified sampling, quotas can be

20 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

Total • The number of variables to be examined simul-

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 21

22 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 23

24 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 25

26 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

Box 13. Use of Percentages in Presenting Data

Or we can present the percentage distribution by district of

Percentage of Fertilizer Purchasers

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 27

Table 3. Variations of Transformed Data

28 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 29

5 7 0 0 49 • The third step involves squaring the devia-

7 16 9 81 256 • The squares from the deviations are

30 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 31

• Transform the data into percentages and

• Present the data in logical order. A commonly

• Provide a title for the table that summarizes its

• Add a note at the end of the table if there is

32 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT

• SPSS includes capabilities for producing charts

• SAS has add-on features that allow researchers

CONDUCTING MINI SURVEYS IN DEVELOPING COUNTRIES 33

Preparing the Report Chambers, Robert. 1981. Rapid rural appraisal:

34 USAID PROGRAM DESIGN AND EVALUATION METHODOLOGY REPORT