VOLUME 2: NO.

1 JANUARY 2005

SPECIAL TOPICS

A Catalog of Biases in Questionnaires

Bernard C.K. Choi, PhD, Anita W.P. Pak, PhD

Suggested citation for this article: Choi BCK, Pak AWP. A inferences from the truth, or processes leading to such a
catalog of biases in questionnaires. Prev Chronic Dis [seri- deviation” (2). Questionnaire bias is a result of unantici-
al online] 2005 Jan [date cited]. Available from: URL: pated communication barriers between the investigator
http://www.cdc.gov/pcd/issues/2005/jan/ 04_0050.htm. and respondents that yield inaccurate results. Bias may
arise from the way individual questions are designed, the
PEER REVIEWED way the questionnaire as a whole is designed, and how the
questionnaire is administered or completed.

Abstract Based on a review of the literature, this paper identifies

and categorizes 48 common types of bias in questionnaires
Bias in questionnaires is an important issue in public and provides an example of each type in addition to a brief
health research. To collect the most accurate data from explanatory comment. This paper goes beyond the general
respondents, investigators must understand and be able to lists of biases that previous authors have provided (3-6) by
prevent or at least minimize bias in the design of their cataloging the types of bias according to their source
questionnaires. This paper identifies and categorizes 48 (Table). Organizing the types of bias in this way keeps
types of bias in questionnaires based on a review of the lit- related problems together and makes analysis of a ques-
erature and offers an example of each type. The types are tionnaire easier. The catalog below is meant to help public
categorized according to three main sources of bias: the health investigators understand the mechanism and
way a question is designed, the way the questionnaire as a dynamics of problems in questionnaire design and to pro-
whole is designed, and how the questionnaire is adminis- vide a checklist for identifying bias in a questionnaire
tered. This paper is intended to help investigators in pub- before it is used as a survey instrument.
lic health understand the mechanism and dynamics of
problems in questionnaire design and to provide a check-
list for identifying potential bias in a questionnaire before Types of Bias in Question Design
it is administered.
Problems with wording

Introduction Ambiguous question. Ambiguous questions lead

respondents to understand the question differently than
Health surveys have been and will continue to be impor- was intended and, therefore, to answer a different ques-
tant sources of information for evidence-based public tion than was intended (1).
health and medicine (1). The principal instrument for col-
lecting data in surveys is the questionnaire. To allow the Example: Is your work made more difficult because you
investigator to collect the most accurate data from respon- are expecting a baby?
dents, the questionnaire must be unbiased. Bias is a per-
vasive problem in the design of questionnaires. This question is ambiguous. A “no” answer may mean,
“No, I’m not expecting a baby,” or “No, my work is not
In this paper bias is defined as a “deviation of results or made more difficult” (7).

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 1
 VOLUME 2: NO. 1
JANUARY 2005

Complex question. Complex and lengthy questions sion’s technical terms may not be understood by the gen-
should be avoided in a questionnaire. eral public and should be avoided.

Example: Has it happened to you that over a long period Example: What was your age at menarche?
of time, when you neither practiced abstinence,
nor used birth control, you did not conceive? What was your age when your menstrual peri-
ods started?
This question, which was used in a survey on family
planning conducted for the Royal Commission on The technical term in the first question may not be
Population (8), is vague (“a long period of time”), too for- understood by many women, so it is preferable to ask the
mal (“Has it happened to you that”), and complex, same question in more common terms, as in the second
because of its length and use of the neither/nor question (9).
construction.
Uncommon word. Uncommon and difficult words
Double-barrelled question (also known as two ques- should be avoided in questionnaires.
tions in one). Questions that are made up of two or more
questions make it difficult for the respondent to know Example: Gowers (11) and Day (12) have produced lists
which part of the question to answer and for the investi- of words that can be replaced by simpler alternatives. For
gator to know which part of the question the respondent example:
actually answered (9).
Uncommon Common
Example: Do you agree that acquired immunodeficiency Assist Help
syndrome (AIDS) can be transmitted by shak- Consider Think
ing hands with a person with AIDS or through Effectuate Cause
other means of physical contact? Elucidate Explain
Employ Use
A “no” answer may mean not by shaking hands, or not Initiate Begin/Start
through other means of physical contact, or both (1). Major Important/Main
Perform Do
Short question. Short questions may not be as accu- Quantify Measure
rately answered as questions that are longer. A ques- Require Want/Need
tion that is short may come across as abrupt in an inter- Reside Live
view situation. Questions that include a transition to State Say
the next topic give respondents more time to gather Sufficient Enough
their thoughts and also more clues to use in formulating Terminate End
their responses (7). Ultimate Last
Utilize Use
Example: Have you had bad sore throats?
Use common words in questionnaires, especially ques-
Now a question about bad sore throats. We’re tionnaires targeted for the general population, to avoid
looking for information about these. Have you misunderstanding.
had bad sore throats?
Vague word. Vague words in vague questions encour-
The first question has been found to be less accurately age vague answers (1).
answered than the second question when compared with
information obtained from the respondents’ physicians Example: How often do you exercise?
(10). The second question includes an introduction that [ ] Regularly
sets up the query. [ ] Occasionally
Technical jargon. Technical jargon and the profes- How often do you exercise?

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
2 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

[ ] twice a week or more often recover the original continuous data or to change cut-off
[ ] once a week criteria for categories (7).
[ ] less than once a week
Example: What is your birth date?
The first question is vague because “occasionally” and
“regularly” are not defined. The meaning can easily be What is your age in years?
made more precise, as in the second question.
Which age category do you belong to?
Missing or inadequate data for intended purpose
For information on age, the first question is the best
Belief vs behavior (also known as hypothetical ques- because it can provide accurate continuous data, followed
tion or personalized question). Questions that ask the by the second question. The third question is the least
respondent about a belief (hypothetical) can yield quite dif- desirable because data are degraded (1).
ferent answers than questions that ask the respondent
about his or her behaviors (personalized) (9). Insensitive measure. When outcome measures make
it impossible to detect clinically significant changes or dif-
Example: Do you think that it is a good idea to have ferences, Type II errors occur (3).
everyone’s chest regularly checked by X-ray?
Example: How important is health to you, on a scale of 1
Have you ever had yours checked? to 3?
(Unimportant) 1 - 2 - 3 (Important)
The two questions generated different results. Ninety-
six percent of the respondents answered “yes” to the first How important is health to you, on a scale of 1
question, but only 54% answered “yes” to the second (13). to 10?
The answers to both questions may be accurate even (Unimportant) 1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 (Important)
though the results are different. The investigator must
determine whether the purpose of the question is to col-
lect data regarding a belief or a behavior and design the The first question may not have sufficient discriminat-
question accordingly. ing power to differentiate the respondents because of the
limited categories. The second question may be better.
Starting time. Failure to identify a common starting
time for exposure or illness may lead to bias (3). Faulty scale

Example: In the last 12 months, have you had an acci- Forced choice (also known as insufficient category).
dent causing head injury? Questions that provide too few categories can force respon-
dents to choose imprecisely among limited options (7,9).
Because a survey is normally conducted over an extend-
ed period, the time frame of “last 12 months” will vary Example: Do you agree? Yes [ ] No [ ]
depending on the date of the interview. The data obtained
therefore cannot be used to estimate incidence rates. The Do you agree? Yes [ ] No [ ] Don’t Know [ ]
following question is better and will provide a common
time frame: The first question, which does not have a “don’t know”
category, may produce a bias because respondents who
From January 1 to December 31 of last year, have no opinion are forced to select an answer that may or
did you have an accident causing head injury? may not reflect their true feelings. The second question is
recommended.
Data degradation. It is better to collect accurate, con-
tinuous data at source instead of degraded data. Once
degraded data have been collected, it is impossible to Missing interval. Missing intervals in response choic-

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 3
 VOLUME 2: NO. 1
JANUARY 2005

es can cause confusion. [ ] An operation that has a 5% mortality.

[ ] An operation in which 90% of the patients
Example: How often does the computer system go down? will survive.
[ ] Less than once per month
[ ] Once per month Patients scheduled for surgery may choose the second
[ ] Once per week option when they see or hear the words “90%” and “sur-
[ ] More than once per week vive,” but in fact a 90% survival rate (or 10% mortality) is
worse than a 5% mortality (1).
Respondents do not have a place to put “once every two
weeks.” The following response categories are recom- Leading question. Different wording of the same ques-
mended: tion can guide or direct respondents toward a different
answer (1,7).
How often does the computer system go down?
[ ] Less than once per month Example 1: Do you do physical exercise, such as cycling?
[ ] Once per month to once per week
[ ] More than once per week This is a leading question because it will likely lead the
respondent to focus only on cycling.
Overlapping interval. Overlapping intervals in
response choices can cause confusion (9). Example 2: Don’t you agree that . . . ?

Example: How many cigarettes do you smoke per day? This negatively worded question leads respondents to
[ ] None [ ] 5 or less [ ] 5-25 [ ] 25 or more answer no (14). The preferred phrasing is, “Do you agree
or disagree that . . . ?”
Respondents smoking exactly 5 or 25 cigarettes per day
do not know in which category to place themselves. The Mind-set. The mind-set of the respondent can affect his
following question is more appropriate: or her perception of questions and therefore can affect
answers.
How many cigarettes do you smoke per day?
[ ] None [ ] 4 or less [ ] 5-24 [ ] 25 or more Example: 1. How many cigarettes do you smoke per week?
2. How many cigars do you smoke per week?
Scale format. An even or an odd number of categories 3. How many beers do you drink per month?
in the scale for the respondents to choose from may pro-
duce different results. The change in wording from “per week” to “per month”
can result in wrong answers for the third question above,
Example: Do you agree? (Agree) 1 – 2 – 3 (Disagree) because of the possible mind-set of the respondents.
Do you agree? (Agree) 1 – 2 – 3 – 4 (Disagree)
Intrusiveness
The first question, with an odd number of categories,
tends to result in neutral answers (i.e., 2), and the second Reporting (also known as self-report response). A
question, with an even number of categories, tends to force respondent may selectively suppress information, such as
respondents to take sides (1). The two approaches produce past history of sexually transmitted disease (2,15).
different results, but there is no general consensus as to
which one is better. Example: In the past five years, have you engaged in anal
intercourse, that is, rectal intercourse?
Leading questions
This question is so direct and up-front that many people
Framing. Some questions may be framed in such a may refuse to answer. The following question may reduce
manner that respondents choose an inaccurate answer. reporting bias by deliberately loading the question to sug-
Example: Which operation would you prefer? gest that others also engage in the behavior (1): People

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
4 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

practice many different sexual activities, and some people and the second question (1995 NCHS-NHIS) (1) has five
practice things that other people do not. In the past five categories. Therefore, the categories may not mean exact-
years, have you engaged in anal intercourse, that is, rectal ly the same in the two surveys and will cause a problem for
intercourse? comparison over time.

Sensitive question. Sensitive questions, such as age, Change of wording. If the precise wording of a ques-
personal or household incomes, sexual orientation, or mar- tion changes in different surveys, the results may not be
ital status, may elicit inaccurate answers and may also comparable (7).
affect the interviewer-interviewee relationship so that all
subsequent answers can be affected. Example: Compared to other persons your age, would you
say your health is excellent, good, fair, or poor?
Example: How old are you?
Would you say your health in general is excel-
In what year were you born? lent, very good, good, fair, or poor?

The first question, which is direct, tends to result in a The first question (1985 NCHS-NHIS) (1) and the sec-
high percentage of refusals to answer. The second question ond question (1995 NCHS-NHIS) (1) use different word-
tends to yield fairly accurate responses (1). ing, namely, “compared to other persons your age” vs “in
general.” This may guide respondents to evaluate their
Inconsistency health in a different context.

Case definition. Definition of cases based on different Diagnostic vogue. The same illness may receive dif-
versions of the International Classification of Disease ferent diagnostic labels at different points in space or
(ICD) codes, for example, or first-ever cases vs recurrent time (3).
cases, may change over time or across regions, resulting in
inaccurate trends and geographic comparisons (16). Example: Do you have bronchitis?

Example: How many bladder cancer cases do you see in a Do you have emphysema?
year?
The terms “bronchitis” and “emphysema” are used in
How many histologically confirmed bladder Great Britain and in North America, respectively, to refer
cancer cases do you see in a year? to the same disease (3). It is therefore important to use the
term that is appropriate in space and time.
The use of two different case definitions can present
problems when comparing results.
Types of Bias in Questionnaire Design
Change of scale. If the measurement scale for a quan-
tity changes in different surveys, the results may not be Formatting problem
comparable.
Horizontal response format. In self-administered
Example: Compared to other persons your age, would you questionnaires, horizontal vs vertical format of the
say your health is excellent, good, fair, or poor? response choices can affect the answers (17).

Would you say your health in general is excel- Example: Your health is:
lent, very good, good, fair, or poor? Excellent ... [ ] Good ... [ ] Fair ... [ ] Poor ... [ ]

The first question, which was used in the 1985 National Your health is:
Center for Health Statistics National Health Interview Excellent . . . . [ ]
Survey (NCHS-NHIS) (1), has four categories of health, Good . . . . . . . [ ]

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 5
 VOLUME 2: NO. 1
JANUARY 2005

Fair . . . . . . . . [ ] a. Your family _________________

Poor . . . . . . . . [ ] b. Your career _________________
c. Your marriage _________________
The horizontal response format (first example) can cause
confusion among the respondents because of poor spacing Left alignment and right alignment. Alignment of
and may result in the wrong answers being checked or cir- the response choices to the left or right side of the possible
cled. The vertical response format (second example) has responses can produce a bias (17).
been suggested as better for listing response options (17).
Example: Your health is:
Juxtaposed scale (also known as questionnaire for- excellent . . .[ ]
mat). Juxtaposed scales, a type of self-report response good . . . . . . .[ ]
scale that asks respondents to give multiple responses to fair . . . . . . .[ ]
one item, may elicit different responses than separate poor . . . . . . .[ ]
scales (18).
Your health is:
Example: 1. Indicate how important and how satisfied [ ] excellent
you are with each of the following using a scale [ ] good
of 1 to 5: [ ] fair
[ ] poor
(Unimportant) 1 - 2 - 3 - 4 - 5 (Important)
(Dissatisfied) (Satisfied) It has been suggested that placing the response choices
Importance Satisfaction to the right side of (i.e., after) the list of possible responses
a. Your family ________ ___________ will result in fewer errors on the part of interviewers in a
b. Your career ________ ___________ personal or telephone interview. This facilitates subse-
c. Your marriage _ _ _ _ _ _ _ _ ___________ quent data input directly from the questionnaire. For
mailed and other self-administered questionnaires, plac-
The above question is in a juxtaposed scale format. The ing the response choices to the left of (i.e., before) the pos-
advantage is that it can force respondents to think and sible responses makes it easier for the respondent to circle
compare the importance and satisfaction for each item or check them (17).
because they are side by side. However, this questionnaire
format has been shown to cause confusion among respon- Questionnaire too long
dents who are less educated, in which case the following
question, with parts A and B in a separate scale format, No-saying (also known as nay-saying) and yes-saying
may be preferred (18): (also known as yea-saying). Some respondents tend to
answer no to all questions or to answer yes to all questions
1A. Indicate how important each of the follow- (1).
ing is to you using a scale of 1 to 5:
Example: What are the reasons why you do not exercise
(Unimportant) 1 - 2 - 3 - 4 - 5 (Important) daily?
Importance Yes No
a. Your family _________________ It takes too much time . . . . . . . . . . . . . . . . . . .[X] [ ]
b. Your career _________________ There is not enough time in my day . . . . . . .[X] [ ]
c. Your marriage _________________ There is no equipment at home . . . . . . . . . . .[X] [ ]
There are no community resources . . . . . . . .[X] [ ]
1B. Indicate how satisfied you are with each of I feel I am not trained to do it . . . . . . . . . . . . .[X] [ ]
the following using a scale of 1 to 5: I feel I do not want to do it . . . . . . . . . . . . . . .[X] [ ]
I am too tired . . . . . . . . . . . . . . . . . . . . . . . . . . . .[X] [ ]
(Dissatisfied) 1 - 2 - 3 - 4 - 5 (Satisfied) It is too difficult . . . . . . . . . . . . . . . . . . . . . . . . .[X] [ ]
Satisfaction In the above example, the respondent chooses yes for all

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
6 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

items. One way to reduce the no- or yes-saying bias is to

use both positive and negative statements about the same Skipping question. Skipping questions may lead to
issue in a battery of items to break the pattern (1), as in the loss of important information because of logical errors
the following example: in the flow of questions.

Yes No Example: 1. Are you self-employed?

1. People with AIDS deserve to [ ] Yes
have the disease . . . . . . . . . . . . . . . . . . . . . . .[ ] [X] [ ] No (Go to question 8)
2. People with AIDS should be
given more help . . . . . . . . . . . . . . . . . . . . . . .[X] [ ] 2. Do you smoke?
[ ] Yes
Open question (also known as open-ended question). [ ] No
Open-ended questions can result in data with differential
quality (14). Also, respondents are likely to be unwilling to 3. . . .
take the time to answer them.
8. Do you use a cellular telephone?
Example: What kind of physical exercise do you do? [ ] Yes
[ ] No
This open-ended question presents a difficult record-
ing task. The interviewer must decide whether to record The above questions, because of errors in the skipping
everything that the respondent says, record only what sequence, will not collect smoking information for those
the interviewer considers relevant, or paraphrase the who are not self-employed. Pretesting of the survey instru-
respondent’s answer. However, in some circumstances, ment should prevent such a bias.
open-ended questions are more appropriate than
close-ended questions, particularly in surveys of knowl-
edge and attitudes, and can yield a wealth of informa- Types of Bias in Administration of
tion through appropriate qualitative methods such as Questionnaire
content analysis (7).
Interviewer not objective
Response fatigue. Questionnaires that are too long
can induce fatigue among respondents and result in uni- Interviewer. Bias can be caused by an interviewer’s
form and inaccurate answers (19). subconscious or even conscious gathering of selective data
(2,4), which can result from inter-interviewer or intrain-
Example: Now we would like to move on to our Question terviewer errors (4).
no. 618, concerning the health of your pet fish...
Example: Do you smoke? Yes [ ] No [ ]
Personal interviews usually last 50 to 90 minutes; tele-
phone interviews typically last 30 to 60 minutes; self- If an interviewer knows that the respondent does not
administered questionnaires typically take 10 to 20 min- have a smoking-related disease, and therefore is unlikely to
utes to complete (19). From field experience, interviewers be a smoker, he or she may rephrase the question and ask
and respondents report these times to be acceptable, and instead, “You don’t smoke, do you?” This is a leading ques-
common sense suggests that much longer times are not tion and is likely to lead to a negative answer (14). Proper
feasible. Respondents are unable to concentrate and give interviewer training is needed to prevent such biases.
correct answers in a lengthy interview, especially if the
topics are not of interest. Toward the end of a lengthy ses- Nonblinding. When an interviewer is not blind to the
sion, respondents tend to say all yes or all no or refuse to study hypotheses, he or she may consciously gather selec-
answer all remaining questions (1). tive data (20).

Flawed questionnaire structure Example: 1. Do you have lung cancer? Yes [ ] No [ ]

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 7
 VOLUME 2: NO. 1
JANUARY 2005

2. Do you smoke? Yes [ ] No [ ] desired by the investigator (3). Socially undesirable

answers tend to be under-reported (7).
The first question reveals to the interviewer the dis-
ease status of the respondent, and this may affect the Example: Did you smoke during your pregnancy?
way he or she asks or records the answer for the second Yes [ ] No [ ]
question. Besides providing interviewer training, it is
also important to ensure that interviewers are blind to Mothers tend to answer no even if they smoked during
the study hypotheses. pregnancy (1).

Respondent’s subconscious reaction Unacceptable disease. Socially unacceptable disor-

ders (e.g., sexually transmitted diseases, suicide, insanity)
End aversion (also known as central tendency). tend to be underreported (1).
Respondents usually avoid ends of scales in their answers.
They tend to try to be conservative and wish to be in the Example: Do you have a sexually transmitted disease?
middle (7).
Ask these questions toward the end of the questionnaire
Example: Do you agree? so that they will not affect other questions. Also consider
[ ] Strongly agree using anonymous, mailed questionnaires instead of face-
[ ] Agree to-face interviews.
[ ] Disagree
[ ] Strongly disagree Unacceptable exposure. Socially unacceptable expo-
sures (e.g., smoking, drug abuse) tend to be underreported
Respondents are more likely to check “Agree” or (1).
“Disagree” than “Strongly agree” or “Strongly disagree” (1).
Example: Do you now smoke cigarettes every day?
Positive satisfaction (also known as positive skew).
Questions on satisfaction may cause problems. A direct and intruding question like the one above may
result in reporting inaccuracy. Instead, when asking about
Example: Yes No undesirable behaviors, it is better to ask whether the per-
1. Are you satisfied with your family? . . . . . . . . .[X] [ ] son had ever engaged in the behavior in the past before
2. Are you satisfied with your career? . . . . . . . . .[X] [ ] asking about current practices, because past events are
3. Are you satisfied with your marriage? . . . . . . .[X] [ ] less threatening (21). For example:

Respondents tend to give positive answers when 1. Have you smoked at least 100 cigarettes in your entire
answering questions on satisfaction (1). life?

Respondent’s conscious reaction 2. Last year, were you smoking cigarettes every day?

Faking bad (also known as hello-goodbye effect). 3. Do you now smoke cigarettes every day?
Respondents try to appear sick to qualify for support (1).
Unacceptability. Measurements which hurt, embar-
Example: Which of the following symptoms do you have? rass, invade privacy, or require excessive commitment
may be systematically refused or evaded (3).
Respondents tend to check more types of symptoms than
they have (1). Example (22): We would now require two urine speci-
mens from you. The first specimen will be collected over a
Faking good (also known as social desirability, obse- 24-hour period, part of which will be while you are in your
quiousness). Respondents may systematically alter ques- natural working environment, probably toward the end of
tionnaire responses in the direction they perceive to be the work week, such as on a Friday. The second specimen

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
8 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

will be taken over another 24-hour period while you are at ly alter questionnaire responses when, during the process
home, out of the work environment for at least 24 hours, of answering the questionnaire, they think they know the
such as on a Sunday. During collection, keep all urine sam- study hypothesis.
ples refrigerated, in the refrigerator at home, or by the
portable thermos bottle and ice-packs at work. When fin- Example: Yes No
ished, please call the taxi company with the instruction 1. Does your child have headaches? . . . . . . . . . . .[ ] [ ]
sheet to deliver the samples to the laboratory. 2. Does your child play with
battery-operated toys? . . . . . . . . . . . . . . . . . . . .[ ] [ ]
Avoid measurements by intrusive means, or consider 3. Does your child play with batteries? . . . . . . . . .[ ] [ ]
using incentives to increase participation rate. 4. How many and which types of batteries do you have at
home?
Underlying cause (also known as rumination). Cases
may ruminate about possible causes for their illness and The respondents, perceiving that the study is about
thus exhibit different recall of prior exposures from those headache and battery use, may overreport the number of
of controls (3). batteries if they have a child with headaches.

Example: Did you have skull x-rays in the past five years? Respondent’s inaccurate recall

In a case-control study of childhood brain tumors, a sig- Primacy and recency. Depending on the type of ques-
nificantly elevated risk was reported by cases for skull x- tionnaire (interviewer-administered questionnaires or
rays compared to controls (23). It is not known whether self-administered questionnaires), respondents may
this was a true effect of x-rays on brain tumors or of cases’ choose answers differently.
thinking that x-rays were the cause of their illness.
Example: (24): Which of the following types of doctors did
Respondent’s learning you see in the past year?
[ ] family doctor
Learning. Completing a questionnaire can be a learn- [ ] pediatrician
ing experience for the respondent about the hypotheses [ ] lung doctor/internist
and expected answers in a study. [ ] allergy doctor/immunologist
[ ] emergency room doctor
Example (24): [ ] some other kind
1. Which of the following investigations would
you order for a patient of yours with asthma- Research has indicated that in mailed surveys, respon-
like symptoms? dents may tend to choose the first few response options on
[ ] spirometry the list (primacy bias), though in telephone or personal
[ ] lung volumes, diffusing capacity interview surveys, they are more likely to respond in favor
[ ] peak expiratory flow rate of the later categories (recency bias) (25,26). These effects
[ ] chest X-ray can be minimized by reducing the number of categories
presented to respondents and by randomizing the order of
2. Under what conditions would you order categories in survey instruments.
spirometry for a patient?
Proxy respondent (also known as surrogate data). For
Having thought about prior questions (such as the first deceased cases or surviving cases (e.g., brain tumors)
question) can affect the respondent’s answer to subsequent whose ability to recall details is defective, soliciting infor-
questions (e.g., the second question) through the learning mation from proxies (e.g., spouse, family members) may
process as the questionnaire is completed. To avoid learn- result in differential data accuracy. In general, it is not
ing bias, it may be necessary to randomize the order of the advisable to ask someone to answer attitudinal, knowl-
questions for different respondents. edge, or behavior questions for others (1).
Hypothesis guessing. Respondents may systematical- Example: 1. What is your wife’s occupation?

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 9
 VOLUME 2: NO. 1
JANUARY 2005

2. Please tell me how afraid your wife is of representing annual, not monthly, income. This question
getting cancer? would be appropriate for a survey in Asia, however, since
Not at all afraid Extremely afraid the culture there is to report monthly income. Pretesting
1 2 3 4 5 the survey instrument should minimize this bias.

The first question is appropriate but the second question

is not. Discussion
Recall. This type of bias is because of differences in Questionnaire bias is an important subject given the less
accuracy or completeness of recall prior to major events or than optimal questionnaires that are produced in the
experiences (3). health research field. This paper can serve as a resource
for health researchers and practitioners using question-
Example: How many diagnostic x-ray examinations did naires. It provides a catalog of types of bias that can be
you have when you were pregnant? used as a checklist for identifying potential problems when
designing and administering questionnaires.
It was found that mothers whose children have had
leukemia were more likely than mothers of healthy chil- This paper focuses on biases specific to questionnaires
dren to remember details of diagnostic x-ray examinations (design and administration). It does not cover such biases
to which these children were exposed in utero (2). as sampling and selection biases (5). Nor is it within the
scope of this paper to discuss such general practices as sur-
Telescope. Respondents usually recall an event in the vey development, interviewer training, or for that matter,
distant past as happening more recently (1). This is a form how to best conduct an adequate survey. For example,
of recall bias. inadequate survey design may sometimes result in biases
in sample selection, such as by not having the question-
Example: In an interview in May, an event which was naire translated in all necessary languages (language
thought to have occurred in March actually barrier bias), by restricting the survey to those subjects
happened in November of the previous year. with telephones (telephone sampling bias) (5), or by select-
ing only those born close to the date of interview (next
Telescope bias can be reduced by the bounded recall pro- birthday bias) (28). There may be other common errors in
cedure in which respondents are interviewed at the begin- survey development that may cause interpretational
ning and end of the time period referenced in a survey problems, including asking for family history of a disease
questionnaire (1). The first interview would serve to iden- (family history bias) (29), not having pilot surveys to
tify events that occurred before the interview period so pretest the questionnaires (lack of pretest bias) (14), or
that they could clearly be eliminated if the respondent using telephone interviews where visual aids cannot be
later reported that they occurred during the period used to illustrate the questions (telephone interview bias)
between the first and second interviews. However, this (19). Furthermore, different kinds of study methods, such
procedure must ask people about the same thing twice. as mailed questionnaires, personal interviews, telephone
interviews, Web surveys, routine data and registries, sur-
Cultural differences veillance systems, and focus groups, should be used
depending on the nature of the study to avoid bias (wrong
Cultural. The culture of the respondents can instrument bias, also known as wrong study method bias)
affect their perception of questions and therefore (1,7). For a simple survey of an educated section of the
their answers (27). population (e.g., a professional group) concerning a subject
of interest to its members, a mailed questionnaire might
Example: What is your gross monthly income? be appropriate. On the other hand, a survey of the gener-
al population on detailed and complicated information
The culture in North America and Europe is to think in would almost certainly call for a personal interview. These
terms of annual income. For the above question, it is are questionnaire problems that affect or are related to
inevitable that some respondents will put down a figure study designs and, strictly speaking, are not questionnaire

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
10 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

biases. They are therefore not included in our catalog of University Press; 1986.
questionnaire biases. 10. Marquis KH. An experimental study of the effects of
reinforcement, question length, and reinterviews on
reporting selected chronic conditions in household
Author Information interviews. Ann Arbor (MI): University of Michigan,
Institute for Social Research, Survey Research Centre;
Corresponding author: Bernard C.K. Choi, PhD, 1969.
Department of Public Health Sciences, Faculty of 11. Gowers EA. The complete plain words. London:
Medicine, University of Toronto, Toronto, Ontario, Penguin (Pelican) Books; 1954.
Canada, Department of Epidemiology and Community 12. Day RA. How to write and publish a scientific paper.
Medicine, Faculty of Medicine, University of Ottawa, 5th ed. Phoenix (AZ): Oryx Press; 1998.
Ottawa, Ontario, Canada, Centre for Chronic Disease 13. David ST. Public opinion concerning tuberculosis.
Prevention and Control, Public Health Agency of Canada, Tubercle (Journal of the British Tuberculosis
Room A147, 120 Colonnade Road, PL# 6701A, Ottawa, Association) 1952;33:78-90.
Ontario, Canada K1A 1B4. Telephone: 613-957-1074. 14. Schlesselman JJ. Case-control studies: design, con-
Email: Bernard_Choi@phac-aspc.gc.ca. duct, analysis. New York: Oxford University Press;
1982.
Author affiliations: Anita W.P. Pak, PhD, Office of 15. Fennema JS, van Ameijden EJ, Coutinho RA, van den
Institutional Research, University of Ottawa, Ottawa, Hoek JA. Validity of self-reported sexually transmit-
Ontario, Canada. ted disease in a cohort of drug-using prostitutes in
Amsterdam: trends from 1986 to 1992. Int J Epidemiol
1995;24:1034-41.
References 16. May DS, Kittner SJ. Use of medicare claims data to
estimate national trends in stroke incidence, 1985-
1. Aday LA. Designing and conducting health surveys. 1991. Stroke 1994;25:2343-7.
2nd ed. San Francisco (CA): Jossey-Bass; 1996. 17. Dillman DA. Mail and telephone surveys: the total
2. Last JM. A dictionary of epidemiology. New York design method. New York: John Wiley & Sons, Inc;
(NY): Oxford University Press; 2001. 1978.
3. Sackett DL. Bias in analytic research. J Chron Dis 18. Hunt DM, Magruder S, Bolon DS. Questionnaire for-
1979;32:51-63. mat bias: when are juxtaposed scales appropriate: a
4. Choi BC, Noseworthy AL. Classification, direction, call for further research. Psychol Reports
and prevention of bias in epidemiologic research. J 1995;77:931-41.
Occup Med 1992;34:265-71. 19. Hartge P, Cahill J. Field methods in epidemiology. In:
5. Choi BC, Pak AWP. Bias, overview. In: Armitage P, Rothman KJ, Greenland S, editors. Modern epidemi-
Colton T, editors. Encyclopedia of biostatistics. Vol 1. ology. 2nd ed. Philadelphia (PA): Lippincott-Raven
Hoboken (NJ): John Wiley & Sons, Inc; 1998. p. 331-8. Publishers; 1998. p. 163-80.
6. Delgado-Rodriguez M, Llorca J. Bias. J Epidemiol 20. Rothman KJ, Greenland S. Types of epidemiologic
Community Health 2004;58:635-41. studies. In: Rothman KJ, Greenland S, eds. Modern
7. Foddy W. Constructing questions for interviews and epidemiology. 2nd ed. Philadelphia (PA): Lippincott-
questionnaires: theory and practice in social research. Raven Publishers; 1998. p. 67-78.
Cambridge (United Kingdom): Cambridge University 21. Sudman S, Bradburn NM. Asking questions: a practi-
Press; 1993. cal guide to questionnaire design. San Francisco (CA):
8. Lewis-Faning E. Report on an enquiry into family lim- Jossey-Bass Publishers; 1982.
itation and its influence on human fertility during the 22. Choi BC, Connolly JG, Zhou RH. Application of uri-
past fifty years. Vol 1. London: Her Majesty’s nary mutagen testing to detect workplace hazardous
Stationery Office, Royal Commission on Population; exposure and bladder cancer. Mutat Res
1949. 1995;341:207-16.
9. Kelsey JL, Thompson WD, Evans AS. Methods in 23. Howe GR, Burch JD, Chiarelli AM, Risch HA, Choi
observational epidemiology. New York: Oxford BC. An exploratory case-control study of brain

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 11
 VOLUME 2: NO. 1
JANUARY 2005

tumours in children. Cancer Res 1989;49:4349-52.

24. Jin R, Choi BC, Chan BT, McRae L, Li F, Cicutto L, et
al. Physician asthma management practices in
Canada. Can Respir J 2000;7:456-65.
25. Ayidiya SA, McClendon MJ. Response effects in mail
surveys. Public Opinion Quarterly 1990;54:229-47.
26. Salant P, Dillman DA. How to conduct your own sur-
vey. New York: John Wiley & Sons, Inc; 1994.
27. van Hemert DA, Baerveldt C, Vermande M. Assessing
cross-cultural item bias in questionnaires: accultura-
tion and the measurement of social support and fami-
ly cohesion for adolescents. J Cross-Cultural Psychol
2001;32:381-96.
28. Salmon CT, Nichols JS. The next-birthday method of
respondent selection. Public Opinion Quarterly
1983;47:270-6.
29. Khoury MJ, Flanders WD. Bias in using family histo-
ry as a risk factor in case-control studies of disease.
Epidemiology 1995;6:511-9.

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
12 Centers for Disease Control and Prevention • www.cdc.gov/pcd/issues/2005/jan/04_0050.htm
 VOLUME 2: NO. 1
JANUARY 2005

Table
Table. Sources of Questionnaire Bias

Source Bias Source Bias

1. Question Design Flawed questionnaire skipping question

structure
Problems with wording ambiguous question
complex question 3. Administration of Questionnaire
double-barrelled question (two
questions in one) Interviewer not interviewer
short question objective nonblinding
technical jargon
uncommon word Respondent’s end aversion (central tendency)
vague word subconscious reaction positive satisfaction (positive skew)

Missing or inadequate belief vs behavior Respondent’s faking bad (hello-goodbye effect)

data for intended (hypothetical question, personalized conscious reaction faking good (social desirability,
purpose question) obsequiousness)
starting time unacceptable disease
data degradation unacceptable exposure
insensitive measure unacceptability
underlying cause (rumination)
Faulty scale forced choice (insufficient category)
missing interval Respondent’s learning learning
overlapping interval hypothesis guessing
scale format
Respondent’s primacy and recency
Leading questions framing inaccurate recall proxy respondent (surrogate data)
leading question recall
mind-set telescope

Intrusiveness reporting (self-report response) Cultural differences cultural

sensitive question

Inconsistency case definition

change of scale
change of wording
diagnostic vogue

2. Questionnaire Design

Formatting problem horizontal response format

juxtaposed scale (questionnaire
format)
left alignment and right alignment

Questionnaire too long no-saying (nay-saying) and yes-saying

(yea-saying)
open question (open-ended question)
response fatigue

The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services,
the Public Health Service, the Centers for Disease Control and Prevention, or the authors’ affiliated institutions. Use of trade names is for identification only
and does not imply endorsement by any of the groups named above.
www.cdc.gov/pcd/issues/2005/jan/04_0050.htm • Centers for Disease Control and Prevention 13