a_Reading and evaluating qualitative research studies
a_Reading and evaluating qualitative research studies
a_Reading and evaluating qualitative research studies
Qualitative research is now published across the fam search design and analysis. By using clinical analogies,
ily practice and medical literature. This article is de the qualitative research process can be better under
signed to help busy family physicians decide which stood. Key words. Family practice; family physicians;
qualitative studies are worth reading and to provide research, qualitative; critical appraisal. ( / Fam Pract
them with the tools to appreciate and evaluate re 1995; 41:279-285)
As family physicians seek information to help them make used by qualitative researchers, exotic for now, will be
decisions about incorporating new therapies or theories in come more familiar as qualitative research becomes more
their practices,1 they are occasionally confronted by clin prevalent. Some of these terms are briefly addressed in this
ical questions that cannot be answered by traditional review, with references provided for those who wish to
quantitative research. It is therefore important that phy explore these concepts more thoroughly in the qualitative
sicians learn how to read and assess qualitative studies. literature.
The medical literature now contains numerous qualitative
studies.2-5
lust as there is no one way to perform a quantitative Study Question and Design
study, neither is there just one way to perform a qualita
The first issue for the reader of any research study is quite
tive one. The language of qualitative research is unfamiliar
basic (Table 1): what is the research question? Is it a
to many family physicians, and the research itself may
practical, important question for the reader?7 Does the
seem “hopelessly subjective” and “ unscientific.” 6 Quali
question look for “ patient-oriented evidence that mat
tative research is an outgrowth of the psychological, so
ters,” or POEMs, as described by Slawson and col
ciological, and anthropological disciplines, and while the
leagues?1 These authors define POEMs as studies that
concepts used in these disciplines are not necessarily new
evaluate “ interventions that patients care about and that
to family physicians, the language with which they have
we, as clinicians, care about for our patients.” Some of
been described may be. these research questions are best answered by a quantita
An analogy to the clinical decision-making process tive approach, some by a qualitative, and some by a com
can be a useful tool in helping family physicians read, bination of the two.
understand, and evaluate the information found in qual What type of study design is best for a particular
itative research studies. This paper describes the qualita research question? An analogy to patient care can be help
tive research process and compares it with similar pro ful in answering this question. Sometimes a clear finding
cesses used by physicians in evaluating and assessing on physical examination or a positive laboratory result can
patients. A series of questions is offered to help the reader cinch a diagnosis. At other times, it is necessary to ask
assess the quality of qualitative studies. The terminology numerous questions, perform an extensive examination,
and look for recognizable patterns, subtle inconsistencies,
Am ited, revised, M ay 18,1995. and hidden agendas before making a diagnosis. In re
trom the D epartm ent o f Fam ily M edicine, Oregon H ealth Sciences University, Port
search, the investigator must ask, “ Given what I want to
e d (N.C.E.), a n d the D epartm ent o f Family Practice, Lehigh Valley Hospital, find out, what is the best research approach?” A quanti
Allentown, Pennsylvania ( W .L.M .). Requests fo r reprints should be addressed to
\oncy C. Elder, M D , M SPH , D epartm ent o f Family M edicine, Oregon H ealth
tative approach seeks to carefully define and measure vari
Sciences University, Portland, O R 97201. ables and outcomes. A qualitative approach seeks to frame
6 1995 Appleton & Lange ISSN 0094-3509
The Journal of Family Practice, Vol. 41, No. 3(Sep), 1995 279
Evaluating Qualitative Research E ld e r and Millet
Table 1. Questions to Ask in Reading Qualitative Research search is done within a specific theoretical framework
1. What is the study question?
However, family physicians, even those with little under
standing of these theories, can read and appreciate quali
2. Is an appropriate research approach taken to the initial study tative studies by looking at these basics of a qualitative
question?
study design: participant selection, data collection, and
3. What is the specific study design? data analysis.
• Who are the participants and how are they chosen?
• H ow are the data collected?
• H ow are the data analyzed? Participant Selection
4. Is the final study design appropriate for the study question? As in any well-done quantitative study, information
should be given on how participants are chosen.8 A ran
domized sample, the standard in quantitative researches
an issue and assess it as a complex whole. For example, to rarely used in qualitative studies. Quantitative studies seek
answer a question about the effectiveness of a new antibi to minimize bias and maximize generalizability by ran
otic to treat urinary tract infections (ie, “what percentage domization. Qualitative studies, on the other hand, aim
of patients are cured?” ), a quantitative approach is pre to maximize the richness of information pertinent to the
ferred. On the other hand, exploring the behaviors and issue being studied. Some of the most common ways for
reasonings behind patient decisions regarding diabetic selecting participants include using key informants, a pur
treatments (eg, “why do many patients with diabetes poseful sample (either random or stratified), a maximum
choose not to follow dietary advice from their physi variation sample, a homogeneous sample, a sample usin?
cian?” ) is better accomplished by a qualitative approach. selective characteristics (eg, extreme cases, important
The study question or purpose, which serves as a cases), or a sample o f convenience.16 Key informants are
“ point of departure” for the reader,8 should be clearly individuals, often opinion leaders or valued members ofa
stated in all research articles. In qualitative research, it is group who can provide insight into a question or issue
possible for the initial question to change while the re being studied.17 Their identification as key informants
search is under way, which may be difficult for the quan should be described in a study using this sampling tech
titatively trained reader to accept. This possibility is based nique. A purposeful or intentional sample includes a
on a central theme in qualitative research: data collection proper mix of participants for the question under studv,
and data analysis proceed together and are performed by perhaps by seeking out individuals who represent both
the same researchers. If the original question is found not extremes of a population spectrum and assuring their in
to be the one of most importance or significance, then the clusion in the study. Once the sample is defined, individ
original research question, study design, or both may uals within that sample may be randomly chosen or tht
change.9’10 In our clinical encounters, it is not unusual for participants may be chosen from within sample sub
the initial “ chief complaint” given by a patient to change groups, with greater or lesser emphasis given to certain
as the clinician asks questions and finds hidden agendas. groups. A maximum variation sample includes individuals
Likewise, in qualitative research, investigators looking at who represent diversity surrounding a chosen area, while
the role of ethnic influences on health beliefs may shift the a homogeneous sample is composed of participants with
focus of their study when they find, after initial interviews, similar characteristics. Some studies, much like case re
that the family influences are more important. ports, may have only one or a handful of participants, who
All studies, qualitative or quantitative, have a struc are often chosen at the time the issue or study question is
ture within which a question or subject is studied.11 The defined. Their identification should include why they
structure o f a qualitative research study may seem fairly were considered important for study. Regardless ofwhich
“ unstructured” to the traditional quantitative researcher sampling method is chosen, justification for such sam
or reader.12 Both quantitative and qualitative research pling should be given.
studies share the same three structural components, how Although it is less desirable than some, a sample ol
ever: sampling, collection, and analysis of data. Qualita convenience is the sampling strategy most often used.
tive studies may seem different because they are flexible Proximity due to location, acquaintance, or ease of com
and changeable, mainly owing to the iterative process, munication often determines who among the population
where repeated questioning (and questioning of the ques of interest are studied.18 As in a clinical encounter witha
tions) occurs. The investigator, rather than trying to elim patient, in which the simplest and most common ques
inate as many biases as possible by using a rigid structure, tions are not necessarily the ones that solve the clinical
actually becomes part of the research, and describes rather dilemma, a sample of convenience in qualitative research
than eliminates known biases.12-15 Much qualitative re does not always represent the best study population. To
280 The lournal of Family Practice, Vol. 41, No. 3(Sep), 199*
Evaluating Qualitative Research Elder and Miller
really understand a patient, physicians must ask difficult as physicians ask themselves similar questions: “ How sure
well as “ convenient” questions. In research, less conve am I of my assessment? Do I need to get more history, do
nient participants might provide new or insightful data. a more extensive examination, or order more tests?”
Unfortunately, in research, participant selection is often When the answers to history questions provide no new
influenced by monetary and time constraints. Working information, and when information from examinations
within these constraints, a qualitative researcher should and tests offers only more confirmation, most physicians
strive to define and use the most appropriate sampling feel comfortable with their diagnosis and proceed to their
strategy for answering the research question. plan. On the other hand, when new information is re
vealed in the history, or tests give disconfirming results, a
physician usually keeps looking for more data. For this
Data Collection
same process to occur in qualitative research, analysis
Data in qualitative studies are usually collected by means must be ongoing during the course of data collection.
of interviews, observation, or review of documents or The criteria the researchers used for stopping data collec
historical records.9’12'18 Because there is wide variation tion, however, must be explicitly stated in their reports.
within these categories, a qualitative study should define Unfortunately, just as in clinical practice, matters of time,
indetail the data collection method(s) used. For example, money, and logistics may be as important as data satura
interviews can be group or individual, long or short, for tion in determining data collection endpoints.
mal or informal, structured or unstructured, single or How the data are collected is also important. Audio-
multiple, and so forth. Since different kinds of data can be taping and videotaping are common in interviews, and are
obtained within each of these types, researchers need to often transcribed for subsequent analysis. It is important
choose the method best suited for their study and ade to assess the accuracy of these transcriptions. Observa
quately describe the type of interview used as well as why tions are documented in researcher “ field notes” or jour
it was chosen. Stories and cognitive processes are often nals.12 The training of the recorder and the timing of the
best understood through interviews, in much the same notes is important in field-note accuracy. Existing docu
way that clinicians use the patient interview to learn the ments themselves may be analyzed using various tech
patient’s story.9 Observation and participation can be a niques. Regardless of the type of data collection tech
valuable tool for collecting data, especially when research nique used, researchers will often keep notes and journals
ers want to examine a phenomenon in its natural setting. of their involvement in the study and a written record of
Observation may be brief or prolonged, and the observer their thoughts as the study progresses.
may take a passive or active role, ultimately even partici
pating in the group being observed. Behaviors, activities
D ata Analysis
ofdaily living, and even conversations are usually studied
by means of observation.9 Clinicians, too, find observa Data analysis in qualitative studies can vary from very
tion helpful. Watching parent-child interactions in the structured and almost “ quantitative” to highly intuitive
examination room or making home visits to see patients in and personal. Data consisting of interview transcripts are
their own surroundings, for example, gives physicians often analyzed for content using “ coding” categories that
new insight into their patients’ lives. Review of written can be decided a priori (before the data are collected) or
records, including charts, journals, and letters, is another ascertained from the data themselves.18’20 These catego
source of qualitative information available to researchers. ries can be counted in a quantitative fashion, and the data
Again, these data may be informally retrieved and studied treated numerically, or they can be used for generating
or formally obtained, and their connection to the study theories or explaining behaviors. Other types of notes or
question may be direct or indirect. Written records can documents may be coded as well, or used in less quanti
ofer insight into previous events and decision-making for tative ways, eg, the researchers immersing themselves in
both researchers and physicians, who often review medi the data with frequent readings and discussions of the
cal records to help them better understand the patient’s data to form theories and explanations. During the data
current story. analysis, qualitative researchers should specifically look for
Unlike quantitative studies, for which sample size discrepant cases and account for them in their study.12
and data collection are often decided before the study is During this time of analysis, reading, discussing, and
undertaken, the decision on “ how much” data to collect theorizing, many of the specific theories of qualitative
isnot as easily decided in qualitative research. The rule of research determine how the researcher addresses the
thumb for qualitative researchers is that when no more data.6’9’10 Those with a grounded-theory background will
new data or ideas are being generated, enough time has attempt to identify the core social processes, or the “what
been spent in the collection phase.19 In a clinical setting, is going on here” within a situation.10’21 Ethnography
The Journal of Family Practice, Vol. 41, No. 3(Sep), 1995 281
Evaluating Qualitative Research E ld e r and MillCr
involves identifying meanings, patterns, and passions of a Table 2 , Questions to Ask in Assessing Qualitative Research
cultural group.22 Phenomenology and hermeneutics both
Trustworthiness
seek to understand the lived experience o f individuals,
1. Who are the investigators and what is happening to them?
and, in the case o f hermeneutics, to better understand the 2. D o the investigators keep following up?
political, historical, and sociocultural context in which it 3. Is there sufficient contact between the investigators and the
participants?
occurs.23 Heuristics places a special emphasis on self
4. Did the investigators becom e too close to their participants?
reflection in the research experience.24 Other researchers 5. Did the investigators cross-check their data?
use a variety o f these traditional approaches to research as 6. Did the investigators “rule ou t” other theories?
they attempt to find meaning in their data to help answer 7. Can the reader follow how the investigators got from the
problem to the plan?
their research question. 8. Were there outside checks?
After reading about the study design, including how 9. D oes the study provide sufficient information for readers to
the participants are chosen and the data collected and determine whether the study applies to them?
Trustworthiness and Believability 1. Who are the investigators, and what is happening tt
them?
Once a reader has explored the issues of study question Knowing oneself and one’s limitations and strengths is
and study design, it is necessary to ascertain if the study important in a physician’s ability to practice the art of
interpretation is valid relative to its intent. In quantitative medicine. Self-questioning or reflexivity is also an impor
research, readers look for validity and reliability. Although tant concept in qualitative research.28 Clinicians do this
these terms are not directly applicable to qualitative re when they examine the physician-patient relationship,
search, the concepts they express are. Quantitatively, re and their role in their patient’s care; researchers do it
liability is related to how reproducible measurements are, when they examine themselves and their roles with their
and validity is related to how the measurements reflect the participants.27’29
“ reality” or the “ truth” of what is being measured.25 In What one knows about another person is important
qualitative research, one looks not so much for validity in evaluating his or her work. In a clinical setting, when a
and reliability as for trustworthiness.13>26-27 Lincoln and colleague or consultant gives an assessment of a patient,
Cuba13 have defined trustworthiness by the terms credi one judges this assessment based in part on what is known
bility, transferability, dependability and confirmability, about the consultant’s strengths and weaknesses as a cli
but family physicians may find it easier to ask nine simple nician. For example, an otolaryngologist, an allergist, and
questions about the “ interpretation” or “ assessment” of a homeopath might each give different assessments of a
282 The Journal of Family Practice, Vol. 41, No. 3(Sep), 199a
Evaluating Qualitative Research Elder and Miller
patient with chronic ear pain. Each specialist sees different important means of ensuring a trustworthy study.12-15-27
“truths.” As outsiders to the actual qualitative research In triangulation, multiple data sources and methods are
project, readers must base their judgment on what they used. Examples include a study that uses interviews with
know of the study investigators based on the investiga participants along with reviews of existing records, or a
tors’ description of who they are and what happened to study that interviews a wide variety of individuals to con
them during the course of the study. firm information from one subgroup.
2. Do the investigators keep following up ? 6. Did the investigators “rule out” other theories?
Iteration, assessing and repeating questions in different In practice, physicians will often look for information that
forms, is a process clinicians use with their patients on a will help prove or disprove a possible diagnosis. When a
daily basis. By asking patients about their complaints in patient has chest pain, and a physician considers a diag
several ways and by following up on previous answers, nosis of myocardial infarction, he or she orders an elec
clinicians are attempting to understand well enough to
trocardiogram and blood test to look for evidence that
diagnose and treat. In qualitative research, by asking
there is or is not an infarction. Qualitative researchers
questions many ways, researchers are endeavoring to suf
must also take an active role in looking for discordant or
ficiently understand the issue being studied.28 As more
disconfirming data.12 It is insufficient to wait for these
information is gained, questions change and gain new
data to show up on their own. Just as a patient who has
dimensions based on what has already been learned.9-10
negative tests for myocardial infarction will need further
3. Is there sufficient contact between the investigators and investigation to find the cause of the chest pain, qualita
theparticipants? tive researchers look for data that might prove their the
What family physicians refer to as continuity of care is ories wrong, so they can then investigate further to find
known as “ prolonged engagement” in qualitative re out what really is going on.
search.27 In clinical practice, it is clear that as a physician
has more encounters with a patient, he or she understands 7. Can the reader follow how the investigators got from the
that patient better. Extended and numerous visits are question to the conclusion ?
especially helpful to physicians in understanding complex Qualitative research, as in clinical practice, rarely unfolds
issues such as coping skills and disease adaptation. As trust as expected. The research design, collection and analysis
is built, there are fewer misunderstandings between phy strategies, and working hypotheses often change in re
sician and patient. Researchers also understand their par sponse to unexpected findings. Clinicians also frequently
ticipants better when they have spent more time with change diagnostic or treatment plans in response to new
them. information. What matters from the patient’s perspective
is that the change, and the physician’s associated decision
4. Did the investigators become too close to their partici making process, are transparent, that is, clearly under
pants? standable and available to the patient.30 The qualitative
While extended contact with patients can help a physi investigator’s decision-making process should be similarly
cian’s understanding, there is always the risk of becoming apparent to the reader.
too involved with a patient and losing objectivity' and
perspective. This balance is a constant struggle for caring 8. Were there any outside checks?
physicians. It is a risk for qualitative researchers as well, Especially with difficult patient problems, physicians talk
who may become so enmeshed with their participants,
with colleagues and consultants for advice or another
that they can no longer look critically at the issues of
opinion. Physicians document their decisions with a com
concern.27-29
plete medical record that clearly states what decisions
were made, tests ordered, specialists consulted, and so
5. Did the investigators cross-check their data ?
forth. When needed for billing, legal, or quality assurance
Inclinical practice, physicians consider findings from both
the history and the examination in making a diagnosis. At reasons, this record serves as an audit of what was done in
times, data are also needed from laboratory, radiography, caring for the patient. In qualitative research, an audit
or other types of tests. Occasionally, input is requested allows other researchers and co-investigators to offer sug
from consultants and other health care professionals. In gestions and input into the process of data analysis and
formation from all these sources helps a physician to refine theory development.27-31 This review by colleagues and
and feel comfortable with an assessment and plan. In other researchers can help clarify issues and point out
qualitative research, “ triangulation” is one of the more overlooked investigator bias. 12-29
The Journal of Family Practice, Vol. 41, No. 3(Sep), 1995 283
Evaluating Qualitative Research E ld e r and Miller
9. Does the study provide sufficient information for the read knowledge of family medicine. Researchers who perform
ers to determine whether the study applies to them ? an impressive study on an important topic but then do not
When presenting patients to colleagues, enough informa address issues of trustworthiness in their report are simi
tion must be provided so that the person listening has a larly ineffective. Finally, a poorly prepared or written study
good understanding o f the patient and the issues in usually attracts only a small audience.
volved. The same is true in qualitative research. By using “ Is it clinically convincing?” is an especially pertinent
what is called a “ thick description,” the researchers give question for practicing physicians to ask. The clinical
the readers enough information for them to determine if questions best answered by qualitative research include
the participants, setting, and information are pertinent to who are our patients, what is it that we do, and what
their practice and work.12-27 difference does it make? In other words, qualitative re
search should help us better understand our patients’
After a reader determines that a study is trustworthy, lives, the context o f our clinical decisions, or the impact of
he or she should look at whether the study is believable. A
our clinical action on patients, families, and the commu
reader can assess the writing, organization, and interpre
nity. Does the story the researchers tell “ click” with a
tation by asking: Does the “ story” make sense? In clinical
clinician’s experience and help explain phenomena that
practice, a “ good” diagnosis is one that offers the most
they themselves have experienced? Does it motivate them
likely explanation for the patient’s problem. It should also
to change or adapt their own practices?
be consistent with known scientific data, clear to other
By understanding study design and questions for the
professionals, and a good interpretation of the patient’s
purpose of assessing trustworthiness, readers of the med
story. A good diagnosis leads to a plan that will help the
ical literature who are not qualitative researchers can bet
patient improve. In qualitative research, this rhetorical
ter understand and appreciate qualitative studies. By in
quality of the written study is also important. The follow
corporating qualitative as well as quantitative research
ing factors are significant in determining how well the
researchers have related their story: articles into journal clubs and an individual physician’s
regular reading program, physicians will be better pre
• Is it parsimonious? A good study should not have to pared to recognize and understand medical research that
make large numbers of assumptions in order to explain addresses “ patient-oriented evidence that matters”
the data; economy of words and ideas is the goal. (POEMs).1 With this understanding, they will be able to
critically analyze qualitative as well as quantitative studies
• Is it consistent? A study should either conform to what is before accepting and applying the findings of a research
already known, or the reasons for its uniqueness and dis study to their own lives and practices.
cordance should be explored and explained.
284 The Journal of Family Practice, Vol. 41, No. 3(Sep), 1995
Evaluating Qualitative Research Elder and Miller
[1 Ward MM. Study design in qualitative research: a guide to assessing 21. Glaser B, Strauss AL. The discovery o f grounded theory. Haw
quality. J Gen Intern Med 1993; 8:1 07-9. thorne, NY: Aldine de Gruyter, 1967.
12. Smith III AC, Noblit GW. The idea o f qualitative research in med 22. Murdock GP, Ford CS, Hudson AE, et al. Outline o f cultural
ical education. Teach Learn Med 1989; l(2 ):1 0 1 -8 . materials. 3rd ed. New Haven, Conn: Human Relations Area Files,
13. Lincoln YS, Guba EG. Naturalistic inquiry. Newbury Park, Calif: 1950.
Sage Publications, 1985. 23. Packer MJ, Addison RB. Entering the circle: hermeneutic investi
14. Steckler A, McLeroy KR, Goodman RM, Bird ST, McCormick L.
gation in psychology. Albany, NY: State University o f New York
Toward integrating qualitative and quantitative methods: an intro
Press, 1989.
duction. Health Educ Q 1992; 19(1): 1 -8 .
24. Moustakas C. Heuristic research: design, m ethodology and appli
15 Kuzel AJ. Naturalistic inquiry: an appropriate model for family
cations. Newbury Park, Calif: Sage Publications, 1990.
' medicine. Fam Med 1986; 18^:369-74.
16. Kuzel A. Sampling in qualitative inquiry. In: Crabtree BE, Miller 25. Kirk ), Miller ML. Reliability and validity in qualitative research.
WL, eds. D oing qualitative research. Newbury Park Calif: Sage Qualitative research methods series 1. Newbury Park, Calif: Sage
Publications, 1 9 9 2 :3 1 -4 4 . Publications, 1986.
17. Gilchrist VJ. Key informant interviews.In: Crabtree BF, Miller WL, 26. Guba EG. Criteria for assessing the trustworthiness o f naturalistic
eds. Doing qualitative research. Newbury Park Calif: Sage Publica inquiries. Educ Community Technol J 1981; 29(2):7 5 -9 1 .
tions, 1 9 9 2 :7 0 -8 9 . 27. Krefting L. Rigor in qualitative research: the assessment o f trust
18. Cobb AK, Hagemaster JN. Ten criteria for evaluating qualitative worthiness. Am J Occup Ther 1991; 4 5 :2 1 4 -2 2 .
research proposals. J Nurs Educ 1987; 2 6 (4 ):1 3 8 -4 3 . 28. Miller WL, Crabtree BF. Qualitative analysis: how to begin making
19. Conrad P. Qualitative research on chronic illness: a commentary on sense. Fam Pract Res J 1994; 1 4 :289-97.
method and conceptual development. Soc Sci Med 1990; 29. Ammon-Gaberson KB, Piantanida M. Generating results from
30:1257-63. qualitative data. Image: J Nurs Scholar 1988; 20(3): 1 5 9 -6 1 .
20. Crabtree BF, Miller WL. A template approach to text analysis: 30. Brody H. The healer’s power. New Haven, Conn: Yale University
developing and using codebooks. In: Crabtree BF, Miller WL, eds. Press, 1992.
Doing qualitative research. Newbury Park, Calif: Sage Publications, 31. Sandelowski M. The problem o f rigor in qualitative research. Adv
1992:93-109. Nurs Sci 1986; 8(3):27 -3 7 .
The Journal of Family Practice, Vol. 41, No. 3(Sep), 1995 285