Chapter 10

Thinking Critically About the Quality

of Critical Thinking Definitions
and Measures

Lily Fountain

Abstract Leading policy and professional organizations are in agreement that

critical thinking is a key competency for health professionals to function in our
complex health care environment. There is far less clarity in professions education
literature about what it really means or how to best measure it in providers and
students. Therefore, in order to clarify terminology and describe the context, def-
initions, and measures in critical thinking research, a systematic review using the
keywords “critical thinking” or the associated term “clinical reasoning,” cross
keywords nurse and physician, resulting in 43 studies is presented in this chapter.
Results indicated that an explicit definition was not provid7ed in 43 % of the
studies, 70 % ascribed equivalency to the terms critical thinking, clinical reasoning,
clinical judgment, problem-solving, or decision-making, and 40 % used
researcher-made study-specific measures. Full alignment of definition and measure
was found in 47 % of studies, 42 % of studies reported validity for obtained scores,
and reliability was documented in 54 % of studies. Keyword analysis identified six
common constructs in nursing studies of critical thinking in nursing: individual
interest, knowledge, relational reasoning, prioritization, inference, and evaluation,
along with three contextual factors: patient assessment, caring, and environmental
resource assessment. Recommendations for future research include increased use of
purposeful samples spanning multiple levels of provider experience; documentation
of effect sizes, means, SD, and n; adherence to official standards for reliability and
validity requiring documentation of both previous and current datasets as appro-
priate; and greater use of factor analysis to validate scales or regression to test
models. Finally, critical thinking research needs to use explicit definitions that are
part and parcel of the measures that operationalize the construct, and alternative
measures need to be developed that line up the attributes of the definition with the
attributes measured in the instrument. Once we can accurately describe and measure
critical thinking, we can better ensure that rising members of the professions can
apply this vital competency to patient care.

L. Fountain (&)
Faculty of Nursing, University of Maryland School of Nursing, Baltimore, MD, USA
e-mail: fountain@son.umaryland.edu

© Springer International Publishing Switzerland 2016 185

P.F. Wimmers and M. Mentkowski (eds.), Assessing Competence in Professional
Performance across Disciplines and Professions, Innovation and Change
in Professional Education 13, DOI 10.1007/978-3-319-30064-1_10
186 L. Fountain

Takeaways
• Performance constructs such as critical thinking should be explicitly
defined and measures should align with these definitions.
• Previous critical thinking research has usually examined learners at only
one level of experience, such as students, new graduates, or experts.
Future studies can be strengthened by examining learners at more than one
level of experience.
• Domain-specific measures are preferable for performance assessment. For
example, in nursing education studies, individual interest, knowledge,
relational reasoning, prioritization, inference, and evaluation, along with 3
contextual factors, patient assessment, caring, and environmental resource
assessment, were identified as common keywords.
• Professions education research is strengthened when AERA/APA/NCME
standards for reliability and validity are followed.

10.1 Introduction

Nurse Jennifer has been a maternity nurse for 5 years and has been fascinated by maternity
nursing since her basic education in nursing. Today, she stares thoughtfully at her patient
Mrs. Gablonsky. Nurse Jennifer sees something surprising. Mrs. Gablonsky’s condition
differs in a way the nurse does not expect for a woman who birthed a baby the previous day.
Nurse Jennifer wonders about what is causing Mrs. Gablonsky’s state and questions the
patient closely to find out if there were any symptoms that could help explain Mrs.
Gablonsky’s condition. Nurse Jennifer compares Mrs. Gablonsky’s condition to the other
postpartum women she has treated in her career. She searches her mental data base for
knowledge about complications that could be consistent with the symptom that surprised
her. After a few moments of contemplation, Nurse Jennifer knows how to help her patient.

As this scenario illustrates, critical thinking using key cognitive processes is cen-
trally involved in the quality of care provided by maternity nurses and other healthcare
professionals. Leading policy and professional organizations such as the Institute of
Medicine, Carnegie Foundation, and American Association of Colleges of Nursing are
in agreement that critical thinking is a key competency for health professionals to
function in our complex health care environment (American Association of Colleges
of Nursing 2008; Institute of Medicine 2010; Cooke et al. 2010). There is far less
clarity in professions education literature about what critical thinking and its analog in
practice, clinical reasoning really mean or how to best measure it to ensure compe-
tence in providers and students. Toward this end, this systematic review was con-
ducted to examine the quality of definitions and measures of critical thinking and
clinical reasoning within the literature pertaining to health care professions.
10 Thinking Critically About the Quality of Critical Thinking … 187

What is meant by critical thinking? Facione (1990) gives one commonly used
definition of critical thinking as “purposeful, self-regulatory judgment which results
in interpretation, analysis, evaluation, and inference” (p. 2). This present study also
examined the term clinical reasoning, defined by Higgs and Jones (2000) as “the
thinking and/or decision-making processes that are used in clinical practice”
(p. 194). Although other terms have been used to describe clinical thinking, such as
clinical judgment, problem-solving, and decision-making, the terms critical think-
ing (CT) and clinical reasoning (CR) were chosen as the basis for this systematic
review because a preliminary electronic database search indicated they were the
most commonly equated terms populating the recent research.
Specifically, from the 1980s to the present, critical thinking and clinical rea-
soning have been areas of intense research and clinical interest in nursing and
medicine, as documented by several recent reviews of the literature (Brunt 2005;
Chan 2013; Norman 2005; Ross et al. 2013; Simpson and Courtney 2002; Walsh
and Seldomridge 2006a). Critical thinking has been identified as a key construct in
core competencies for interprofessional collaborative practice and consensus
statements on critical thinking in health professions education (Huang et al. 2014;
Interprofessional Education Collaborative Expert Panel 2011). The most important
reason for studying critical thinking in the health professions is that health care
providers, educators, researchers, and policy-makers believe it leads to better health
care. It has been argued that critical thinking has the potential to reduce morbidity
and mortality for patients, and increase patient satisfaction with care. It can reduce
health care costs by avoiding mistakes, unnecessary procedures, and unnecessary
use of supplies (Benner et al. 2008; Kataoka-Yahiro and Saylor 1994).
However, problems with the definitions and measures used in critical thinking
and clinical reasoning have been recognized (Krupat et al. 2011; Walsh and
Seldomridge 2006a). Further, the quality of definitions and measures used in
educational research on critical thinking and clinical reasoning affects the ability to
evaluate which educational strategies are effective at promoting these skills (Brunt
2005). In addition, the context of health education research affects the quality of
research (Ovretviet 2011). Merriam-Webster defines context as “the interrelated
conditions in which something exists” (Context n.d.). The types of participant
samples and research designs used in health education affect the ability of
researchers to clearly define and measure constructs in the health professions (Waltz
2010).
In order to function with the same level of research rigor as clinical practice
(Harden et al. 2000), clear definitions (Creswell 1994, 2014) and measures
(Ratanawongsa et al. 2008) have been identified as essential steps in producing
quality professions education research. With this increased focus on evidence-based
practice during education and clinical practice, in combination with the other
pressures on health care and education, it is vital that a shared base of terminology
and psychometrically sound assessment tools be identified.
There is long-standing literature on the problems with definitions of CT terms,
which fall into the categories of clarity, domain specificity, and equivalency of term
usage. The lack of explicit, clear definitions of terms has been cited as a
188 L. Fountain

methodological issue by other researchers in educational psychology (Alexander

et al. 2011). Alexander et al. (2011) found that many studies were not explicitly
defining terms for higher order learning. Similarly, Brunt (2005) stated that “to
evaluate CT, educators need a clear definition or framework to use” (p. 255), noting
the deficit specifically for critical thinking in nursing studies. Carpenter and Doig
(1988) also advocated for the importance of clear critical thinking definitions in
order to measure it. Lack of domain specificity has also been cited as a problem
with definitions of these constructs (Ennis 1991; Pintrich et al. 1991). Walsh and
Seldomridge (2006b) concluded that domain-specific standardized critical thinking
instruments produced mixed results in nursing studies due to domain-general
definitions.
The tendency to equate different terms for critical thinking in the professions
with each other has been noted by many others (Patel et al. 2004; Simmons 2010;
Simpson and Courtney 2002). For one, Patel et al. (2004) noted that the terms
clinical reasoning, problem-solving, and decision-making have been used in
medicine to describe how physicians make decisions (p. 2). Simmons described
decision-making, problem-solving, clinical judgment, and clinical reasoning as
being used synonymously (p. 1152). Likewise, Simpson and Courtney (2002)
stated that “the multiplicity of definitions of critical thinking proved to be a hin-
drance” (p. 7).
In addition to these problems of clarity of definitions of critical thinking and
clinical reasoning, problems in the psychometric properties of measures of critical
thinking have also been identified (Ennis 1989). The features of measures that have
been examined in studies of methodological quality include alignment (Simmons
2010), reliability, validity (Cook and Beckman 2006), and commonality (Abrami
et al. 2008). For example, alignment, the degree of match between a construct’s
definition with operationalized measures in a study, has been examined for
self-regulated learning in education (Dinsmore et al. 2008). Ratanawongsa et al.
(2008) noted that “educators should consider whether previously developed
instruments are valid for their targeted objectives.”
Concerns regarding reliability and validity of data obtained from measures of
critical thinking have been identified; Cook and Beckman (2006), for example,
criticize the practice of citing evidence from a previous study only, in order to
support validity. Watson et al. (2002) found that the lack of agreement on defini-
tions contributed to the lack of validity of resulting data (p. 423). The Standards for
Educational and Psychological Testing (AERA 1999) recommend that researchers
document reliability and validity.
The final area used to examine the quality of measurement tools is commonality,
used here to describe the degree to which a measure is shared in use by other
researchers. This was coded in a meta-analysis of critical thinking instructional
interventions: Abrami et al. (2005) categorized the measures used as: standardized
tests; tests developed by a teacher; tests developed by a researcher; tests developed
by teacher–researchers; and secondary source measures, which are measures
adopted or adapted from other sources. One of the reasons that type of measures has
been examined in critical thinking research is that standardized test scores in
10 Thinking Critically About the Quality of Critical Thinking … 189

particular did not show expected improvements in critical thinking over the course
of professional education (Walsh and Seldomridge 2006a).
Thus, the need exists for a detailed and systematic description of the types and
quality of definitions and measures of critical thinking and clinical reasoning used
in the health professions. This study seeks to address that gap. The aim of this
review is to lay the foundation for improving the conceptualization and opera-
tionalization of critical thinking and clinical reasoning in the health professions by
addressing the following research questions:
1. What is the nature of the context in which critical thinking and its analog clinical
reasoning have been examined?
2. How and how well are definitions for critical thinking and clinical reasoning
specified in the literature?
3. How and how well are measures of critical thinking and clinical reasoning used
in the literature?

10.2 Methods

10.2.1 Data Sources

I conducted a systematic review of the literature, using specific terms, delimiters,

and selection criteria. A combination of electronic and hand searching were used to
minimize bias in article selection (Cooper et al. 2009).

10.2.2 Study Selection

Inclusion criteria were peer-reviewed, empirical journal articles published between

2006 and 2011 using the terms clinical reasoning or critical thinking, searching the
PsycINFO database. PsycINFO, as an interdisciplinary database for behavioral and
social sciences research, that includes psychology, nursing, and education, was
chosen as the database for this study in order to more closely examine the edu-
cational psychology research on this topic and maintain an educational focus, not a
clinical one. Delimiters included the database: PsycINFO; keywords: clinical rea-
soning or critical thinking; cross keywords: nurs* and doctor, physician; language:
English; type of article: empirical research; publication time period: June 2006 to
June 2011; population: human; and publication type: peer-reviewed journals. To
focus on the evaluation of critical thinking in practice, selection criteria required
that included articles (a) studied direct patient care, or education to provide patient
care, in the health sciences and (b) concerned the thinking of individual human
health professionals (not computers), and (c) that the research directly measured
190 L. Fountain

thinking, not dispositions, confidence, or self-efficacy. The search strategy is

summarized in Table 10.1.
In addition to the electronic search, footnote chasing was used to identify rel-
evant studies in the reference lists of the study pool of articles. Eight articles that
met the criteria that were identified by this method were included. The tables of
contents of the electronic search’s most commonly identified journal, Journal of
Advanced Nursing, was also physically searched for the year 2011, but no new
articles were identified. The resulting pool of articles comprised the data source for
this systematic review. Figure 10.1 summarizes the disposition of articles in the

Table 10.1 Search strategies 1. EBSCO online research platform

for literature
2. PsycINFO Database
3. Advanced string search—Boolean/Phrase
4. Critical thinking in title (or)
5. Critical thinking in abstract (or)
6. Clinical reasoning in title (or)
7. Clinical reasoning in abstract (and)
8. Nurs* (or)
9. Physician (or)
10. Doctor
11. Narrow results by source type: empirical periodicals
12. Narrow results by year (2006–2011)

Electronic database
Reasons for Exclusion at Title/Abstract
PsycINFO Level:
224
Duplicates Violated Selection Criteria:
1)Not Health Sciences research and about
46
the thinking of health professionals: 56
Title and Abstract 2)Not Analysis of thinking skills used by
individual humans (not computers or
Review
groups) on direct patient care or during
178 education to provide patient care: 34
3)Did not measure thinking as indicated by
Excluded measures, or measured dispositions
105 including confidence or self-efficacy: 11
Article Review 4)Violated Delimiter: 4
73

Duplicates
Hand 9 Reasons for Exclusion at Article Level:
searching Violated Selection Criteria:
from 1)Not Health Sciences research and about
reference the thinking of health professionals: 6
2)Not Analysis of thinking skills used by
lists
individual humans (not computers or
8 Excluded
groups) on direct patient care or during
29 education to provide patient care: 3
3)Did not measure thinking as indicated by
measures, or measured dispositions
including confidence or self-efficacy: 12
Included Studies 4)Violated Delimiter: 8
43

Fig. 10.1 Summary of literature search and review process for primary literature
10 Thinking Critically About the Quality of Critical Thinking … 191

review process. As the figure indicates, 224 abstracts were produced by the search
terms; after title, abstract, and full article review, 43 articles met the criteria for
inclusion in the review.

10.2.3 Coding Protocol

In order to clarify the definitions and measures used for critical thinking research,
an explicit coding scheme was used. I developed a protocol based on the coding
typology used in prior research (Alexander and Murphy 2000; Dinsmore et al.
2008), and the recommendations of the Best Evidence Medical Education (BEME)
collaboration (Harden et al. 2000). The coding typology for Dinsmore et al. (2008)
was adapted to classify explicitness or clarity of definitions, and the alignment of
measures with the definitions. The categories of study design, study size, target
outcomes, and level of experience of participants were chosen from the BEME
categories to evaluate the quality of medical education research studies.
Overall, the study variables were categorized as relating to the contextual aspects
of the studies, the definitions, or the measures. A general description of the coding
is given here, and Appendix A details the components of the codebook that was
developed, that specified all resulting codes and was used to establish interrater
agreement on the coding scheme.
Context. Variables coded for context include purpose, participant variables, and
research design.
Purpose. The constructs critical thinking and clinical reasoning were examined
for several purposes. McCartney et al. (2006) caution that care must be exercised
when a measure is used differently than its intended purpose, such as either clinical
evaluation, screening, or research. In this study, the purposes were categorized into
four types: 1—evaluation of a program of education; 2—evaluation or description
of a teaching technique; 3—evaluation of admission, course performance, or pro-
gression decisions; 4—evaluation or description of students, faculty, or providers.
Participants. The participants in the study pool were coded by professional
domain, level of experience, and number. For this study, professional domain was
determined at the level of licensure, such as nursing or medicine; the study may
have focused on a subspecialty of the domain, such as emergency nursing or cardiac
medicine but these were coded according to the overreaching domain category. In
this study, medicine refers to the profession of doctors or physicians. Nursing refers
to the profession of nurses, and includes registered nurses at all education levels. In
addition to medicine and nursing, articles from veterinary medicine, kinesiology,
health sciences, and occupational therapy were produced by the search. Since the
goal for this study was an examination of terms used in critical thinking across
multiple health care domains, these articles, which included references to medicine
or nursing, were retained if they met the delimiters and selection criteria. In addi-
tion, two articles examined more than one profession, and were labeled multidis-
ciplinary. Each study was coded for the level of experience of participants, either
192 L. Fountain

(a) student or prelicensure, (b) new graduate or residents, (c) practicing provider, or
(d) multiple levels. The number of participants or sample sizes were categorized as
small, with less than 30 participants, moderate, with 31 to 100 participants, or large,
with over 100 participants.
Research designs. The studies were categorized by type of research design.
Preexperimental designs included one group pretest/posttest and cross-sectional
studies. Quasi-experimental designs included separate sample pretest/posttest
design, and separate sample pretest/posttest control design. Examples of experi-
mental designs include pretest/posttest control group design and Solomon
four-group design. Case studies were coded as qualitative.

10.2.4 Definitions

For each study, the definition or descriptive data about of critical thinking or clinical
reasoning was coded for clarity, domain specificity, and equivalency.
Clarity. In this study, clarity refers to whether the definition was explicitly or
implicitly defined in the study. A definition was coded as explicit if the author
explicitly stated the definition of critical thinking used in the study. For example, in
Funkesson et al. (2007), the following definition for clinical reasoning was
explicitly stated: “In this paper, clinical reasoning is seen as a cognitive process,
where both theoretical knowledge and personal experience are used in a unique care
situation aiming to achieve a desired outcome for the person in focus” (p. 1110).
In order to analyze the lack of distinction in CT/CR term usage, the explicit
category was further delineated. If the definition was explicitly stated, the definition
was analyzed as to whether it was a definition shared by other researchers in
published research, or an idiosyncratic definition used by the researcher for this
specific study. For example, Blondy (2011) stated this definition: “We understand
critical thinking to be purposeful, self-regulatory judgment which results in inter-
pretation, analysis, evaluation and inference… and inductive and deductive rea-
soning” (p. 182), and this was coded as explicit shared. Appendix B contains a list
of shared definitions.
Forneris and Peden-McAlpine (2007), on the other hand, explicitly stated her
own unique definition: “Grounded in these theoretical views, critical thinking is
defined as a process of reflective thinking that goes beyond logical reasoning to
evaluate the rationality and justification for actions within context…Using the work
of these theorists, four attributes of critical thinking: reflection, dialog, context, and
time (p. 411). This definition was coded as explicit idiosyncratic. Idiosyncratic
explicit definitions were definitions that contained components that were specific to
this study, not captured by previously published definitions.
Another example of an explicit but idiosyncratic definition is “critical thinking:
problem identification, problem definition, exploration, applicability, and integra-
tion” (Schnell and Kaufman 2009). The keywords “applicability” and “integration”
10 Thinking Critically About the Quality of Critical Thinking … 193

were not found in common CT definitions, so this study was coded as explicit
idiosyncratic.
If the definition was only implicitly defined, the definitional data were further
analyzed as to the manner in which the construct was discussed and was coded as
conceptual, referential, or measurement. If the construct was discussed through the
use of related concepts, it was coded as implicit conceptual. For example, Mamede
et al. (2007) stated:
Judicious judgements [sic] and effective decision making define successful clinical problem
solving. Two different approaches for processing clinical cases, nonanalytical and analyt-
ical, have been shown to underlie diagnostic decisions. Experienced doctors diagnose
routine problems essentially by recognizing similarities between the actual case and
examples of previous patients. This pattern-recognition, non- analytical form of clinical
reasoning is largely automatic and unconscious. In the second, analytical form of case
processing, clinicians arrive at a diagnosis by analyzing signs and symptoms, relying on
biomedical knowledge when necessary. (p. 1185)

Mamede’s discussion used many concepts that are part of the discussion of
clinical reasoning but no definition was stated. Thus, this definition was coded as
implicit conceptual.
If the author did not clarify which definition was being used and cited definitions
used in other literature, the definition was coded as implicit referential. For
example, Göransson et al. (2007) stated that, “deductive content analysis was fol-
lowed, using the thinking strategies described by Fonteyn and Cahill (1998) from a
long-term TA study” (emphasis added). If the author only defined the construct
through the use of the measure, the clarity was coded as implicit measurement. For
example, Wolpaw et al. (2009) measured the following outcomes: summarizing
patient findings; providing a differential diagnosis; analyzing possibilities in dif-
ferential diagnosis; expressing uncertainties and obtaining clarification; discussing
patient management; and identifying case-related topics for further study. Because
they did not define clinical reasoning, but measured these aspects of clinical rea-
soning, so this study was coded as implicit measurement.
Domain specificity. Domain specificity refers to whether the definition used was
specific to a domain or was generally applicable. In this study, if a definition was
specific to a domain, it was coded as domain-specific. By contrast, if the definition
could be used by other professions it was coded domain-general. For instance,
Johansson (2009) defined clinical reasoning as “the cognitive processes and
strategies that nurses use to understand the significance of patient data, to identify
and diagnose actual or potential patient problems, to make clinical decisions to
assist in problem resolution and to achieve positive patient outcomes” (p. 3367).
The specification of the definition as pertinent to nurses rendered this a
domain-specific definition. On the other hand, Ajjawi and Higgs (2008) stated
“clinical reasoning is defined as the thinking and decision-making processes
associated with professional practice,” and this was coded as domain-general.
Equivalency. Equivalency refers to how the numerous terms used to describe
critical thinking were used in the study. Up to 43 terms for critical thinking have
been identified (Turner 2005). Often authors would state that one term was
194 L. Fountain

equivalent to another by using such phrases as “also known as” and “or.” This
intermingling of terms was analyzed and coded as “equivalency.” For example,
Funkesson et al. (2007) stated “clinical reasoning…can be named critical thinking,
reflective reasoning, diagnostic reasoning, decision making, etc.” (p. 1110). This
was coded as equivalency present. For the purposes of this study, the terms ana-
lyzed for statements of equivalency were critical thinking, clinical reasoning,
clinical judgment, decision-making, and problem-solving.

10.2.5 Measures

In addition to definitions, the instruments or measures used in studies during this

time period were coded for definition—measure alignment, reliability, validity, and
commonality of measures used.
Alignment. Alignment of the definition under discussion with the
measure/instrument used is valued in educational research. I used the typology
described in Dinsmore et al. (2008) to categorize the amount of congruence of this
operationalization of the definition and the measure for each article in this review.
Only articles with explicit or implicit through concepts definitions were analyzed
for alignment. Categories were defined as full (clear match between definition and
measure), partial (partial match; only some of the items measured by the instrument
were in the definition), and not applicable (referential or measurement definitions).
For example, per the instrument development description, the components of the
California Critical Thinking Skills Test are based on Facione (1990) definition of
critical thinking, so there was full alignment of definition and measure. By com-
parison, the study by Nikopoulou-Smyrni and Nikopoulos (2007) used the
Anadysis Critical Reasoning Model to measure CR. The definition of CR stated that
treatment should be “in collaboration with a patient or family (if possible).”
However, the steps in the model (i.e., gather and evaluate data, predict risk, develop
treatment plan, monitor progress) did not explicitly include the patient input in
measurement so the study was coded as partial for alignment of definition with
measure.
Keywords. In order search for common ground among the many definitions, the
attributes described in definitions were summarized in key words or phrases, and a
list compiled (see Appendix C). Keywords were chosen from the actual words used
in the definition or a close synonym or a word or phrase given equivalence in the
studies. For example, many studies used some phrase including data, such as data
collection, data gathering, or noticing. Significance of data was a different key-
word. If the definition was an explicit shared definition, the keywords were assumed
to remain the same. The explicit idiosyncratic and implicit conceptual definitions
were also coded as to the keywords used. If the definition was implicit through
measurement or referential to another researcher, keyword analysis was not done.
This enabled the categorization of the amount of alignment between definitions and
their operationalization through measures.
10 Thinking Critically About the Quality of Critical Thinking … 195

Reliability. According to the Standards for Educational and Psychological

Testing (APA/AERA/NCME 1999, 2014), reliability of data should be established
for the current dataset except in the case of a standardized measure. Presence of
stability, internal consistency and interrater reliability are a necessary condition for
quality studies (Kimberlin and Winterstein 2008). The presence or absence of such
documentation was coded for each study. In the case of qualitative studies, a
discussion of evidence of rigor, such as triangulation, verbatim participant tran-
scripts, and multiple researcher analysis, was used to indicate the presence of
reliability (Bashir et al. 2008). Reliability was coded as (a) present for the current
dataset except in the case of a normed test, (b) present from previous research,
(c) present from a combination of current and previous datasets, or (d) absent.
Validity. For defensible inferences, valid measures must be used (Crocker and
Algina 2006). Cook and Beckman noted that “validity is a property of inferences,
not instruments; valildity must be established for each intended interpretation”
(p. 166.e8). If measures of content, predictive, concurrent, or construct validity
were documented, the validity was coded as present versus absent. The validity was
coded as (a) present for the current dataset, (b) present from previous dataset,
(c) present for both current and previous studies, or (d) absent.
Commonality of measure use. The types of measures were categorized as to
commonality, defined as whether or not the measure was created by the researcher
specifically for this study, or was a previously used measure. This is important
because using reliable, previously validated measures, instruments, and scales is
valued in research (Kimberlin and Winterstein 2008). Each measure was coded as
to whether it was study-specific by the researcher, standardized, or a commonly
used measure. An instrument that was used in at least one other published study
was coded as commonly used; a study that used a standardized, normed, com-
mercial instrument was coded as standardized; a study that was designed for the
study was coded as study-specific. An example of a researcher-made study-specific
measure was the Cognitive Performance Instrument developed by Johnson et al.
(2008) for their study of critical thinking during chemical warfare triage.
Researchers McAllister et al. (2009) developed a study-specific evaluation tool with
criteria for clinical reasoning in emergency nurses for their study. The California
Critical Thinking Skills Test was a standardized test in this coding scheme.

10.3 Results and Discussion

10.3.1 Research Question 1: What Is the Nature

of Contextual Variables in Critical Thinking Studies?

In order to evaluate the findings regarding the contextual aspects, the purpose,
participants, and research design findings will be reported and discussed.
Purpose. Nearly half (49 %) of the articles in the resulting pool of 43 articles
were about critical thinking, and 51 % had clinical reasoning as the focus.
196 L. Fountain

Evaluation or description of a teaching technique was the most common category of

purpose studied in the pool, with 63 % of the studies, followed by 26 % of studies
done for evaluation or description of students, faculty or providers, 7 % for eval-
uation of a program of education, and 5 % for evaluation of admission, course
performance, or progression decisions.
The purpose of most of these studies was to evaluate teaching strategies. The
lack of explicit, nonequivalent, domain-specific definitions and standardized or
commonly used domain-specific measures may be limiting the purposes that can be
researched.
Participants. The domains of the participants examined by the studies were
distributed as follows: 53 % nursing, 26 %, medicine, 5 % Occupational Therapy,
Interdisciplinary 5 %, Veterinary 5 %, and Other 5 %. The level of professional
experience of the participants varied across the professional life span, ranging from
student to resident/new grad to practicing professional. Over half of the studies
examined participants at the student level (51 %), 35 % studied participants at the
provider level of experience, 12 % examined residents or new grads, and 2 %
studied participants at more than one level of experience. The sample sizes ranged
from 6 to 2144 participants. Thirty-seven percent of the articles had small sample
sizes, 40 % had moderate sample sizes, and 23 % had large sample sizes.
There were several aspects of the participants in the samples that were notable.
There were differences in the use of the main constructs between the domains.
There is a difference in the amount of usage of CT compared to CR between
medicine and nursing domains, in that medicine is mostly focused on CR, whereas
nursing research included both terms. Possible explanations might be that nursing
research in an earlier developmental stage, or that nursing include more categories
of behavior outside diagnosis, such as resource allocation, interpersonal negotia-
tion, stronger patient relationships, as well as clinical knowledge of best practice for
the situation. Nursing protests the fact that caring and integrative functions of
nursing care are often not included in measures (Tanner 1997), and medicine
complains of the focus on diagnostic reasoning and not other aspects of medicine
(Norman 2005).
To really understand critical thinking we must look at its initial development and
refinement across the professional life span; however, most of the studies focused
on students only. This is tied to a sampling problem; very little purposive sampling
was done; most studies were conducted at only one site, and researchers were
investigating participants at their own institution.
Research Designs. The studies were categorized according to the research
design categories described in Creswell (1994, 2014): experimental,
quasi-experimental, preexperimental, and qualitative. Experimental studies com-
prised 16 % of the studies, quasi-experimental studies comprised 28 % of the
studies, preexperimental studies comprised 33 % of the studies, and qualitative
studies comprised 23 % of the studies.
This finding quantifies a trend noted in previous studies (Johnson 2004) of a very
small number of experimental studies. Without experiments it is difficult to test
pedagogies to produce an evidence base for teaching.
10 Thinking Critically About the Quality of Critical Thinking … 197

10.3.2 Research Question 2: How and How Well Are

Definitions for Critical Thinking and Clinical
Reasoning Specified in the Literature?

Clarity: Explicit definitions were used in 58 % of the studies (Table 10.2).

Sixty-eight percent of the explicit definitions were shared, and 40 % of all defini-
tions were explicit shared, so this was the largest subcategory of definitions.
Thirty-two percent of the explicit definitions were idiosyncratic. For all of the
studies using implicit definitions, 39 % used implied-conceptual definitions, 33 %
used implied-referential, and 28 % used implicit through measurement definitions.
Interrater reliability was established for the coding of clarity of the definitions.
Interrater reliability was established using two raters, the author and a second rater
who is a doctorally prepared scientist. After discussion of the codebook, a random
sample of 10 % of the studies was examined until high interrater agreement was
obtained (α = 0.90).
As noted in Table 10.2, the results indicated that for clarity of definitions, a very
large minority of studies, 42 %, did not explicitly state the definition used in the
study. Although this has been mentioned in the literature as a problem
(Ratanawongsa et al. 2008), this review documents the great extent to which this is
an issue for the critical thinking literature. It also makes clear that there is a greater
consensus on explicit definitions of critical thinking (73 % of explicit definitions
were shared compared to 60 % for clinical reasoning), but this is largely because
standardized instruments were used to a much greater extent in CT studies. In the
studies where definitions were implicit, half of the CT studies were referential,
using a previous source for the definition, whereas CR implicit studies were 50 %
conceptual. It may be that using concepts as opposed to the instrument or an outside
source for the definition is more appropriate and better quality strategy to indicate a
study’s definition of critical thinking, so this trend may be one of several that may
indicate greater development in study quality for CR studies.

Table 10.2 Frequency and Definitional Construct

relative percent of definitions category Critical thinking Clinical reasoning
by clarity category and
construct n Percent (%) n Percent (%)
Explicit 15 – 10 –
Shared 11 73 6 60
Idiosyn 4 27 4 40
Implicit 6 – 12 –
Conceptual 1 17 6 50
Referential 2 33 4 33
Measure 3 50 2 17
Total all 21 100 22 –
Note Idiosyn is the abbreviation for idiosyncratic
198 L. Fountain

Table 10.3 Frequency and percentage of domain specificity of clinical thinking studies by
construct
Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Specific 24 56 5 24 19 86
General 19 44 16 76 3 14
Total 43 100 21 100 22 100

Domain specificity. The domain specificity of 44 % of definitions for all studies

were domain-general, 56 % were domain-specific. Within the constructs of CR and
CT the distribution was very different; 86 % of clinical reasoning studies had
domain-specific definitions, whereas 76 % of critical thinking studies were
domain-general.
This may be because although CT and CR are often equated, to really measure
CR a domain-specific definition is usually adopted as more aspects of patient care
as opposed to just cognitive functions are captured. This was seen in the definition
data entries; for example, Kuiper et al. (2008) defined CR as “gathering information
about many different aspects of the clinical situation, perceiving and interpreting
salient cues, and then, on the basis of those cues and relevant knowledge, choosing
the treatment strategy that is most likely to facilitate goal achievement (Rogers and
Holm 1983),” which includes a lot of patient specific data, whereas a typical CT
definition was Drennan’s (2010), “identify central issues and assumptions in an
argument, recognize important relationships, make correct inferences from data,
deduce conclusions from information or data provided, interpret whether conclu-
sions are warranted on the basis of the data given, and evaluate evidence…”
(p. 423), which is less patient-centric (Table 10.3).
This analysis of domain specificity of definitions can advance the conversation
in measuring critical thinking by ensuring researchers consider this important aspect
of the definition they choose to operationalize. This provides measurement of a
concept that was a frequent explanation for the lack of consistency in the results for
critical thinking. As cited in the introduction, many authors have cited a need for
domain-specific instruments.
Equivalency. Most of the studies reported some equivalence between the def-
initions of critical thinking terms (Table 10.4). For example, Cruz et al. (2009)
stated “the hypothesis that a 4-day course consisting of 16 h of content with
discussion of critical thinking and clinical reasoning would positively influence
participant’s diagnostic accuracy scores.” Further discussion made it clear that CR
was seen as equivalent to CT. Seventy percent reported the presence of equivalency
of definitions between at least two of all the terms, and this was even more true for
clinical reasoning (77 %) than for critical thinking (62 %) studies.
Although the interchangeable use of critical thinking terms has been frequently
cited in the literature, it was surprising that a full 70 % of studies attributed
10 Thinking Critically About the Quality of Critical Thinking … 199

Table 10.4 Frequency and percentage of equivalency of term usage presence in clinical thinking
studies by construct
Equivalency presence Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Present 30 70 13 62 17 77
Absent 13 30 8 38 5 23
Total 43 100 – – – –

some degree of equivalency between critical thinking, clinical reasoning,

problem-solving, decision-making, and clinical judgment, especially in light of the
findings above about differences in clarity and domain specificity of the definitions.
It is notable that the equivalency was noted to an even larger degree in CR studies.
Further study is warranted to clarify exactly which terms are being considered
equivalent.

10.3.3 Research Question 3: How and How Well Are

Measures of Critical Thinking and Clinical
Reasoning Specified in the Literature?

Commonality. Study-specific measures were used for 40 % of the studies

(Table 10.5). Commonly used measures were used 37 % of the time. Commonly
used measures included Script Concordance Test, Key Feature Exam, and Health
Sciences Reasoning Test. Standardized measures were used 23 % of the time.
These included the California Critical Thinking Skills Test, the ATI Critical
Thinking Test, and the Watson-Glaser Critical Thinking Test. For the CT articles,
standardized measures were used 48 % of the time, but 0 % of the time for the CR
studies. Also, 55 % of the studies used study-specific measures in CR as compared
to 24 % for CT.

Table 10.5 Frequency and percentage of commonality of measure usage among clinical thinking
studies by construct
Measure commonality Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Study-Specific 17 40 5 24 12 55
Commonly used 16 37 6 29 10 45
Standardized 10 23 10 48 0 0
Total 43 100 21 100 22 100
200 L. Fountain

Table 10.6 Frequency and percentage of alignment of definitions with measures of clinical
thinking
Category Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Full 20 47 11 52 9 41
Partial 10 23 4 19 6 27
NA 13 30 6 29 7 32
Total 43 100 21 100 22 100

Researchers are probably creating new measures more than necessary or opti-
mal; perhaps searching for instruments from other domains could lead to a larger
stable of domain-specific, well-validated instruments.
Alignment. For the measure of alignment adapted from Alexander and Murphy
(2000) and Dinsmore et al. (2008), less than half the studies, 47 %, had a full
alignment between the definition of critical thinking and the operationalization
through the measure. As Table 10.6 shows, partial alignment occurred in 16 % of
the studies, and minimal alignment occurred in 7 % of the studies. Thirty percent of
the studies were excluded because the definition was implicit referential or implicit
measurement, or unable to be determined because the measurement was not ade-
quately described. Interrater reliability was established for the coding of alignment
of the measures at α = 0.90.
Operationalization of the construct was a big problem for this pool of studies,
with less than half having complete alignment. This distribution was about the same
for both CR and CT. The lack of clarity in definitions, or lack of explicit definitions,
is probably driving this. Researchers often listed a menu of definitions or attributes
of critical thinking without specifying which definition or model of thinking they
were using to measure critical thinking.
Reliability. Reliability documentation was absent in 7 % of studies (Table 10.7).
Reliability was obtained from a previous dataset in 23 % of studies. Reliability was
obtained for the current sample in 54 % of studies. Reliability was obtained for both

Table 10.7 Frequency and percentage of reliability categories of studies by construct

Reliability documentation Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Absent 3 7 1 5 2 9
Previous sample only 10 23 7 33 3 14
Current sample only 23 54 8 38 15 68
Both current and previous 7 16 5 24 2 9
samples
Total 43 100 21 100 22 100
10 Thinking Critically About the Quality of Critical Thinking … 201

Table 10.8 Frequency and Percentage of validity of studies by construct

Validity documentation Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Previous sample only 21 49 15 71 6 27
Current sample only 18 42 4 19 14 64
Both current and previous 4 9 2 10 2 9
samples
Total 43 100 21 100 22 100

Table 10.9 Frequency and percentage of studies at different levels of analysis

Analysis level Construct
All studies Critical thinking Clinical reasoning
n Percent (%) n Percent (%) n Percent (%)
Descriptive, narrative 12 28 7 32 5 24
t-test, ANOVA 30 70 15 68 15 71
Regression, factor 1 2 0 0 1 5
analysis
Total 43 100 22 100 21 100

the current and a previous sample in 16 % of studies. A larger percentage reported

reliability for both the current and previous datasets for CT studies (24 %) com-
pared to CR studies (9 %), again driven by the use of standardized tests with
documented reliability.
Validity. Validity was obtained from a previous sample in 49 % of studies
(Table 10.8). Validity was obtained for the current sample only in 42 % of studies.
And validity was obtained for both the current and a previous sample in 9 % of
studies. All studies documented validity in some way. Nearly half based the
evaluation of validity on a previous sample. There was a notable difference in the
constructs; most of the CR studies used validity from the current sample only,
whereas for CT most of the studies used validity from previous research only, again
probably driven by the extensive use of normed, standardized tests in CT research.
Only 9 % of the studies documented validity from both previous and current
datasets, which is best practice in research reporting (APA/AERA/NCME 1999,
2014) (Table 10.9).

10.4 Conclusions and Future Directions

There were several limitations to the generalizability of this study. It was conducted
by a single researcher, with a second researcher for interrater reliability of key
variables. Systematic review technology calls for a team of researchers at each step
202 L. Fountain

of the process from electronic searching to conclusions, as well as a technical

advisory panel to provide input.
The search terms were limited to nursing and medicine. More professions could
be included, and the differences and similarities between the professions could be
more closely examined. There is a current professional groundswell to promote
more interprofessionalism, and knowledge of interprofessional differences in
research approaches to measuring critical thinking and clinical reasoning would be
part of a foundation for that. The experience level of practicing professionals not
always clearly described in the studies; this could well have an influence on find-
ings if experts are combined with more novice providers. Also, there may be
international differences related to scope of practice and cultural differences among
countries.
It was not possible to compute effect size due to limitations in reporting: Very
few studies reported means, SD, and n, needed in order to have higher quality
evidence with meta-analyses. Although most studies used inferential statistics such
as t-test and ANOVA in the analysis, regression analysis and factor analysis to
validate scales or regression to test models were almost never used. The lack of
sampling in nearly all the studies also decreases the quality and generalizability of
results.
The limitation of the search terms to clinical reasoning and critical thinking
could have provided a bias in the findings; future systematic reviews of critical
thinking should include problem-solving, clinical judgment, and decision-making
to help determine the key attributes of each, and move toward less equivalency in
the use of the terms. Finally, only one database was used, PsycINFO. Although
PsycINFO is a highly respected comprehensive database, it is possible that an
emphasis on studies of students resulted due to the journals included. Future sys-
tematic reviews of this topic could include MEDLINE and CINAHL databases also.
Bias could be further reduced with hand searching that included key journals and
work of key researchers.
It is clear that critical thinking and clinical reasoning are still important terms of
study for researchers in the health professions. This systematic review of studies of
critical thinking in the health professions suggests that definitional conflation and
lack of measurement alignment limited the generalizability and value of the study
pool findings. Based on this research, some suggestions for future research can be
made.
More use of purposive samples spanning multiple levels of experience (student,
new grad, very experienced provider) may lead to increased generalizability of
findings. Researchers should explicitly state the conceptual and operational defi-
nitions used for measures of critical thinking, and be sure that the attributes mea-
sured in the instrument line up with the attributes of the definition. Researchers
should be aware of the implications of domain-specific or general definitions and
consider the impact on outcomes. The role of knowledge may be key in the dis-
tinction, as noted in Norman (2005); because many of the strategies to improve CR
assess different kinds of knowledge, such a conceptual, procedural, or conditional.
If standardized reference definitions for critical thinking and similar terms were
10 Thinking Critically About the Quality of Critical Thinking … 203

developed and disseminated, a subsequent analysis of attributes could allow for

distinctions between the attributes of the terms.
Implications for assessments of reliability and validity documentation should
include both previous and current datasets as appropriate. If journals that do not
currently adhere to the AERA/APA/NCME standards for reliability and validity
changed their policies, this could change quickly. Institutional supports for ade-
quate research time, and faculty development in psychometrics could also help
increase the methodological quality of research. Kane outlines an argument-based
approach to validity, which could be used to ensure better alignment of definitions
with measures (Kane 1992). An analysis of keywords to clarify exactly which
attributes are being used to define CT or CR would help illuminate which
dimensions are being measured. It is also possible that critical thinking components
will vary by subdomains such as maternity nursing or surgical medicine, so sub-
domain analysis would be helpful. Future studies should rate and grade the evi-
dence, not just describe it. Coding schemes such as developed for this study could
be used. There could be international differences in use of terms.
In conclusion, articulation of the attributes of conceptual and operational defi-
nitions, as well as reforms in research designs and methodologies, may lead to more
effective measurement of critical thinking. This could lead to more advanced
research designs and more experimental studies. Faculty could target certain aspects
of critical thinking in their pedagogy, such as compiling evidence, or making
inferences. As discussed elsewhere in this volume, domain-specific measures,
precise operationalization of constructs, and curricular alignment should be
embedded in professions education. More effective teaching strategies, curricular
changes, and ultimately patient care solutions could result. These principles of
improved conceptualization and measurement to improve professional competence
can also be applied to professions outside of healthcare. For example, one engi-
neering study examined a critical thinking curriculum in the engineering context
that used a strong theoretical base and reliable rubrics to evaluate critical thinking in
engineering projects (Ralston and Bays 2013). Professionals can then strive to
deliver higher quality outcomes in a complex society that demands not rote
thinking, but the highest levels of critical thinking possible.

Issues/Questions for Reflection

• How can the measures used in professions education research be designed
to align more effectively with definitions of constructs used in the study?
• How can reliable and valid measures be shared more extensively among
researchers?
• What will be the benefits of sharing definitions and measures of perfor-
mance between professions and disciplines?
204 L. Fountain

References

Abrami, P. C., Bernard, R. M., Borokhovski, E., Wade, A., Surkes, M. A., Tamim, R., & Zhang,
D. (2008). Instructional interventions affecting critical thinking skills and dispositions: A stage
1 meta-analysis. Review of Educational Research, 78, 1102–1134. doi:10.3102/
0034654308326084
Ajjawi, R., & Higgs, J. (2008). Learning to reason: A journey of professional socialization.
Advances in Health Sciences Education, 13, 133–150. doi:10.1007/s10459-006-9032-4
Alexander, P. A., & Murphy, P. K. (2000). The research base for APA’s learner-centered
psychological principles. In N. Lambert & B. McCombs (Eds.), How students learn (pp. 25–
60). Washington, DC: American Psychological Association.
American Association of Colleges of Nursing. (2008). Essentials of baccalaureate education for
professional nursing practice. Washington, DC: American Association of Colleges of Nursing.
American Educational Research Association, American Psychological Association, National
Council on Measurement in Education. (1999). Standards for educational and psychological
testing. Washington, DC: American Educational Research Association.
American Educational Research Association, American Psychological Association, National
Council on Measurement in Education. (2014). Standards for educational and psychological
testing. Washington, DC: American Educational Research Association.
Bashir, M., Afzal, M. T., & Azeem, M. (2008). Reliability and validity of qualitative research.
Pakistani Journal of Statistics and Operation Research, IV(1), 35–45.
Benner, P. E., Hughes, R. G., & Sutphen, M. (2008). Clinical reasoning, decision-making, and
action: Thinking critically and clinically. In R. G. Hughes (Ed.), Patient safety and quality: An
evidence-based handbook for nurses. AHRQ Publication No. 08-0043. Rockville, MD: Agency
for Healthcare Research and Quality.
Blondy, L. C. (2011). Measurement and comparison of nursing faculty members’ critical thinking
skills. Western Journal of Nursing Research, 33, 180–195. doi:10.1177/0193945910381596
Brunt, B. A. (2005). Critical thinking in nursing: An integrated review. Journal of Continuing
Education in Nursing, 36, 60–67.
Carpenter, C. B., & Doig, J. C. (1988). Assessing critical thinking across the curriculum. New
Directions for Teaching and Learning, 34, 33–46. doi:10.1002/tl.37219883405
Chan, Z. C. Y. (2013). A systematic review of critical thinking in nursing education. Nurse
Education Today, 33, 236–240.
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for
psychometric instruments: Theory and application. The American Journal of Medicine, 119,
166.e7–166.e16.
Cooke, M., Irby, D. M., & O’Brien, B. C. (2010). Educating physicians: A call for reform of
medical school and residency. San Francisco: Jossey-Bass.
Cooper, H., Hedges, L. V., & Valentine, J. C. (2009). The handbook of research synthesis and
meta-analysis (2nd ed.). New York: Russell Sage Foundation.
Creswell, J. W. (1994). Research design: Qualitative and quantitative approaches. Thousand
Oaks, CA: Sage Publications.
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods
approaches. Thousand Oaks, CA: Sage Publications.
Crocker, L. M., & Algina, J. (2006). Introduction to classical and modern test theory. Mason, OH:
Wadsworth Group/Thomas Learning.
Cruz, D. M., Pimenta, C. M., & Lunney, M. (2009). Improving critical thinking and clinical
reasoning with a continuing education course. The Journal of Continuing Education in
Nursing, 40, 121–127. doi:10.3928/00220124-20090301-05
Dinsmore, D. H., Alexander, P. A., & Loughlin, S. M. (2008). The impact of new learning
environments in an engineering design course. Instructional Science, 36, 375–393. doi:10.
1007/s11251-008-9061-x
10 Thinking Critically About the Quality of Critical Thinking … 205

Drennan, J. (2010). Critical thinking as an outcome of a master’s degree in nursing programme.

Journal of Advanced Nursing, 66, 422–431. doi:10.1111/j.1365-2648.2009.05170.x
Ennis, R. H. (1989). Critical thinking and subject specificity: Clarification and needed research.
Educational Researcher, 18, 4–10. doi:10.3102/0013189X018003004
Ennis, R. H. (1991). Critical thinking: A streamlined conception. Teaching Philosophy, 14, 5–24.
Evidence for Policy and Practice Information (EPPI). (N.D.). Different types of review. Retrieved
from http://eppi.ioe.ac.uk/cms/default.aspx?tabid=1915&language=en-US
Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of
educational assessment and instruction. Millbrae, CA: The California Academic Press.
Forneris, S. G., & Peden-McAlpine, C. (2007). Evaluation of a reflective learning intervention to
improve critical thinking in novice nurses. Journal of Advanced Nursing, 57, 410–421. doi:10.
1111/j.1365-2648.2006.04120.x
Funkesson, K. H., Anbäckena, E.-M., & Ek, A.-C. (2007). Nurses’ reasoning process during care
planning taking pressure ulcer prevention as an example: A think-aloud study. International
Journal of Nursing Studies, 44, 1109–1119. doi:10.1016/j.ijnurstu.2006.04.016
Göransson, K. E., Ehnfors, M., Fonteyn, M. E., & Ehrenberg, A. (2007). Thinking strategies used
by registered nurses during emergency department triage. Journal of Advanced Nursing, 61,
163–172. doi:10.1111/j.1365-2648.2007.04473.x
Harden, R. M., Grant, J., Buckley, G., & Hart, I. R. (2000). Best evidence medical education.
Advances in Health Sciences Education, 5, 71–90. doi:10.1023/A:1009896431203
Higgs, J., & Jones, M. (Eds.). (2000). Clinical reasoning for health professionals. Boston:
Butterworth Heinemann.
Huang, G. C., Newman, L. R., & Schwartzstein, R. M. (2014). Critical thinking in health
professions education: Summary and consensus statements of the millennium conference 2011.
Teaching and Learning in Medicine, 26(1), 95–102.
Institute of Medicine. (2010). The future of nursing: Leading change, advancing health.
Washington, D.C.: National Academies Press.
Interprofessional Education Collaborative Expert Panel. (2011). Core competencies for interpro-
fessional collaborative practice: Report of an expert panel. Washington, D.C.:
Interprofessional Education Collaborative.
Johansson, M. E., Pilhammar, E., & Willman, A. (2009). Nurses’ clinical reasoning concerning
management of peripheral venous cannulae. Journal of Clinical Nursing, 18, 3366–3375.
doi:10.1111/j.1365-2702.2009.02973.x
Johnson, M. (2004). What’s wrong with nursing education research? Nurse Education Today, 24,
585–588. doi:10.1016/j.nedt.2004.09.005
Johnson, D., Flagg, A., & Dremsa, T. L. (2008). Effects of using human patient simulator (HPS™)
versus a CD-ROM on cognition and critical thinking. Medical Education Online, 13(1), 1–9.
doi:10.3885/meo.2008.T0000118
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–
535. doi:10.1037/0033-2909.112.3.527
Kataoka-Yahiro, M., & Saylor, C. (1994). A critical thinking model for nursing judgment. Journal
of Nursing Education, 33, 351–356.
Kimberlin, C. L., & Winterstein, A. G. (2008). Validity and reliability of measurement instruments
used in research. American Journal of Health-System Pharmacy, 65, 2276–2284.
Krupat, E., Sprague, J. M., Wolpaw, D., Haidet, P., Hatem, D., & O’Brien, B. (2011). Thinking
critically about critical thinking: Ability, disposition or both? Medical Education, 45(6), 625–
635.
Kuiper, R. A., Heinrich, C., Matthias, A., Graham, M. J., & Bell-Kotwall, L. (2008). Debriefing
with the OPT model of clinical reasoning during high fidelity patient simulation. International
Journal of Nursing Education Scholarship, 5(1), Article 17. doi: 10.2202/1548-923X.1466
Mamede, S., Schmidt, H. G., Rikers, R. M., Penaforte, J. C., & Coelho-Filho, J. M. (2007).
Breaking down automaticity: Case ambiguity and the shift to reflective approaches in clinical
reasoning. Medical Education, 41, 1185–1192. doi:10.1111/j.1365-2923.2007.02921.x
206 L. Fountain

McAllister, M., Billett, S., Moyle, W., & Zimmer-Gembeck, M. (2009). Use of a think-aloud
procedure to explore the relationship between clinical reasoning and solution-focused training
in self-harm for emergency nurses. Journal of Psychiatric and Mental Health Nursing, 16,
121–128. doi:10.1111/j.1365-2850.2008.01339.x
McCartney, K., Burchinal, M. R., & Bub, K. L. (2006). Best practices in quantitative methods for
developmentalists. Monographs of the Society for Research in Child Development, 71, 285.
Chapter II, Measurement issues. doi: 10.1111/j.1540-5834.2006.00403.x
Nikopoulou-Smyrni, P., & Nikopoulos, C. K. (2007). A new integrated model of clinical
reasoning: Development, description and preliminary assessment in patients with stroke.
Disability and Rehabilitation, 29, 1129–1138. doi:10.1080/09638280600948318
Norman, G. (2005). Research in clinical reasoning: Past history and current trends. Medical
Education, 39, 418–427. doi:10.1111/j.1365-2929.2005.02127.x
Ovretviet, J. (2011). BMJ Qual Saf 2011. 20(Suppl1), i18ei23. doi:10.1136/bmjqs.2010.045955
Patel, V. L., Arocha, J. F., & Zhang, J. (2004). Thinking and reasoning in medicine. In Keith
Holyoak (Ed.), Cambridge handbook of thinking and reasoning. Cambridge, UK: Cambridge
University Press.
Pintrich, P. R., Smith, D. A., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the
Motivated Strategies for Learning Questionnaire (MSLQ). Ann Arbor, MI: University of
Michigan National Center for Research to Improve Postsecondary Teaching and Learning.
Ralston, P. A., & Bays, C. L. (2013). Enhancing critical thinking across the undergraduate
experience: an exemplar from engineering. American Journal of Engineering Education
(AJEE), 4(2), 119–126.
Ratanawongsa, N., Thomas, P. A., Marinopoulos, S. S., Dorman, T., Wilson, L. M., Ashar, B. H.,
et al. (2008). The reported reliability and validity of methods for evaluating continuing medical
education: A systematic review. Academic Medicine, 83, 274–283. doi:10.1097/ACM.
0b013e3181637925
Rogers, J. C., & Holm, M. B. (1983). Clinical reasoning: The ethics, science, and art. American
Journal of Occupational Therapy, 37, 601–616. doi:10.5014/ajot.37.9.601
Ross, D., Loeffler, K., Schipper, S., Vandermeer, B., & Allan, G. M. (2013). Do scores on three
commonly used measures of critical thinking correlate with academic success of health
professions trainees? A systematic review and meta-analysis. Academic Medicine, 88(5), 724–
734.
Schell, R., & Kaufman, D. (2009). Critical thinking in a collaborative online PBL tutorial. Journal
of Educational Computing Research, 41, 155–170. doi:10.2190/EC.41.2.b
Simmons, B. (2010). Clinical reasoning: Concept analysis. Journal of Advanced Nursing, 66,
1151–1158. doi:10.1111/j.1365-2648.2010.05262.x
Simpson, E., & Courtney, M. D. (2002). Critical thinking in nursing education: A literature review.
International Journal of Nursing Practice, 8(April), 89–98. doi:10.1046/j.1440-172x.2002.
00340.x
Tanner, C. A. (1997). Spock would have been a terrible nurse and other issues related to critical
thinking. Journal of Nursing Education, 36, 3–4.
Turner, P. (2005). Critical thinking in nursing education and practice as defined in the literature.
Nursing Education Perspectives, 26, 272–277.
Walsh, C. M., & Seldomridge, L. A. (2006a). Critical thinking: Back to square two. Journal of
Nursing Education, 45, 212–219.
Walsh, C. M., & Seldomridge, L. A. (2006b). Measuring critical thinking: One step forward, one
step back. Nurse Educator, 31, 159–162.
Waltz, C. F. (2010). Measurement in nursing and health research (4th ed.). Philadelphia: Springer.
Watson, R., Stimpson, A., Topping, A., & Porock, D. (2002). Clinical competence assessment in
nursing: A systematic review. Journal of Advanced Nursing, 39, 421–431.
Wolpaw, T., Papp, K. K., & Bordage, G. (2009). Using SNAPPS to facilitate the expression of
clinical reasoning and uncertainties: A randomized comparison group trial. Academic
Medicine, 84, 517–524. doi:10.1097/ACM.0b013e31819a8cbf
10 Thinking Critically About the Quality of Critical Thinking … 207

Context (n. d.). In Merriam-Webster.com. Retrieved April 11, 2012 from http://www.merriam-
webster.com/dictionary/context.
Alexander, P. A., Dinsmore, D. L., Fox, E., Grossnickle, E., Loughlin, S. M., Maggioni, L.,
Parkinson, M. M., & Winters, F. I. (2011). Higher order thinking and knowledge:
Domain-general and domain-specific trends and future directions. In G. Shraw & D.
H. Robinson (Eds.), Assessment of higher order thinking skills (pp. 47-88). Charlotte, NC:
Information Age Publishing.
Fonteyn, M. E., & Cahill, M. (1998). The use of clinical logs to improve nursing students’
metacognition: A pilot study. Journal of Advanced Nursing, 28(1), 149-154.