Chapter 11:

Understanding the
Assessment of Clinical
Reasoning

Omar S. Laynesa
ABSTRACT
• Clinical reasoning ability depends on a health professional’s knowledge and knowledge organization rather
than a general thinking process.

• Clinical reasoning is context-specific; clinician or trainee characteristics account for only a small amount of
the variance in diagnostic accuracy.

• Determining the validity of any clinical reasoning assessment method is challenging due in part to the
situation specific nature of clinical reasoning (context specificity).

• Most clinical reasoning assessment methods provide adequate reliability for a high stakes examinations
provided adequate sampling.

• No gold standard for the clinical reasoning process exists.

• High-stakes assessment focus on the accuracy of diagnostic and therapeutic choices because “process”
checklists can downgrade advanced trainees or experienced health professionals.
• There is no “magic bullet” for assessing clinical reasoning; therefore, multiple assessments by multiple raters
in diverse contexts is key to adequate clinical reasoning assessment.
• We believe that a combination of knowledge assessment (e.g., multiple choice question examination) and
clinical skills assessment (e.g., 10 station objective structured clinical examination) should be used in high-
stakes conditions.
11.1Defining Clinical Reasoning
• Clinical reasoning is the cognitive and physical processes by
which a health care professional consciously and
unconsciously interacts with the patient and environment to
collect and interpret patient data, weigh the benefits and risks
of actions, and understand patient preferences to determine a
working diagnostic and therapeutic management plan whose
purpose is to improve a patient’s well-being.
11.2 Historical and Contemporary Understanding of the
Cognitive Processes of Clinical Reasoning
THE GENERAL PROBLEM SOLVING ERA EXPERTISE AS KNOWLEDGE ORGANIZATION:
THE ERA OF INFORMATION PROCESSING
In the 1960s, medical education scholars were convinced that
clinical reasoning was based on general problem solving skills, Information processing theory became the preeminent conceptual
which would be superior in experts than in novices. These framework for researchers of clinical reasoning, but this approach
predictions were shown to be incorrect. encountered its own limits. Chief among these was its inability to
explain the variance in performance by an individual clinician on
similar cases requiring the same medical knowledge.
Performance was discovered to be highly dependent on
knowledge content within a given domain, thus establishing
This work led to the era of context specificity.
the concept of content specificity. This discovery led to the
next era.

CLINICAL REASONING EXPERTISE AS A STATE: CONTEXT SPECIFICITY AND

SITUATIVITY THEORY

Expertise as a state argues that clinical reasoning performance is specific to the patient, other health professionals on the
team, the environmental and their emergent interactions (i.e., the specific situation). Thus, content specificity that was first
described by Elstein et al. (1978) has been renamed context specificity to capture the notion that something besides the
clinical content of a case is influencing diagnoses and therapy.

Situativity theory has emerged to explain context specificity and expand understanding of clinical reasoning beyond the
limits of information processing theory. Situativity theory posits that the knowledge of the health professional is only one
of several, rather than the sole factor that predicts clinical reasoning success.
 11.3 The Construct and Process of Clinical Reasoning

DATA COLLECTION PROBLEM REPRESENTATION

• A health professional obtains a patient’s history, performs • With potentially 20–30 items of patient data to process,
a physical examination, and orders studies (when distinguishing salient data from “red herrings” requires
necessary) to determine the exact nature of the patient’s significant content knowledge and organization.
problem(s).

HYPOTHESIS GENERATION HYPOTHESIS REFINEMENT

• Often hypotheses are triggered by bits of clinical • This data collection may include additional history, physical
data through nonanalytic reasoning when clinicians examination, laboratory, or radiological studies. In addition,
a diagnostician may use point of care resources to look up
recognize a pattern of symptoms in a patient that information that would aid in the reasoning process (e.g.,
matches a mental disease representation (i.e., the typical presentation of the disease, tests of choice, etc.)
“illness script”) within long-term memory.

SCRIPT SELECTION AND

CONFIRMATION WORKING DIAGNOSIS
• A The reasoning exercise of diagnosis seeks to • If the clinical findings do not match, then the
select and confirm that a health professional’s hypothesis is falsified and the process repeats until
mental construct of the leading disease hypothesis enough diagnostic certainty is achieved that treatment
matches the patient’s problem representation can begin.
11.4 Foundational Issues in Clinical Reasoning Assessment

ASSESSMENT OF VERSUS FOR CRITERIA FOR GOOD ASSESSMENT

LEARNING Good assessment has the following characteristics: validity
or coherence, reproducibility or consistency, equivalence,
feasibility, educational effect, catalytic effect.
There is little debate with the aphorism “assessment drives
It should be clear that any assessment of clinical reasoning
learning” and assessments signal very clearly what
should be “fit for purpose” to maximize its utility and
programs and educators believe is important.
impact, and the importance of purpose should therefore
A number of continuous professional development
guide what should be measured in regards to clinical
programs are increasingly incorporating assessment for reasoning.
learning to enhance clinical reasoning.

WHAT SHOULD WE MEASURE? HOW CAN WE MEASURE?

Physiologic measurements are beginning to help us better

When the primary purpose of assessment is to ensure understand how the neuro-circuitry functions in clinical
healthcare professionals possess a minimal level of reasoning in medicine. For example, Durning et al. (2014),
competence for promotion decisions and public using functional MRI imagining, have identified areas of the
accountability, the emphasis has been on measuring brain are active during the clinical reasoning process, raising
the end product of the clinical reasoning process. hopes that we may indeed soon have the capacity to actually
“see” what is happening in clinical reasoning physiologically.
11.5 Tools for Assessing Clinical Reasoning
STANDARDIZED ASSESSMENT METHODS

Multiple Choice and Extended Matching Structured (Standardized) Oral

Questions Examination
MCQ questions typically embed clinical context into the question
through use of a clinical stem, or vignette, that includes a description
of a patient, key clinical features, and “distractors”. In traditional oral examinations, the examinee performs a
history and physical examination of a patient, with or
EMQ are any matching format with more than five options with items without supplemental laboratory or radiological material,
are grouped into sets, with a single option list used for all items in a followed by a series of questions from an examiner.
set. A well-constructed extended matching set includes four elements:
a theme; a lead-in statement; an option list; and two or more item
stems.

Key Features Script Concordance Testing

The key features (KF) approach is designed to
specifically target a learner’s decision-making skills The script concordance test (SCT) was specifically
with less emphasis on factual recall. KF focuses on developed to assess clinical reasoning that
how incorporates, to the extent possible, practice-like
knowledge is used to create a differential diagnosis, conditions (Dory et al. 2012).
decide on what tests or procedures to pursue, choose
a management approach, and so on.
11.6 Situated Assessment
1. Work-Based Assessments

“Expert” Assessments Direct Observation

Sometimes called “global summaries.” In Assessing clinical judgment through observation
They typically occur at the end of a learning experience commonly incorporates an assessment of data gathering
(e.g., “a two-week rotation”) and are “overall” or skills (medical interviewing and physical examination),
“summative” assessments performed by raters, typically which is lacking in the other tools described. The
faculty. These judgments have traditionally been advantage of direct observation, when combined with an
expressed on some type of rating scale (e.g., a five-point assessment of data gathering skills, allows for faculty to
Likert scale). assess integration of all the key components.

Chart Stimulated Recall

Chart stimulated recall (CSR) can be considered as a
more structured “oral exam” that uses the medical
record of an actual patient encounter to
retrospectively review the clinical reasoning process
of healthcare professional (Maatsch et al. 1983).
11.6 Situated Assessment (cont.)
2. Clinical Simulation-Based Assessments

Objective Structured Clinical

Examination (OSCE)

The most common format when standardized patients

(SP) are employed to assess clinical skills. Standardized
patients are live actors trained to portray a range of
clinical scenarios.

Structured (Standardized) Oral

Examination

High fidelity simulation is also increasingly incorporating

assessment of clinical reasoning into the assessment
process. High fidelity simulations often do not involve an
SP, but instead employ sophisticated mannequins, virtual
reality, and other computer-based simulations.
11.7 Emerging Methods in Clinical Reasoning Assessment

CONCEPT MAPPING SELF-REGULATED LEARNING (SRL)

- a technique for visually representing a learner’s thinking - defined as a set of processes that learners use to
or knowledge organization. In a concept map, the learner moderate their own learning and performance, which is
connects a number of ideas (concepts) with specific typically divided into a number of elements in each of
phrases (linking words) to demonstrate how they put their three stages: forethought (before), performance (during),
ideas together. and reflection (after).

NEUROBIOLOGICAL CORRELATES

- exploring clinical reasoning assessment through

more direct means such as functional MRI and EEG
(Durning et al. 2014). These means may show
particular promise for nonanalytical reasoning that is
not believed to be completely subject to
introspection.
11.8 Conclusion
• In a competency-based world, assessment for learning is more important than
assessment of learning. Such formative assessment will therefore need to focus
more on clinically based tools like direct observation, meaningful simulation as
starter and reinforcing activities, CSR, and new forms of audit.

• In a competency-based world, assessment for learning is more important than

assessment of learning. Such formative assessment will therefore need to focus
more on clinically based tools like direct observation, meaningful simulation as
starter and reinforcing activities, CSR, and new forms of audit.
Chapter 12:
Assessment of
Interprofessional Education:
Key Issues, Ideas, Challenges,
and Opportunities
ABSTRACT
• The assessment of interprofessional education (IPE) is a complex activity due to its involvement of both
individuals as well as interprofessional teams/groups.

• In designing an IPE assessment a series of key questions need to be posed and addressed, including, what is
the purpose of the assessment? What is one going to assess? How is the assessment to be performed?

• Development of assessment blueprint is vital to linking proposed learning outcomes with methods of
assessment.

• A focus on collaborative performance using competency domains such as communication, collaboration and
professionalism can be an effective approach to IPE assessment.

• The use of an assessment matrix can effectively collate key elements related to the assessment of IPE.
• Entrustable professional activities and milestones are promising techniques to use in IPE assessment.
INTERPROFESSIONAL EDUCATION (IPE)

IPE focuses on learning activities designed to enhance the attitudes, knowledge, skills, and
behaviors for effective interprofessional practice (Barr et al. 2005).

Through the use of IPE, it is anticipated that improvements in the quality of care delivered
to patients/clients and families will be achieved (e.g., Reeves et al. 2010, 2013; Institute of
Medicine 2013, 2014).
12.2 Assessment Development: Key Principles
1. COLLABORATIVE PERFORMANCE 2. WHAT TO ASSESS
• Within an IPE program or curriculum, working • Within each learning activity, assessment must
together collaboratively is the core issue. In include knowledge, application of knowledge,
introducing an IPE curriculum, agreement was performance of the knowledge, and what we do
needed for both examination standards and in reality which would be to develop a
assessment criteria—these should be equivalent professional competence in practice: ‘knows’;
across all programs. ‘knows how’; ‘shows how’, and ‘does’ (Miller
• As these assessment issues are complex, it was 1990).
not surprising that the leaders of the different • This approach can also be defined as content-
programs initially agreed on a formative specific assessment or domain-specific
approach to assessment. assessment (van der Vleuten 2008).
4. ASSESSING COLLABORATIVE
3. DEVELOPMENT OF A BLUEPRINT PERFORMANCE
• A blueprint links learning outcomes with • Attempting to assess collaboration can be
methods of assessment, the target phase of problematic, especially when a group of learners are
learning in which the learning outcome should be brought together, often without preparation, and
achieved and also maps the assessment to required to perform together as an interprofessional
team.
practice.
• In practice, however, interprofessional teams may
have worked together for many years.
 12.2 Assessment Development: Key Principles

5. DESIGNING AND PILOTING AN 6. A NEW APPROACH TO ASSESSING

INTERPROFESSIONAL ASSESSMENT INTERPROFESSIONAL EDUCATION
• Based on this literature, an interprofessional • Based on these experiences with assessing IPE, in this
OSCE (iOCSE) was developed at the University section a new possible approach to IPE assessment is
of Toronto, Canada in which we designed a three- described in which a range of interlinked concepts
station assessment with five students from and activities is explored.
different professions (Simmons et al. 2012).
6. A NEW APPROACH TO ASSESSING INTERPROFESSIONAL EDUCATION

1. Structure-Function-Outcome

2. Individuals-Team-Task

3. Focusing the Assessment

4. The Use of a Matrix

6. A NEW APPROACH TO ASSESSING INTERPROFESSIONAL EDUCATION

5. Mapping Assessment Options: An

Interprofessional Blueprint

6. Entrustable Professional
Activities

7. The Use of Milestones

12.3 Conclusion
• The assessment of IPE is fraught with complexity due to its application to
individuals
as well as interprofessional teams/groups. Although knowledge-based IPE
assessments are a useful way of assessing individual understanding of IPE, there are
limited as they fail to assess possible changes in behavior (performance).
Thank You!