MCQ CONSTRUCTION &

DESIGN
Dr. Nadia Al Wardy
College of Medicine & Health Sciences

Medical Education Unit College of Nursing Workshop March 2014

 A Simple Model of Competence
2

Performance
Does or hands on
assessment
Shows how
Written,
Knows how Oral or
Computer
Knows based
assessment
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S7. 2
3

Performance assessment in vivo:

Does
Does Masked SPs, Video, Audits…..

Performance assessment in vitro:

Shows how
Shows how OSCE, SP-based test…..

(Clinical) Context based tests:

Knows how
Knows how MCQ, essay type, oral…..

Knows
Knows Factual tests:
MCQ, essay type, oral…..

3
 Instruments of Written Assessment
4

 Open-ended question types

 Modified Essay Questions (MEQs)
 Short-answer questions (SAQs)
 ….

 Multiple-choice question types

 Single-best answer
 True/false
 Extended-matching items (EMIs)
 Key features
 ….
 Multiple-Choice Question Formats

5
A-type

One-best-answer
Questions
R-type
Extended matching
Multiple-choice
Questions
Standard
True-false-type
True-false
Questions
K-type
Complex True False
 Chapter 2
6

Multiple-Choice Item Formats

Page 13

From Case & Swanson: Constructing Written Test Questions for the
Basic and Clinical Sciences
http://www.NBME.ORG/about/itemwriting.asp
 MULTIPLE-CHOICE
QUESTIONS
A-Type (Single Best Answer)
 Components of an A-type Question
A 35-year old man has had a stomach
8
ache all afternoon. He ate the following Vignette,
lunch: two big McDonalds hamburgers, Scenario,
an ice cream shake, and large fries.
Or Stem

What is the most likely diagnosis? Lead-in

A. Abdominal aneurysm
B. Appendicitis
C. Bowel obstruction Options A, B, C, D, E & F are
D. Cholecystitis distractors
E. Colon cancer
F. Pancreatitis Option G is the key
G. Too much lunch
Sample Good Question & ‘Cover The Options’ Rule
9

A 32-year old man has a 4-day history of progressive weakness in his

extremities. He has been healthy except for an upper respiratory tract infection 10
days ago. His temperature is 100 F, BP 130/80, pulse 94, respirations 42 and
shallow. He has symmetric weakness of both sides of the face and the proximal
and distal muscles of the extremities. Sensation is intact. No deep tendon reflexes
can be elicited; the plantar responses are flexor. What is the most likely
diagnosis?

A. Acute disseminated encephalomyelitis

B. Guillain-Barré syndrome
C. Myasthenia gravis
D. Poliomyelitis
E. Polymyositis
 Diagram of Rank-Ordered Options
for a Good One-Best-Answer Item
10

D C A E B

Least Most
Correct Correct
Answer Answer

Note: Options must be homogeneous (eg, all diagnoses, all

muscles).
 Sample Good Question & ‘Cover The Options’ Rule
11

A 40 year old man is admitted to the Emergency Room with

renal calculi and is complaining of moderate to severe flank
pain and nausea. His temperature is 38.20C. What is the
priority nursing goal for this patient?
A. maintain fluid and electrolyte balance
B. Control nausea
C. Manage pain
D. Prevent urinary tract infection
 Sample Flawed Question &
‘Cover The Options’ Rule
12

Which of the following is true about pseudogout?

A. It occurs frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with a finding of
chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
 Sample Flawed One-Best-Answer Item
13

Which of the following is true about pseudogout?

A. It occurs frequently in women.
B. It is seldom associated with acute pain in a joint.
C. It may be associated with chondrocalcinosis.
D. It is clearly hereditary in most cases.
E. It responds well to treatment with allopurinol.
Gender A

Rx E D Inheritance

B C
Associations
False True
 The #1 Rule For One-Best-Answer Questions
14

A 24-year old man comes to the urgent care

unit….. What is the most likely cause of his pain?
A. cholecystitis
B. diverticulitis
C. kidney stone
D. pancreatitis
E. peptic ulcer disease
15

If you want your students

to go beyond memorizing facts,
your tests must require more
 Two Rules
16

1. Ensure that each question satisfies the ‘cover

the options’ rule.

2. Assess examinee ability to apply their

knowledge rather than simply their ability to
recall isolated facts.
 Item Assessing Recall of an Isolated Fact
17

What arterial blood gas analysis is most likely to show in patients

with cardiogenic shock?

A. Hypoxemia with normal pH

B. Metabolic acidosis
C. Metabolic alkalosis
D. Respiratory acidosis
E. Respiratory alkalosis
 Alternate Item Assessing
Application of Knowledge: Diagnosis
18

A 74-year-old woman is brought to the emergency department

because of crushing chest pain. She is restless, confused, and
diaphoretic. On admission, temperature is 36.7 C, blood
pressure is 148/78 mm Hg, pulse is 90/min, and resp are
24/min. During the next hour, she becomes increasingly
stuporous, blood pressure decreases to 80/40 mm Hg, pulse
increases to 120/min, and respirations increase to 40/min. Her
skin is cool and clammy. An ECG shows sinus rhythm and 4
mm of ST segment elevation in leads V2 through V6. Arterial
blood gas analysis is most likely to show
 Alternate Item Assessing
Application of Knowledge: Diagnosis
19

A 74-year-old woman is brought to the emergency department

because of crushing chest pain. She is restless, confused, and
diaphoretic…..
Arterial blood gas analysis is most likely to show

A. Hypoxemia with normal pH

B. Metabolic acidosis
C. Metabolic alkalosis
D. Respiratory acidosis
E. Respiratory alkalosis
 Sample MCQ Items:
Isolated Fact vs Case-Based Question
20

What is the most likely renal abnormality in children with nephrotic

syndrome and normal renal function?

OR

A 2-year-old boy developed swelling of his eyes and ankles over the
past week. BP 100/60; pulse 110; R 28. PE shows eyelid swelling; 2+
pitting edema of his ankles; abdominal distension with positive fluid
wave. Serum: creatinine 0.4; albumin 1.4; cholesterol 569 mg/dL.
Urinalysis: 4+ protein and no blood. What is the most likely
diagnosis?
 How To Develop Better Multiple-Choice Questions
21

A 29-yr-old woman has a one-day

What are history of (insert description of
the signs presenting complaint, history, physical
and symptoms exam here).
of multiple
sclerosis? What is the most likely diagnosis?
 How To Develop Better Multiple-Choice Questions
22

A 6-yr-old boy has a one-day

history of (insert description of
presenting complaint, history, physical
What is the best
exam here).
drug to treat
Otitis media? What is the most appropriate
therapy?
 How To Develop Better Multiple-Choice Questions
23

A 37-yr-old man comes to the

physician for a routine health
What maintenance visit (insert relevant
history, physical exam here).
immunizations
should be given What immunizations should
to adults? be administered?
 Structure of a Patient Vignette
24

Age, Gender (eg, A 45-year-old man)

Site of Care (eg, comes to the emergency dept.)
Presenting Complaint (eg, because of headache)
Duration (eg, that has continued for 2 days)

Patient History (with Family History ?)

Physical Findings
+/- Results of Diagnostic Studies
+/- Initial Treatment, Subsequent Findings, etc.
 Stages in writing MCQs
25

1. Plan the question

2. Write the question
3. Test the question
1. Plan the question
 Ensure question deals with an important and
useful aspect of the subject
27

 Avoid trivia
 Include:
 Common issues
 Serious conditions
 Important or frequent misunderstandings
 Appropriate to expected level
 Test knowledge around a central theme
28

 Be clear about exactly what fact, concept or

generalization is being tested
 This central theme should be clearly defined
in the stem
 All options should refer to this theme
 Decide on the ability you wish to test
29

 Recall or recognition
 Interpretation
 Problem-solving
 Consider how the information should be presented
30

 Word description
 Photograph
 Radiograph
 Tracing (e.g. ECG, visual field)
 Diagram (e.g. anatomical)
 Movie film
 Amplified sound (e.g. heart sounds)
2. Write the Question
 Use simple and clear language
32

 Avoid unfamiliar technical or unusual words

 Check for ambiguity amongst fellow
examiners or students
 Anxious examinees imagine traps in
questions
 Write stem as a question or incomplete statement
33

 If a question, each option must be a possible answer

 If an incomplete statement, each response must follow
grammatically and logically from the stem; i.e. stem + each
option must be a complete statement
 Avoid stems with just one or two words:
Hair:
A. growth in man shows the phenomenon of
asynchronous cyclic growth.
B. ……………
 Keep reading time to a minimum
34

 Include only material or explanations necessary

for answering the question
In considering your management of a 28-year-old
fashion designer with bleeding from the nose, you
would be aware that the most frequent cause of
epistaxis at any age is
A. ………………
 Make the responses clear and concise
35

 If a response requires a large number of words to qualify

what is meant by it, the questions is probably too complex
for an MCQ
 Include all common elements in the stem
The third part of the duodenum
A. is anterior to the uncinate process of the pancreas
B. is anterior to the common bile duct
C. is anterior to the right ureter
D. is anterior to the superior mesenteric artery
E. is anterior to the inferior mesenteric vein
 Avoid clues to the correct answer
36

 Use of stereotyped or standard phraseology

“the pituitary gland is the master of the endocrine orchestra”
 Unequal length of alternatives
 Obvious inconsistencies between stem and a response
An intercostal tube may be placed in a patient with a flail chest but
without a pneumothorax or haemothorax to
A. remove air
B. remove blood
C. prevent onset of pulmonary contusion
D. prevent development of a tension pneumothorax when the patient is
placed on positive pressure ventilation
E. eliminate paradoxical movement of the flail segment
 Avoid clues to the correct answer [contd]
37

 Using opposite of the correct answer as a distractor

Paralysis of a vocal cord in a patient with bronchogenic carcinoma is
most likely to be:
A. bilateral
B. on the right side
C. on the left side
D. due to metastasis to the larynx
E. due to cigarette smoking

 Use of absolutes (e.g. “always”, “only”, “never”)

Adamantinoma of the mandible
A. never metastasize
B. usually metastasize to lymph nodes
C. …………
 Avoid clues to the correct answer [contd]
38

 Use of synonyms or overlapping alternatives

 Use of similar distractors, making the correct
response more conspicuous
Antidiuretic hormone is synthesized in
A. the anterior lobe of the pituitary gland
B. the posterior lobe of the pituitary gland
C. the pars intermedia of the pituitary gland
D. the pars nervosa of the pituitary gland
E. the hypothalamus
 Use plausible and logical distractors
39

 Obvious distractors increase chances of guessing

 Derive distractors from common errors made by
students
 Derive distractors from common misconceptions
held by students
 If legitimate distractors cannot be found, recast
question in a different format
 Use eponyms, acronyms and abbreviations
only if you expect students to know them
40

 Eponym: Named after a person (e.g. pouch of

Douglas; Meckel’s diverticulum)
 Acronym: Combination of first letters (e.g. ACTH,
CK)
 Abbreviation: Shorter form of a word (e.g. ht.
sd.)
 Specify units
41

 Particularly when using laboratory units, as they

can vary from one laboratory to another
 The normal range for a given laboratory may
have to be quoted
 Indicate opinion or authority if dealing
with unestablished facts
42

 Indicate whose opinion or what authority the question is

based on
According to the passive equilibrium model of Kokko (1975)
the main factor responsible for the concentration of urine in
the loop of Henle is
A. the relative impermeability of the descending limb
of the loop of Henle to sodium chloride
B. the movement of urea out of the apex of the loop of
Henle
C. …………
 Minimize negatively phrased questions
43

 Avoid double negatives

 They confuse examinees
 If absolutely necessary, emphasize the word(s)
conveying a negative aspect by using capitals or
underlining
Haematemesis is not unexpected in each of the following,
except
A. ………
B. ………
C. ………
 Beware of unintentionally asking two
questions in the same option
44

 Avoid especially if the answer to one is ‘true’ and

to the other is ‘false’.
Psychological process and gastrointestinal function are
A. clearly separable and rarely correlated
B. ………
 Avoid “all of the above” and beware of “none
of the above”
45

 After partial gastrectomy, gastric emptying is

abnormal because
A. the pylorus is removed
B. there is interference with the pacemaker
C. the gastric remnant is smaller
D. all of the above
E. none of the above
3. Test the question
 Get agreement from fellow examiners that:
47

 the question is worth asking

 it is clearly and unambiguously stated
 distractors are reasonable and plausible
 Test the question on students if possible
48

 may represent a different interpretation of the

question
 may reveal what the question was actually
testing
 may have an opinion on importance and validity
 can indicate their answering strategy
 may point out errors in the question
MULTIPLE-CHOICE
QUESTIONS

R-Type (Extended-Matching Items, EMIs)

 Chapter 6
50

Extended Matching
Page 69

From Case & Swanson: Constructing Written Test Questions for the
Basic and Clinical Sciences
http://www.NBME.ORG/about/itemwriting.asp
 Components of an R-type Question
51
Theme: Cardiovascular drugs
Options: A. Alpha agonists
B. Alpha blockers
C. ACE inhibitors
D. Beta agonists…….
K. Vasoconstrictors
Lead-in: For each patient, select the most appropriate drug class
Item stems:
1. A 40-year-old man with sudden onset of severe headache, dizziness
and vomiting, BP 260/130 mmHg; encephalopathy and grade IV
retinopathy

2. A 55-year-old man with mild, chronic hypertension, tachycardia and

other symptoms of mild congestive cardiac failure. Previous history of
asthma and gout.
 Writing R-Set Extended Matching Items
52

1. Decide on the theme of your set

(eg, Establishing a diagnosis of patients with abdominal pain)

2. Develop the lead-in for your set

3. Write the options for your set

(eg, appendicitis, cholecystitis)

4. Write a patient description for each of the diagnoses in the list

5. Ask a colleague to review your items (patient descriptions) without

the correct answers noted
 Topic: Diagnosis of Patient With Abdominal Pain
53

Lead In:

For each patient with abdominal pain, select the

most likely diagnosis.
 Options and Lead-in: Diagnosis
54

A. Abdominal aneurysm K. Kidney stone

B. Appendicitis L. Mesenteric adenitis
C. Bowel obstruction M. Mesenteric artery thrombosis
D. Cholecystitis N. Ovarian cyst - ruptured
E. Colon cancer O. Pancreatitis
F. Constipation P. Pelvic inflammatory disease
G. Diverticulitis Q. Peptic ulcer disease
H. Ectopic pregnancy - ruptured R. Perforated peptic ulcer
I. Endometriosis S. Pyelonephritis
J. Hernia T. Torsion

For each patient with abdominal pain, select the most likely
diagnosis.
 Structure of a Patient Vignette
55

Age, Gender (eg, A 45-year-old man)

Site of Care (eg, comes to the emergency dept.)
Presenting Complaint (eg, because of headache)
Duration (eg, that has continued for 2 days)

Patient History (with Family History ?)

Physical Findings
+/- Results of Diagnostic Studies
+/- Initial Treatment, Subsequent Findings, etc.
 Sample Diagnosis Items: Abdominal Pain
56

1. A 25-year-old woman has sudden onset of persistent right

lower abdominal pain that is increasing in severity. She has
nausea without vomiting. She had a normal bowel
movement just before onset of pain. Examination shows
exquisite deep tenderness to palpation in right lower
abdomen with guarding but no rebound; bowel sounds are
present. Pelvic examination shows a 7-cm, exquisitely tender
right sided mass. Hematocrit is 32%. WBC is 18,000/mm3.
Serum amylase activity is within normal limits. Test of the
stool for occult blood is negative.
Ans: B
 Sample Diagnosis Items: Abdominal Pain
57

2. A 27-year-old woman has had abdominal pain for the past

18 hours. The pain was first noted in epigastrium and has
moved to the right lower quadrant of the abdomen. She has
mild nausea and anorexia, but no vomiting. She has a low-
grade fever. Her last bowel movement was normal
yesterday. Her last menstrual period was 4 weeks ago.
Examination shows marked right lower quadrant abdominal
tenderness with guarding and rebound. Pelvic exam reveals
marked tenderness on right side without masses. Hematocrit
is 37%. WBC is 14,500/mm3. Serum amylase activity is
within normal limits. Test of the stool for occult blood is
negative.
Ans: C
 Sample Diagnosis Items: Abdominal Pain
58

3. An 84-year-old man in a nursing home has increasing

poorly localized lower abdominal pain recurring every
3-4 hours over the past 3 days. He has no nausea or
vomiting; the last bowel movement was not recorded.
Examination shows a soft abdomen with a palpable,
slightly tender, lower left abdominal mass. Hematocrit
is 28%. WBC is 10,000/mm3. Serum amylase activity is
within normal limits. Test of the stool for occult blood
is positive.

Ans: E
 Rules for Use in Reviewing Sets
59

Options should be single terms, or very short phrases. Make sure

the options are homogeneous.

A lead-in must be specified for each set to make the examinee

task explicit by specifying the relationship between the items
and the options.

Items should assess application of knowledge, not just one-word

associations.

Distractors should be plentiful. Add distractors that would be

used in a traditional item if necessary.
 Sample Extended Matching Set: Flawed
60

A. is motion sickness I. are completely controlled

B. have no effects on people J. cause plant & eye damage
C. indirectly increase CO2 K. are negligible
D. cause death L. increase risk of skin cancer
E. increased odor sensitivity M. cannot be controlled
F. is a reduction in visibility N. excess acute respiratory
G. esthetics, economics, health illness among children
H. products of fossil fuel O. contrary to public opinion
combustion

1. Factors that people consider when evaluating air quality

2. A principal effect of particulate matter in air
3. The products of photochemical smog
 Options and Lead-in: Microbiology
61

A. Adenovirus L. Haemophilus influenzae

B. Aspergillus fumigatus M. Histoplasma capsulatum
C. Bacillus anthracis N. Mycobacterium tuberculosis
D. Candida albicans O. Mycoplasma pneumoniae
E. Chlamydia psittaci P. Neisseria gonorrhoeae
F. Coccidioides immitis Q. Neisseria meningitidis
G. Coronavirus R. Pneumocystis carinii
H. Corynebacterium diphtheriae S. Rhinovirus
I. Coxiella burnetii T. Streptococcus pneumoniae
J. Coxsackievirus U. Streptococcus pyogenes
K. Epstein-Barr virus (group A)

For each description, select the associated pathogen.

 Sample Poor Item: Microbiology
62

An encapsulated gram-positive organism that usually grows in

pairs or short chains.
T

This stem assesses recall of isolated facts, rather than application of

knowledge. It is more like a crossword puzzle question than an item
for a medical school examination.
 Sample Good Items: Microbiology
63

1. A 7-year-old girl has a high fever and a sore throat. There is

pharyngeal redness, a swollen right tonsil with creamy exudate,
and painful right submandibular lymphadenopathy. Throat
culture on blood agar yields numerous small b-hemolytic
colonies that are inhibited by bacitracin. U

2. For the past week, an 18-year-old man has had fever, sore
throat, and malaise with bilaterally enlarged tonsils, tonsillar
exudate, diffuse cervical lymphadenopathy, and splenomegaly.
There is lymphocytosis with atypical lymphocytes. The patient
tests positive for heterophil antibodies.
K
 Defining Content Domain Using Chief Complaints
64

Wheezing Palpitations Chest Pain

Itching Diarrhea Easy Bruising
Fatigue Anemia Lymphadenopathy
Jaundice Back Pain Confusion
Ascites Joint Pain Hematemesis/Melena
Cough Weight Loss Shortness of Breath
Fever Dizziness Abdominal pain
Syncope GI Bleeding
65

Are There Advantages of Extended Matching R-

Sets over Traditional 5-Option Single Items?
66

Percentage of Students Selecting Each Option When Item

Presented in a 5-Option and 15-Option Format

A B C D E F G H I J K L M N O
5-option 81 * 4 0 14 * 1
15-option 2 59 2 13 1 2 1 0 1 7 2 2 8 0 0
 Frequency Distribution of Scores
Comparing Sets with 5 or 15 -Options
67

20
5-Option
15-Option

15
Percent of Students

10

5
Case SM, Swanson DB. Evaluating
diagnostic pattern recognition: a
psychometric comparison of items
0
with 15, 5, and 2 options. Paper
30 40 50 60 70 80 90 100 presentation, Annual Meeting of
the AERA. San Francisco, CA:
Percent Correct Score March 1989.
ACTIVITY
 Checklist
69

 Does it assess application of knowledge?

 Does it assess key concepts and principles that
are essential information?
 Does it assess relevant material?
 Does it satisfy the “cover the options rule”?
 Checklist
70

1. Decide on the theme of your set

(eg, Establishing a diagnosis of patients with abdominal pain)

2. Develop the lead-in for your set

3. Write the options for your set

(eg, appendicitis, cholecystitis)

4. Write a patient description for each of the diagnoses in the list

5. Ask a colleague to review your items (patient descriptions) without

the correct answers noted