Guia para Investigación Residentes

J:4t~() «:..
'P~ ~0Av'1í:-'
p~ 0~.~~~~
e Research Gu· e ~/('~('~~

c:.,)'. ~/)'. ~
A primer for residents, other health care ~oo;~~ a'
trainees, and practitioners

Editors
Bart J. Harvey Eddy s. Lang Jason R. Frank
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Excellent VeryGood Good Far Poor
SCHOlAR
Royal College of Physicians and Surgeons of Canada
774 Echo Orive TEL 613-730-8177

Ottawa ON K1S SN8 TOLL FREE 1-800-668-3740
Canada EMAIL canmeds@royalcollege.ca
royalcollege.ca
The Research Guide: A primer for residents, other health care trainees, and practitioners

774 Echo Orive
Ottawa, Ontario Krs:iN8 Canada
Telephone: 613-730-8177
Toll-Free: 1-800-668-3740
Website: royalcollege.ca/canmeds
E-mail: canmeds@royalcollege.ca
© 2011 Royal College of Physicians and Surgeons of Canada. AII rights reserved.
Reprinted with an index February 2012 in Ottawa, Ontario, Canada.
Received a special recognition in the Physicians category of the American Medical Writers Association's 2012 Medical
Book Awards.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical,
including photocopying, recording, or any information storage or retrieval system, without permission in writing from
the publisher. Permissions may be sought directly from the CanMEDS and Faculty Development Unit at the Royal
College of Physicians and Surgeons of Canada at canmeds@royalcollege.ca
ISBN: 978-1-926588-09-4
How to reference this document:

Harvey BJ, ES Lang, JR Frank, editors. The research guide: a primer for residents, other health care
trainees, and practitioners. Ottawa: Royal College of Physicians and Surgeons of Canada; 2011.
--------------- - - - -- .- -----_. ------------~~~--

AI~':Di~¡Úl OZ¡:\~jO IjA,RG/ó
V!:~(\ic[' . Psinl.:i3líJ
e "';., ':7300 RrJf: 2·3?3'.¡
The Research Guide: A primer for residents,

other health care trainees, and practitioners
Editors
Bart J. Harvey, MD, PhD, MEd, FACPM, FRCPC

Eddy s. Lang, MDCM, CCFP(EM), cSPQ
Jasen R. Frank, MD, MA(Ed), FRCPC
'~ú:;Tij·~·ji6·Léiz~:t,¡6v.~i~·
Table of Contents Illél~CO - Psiquiatra
CMP. 473í)IJ FINE 2323f'
Preamble / IV
Contributors / vi
Starting
1 A research road map Fifteen steps to a successful research project (and ten pitfalls to avoid) / 1
2 Research in residency, other health care training, and practice: Why, when and how? / 11
3 Finding a research supervisor / 15
4 Research teams and networks / 25
5 Looking for an academic mentor / 29
Planning
6 Conceiving and formulating the research question / 35
7 Searching the literature / 41
8 Data: Variables and levels of measurement / 51
Designing
9 Research methods and design: Bringing the research question to life / 57
10 Developing and assessing health measurement items and scales / 71
11 Surveys / 81
12 Medical record review studies / 91
13 Administrative database research / 97
14 Health professions education research / 103
15 Systematic reviews / 109
16 An introduction to qualitative research / 119
Proposing
17 Writing a research protocol / 131
18 Getting to "approved": Concepts, guidelines and processes in ethics applications / 141
19 Research funding and other resources / !55
Conducting
20 Sampling, recruiting and retaining study participants / 163
21 Pilot studies / 169
22 Data collection and data management / 179
23 Managing and monitoring a study / 187
24 Data analysis: Descriptive statistics / 197
25 Data analysis: Hypothesis testing, sample size and study power / 213
Reporting
26 Interpreting your research findings / 227
27 Writing effective abstracts / 233
28 Communicating your research with slide and poster presentations / 247
29 How to write a health science paper / 255
30 Knowledge translation / 269
31 Responsibility and integrity Disseminating research findings to the public / 277
Reflecting
32 So, you've finished your first research projecLnow what? / 287
Reviewers / 292
Index / 294
Preamble
• The motivation to create this guide contacted several Canadian and American hea
arose from our efforts as dinician-educators to support training programs and specialry associarions. Althot
and supervise trainees-predominandy, residents from a Out discussions did nor result in rhe discovery o
variery of medical specialty training programs-who suirable resource, many of the dinician-educarors
wished or were required to complete a formal research spoke with highlighred the potential value of sud
project. Our experiences repeatedly highIighted the fact resource. Ar rhe same rime, we were encouraged by 1
thar many health trainees Iack formal educarion in the °example of some successes in research rraining, such
design and conducr of research. As a resulr, a Iarge portion the well-received two-day "inrroducrion ro researc
of rhe inirial srages of each trainee's research project was course that has been otfered for several years ro residents
spent addressing a variery of knowledge gaps, often Obsrerrics and Gynaecology by rhe Associarion
rhrough a series of one-to-one rutorials. Moreover, as we Professors of Obsterrics & Gynaecology of Canada. T
endeavoured ro provide effective guidance on a diverse organizarion and contenr of rhis course informed o
range of research efforts we ofren became aware of the rhoughrs about the possibiliry of crearing a one-sto
limitarions of our own knowledge. Over the course of Out comprehensive guide ro research for rrainees in al! heal
careers, ir has become clear thar this individuaIized care professions and specialties. We are grareful ro th
teaching of the fundamemals of research is neither course's key faculry members, Roberr Reíd and Pl
efficient nor sustainable. Even more importanr, we Hahn, for their helpful discussions, their generous inp
became ouly too aware thar, for a variety of reasons, many rhroughour rhe development of this guide, and the
promising research initiatives are ncver complered. This is conrribution as co-aurhors, collecrively, of three of i
regretrable on at least rwo levels. First, progress in research chaprers.
is a keystone of contemporary health care: patients and
health care providers alike demand continua], evidence- Once we were persuaded of rhe merits-and feasibility-
based improvernenr in the qualiry and effecriveness of of crearing a research guide for health professions trainee
careo Second, the abiliry ro crirically appraise research, we drafied an outline, a working rabie of contents and
along wirh ar least a basic farnlliarity with the conducr of prospective "abstraer" for each of the proposed chapter.
research, is an importanr dimension of rhe comperencies \V'e rhen set out ro identify and invite porential authors c
of a11 healrh care professionals. \V'ithin the domain of co-authors of whar would eventually becorne
medical pracrice, these research comperencies have been complement of 32 chaprers. Clearly, rhe hnal product-
articulared in terrns of the CanMEDS Scholar Role. The Research Guide: A primer for residenrs, orher healr
care trainees, and pracririoners-would nor have beei
Assernbling a guide such as this one is not a challenge ro possible wirhout the generous contriburions of thi
be undertaken idly, and so Out Iirst step was ro determine capabIe and diverse group of authors! We are also ver:
wherher the kind of resource we envisioned already grareful ro rhe dozens of reviewers who kindly provide:
exisred. Was therc a rexr thar could guide healrh care feedback on early draíts and hdped ro srrengrhen th
trainees rhrough al! rhe sreps of successful!y designing organization, content and relevance of each chapter. Th:
and complering a research projecr? Although several narnes and affiliations of rhese chaprer aurhors and reviewer.
excellenr texrbooks on research rnerhods are available- are listed on pages vi-vii and 294-295, respectively . W(
and many are referenced in rhis guide-none otfered the are particularly appreciative of rhe contributions ofTorr
kind of"self-study" resource we had in mind. To test our Lang, David Streiner and Ross Upshur, who not oniy
impression of the scope of existing resources we also served as chapter authors but also provided rapid ano
informarive peer review of several chaprers.
iv © 2011 The Royal College of Physicians and Surgeons of Canad

The Research Cuide is organized into seven sections, each basis of a research handbook. If you need short readings
comprising up ro eight chapters. Core componenes recur to make your supervision more efficiene, you can direct
through the book to facilitare reading: chus, where your trainee to rhe relevanr chapters in the Cuide.
appropriare, each chaprer begins with an illusrrative case,
a ser of learning objecrives and a list of "key rerrns" ro ser This project would not have been possible wirhour the
rhe contexr and relevan ce of rhe topic being addressed, suppon and "horne" provided by rhe CanMEDS Office
and concludes with a "postscript" to the illusrrarive case, at the Royal College of Physicians and Surgeons of
ene or more exercises, and a summary checklist. For Canada. \Y/e are particularly indebred to Wendy Jemmert
readers who would like to explore sorne aspecrs of a ropic from the CanMEDS Office, who very capably managed
further, an annorared lisr of additional resources is this projecr throughout. We would also like to express our
included at the end of each chapter. gratitude to Anne Marie Todkill for her editorial suppon
and expertise in helping to transform each draft chaprer
lf you are new to research, The Research Cuide will ease inro a more efficient, readable and saristying formo
you into the journey. Each chapter is in tended to walk
you through what you need to know and what you need The Research Cuide is meanr to make the lives of novice
ro do to ensure success in your project. The sections of the researchers and rheir supervisors easier. We would like ro
Cuide are meant to presene a logical progression in step hear from you, the reader, about how well we have
wirh rhe development of your project, but each srands achieved this goal. Please tell us how fmure editions of
alone. You can use rhe Cuide as a personal modular this guide could be improved. Are rhere gaps ro address?
curriculum fOI"rhe development of research skills, or as an Ways to make the Cuide more relevant and easy ro use!
inrroductory reference to be consulted as needed. The \'{1ewelcome all comrnents and suggestions at canmeds@
universe of research could nor possibly be compressed royalcollege.ca
into a brief rexr, and so we have designed rhis book ro be
accessible, practica! and succinct. We wish you an eye- In the mean time, we hope you will enjoy the guide and
opening, informarive and rewarding experience. This is your research journey!
your opporruniry to berrer understand and even
con tribute to progress in the vast world of modern health
careo
lf you are a supervisor of new researchers, this guide is Bart ]. Harvey, Toronto
also for you, and has been written by your peers, who Eddy S. Lang, CaIgary
know whar it takes to suppon and mentor those less ]ason R. Frank, Ottawa
experienced in rhe process of a scholarly inquiry. If you
need an accessible curriculum to help your trainees
undersrand what needs ro be done, rhis text can form rhe September 1, 2011
© 2011 The Royal College 01 Physicians and Surgeons 01 Canada

v
:=
Contributors
Stacy Ackroyd-Stolarz, MSc, PhD lan D. Graham, PhD
Queen Elizabeth II Health Sciences Centre Halifax Infirmary Canadian Institutes of Health Research
Dalhousie University Ottawa, Ontario
Halifax, Nova Scotia
Stefan Grzybowski, MD, FCFp, MCISc

Susan J. Bondy, BA, MSc, PhD, FACE Department of Family Practice
Dalla Lana School of Public Health University of British Columbia
University of Toronto Vancouver, British Columbia
Toronto, Ontario
Philip M. Hahn, MSc

Carolyn Brown, ELS Department of Obstetrics and Gynecoiogy
Science and medicine writer and editor Queen's University
and publishing consultant Kingston, Ontario
Ottawa, Ontario

G. Mark Brown. MSc Dalla Lana School of Public Health
University of Calgary University of Toronto
Calgary, Alberta Toronto, Ontario
June C. Carroll, MD, CCFp, FCFP Grant Innes, MD, FRCPC

Department of Family and Community Medicine Emergency Medicine
Mount Sinai Hospital, University of Toronto Alberta Health Services
Ioronto. Ontario University of Calgary
Calgary, Alberta
Steve Choi, MD, FRCPC
Departrnent of Emergency Medicine Monika Kastner, PhD
The Ottawa Hospital, University of Ottawa Li Ka Shing Know!edge Institute
Ottawa, Ontario St. Michael's Hospital, University ofToronto
Ioronto, Ontario
Kaberi Dasgupta, MD, MSc, FRCPC
Department of Internal Medicine and Endocrinology Karim Khan, MD, PhD
McGill University Health Centre, McGill University Department of Family Practice
Montreal, Quebec University of British Columbia
Vancouver, British Columbia
Jason R. Frank, MD, MA(Ed), FRCPC
Office of Education Terry P. Klassen, MD, MSc, FRCPC
Royal College of Physicians and Surgeons of Canada Manitoba Institute Chi/d Health
Ottawa, Ontario Winnipeg, Manitoba
Scott Garrtson, MD Lorie A. Kloda, MLlS, AHIp, PhD (candidate)

Department of Family Practice McGili Life Sciences Library
University of British Columbia McGill University
Vancouver, British Columbia Montreal, Quebec
vi © 2011 The Royal College 01 Physicians and Surgeons 01 Ca-a.::a

Contributors
Eddy S. Lang, MDCM, CCFP(EM), CSPQ Sharon E. Straus, MD, MSc, FRCPC
Alberta Health Services Li Ka Shing Knowledge Institute
University of Calgary SI. Michael's Hospital, University of Toronto
Calgary, Alberta Toronto, Ontario
Thomas A. Lang, MA David L. Streiner, PhD, CPsych

Tom Lang Communications and Training Department of Psychiatry and Behavioural Neurosciences
Kirkland, Washington McMaster University, Hamilton, Ontario
Department of Psychiatry
A. Curtis Lee, PhD University of Toronto, Toronto, Ontario
Educational Evaluation and Analysis
Royal College of Physicians and Surgeons of Canada Vicky Tagalakis, MD, MSc
Ottawa, Ontario Department of Generallnternal Medicine
Jewish General Hospital, McGi11 University
Sarah Jane Lusina, MSc Montreal, Quebec
Centre for Hip Health and Mobility
University of British Columbia Jacqueline Tetroe, MA
Vancouver, British Columbia Canadian Institutes of Health Research
Ottawa, Ontario
Fiona Alice Miller, MA, PhD
Department of Health Policy, Management and Evaluation Ross E.G. Upshur, MD, MA, MSc, CCFp, FRCPC
University of Toronto Department of Family and Community Medicine
Toronto. Ontario Sunnybrook Health Sciences Centre, University of Toronto
Toronto, Ontario
Jeffrey J. Perry, MSC, MD, CCFP-EM
Department of Emergency Medicine Christian Vaillancourt, MD, MSc, FRCPC, CSPQ
Department of Epidemiology and Community Medicine Department of Emergency Medicine and
University of Ottawa Ottawa Hospital Research Institute
Ottawa, Ontario The Ottawa Hospital, University of Ottawa
Ottawa, Ontario
Robert L. Reid, MD, FRCSC
Division of Reproductive Endocrinology and Infertility Carl van Walraven, MD, MSc, FRCPC
Department of Obstetrics and Gynecology Department of Medicine
Kingston General Hospital, Queen's University The Ottawa Hospital, University of Ottawa
Kingston, Ontario Ottawa, Ontario
David L. Sackett, OC, MD, FRSC, FRCP Andrew Worster, MD, MSc, CCFP(EM), FCFP
Kilgore Trout Research and Education Centre Division of Emergency Medicine
Hamilton, Ontario McMaster Un iversity
Hamilton, Ontario
Julie M. Spence, MD, MSc, FRCPC
Department of Emergency Medicine
St. Michael's Hospital, University of Toronto
Toronto, Ontario
© 2011 The Royal College 01 Physicians and Surgeons 01 Canada vii

5. Develop a research outllne."

for which the data already exist) generall
• .- • Conduct a thorough literature search (ch. 7).

• If your project is designed to examine a therapeutic
less time and fewer resources than a pro:
study (such as a cohort study or randomi
intervention, check for applicable systematic reviews in
for which primary data collection will be
The Cochrane tibrary-
required).
• Collaborate with content and methodological experts.
Consider conducting a clinical audit, syst:
Your supervisor should be able to suggest suitable
review or survey as a manageable project
individuals and to facilitate an introduction to them.
first venture into research.
• Determine which study design is the best fit and the
most practical approach to framing your research • As necessary, estimate an appropriate sample s
project (ch. 9). Focus your energy on addressing the prirr
• Write an outline. research objective/question with just the r
Focus on the primary objective or question of the number of subjects. A study with too mar
proposed research. subjects can expose participants needlessl
Include a brief background statement highlight- potential risks. A study with too few subje
ing the importance of your research question. might not have sufficient statistical power
Estimate how much time you will need to detect a clinically important difference.
complete each of the anticipated stages of the Design an appropriately sized, simple stud
research. than a small, complicated ene.'
List available and required resources and, if
• Discuss the anticipated data collection and ana!
applicable, provide a budget estimate.
methods (chs 22,24 & 25)
If you already have a variety of interesting ideas to
• Select the tools (calculators, databases, and stat
talk about, be sure to keep track of them, but for
and data-entry software) that will be approprian
the purposes of your study outline you will need
your analysis of the study data.
to boil them down to a single, focused, primary
• Investigate which file formats for data recording
study objective.
will allow your study data to be imported into al
Make sure your outline is concise-a maximum of
appropriate application for statistical analysis (e.:
two pages is best at this point.
Excel spreadsheet for irnport into SPSS).
With the outline of your proposed research
project in hand, arrange to meet with method- 7. Develop the research protocol" (ch.
ological specialists (Step 6) in preparation for
• Create an expanded version of the outline creat:
writing a more detailed protocol (Step 7). Step 5.
• Include details on:
6. Meet with methodological (especially
research team members and their roles (eh:
biostatistical) specialists with particular
recruitment of study participants (ch 20)
expertise in your area of study.
inclusion and exclusion criteria
• Refine the study's primary objective. design features such a criterion standard fo
• Discuss pertinent design issues, focusing on the clinical practice audit.' the sampling technn
primary study objective. a survey" or the random allocation tecbniqi
• Se realistic. creation of placebos for a randomized cont
Sear in mind that planning and completing a trial.?
retrospective study (such as a systematic review, any secondary objectives, questions and ou
case-control study, chart review or practice audit, comes
statistical issues such as estimating sample ~
and methods to analyze the data (chs. 24 &
a 5ee www.thecochranelibrary.com
2 © 2011 The Royal College 01 Physicians and Surgeons e

A research roa ca: Fiteen steps to a successful research project
your timetable for starting and finishing the

10. If you are conducting a clinical trial,
ensure that it is registered with
,,
project, as well as the anticipated time frame for
ClinicalTrials.govd (ch. 18). ~
each phase of the research project.
ethical considerations such as safety, confidenti- • Member journals of the International Committee
,••
ality and inforrned consent'? (ch. 18) of Medical Journal Editors (ICMJE), also called •
•
• The proposal should contain as much detail as possible the Vancouver Group, require, as a condition of
and provide the framework for ethics submission (ch consideration for publication, registration of al!
18) and ultimately drafting a manuscript (ch. 29) clinical trials in a public trials reqistry." Complete the
trial registration befo re any study participants are
8. As applicable, obtain institutional and
recruited and enrolled.
research ethics approval (ch. 18).
11. Collect and analyze the data (eh s 22,
• Obtain approvals specific to your institution or
24 & 25).
research centre.
• Seek help from your research supervisor and • Follow the data collection and analysis plan described
other university and hospital personnel who are in the research protocol
knowledgeable about and experienced with approval • Report confidence intervals, if appropriate, in
processes and requirements. addition to P values for the results of the primary
• Find out whether your project is eligible for an and any secondary research questions. As Martin
expedited review, which will take less time than a full Gardner and Douglas Altman note, "Overemphasis
Research Ethics Board (REB) review. A clinical practice on hypothesis testing-and the use of P values to
audit, for example, might be eligible for an expedited dichotomise significant or non-significant results-
review. has detracted from more useful approaches to
• If your research involves humans or human tissues, interpreting study results .... In medical studies
review the Tri-Council policy staternent" and complete investigators are usually interested in determining the
the Tri-Council tutorial on the ethical conduct for size of difference of a measured outcome between
research involving humans." groups, rather than a simple indication of whether or
not it is statistically significant." 14
9. Seek necessary funding (ch. 19)
12. Present your findings (chs 27 & 28).
• Consult with your research supervisor and
collaborators to identify potential funding sources for • Remember that one of your responsibilities as a
your project. researcher is to communicate your results.
• Seek departmental and university resources first. • Find appropriate venues in which to present your
• If appropriate, submit a grant to an external funding work as soon as possible, such as your department's
agency such as the Physicians' Services Incorporated annual research day and relevant local, national and
Foundation." or the Canadian Institutes of Health international meetings.
Research.'
13. Prepare and submit a manuscript
describing the study and its results to
a suitable journal (ch. 29).
• Be prepared for some hard work on preparing your

research report for publication In an excellent review
on preparing manuscripts for submission to medical
journals, Gill Welch outlines a systematic approach
b See the PSI Foundation website at www.psifoundation.org/

ResidentResearchPrizes.html
c See the CIHR website at www.dhr.ca/ d The registry can be accessed at wwwclinicaltrials.gov/
© 2011 The Royal College 01 Physicians and Surgeons 01 Canada 3

The Research Guide: A primer for residents, other heal h care trainees, and practitioners
for making it easier to prepare a readable paper." His will be listed. The first author is generally t
take-home points are: individual who has, overall, contributed th

most, while the last (or second) author is
.- Start writing before your project is completed .
Focus your attention on what readers are most generally the project supervisor.
likely to look at: the title, abstract, tables and The ICMJE criteria for authorship credit sta
figures. that authorship requires: (1) substantial
Develop a systematic approach to the introduc- contributions to conception and design, or
tion, methods, results and discussion. acquisition of data, or analysis and interpre
Improve the paper by learning how to obtain tion of data; (2) drafting the article or revisi
and incorporate useful feedback. critically for important intellectual content;

(3) final approval of the version to be pub-
• Familiarize yourself with the guidelines and checklists lished."
that you will need to follow in reporting your Be prepared to describe the contribution of
methods and results and in disclosing competing each author. Many journals require on subn
interests. An awareness of these standards before sion a description of each author's contribuí
you begin will guard you against any deficiencies
and publish a contributors' statement at the
that could make your report ineligible for
end of each article.
publication.
Also consider those whose contribution sho
For examples of guidelines for the reporting of
be acknowledged in the published article.
findings, see the CONSORT Statement on the
Many journals require written permission frc
reporting of randomized controlled trials as well
anyone identified in an acknowledgement.
as its extensions for other study types; the
PRISMA Statement for the reporting of system-
14. If your manuscript is accepted, revis
atic reviews and meta-analyses; and the STARD it according to the editors' and
Statement on the reporting of diagnostic reviewers' comments.
accuracy studies.'
• If your manuscript is rejected, take into account
For an example of a checklist for reporting
the editors' and reviewers' comments and considr
competing interests, see the Financial Conflicts
submitting the revised paper to another appropriz
of Interest Checklist 2010 for Clinical Research
journal (ch. 29).
Studies developed by Rochon and colleaques."
A fuller list of reporting guidelines is available
on the website of the EQUATOR Network, an 15. Celebrate with your coauthors.
international initiative to improve the reporting
• You have just completed a research project and
of health sciences research.'
have contributed to the creation and disseminatio
• Familiarize yourself with the Uniform Requirements of new health science knowledge. Don't let this
for Manuscripts Submitted to Biomedical Journals accomplishment pass without taking time to
outlined by the ICMJE;9 these requirements are celebrate with your co-investigators and others wl
applied by most peer-reviewed medical journals. have supported you and the project.
• Establish authorship, and the order of authorship.
As early as possible, establish authorship
credits, including the order in which authors
e The most recent guidelines for CONSORT, PRISMA and STARD can be
viewed al wwwconsort-stalement.org/consort-statementl, www.prisma-
statement.org/ and www.stard-statement.org/
See the EQUATOR website al www.equator-nelwork.org/resource-centre
9 Available at www.icmje.orglurm_main.hlml
4 © 2011 The Royal College 01 Physicians and Surgeons 01 Cal

7 A research roa a _--'-;_een steps to a successful research project
TEN COMMON RESEARCH PITFALLS
1. Not establishing a focused, answerable question.

2. Enlisting a research supervisor who doesn't make sufficien time o advise and help you throughout the stages
of the project.
3. Picking a topic about which you have little interest.
4. Planning a small, complicated study that attempts to answer many questions, rather than an appropriately sized,
simple study focused on one primary objective/question.
5. Not taking the time to draft a research outline to keep your research team coordinated and on schedule.
6. Not being realistic about how much time and effort your project will take (see "Tips for Staying on Track").
7. Basing a prospective study on outcomes that are rare or take a long time to occur.
8. Entering your data into an Excel spreadsheet using formats that are not compatible for importing into a statistical
application such as SPSS.The only thing worse than entering data is having to enter it twice.
9. Not meeting with a statistician to talk about the analysis before you begin collecting the data.
10. Waiting too long to begin (see "Tips for Staying on Track").
TIPS FOR STAYING ON TRACK
As you've no doubt already recognized, your biggest obstacle to successfully completing a research project will likely be
finding the time.'B-20 Here are some tips for staying on track to finish a research project.
1. Carving out one or more blocks of protected time is key." Use this time to develop your research proposal and start
off on the right foot; additional blocks of time can be used for data collection, analysis or vvrite-up.
2. If you are in a two- or three-year program, consider a study design that will allow you to finish on time, such as a
medical record review or practice audit, for which the data should be comparatively easy to access and for which an
expedited ethics review might be feasible.
3. If you are in a longer program, such as a five-year medical or surgical specialty, consider the following timelines and
milestones.
• In your first year, introduce yourself to the basic concepts of research methodology by taking a dedicated course or
intensive workshop. Identify a research supervisor.
• Identifya methodological specialist to help you develop your research question, study design and research protocol
in your second year. Submit your study for ethics approval and funding opportunities. If you are conducting a
clinical trial, remember to reqister your study before you begin to enrol participants.
• Collect and analyze your data, and then present your findings locally, nationally or beyond by the end of your third
or fourth year.
• Begin drafting your manuscript. aiming for completion in year four .
• Submit your manuscript to a suitable journal early in year five, leaving your final term free to prepare for your
certification exam and life after graduation.
© 2011 The Royal College of Physicians and Surgeons of Canada 5

REFERENCES
en
_
.e
......,
1. Thabane L, Thomas T, Ye C, Paul J. Posing the research question: not so simple. Can J Anaesth. 2009;56(1):71-9.
2. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Oesigning clinica/ research: an epkiemiolock: approach. 3rd
ed. Philadelphia: Lippincott Williams & Wilkins; 2007. Appendix 1.1; p. 15.
~
ca
....,
3. Altman DG. Statistics and ethics in medical research: 111 How large a sample? Br Med J. 1980;281 (6251): 1336-8.
4. Schulz KF,Grimes DA Sample size calculations in randomised trials: mandatory and mystical. Lancet. 2005;365(9467): 1348-53
V\ 5. McAlister FA,Straus SE, Sackett DL. Why we need large, simple studies of the clinical examination: the problem and a propos
solution. CARE-COAD1 group. Clinical Assessment of the Reliability of the Examination-Chronic Obstructive Airways Disease
Group. Lancet. 1999;354(9191) 1721-4
6. Cummings SR, Hulley SB. Writing and funding a research proposal. In: Hulley SB, Cummings SR, Browner WS, Grady DG,
Newman TB, editors. Oesigning c/inica/ research: an epidemiolcqi: approach. 3rd ed. Philadelphia: Lippincott Williarns &
Wilkins; 2007. p. 301-16.
7. Godwin M. 2001. Conducting a clinical practice audit. Fourteen steps to better patient care. Can Fam Phys. 2001 ;47(11 ):2331-~
8. Salant P,DA Dillman. How to conduct your own survey New York: John Wiley & Sons; 1994. Chapter 5, When and how to
select a sample; p. 53-72.
9. Pocock SJ.Clinica/ tria/s: a practica/ approach. Toronto: John Wiley & Sons; 1983. Chapter 5, Methods of randomization, and
chapter 6, Blinding and Placebos; p. 66-89, 90-9.
10. Pocock SJ.Clinica/ tria/s: a practica/ approach. Toronto: John Wiley & Sons; 1983. Chapter 7, Ethical issues: p. 100-9.
11. Canadian Institutes of Health Research, Natural Sciences and Engineering Council of Canada, Social Sciences and Humanities
Research Council of Canada. Tri-Counci/ po/icy sta temen t: ethica/ conduct for research invo/ving humans (December 2010).
Ottawa: Interagency Secretariat on Research Ethics; 1998. Available from: www.pre.eth¡cs.gc.ca/pdf!eng/tcps2frCPS_2_FI~jAl..
Web.pdf
12. Panel on Research Ethics. Introductory tutoria/ for the Tri-Council policy statement: ethica/ conduct for research invo/ving
humans [Iast modified 2009 Aug 30; cited 2009 Aug 30]. Available from http://pre.ethicsgc.ca/english/tutorial/
13. DeAngelis C, Drazen Jfv1,Frizelle FA, Haug C, Hoey J, Horton R, et al. 2004. Clinical trial registration: a statement from the
International Committee of Medical Journal Editors. CMAJ. 2004; 171 (6):606-7
14. Gardner MJ, Altman DG. Confidence intervals rather than P values: estimation rather than hypothesis testing. BMI
1986;292(6522):746-50.
15. Welch HG. Preparing manuscripts for submission to medical journals the paper trail. Eff Clin Pract. 1999;2(3): 131-7.
16. Rochon PA, Hoey J, Chan AW, Ferris LE, Lexchin J, Kalkar SR, et al. Financia! conflicts of interest checklist 2010 for clinical
research studies. Open Med. 2010;4(1 ):69-91. Available from: www.openmedicine.ca/article/view/356/318
17. Hoey J. Who wrote this paper anyway7 The new Vancouver Group statement refines the definition of authorship. CMAI
2000; 163(6):716-7
18. Chan RK, Lockyer J, Hutchison C. Block to succeed: the Canadian orthopedic resident research experience. Can J Surg.
2009;52(3) 187-95.
19. Silcox Le. Ashbury TL, VanDenKerkhof EG, Milne B. Residents' and program directors' attitudes toward research during
anesthesiology training a Canadian perspective. Anesth Ana/g. 2006; 102(3):859-64
20. Gill S, Levin A, Djurdjev O, Yoshida EM. Obstacles to residents' conducting research and predictors of publication. Acad Med.
2001 ;76(5):477
6 © 2011 The Royal College of Physicians and Surqeons 01 Canada

A research roa eo: ~;Leen steps to a successful research project
ADDITIONAL RESOURCES
I recommend the following books to busy professionals who want to impro e their understanding of key concepts in biostatistics
and epidemiology
•
I
Biostatistics •
•
Altman DG. Practical statistics for medical research. London (UK): Chapman & Hall; 1991.
•
• Douglas Altman is Director of the Centre for Statistics in Medicine in Oxford, England. By discussing both the use and misuse
of statistics this book equips the reader to judge the appropriateness of the methods and interpretations presented in papers
published in medical journals.
Altman DG, Machin D, Bryant TN, Gardner JM, editors. Statistics with confidence: confidence intervals and statistical guidelines
2nd ed London (UK) BMJ Books; 2000.
• As Gardner and Altman say elsewhere." "Overemphasis on hypothesis testing-and the use of P values to dichotomise
significant or non-significant results-has detracted from more useful approaches to interpreting study results, such as
estimation and confidence intervals." This book gives guidelines for calculating and using confidence intervals around just
about any point estímate.
Norman GR, Streiner DL. PDQ statistics. 3rd ed. Hamilton (ON): B.C Decker; 2003.
• That's PDQ for "pretty darn quick ": with this book, you can rapidly find concise descriptions of statistical tests that you might
come across while reading journal articles.
Epidemiology
Sackett DL, Haynes RB, Guyatt GH, Tugwell P. C1inical epidemiology: a basic science for clinical medicine. 2nd ed. Toronto (ON):
Little Brown and Company; 1991.
• This text applies the principies learned in epidemiology to the complex clinical decisions that must be made every day.
Streiner DL, Norman GR. PDQ epidemiologr. 2nd ed. Hamilton (ON): B.C. Decker; 1998.
• This efficient study guide in the "pretty darn quick" series introduces the reader to the world of epidemiology; it covers data-
gathering, sampling procedures, study designs, biases, measuring reliability and validity, and much more.
Evidence-based medicine
Godwin M, Hodgetts G. The Bedford murder. an evidence-based clinical mystery Philadelphia (PA) Hanley & Befus; 2003.
• This murder mystery novel reveals clinical pearls and the concepts of evidence-based medicine.
Greenhalgh T How to read a paper: the basics of evidence-based medicine. 3rd ed. Oxford (UK): Blackwell Publishing; 2006.
• One of the best-selling texts on evidence-based medicine, this book is used by health care professionals and medical students
worldwide. With chapters on topics such as "statistics for the non-statistician," it serves as a good critical appraisal primer
for journal clubs.
rvlcKibbon A, Wilczynski N. PDQ evidence-based principies and practice. 2nd ed. Shelton (CT): People's Medical Publishing House;
2009.
• "Provides a plain-Ianguage approach to basic principies of evidence generation and application" (Brian Haynes, Chief, Health
Information Research Unit at McMaster University). This text has a good section on systematic reviews.

en
._
t: SUMMARV
o
CHECKLlST
Meet with your program director or departmental research coordinator as soon as possible.
O Look for resources that provide an introduction to the basic concepts of research methodology and critical apprais,
O Find a research supervisor.
O Pose a focused and specific research question.
O Develop a research outline.
O Meet with methodological (especially biostatistical) specialists with particular expertise in your area of study.
O Develop the research protocol.
O As applicable, obtain institutional and research ethics approval.
O Seek necessary funding.
O If you are conducting a clinical trial. ensure that it is registered with ClinicalTrials.gov
O Collect and analyze the data.
D Present your findings.
O Prepare and submit a manuscript describing the study and its results to a suitable journal.
O If your manuscript is accepted, revise it according to the editors' and reviewers' comments.
O Celebrate with your coauthors.
10 © 2011 The Royal College 01 Physicians and Surgeons 01 Canad

2
Research in residency, other health care training,
and practice: Why, when and how?
--
,.
Robert L. Reid, MD, FRese
--
ILLUSTRATIVE CASE
A second-year Obstetrics and Gynecology resident is told by her program director that she must present a research project
at the annual resident research day next year. She is swamped with clinical work and is using every spare moment to study
for an upcoming departmental exam. The last thing she needs to add to her workload is a research project, and so she
asks her director if she can duck the "researcher role" and spend the extra time and energy becoming a better clinician.
•
Competency
The CanMEDS 2005 Physician
Frarnework' identihes seven overlapping core
This guide has been designed to provide a framework ro
help learners acquire basic skills as researchers, in keeping
cornpetencies as essenrial in the preparation of physicians with the fourrh component of the Scholar Role. However,
ro meet rhe needs of patients in rhe 21st cenrury. Among research experience enhances the other comperencies of rhe
rhese, rhe Scholar Role serves to anchor conremporary Scholar by fostering lifelong learning along wirh skills in
medical pracrice in continuing professional developmenr, critical evaluation and the translation of new knowledge ro
research literacy, crirical appraisal skills and rhe abiliry ro orhers. Experience in the conduce of research helps clini-
educate orhers. The best practice is informed by cians and other healrh care practitioners to more capably
scholarship. A failure ro adequately equip graduares wirh challenge clairns based on faulry research or irnproper
skills in critical appraisal and an understanding of research interpreration, and ro confidendy incorporare inro their
methodology has been idenrified as a common deficiency practice those innovations that have been dernonstrared ro
of training prograrns.? and examples of the devastating irnprove the qualiry of careo
effecrs of this scienrihc illiteracy abound.:'
To fulfill rhe Scholar Role, physicians must dernonstrate
CHAPTER OBJECTIVES
that rhey:
After reading this chapter, you should be able to
• rnaintain and enhance professional activiries through • describe why scholarly activity, including research, is an
ongoing learning; essential prerequisite for residents, other health care
• crirically evaluare information and irs sources, and trainees, and practitioners;
apply this appropriarely ro practice decisions; • list the key components of the CanMEDS Scholar Role;
• facilitare rhe learning of patienrs, families, srudents, and
residenrs, other health professionals, rhe public and • highlight the two most critical choices in planning a
orhers, as appropriare: and, research project-especially one's first.
• conrribure ro the crearion, dissemination, applicarion
and translation of new medical knowledge and
practices. l

The chapters of this guide provide a road map for the critical appraisal skills if they hope to eleva te discussio
novice health care researcher. They examine what consti- with patients aboye the level of what they hear from pOF
tutes a scholarly project; emphasize the importance of find- lar media. New health care practice subscription servir
.- ing a qualified supervisor or mentor and a working are emerging on the Internet that identify and summar
environment that supports research; and explore rhe funda- important discipline- and specialry-speciric research fin
mentals of conceiving and formulating a research question, ings while providing a short critical appraisal of rhe qual
conducting literature searches, choosing an appropriate of rhe research. Although these services can be extrem:
study design, obtaining institutional approval, collecting helpful to busy health care practitioners, rhey genera
and analysing the data, and reporting the results. Annotar- require a basic understanding of key research concepts al
ed lists of references and resources are provided to assist in suflicient skill in critical appraisal to judge the quality
the planning and implementation of a research project, evidence and facilitate meaningful inrerpretation and d
Throughout the guide, the authors-all of whom are expe- cussion. In addirion, practitioners who acquire ba:
rienced invesrigators-offer pracrical advice on building a knowledge and skills with regard ro health research will
solid foundarion for successful health research. betrer equipped to contribute to relevant research rher
selves, such as by identifying patients who meet rhe enrr
ment criteria for a clinical rrial or even by serving as a stu
Why me?
co-invesrigator.
No doubt, many resident physicians, other health care
trainees and practitioners reading this introduction will
When can I find time?
think, "That's all fine, but 1 didn'r decide to become a
healrh professional so rhat 1 could do scienrific research. The trainee in our case example is no doubt experienci:
I'rn not looking for a Nobel Prize. 1 just want to be a corn- rhe same worry about participating in research as many
petent and caring health care practitioner." her peers have before her: she wonders where in the wor
The answer ro "\V'hy me?" lies in the many career paths she will find the time-and the energy-ro fir a resean
open to graduares of residency and other health care train- project around all ofher clinical responsibilities.
ing programs. Craduaring pracritioners are charged wirh Within rhe time frame of a healrh care practitioner trai
rwo responsibilities: (l) ro provide the highest possible ing program, ir is generally unrealisric to expecr to have
quality of healrh care ro patients and popularions; and (2) significant period of protected time in which ro condu
ro educare rhe next generations ofhealrh care practirioners, research. Exceptions are the Royal College Clinical Inves
Research is essential to evaluaring and improving perfor- gator Program and a few residency programs rhat are al:
mance in both of these roles. to offer a one- or rwo-year deferral for rhe complerion 01
You may think at rhe start of your training thar a research Masrers or PhO degree. For most trainees, howevcr, al
role would be better lefr to your colleagues who received research project will need to be underraken simulraneous
Masters or PhO degrees before they entered health pracrice with training activities. A short elecrive devored to rl
training. Although it is true rhat such individuals bring spe- development of a research projecr can be ideal, althouj
cialized knowledge and research skills ro rheir training, this is not available in all prograrns. Given the demands
many excellent researchers were not "rurned on" to a any training program, ir is imponanr to plan a resear:
research career until they encountered a challenging prac- project realistically and at an early srage. Time will be nee
tice-based problem during their residency, other health ed to identiíy and meet with a qualified supervisor or me
training, or practice. Moreover, ofren because of lifesryle tor, to develop the research idea, establish necessa
factors or rhe iníluence of mentors, many health science collaborations, and prepare the proposal and orher doc
students and rrainees change career course during clinical rnentation for ethics revicw, Care in [he planning stage w
training, and so ir is wise to keep one's options open as long pay big dividends larer on.
as possible for advanced supplernentary training-where For a first foray into research, ir often makes sense
research involvemenr is otten cxpecred, design, in consultarion with a supervisor or mentor, a prc
Those who choose to enter practice as soon as rhey com- ecr that examines data retrospectively, examples are a sy
plete their training will need to acquire at leasr rhe basics of tematic review, survey-based study or chart audir. Mo
© 2011 The Royal College 01 Physicians and Surgeons 01 Can,

12
2 Research in residencr, other hea ~ cere rraining and practice. Whr, when and how?
ambitious projecrs rhat collect data prospectively, as in a ernerze hom rhose who are willing ro take the rime ro pur-
clinical study of a new test or rrearmenr or a mulri-centre sue a new idea, accepr failure if ir occurs and move on wirh
trial, require considerably more preparation and, of course, orher attemprs ro solve the problem.
the requisite funding. Ir is better, at íirst, ro rhink small and
see a project rhrough than ro start an enormous project rhar
,•
falls apart for want of time, fllnding or subject recruitrnenr, "A discovery is said to be an accident meeting
or because of investigaror burnou t. a prepared rnind.:"
Albert Szent-Gyórgyi (1893-1986)

Where do study ideas come from?
Nobel Prize laureate in 1937
Dramatic innovarions in healrh and science over rhe past
60 years have propelled us into rhe 21 st century wirh a
mornentum that is already revolurionizing the delivery of
How do I get started?
healrh care and the way we live our lives. Did these discov-
eries happen by accidenr? Although rhere are many seren- When you have an idea for a solurion or a new approach ro
dipirous findings in science, discovery and "happy a problem, write ir down before ir vanishes among all rhe
accident" ofren occur ro those who are the most alert and orher issues arising from the counrless health care concerns
prepared. Many of rhe most exciting revelations result you encounrer in the course of a day. Then, when you have
when discoveries in rhe basic sciences are brought ro bear more rime, do a rhorough lirerature search ro see wherher
on pressing clinical problems by those with an open mind sorneone has rested your idea befo re. Make sure, before you
and a willingness ro crirically challenge existing dogma. ernbark on a research projecr, rhat you won't end up simply
Witness rhe discovery by Australian clinician-scientists reinvenring the wheel.
Robin Warren and Barry Marshall thar peptic ulcers are lf, like many healrh trainees and practirioners, you feel
not, in fact, caused by stress or the consumprion of alcohol stuck for a clever or feasible research idea, turn ro your
or spicy food, but by infecrion wirh rhe bacterium Helico- supervisor or mentor for help. Qualified supervisors and
bacter pylori. Marshall and Warren challenged conventional menrors have a track record of productive research. Typi-
rhinking on rhe subjecr, "rewrore the book" on pepric ulcer cally, they have a host of ideas relating ro rheir own research
disease, and were awarded rhe 2005 Nobel Prize for Medi- bur not enough time ro pursue each one. A keen trainee
cine or Physiology for their insight and eflorts. who is willing ro tackle one of rhese projecrs can forge a
Those who work in the rrenches of health practice face long-term collegial relationship rhat can potentially benefir
common problems for which rhere appears ro be lirrle borh parries for years ro come.
progress roward a solution. An unwillingness ro explore
new srrategies is ofren explained by a shortage of time,
Conclusion
which seems ro lead ro complacency. Unfortunately, many
accepr whar appears ro be a suboprimal approach sirnply An undersranding of the basics of research merhodology
because "that's the way it's always been." and pracrice are essential for al! health trainees and
Novel ideas arise when old problems are considered practitioners ro allow thern ro critically evaluate new healrh
from a new perspecrive. However, new ideas are ofren dis- science developrnenrs. During your health pracrice
missed with a quick rebuke: "lf rhe solurion were thar obvi- training, you will have valuable opportunities to enhance
ous, someone else would have done ir already." Good these skills by entering research training programs,
hunches can be discarded for any number of reasons. Albert studying rexrs such as chis one, participating in activities
Szenr-Cyorgyi, the firsr scienrist ro isolare viramin e, rnade such as journal clubs and research projects, and interacring
imporrant discoveries by paying attention ro resulrs that with experienced researchers at your universiry. Taking
another researcher had lirerally poured down rhe drain. advantage of rhese opportunities at an early srage of your
Nor all ideas work out. BU(, ro use a phrase artributed ro career will help you to become a well-rounded practirioner
Nobel Prize winner Linus Pauling, "The best way ro have a and ro develop íurrher the irnportant comperencies of rhe
good idea is ro have a lot of ideas." Healrh care leaders Scholar Role. 1.1

The Research Guide: A primer for residents, ather health care trainees, and practitianers
CASE POSTSCRIPT
._ Searching her department's

progress within the department
website, the resident informed herself about the range of research projects that were in
and about the research interests and activities of individual staff. She asked senior
residents about the quality of supervision, support and mentorship offered by different staff. She then approached a
qualified and supportive faculty member to see if he would be willing to serve as her research supervisor. The resident
spent a one-month elective later in the year developing a research idea and writing her proposal. This involved several
meetings with her faculty supervisor. Her supervisor facilitated a meeting with a statistician during the planning stages
to ensure that the study's sample size would be adequate to demonstrate a difference between twa labaur-induction
protocals. Knowing that the project was well planned and adequately powered, the resident and her supervisor submitted
the protocol to the university and hospital Research Ethics Boards and received approval. The resident then involved several
other residents from the labour and delivery floor in data collection over the next year, and all contributed as co-authors to
a prize-winning paper presented at the departmental research day one year later. The resident was also gratified to have
her findings published in a leading journal of Obstetrics and Gynecology several months later.
REFERENCES
1. Royal College of Physicians and Surgeons of Canada. CanME05 2005 Physician Competency Framework. Ottawa: The
College; 2005. Available from: http//rcpsc.medical.orglcanmeds/CanMEDS2005/index php
2. Mason AD, Biehler JL, Linares MY, Greenberg B. Perceptions of pediatric emergency medicine fellows and program directors
about research education. Acad Emerg Med. 1991;6(10):1061-5.
3. Jordan B, Mooney C. Why we need to truly understand the medicalliterature. Contraceptian. 2007;75(6):405-6.
4. Szent-Gybrgyi A. Laaking back. Perspect Bial Med. 1971; 15(1): 1-5. Available from: http//profiles.nlm.nih.gov/WGNiews/
AlphaC hron/date/l 0005/
SUMMARY CHECKLlST
o Describe at least three reasons to get involved in the world of research.

O Consider, personally and professionally, what you want to get out of being involved in research.
O List three people you can talk to about research.
O Get started. Let this book be your guide!
14 © 2011 The Royal College 01 Physicians and Surgeons oí Canada

3
Finding a research supervisor
Sarah Jane Lusina, MSc
Scott Garrison, MD
Stefan Grzybovvski. MD, MCISc, FCFP
Karim Khan, MD, PhD
ILLUSTRATIVE CASE
A family medicine resident is required to complete a research project but isn't sure where to start. Her program director
suggests that she start by trying to identify a faculty member to serve as her research project supervisor. Although that
seems reasonable, she isn't sure what she should be looking for in a potential faculty supervisor.
11 You will read about designing, irnporranr part of your training and professional develop-
conducring and interprering research in orher chapters of ment rhat will enhance your understanding of the health
this guide. Before any of that can begin, however, you will sciences and of the practices you apply every day. Perhaps, if
need ro find a research supervisor. In this chapter, we will the years of clinical experience ahead of you generare new
ourline various aspecrs of supervisión for you to consider. insights and research questioris, you will find yourself con-
Specifically, we will highlighr the benefits of selecring a tinuing ro include research as a rewarding dimensión of
supervisor who can provide access ro a research team, in your professional career (see ch. 32).
corirrast to basing thar selecrion solely on what a prospec- Before you can seek out a suitable supervisor, you musr
rive supervisor can offer individually. 1his is irnportant first evaluare your personal priorities and goals. Is ir your
beca use research in rhe real world is rarely an individual prioriry ro advance your contenr knowledge in a particular
pursuir. area (e.g., prevention of Chiamydia infecrion in reenaged
snowboarders)? If so, finding a supervisor with experrise in
Choosing your topic: What comes

first-the project or the process? CHAPTER OBJECTIVES
1he research intereses of many healrh practitioners and After reading this chapter, you should be able to:
rrainees are inspired by somerhing rhey have seen in prac- • describe the issues to consider in the selection of a
rice or have experienced. For others, the research compo- research supervisor
nenr of professional training is merely a hurdle rhey rnust • discuss the qualities to look for in a research supervisor
clear ro meer program requirements. Regardless of which • describe the concept of a team approach to research and
case applies ro you, gaining experience in research is an research supervision
KEY TERMS
Authorship Informallearning Research environment
Collaborative Interdisciplinary Supervisory team
Communication Norms Team ski lis
Conflict resolution Priorities
Expectations RACI management tool

that area will be the rnost imporrant aspecr of your search.

Selecting supervision and teaming up
Or is rhe research process more imporrant ro you, such that,
for success
.- regardless of the topic, you are seeking a supervisor who will

provide you wirh an environrnent for superior research
Finding a supervisor with skills, interests, resources ar
trairs rhat are matched ro your learning needs, expectatioi
training? Ideally, you will find a supervisor who can support and priorities is the key ro success. A number of conside
a research project in an area that you are keenly interested in ations should be taken into a accounr in the selection of
and offer an effective environrnenr for training in research supervisor: we highlighr a few essenrial facrors here.
processes and rnerhods. You may, however, have ro choose First, the purpose and responsibiliry of a supervisor is t
berween the rwo. provide guidance and support ro a research trainee as his e
Let's say you choose ro pursue the ideal training environ- her work progresses. Thus, finding a supervisor who wi
ment alrernative. Firringly, the first step is ro research your actually be available and able ro perform this oversight OVé
options. Scan your departrnent and your wider research rhe course ofyour projecr is fundamental.
and healrh care nerwork ro find out which professors or Second, the supervisor needs ro have skills and experi
research groups have a reputarion for good supervisión and ence in your research area, Ideally, rhar experience \ViIItak
training as well as for producing high-qualiry research. For many forms: knowledge of the content arcas and of researcl
exarnple, suppose you discover rhat a group of renowned merhodology and process, a hisrory of conducring research
clinician-scientisrs wirh full professorships are working and a rrack record of research success. Your supervisor's jol
with a diverse ream of researchers ro examine the risks of is ro ensure thar your work is val id, is bascd on sound idea:
shorter posrpartum srays on rhe marerniry ward, but you and merhodology, and advances roward rhe idenrifiec
have limired knowledge of and have never been particular- research goal. Also, the ideal supervisor will have experience
Iy interested in, materniry careo Because rhe training envi- in supervising trainees, preferably across a broad spectrum
ronment is your prioriry, you pursue this opportunity in of levels from undergraduate students ro posrdocroral fel-
view of rhe benefirs of working wirh rhis expert tearn and lows.
the excepcional research skills and experience rhat you will In addition ro availabiliry and cxperience, your supervi-
gain in rhis environrnent. Of course, this productive and sor should have access ro resources ro suppon your work.
successful team of researchers will be soughr afrer by many These resources can also come in many forms: laborarory or
trainees, and so you will have to convince thern that you are office space, research rools or equipment, access ro databas-
a worrhy candidare. 111is is where your preparation and es, or access ro porenrial srudy parricipants or recruirment
negoriaring skills come inro play: Be sure ro update your possibilities. However, among rhe most valuable of rhese
résumé and curriculum vitae before beginning rhe search resources are nerworks and teams of skilled expens who can
for a supervisor, and be prepared ro write a cover letter (or assist with and facilitare your research projecr.
equivalenr émail) ro make your requesr for supervision and In our opinion, the ideal supervisor would provide access
ro highlighr the skills and strengths you can bring ro the ro a tearn who can add ro rhe supports your supervisor can
team. offer as an individual: thar is, addirional guidance, exper-
\'>V'hetheryour approach is process-focused or content- tise, training in specific research techniques and facilitation
focused, developing an excellent research quesrion should of your projecr's goals. Alrhough individual tearn mernbers
be of paramount prioriry. \'V'herher you are keenly inrer- willlikely be interested only in certain aspecrs of a projecr
ested in the ropic or nor, a good research experience will (e.g., rhe biosraristical rnerhods) or rhe applicarion of resulrs
be facilirared by a thoughrful, well-formed research ques- (e.g., effecrive srraregies for Chlamydia prevention), the
tion. If your goal is to learn how to perform research, to team approach harnesses all of these interests ro enhance
benefit from rhe guidance of your supervisor, and ro pro- your experience and nerwork of supporr. Thus, a collabor-
duce a publicaticn, a well-formed research question is ative group with a range of rechnical and adrninistrarive
essential (see ch. 6). Make sure rhat you review your skills can be extrernely beneficial ro your research experi-
quesrion wirh your supervisor and your supervisory rearn ence. Research rearns come in many shapes. You may be
ro ensure that you are 011 track ro produce inreresring, working in a tearn comprising other rrainees, research assis-
worthwhile resulrs rhar will add ro the knowledgc base of tants or sraff. Alrernarively, you may hnd yourself working
your field. in a team rhat includes a few mid-Ievel and senior faculry
16 © 2011 The Royal College 01 Physicians and Surgeons 01 Canada

3 Finding a research supervisor
members. Whatever the rearn's cornposition,

to function effectively within ir, taking advantage of whar
you will need inter onnecrion with orher topic areas? Finding out
how long he or she has been active in the area and
,,
can be learned from rhe differem members and undersrand- wherher he or she teaches courses on rhe topic should
~
ing the unique contributions of each. 1his also means that
you will need to develop rhe skills w work producrively O
give you a good sen se of this.
Methodological expertise. Make sure rhat the super-
,••
wirhin a reamo If ir seems difficulr at rimes ro work collab- visor you have in mind is well versed in the approach- •
•
orarively, rake cornforr in knowing rhat rhe team skills you es rhat you will need ro use for your project, ranging
acquire nowwill beneíit you in your clinical pracrice, which from laborarory techniques ro biostatistics,
undoubtedly requires a rearn approach. 1he notion of a O Management experience, Does rhe supervisor you
supervisory team is discussed at grearer lengrh later in the have in mind have a good crack record in organizing
chapter, but keep ir in mind as you read on. and managing people, resources, data, space and
An irnportant rrait worth being acutely aware of as you multiple responsibilities?
seek a research supervisor is artenrion ro deadlines. A super- O Enriching environment. Wil! your prospective
visor who is mindful of your tirneline will ensure thar you supervisor be able w enhance your research experience
sray on track wirh your desired projecr milesrones and wil! rhrough access ro educational workshops and confer-
be receptive ro queries and unexpected obsracles thar ences, or key national and internacional experrs?
require rroubleshooting. A grear supervisor will ensure that O Matching expectations, Do your expecrations for
all members of the tearn are productive and rhat they have your research project match those of the supervisor
opportunities not only ro use their current skills, but also to you have in mind? To broach this question, when you
develop new ones. Further, a good supervisor wil! ensure meet with a potential supervisor, come prepared with
rhat the rearn has the capaciry ro resolve problems as they a list of your expecrarions and offer ir as something
arise, and thar they feel empowered ro do so. Lastly, and that you can work on together. 1his gesrure should
arguably most irnportantly, the ideal supervisor will have open up the discussion nicely. Before your meeting,
enrhusiasrn for the topic and the research process as well as be sure ro discuss your expectations wirh respected
an appreciarion for rhe tearn approach ro research. Clearly, colleagues, especially those with research experience,
finding a good research supervisor requires a fair bit of to ensure rhat they are realistic and appropriate.
research on irs own. O Funding track record. Can you r porenrial su pervisor
In summary, think abour what research supervision provide rhe resources ro carry out rhe proposed
means ro you and where you expect ro need rhe most sup- project? Biographies posed 011 your departrnenr's
porr. 1he following checklist outlines sorne considerations websi te can give you a sense of whar faculry or staff
rhat might help you ro find the best person possible to can offer. You may hnd curriculum vitae and infor-
supervise your research project; rhese apply w borh supervi- mation on their collaborations, resources and recenr
sors and supervisory teams. grants.
O Publication track record and scientific writing
o Other students and trainees. Talk with orher skills. Is your porential supervisor well published?
research studenrs or graduare srudenrs who have Search lirerature databases such as PubMed, or [he
rrained with rhe supervisor you are considering. 1hey person's own websi te, if available, for his or her
are the besr source of information on what your publication record. Is he or she able and willing ro
experience rnight be like. coach you rhrough the writing and submission
O Research process experience, Does rhe supervisor process, and knowledgeable about possible journals
you have in mind undersrand the challenges one faces rhat might be inrerested in publishing your research?
in research, and can he or she offer practica] tips for
rroubleshooting problems that arise? Currenr and You might think of other items of particular concern w you
former studenrs and trainees should be able to ro add to rhis list.
provide insight here.
O Content expertise. Does your potential supervisor
know the topic area well and its application to and
© 2011 The Royal College 01 Pl1ysicians and Surgeons 01 Canada 17

Milestones for management, to discuss progress, de!iverables and timelines? Or do they

and establishing norms prefer a more informal working sryle in which you fee! you
For the trainee, a first research project is, among other can "pop in" to ask a quick question and provide "side bar"
things, an exercise in independent adult learning, project progress updates?
management and time management. Establishing Authorship is another irnportant expectarion to discuss.
norms-standard, agreed-upon rules for the conduct of Before the project commences, identify who will be authors I
rhe research team-before the work begins can help to on any publications arising from your project, in addition I
ensure that the project runs srnoothly. to any other produces rhat might flow from the work. If you
You may find that the approach to the research cornpo- are unsure of what is appropriate, seek out sources rhar
nenr of your training will be different from approaches you define authorship, such the Internacional Committee of
have grown accustomed ro in your training so faro Your Medical Journal Editors criteria or orher standards corn-
research experience will be gready enhanced if you quickly monly accepted at biomedical journals." Also, decide col-
beco me comfortable with rhe idea that you, for the most lectively rhe order of aurhorship on any publications and
part, will be directing the course and pace of your project. ensure that everyone underscands and is comfortable with
Acquiring a certain savvy in project management and time the plan. For example, research supervisors are ofren listed
management will he!p you stay on track (see ch. 23). Set last (but sometirnes second), indicating thar rhey are the
timelines and miles tones to keep you on course. most senior invescigaror. Makc che topic of aurhorship a
Before the project begins, certain norms should be dis- discussion point with colleagues and rnenrors ro learn how
cussed and agreed on by you, your supervisor and your they have approached this topic and applied aurhorship cri-
team. You may find that you have to be the one to bring up teria in the pasto Be sure to rework the plan if roles change.
the topic of establishing norms, and you should fee! Research projects are dynamic, and it is highly likely rhat
empowered to do so. Before you do, though, consider what you will end up being responsible for tasks that you and
processes and srandards ofbehaviour you are most cornfort- your tearn never anticipated.
able with, what you think is fair, and what you need, con- Ir is also irnportant ro address expectations about how
sidering your skill set. Although it may fed awkward the work will get done, by whom, and within what time
inirially, you will be thankful that you initiated the norms frame. Be aware of what you would like, and what your
conversarion at rhe outset of the project: Ir will save you supervisor and tearn can actually provide, so that expecca-
sorne headaches (and potenrial conílicrs) down the road. tions are aligned. Be prepared: by providing thoughtful
Also, by doing so, you will dernonstrate your conscientious- background material, clear ourlines, and the Hexibiliry to
ness, maturiry and foresight. Sorne of the rnost important accommodate team mernber's schedules, you can enhance
norms ro discuss include communication, aurhorship, your chances of success. For example, get a time cornmit-
expectations and conilict resolution, although you and ment on de!iverables from your supervisor. A promise such
your team may come up with more. as "1 will read over your draft of the protocol so you can puc
Establishing norms for communication is important. ir in for ethics review" is meaningless without a date.
Knowing how, in what manner and how frequendy (weekly Encourage follow-through on promises wirh patience, per-
or daily) to communicate is essential to moving your proj- sistence and smiles: Negativiry won't get you anywhere. Ser
ect forward. Does your supervisor prefer face-ro-face, an example by meeting rhe timelines you proposed and
phone or email communication? How casual or assertive a being accountable to your commitments. This also means
communicarion sryle is everyone happy wirh? How long is ensuring that your proposed timelines are realistic. When
an appropriate wait time for a reply? Do the team members rirnelines aren't being met, it will be he!pful if you have
work over rhe weekend and into rhe evenings and expect established a working culture within rhe tea m in which
cornmunication about the project ro extend into rhis time checking in and following up are seen as appropriate and,
(it may be the best time to get their attention, especially if indeed, an essential e!ement of accountability. Of course,
they are busy during "regular" hours). Explicitly ask your this is easier to say than do when rhe recidivist is the dean of
supervisor and tea m members what the best methcds of medicine, but the principie is important to keep in mind.
comrnunication are for each of rhern. Do they prefer a
strucrured process involving regulady scheduled meetings
a For the ICMJEcriteria, see www.icmje.org/ethical_lauthor.html

18
Although the trickiest norms to talk abo U( are likely seek me cornrnirrnenr of an "ofhcial" supervisor, but you
those concerning confllct resolution, they are arguably should also be open ro supervision, guidance and mentor-
,.
~
--
(pun intended) rhe mosr irnporranr. You might not be able ship rom orhers.
Q
to predict where coníiict will arise, but you can make sure
thar you have a plan for dealing wirh ir when ir does, For
These days, resident research projects are rarely carried
OL![by one trainee at a bench or computer with suPPOrt
,.
exarnple, berween you, your supervisor, and your ream you
mighr agree that if a disagreemenr arises rhe parties involved
from a single supervisor in an adjacent office. Ideally, you
\ViII find a supervisor who works with a variery of expertS,
ro whom you will have access and who may become pan of
-
will nrst try to son it out (privately) berween themselves. If
rhar rurns out to be unsuccessful, a mediator will be a supervisory team focused, at least in pan, on you and
sought-ideally, someone who is not tethered to the your project. Other people mighr link into )'our project
project or source of confiicr. Many studenr services offices because of a keen interest in rhe topic area or rhe panicular
and deparrmenrs have information on where to nnd a merhodological approach )'ou plan ro use. Embrace and
mediator. If conílict does arise, address ir eady; orherwise, seek out rhese team opportunities in your research experi-
it could snowball, leaving you in a bit of a predicament. ence. You and your research will certainly benefir from
This brings us to the importance of identifying a super- doingso.
visor or tearn of supervisors rhar you feel comfortable with As with many of rhe skilis and technigues acquired dur-
and with whom you believe you \ViIIhave a pleasant working a health sciences training program, prohciency in the
ing relationship. Does rhe chernistry feel right? If not, you research process reguires specific training. With effective
may need to look funher. The selection is irnportanr. Nor training and experience you can acquire the necessary set of
only will you learn better in a positive environmenr where critical thinking skills, analytical approaches and perspec-
you feel respected and induded, bur you may find that tives necessary to conduct high-qualiry research. Good
supervision around a project cvolves inro irnportant profes- supervision can facilitare the developmenr of these skills,
sional and career mentorship. For all of these reasons, ir is but rhe successful hearh research trainee wil! be open ro
irnporranr to choose well (see also ch. 5). obtaining rhem from a variery of people and in a range of
Alter the norms conversaticn, wrire up what you under- settings.
stand to be the decisions and share thern with your supervi- In health research, where rhe question under investiga-
sor and orhers who will be involved in your projecr. 5eek tion is complex and multi-lacered, invesrigarors ofren col-
confirrnarion that the norms document represents what laborare across disciplines. For exarnple, a research tearn
they understood to be rhe agreement; if changes and addi- interested in prevenring falls among seniors rhrough the
tions are needed, rhis is a great time to make rhern. The examination and resring of various exercise rraining pro-
norms document will hopefully serve as a compass for your grams could indude collaborarors with experrise in kinesi-
research project that you can refer back to in the event of a ology, medicine, physical rherapy, epidemiology,
misundersranding. biostaristics, healrh economics and program evaluarion. The
roles of the rearn members might range from lead invesriga-
tors, to masters, doctoral and posrdoctoral srudents, research
Parallels between health care practice
assistants, data adrninistrarors, and )'0 u! This interdisci-
and the research environment
plinary approach provides an ideal wa)' ro learn about alter-
As in healrh care practice, achieving high-qualiry results in narive approaches relevant to your research topic. And,
research requires a tearn effort, and thus you shouldnr rely because most new researchers have a lirnired research back-
solely on your supervisor. Research is more of a team enter- ground and experience working in an interdisciplinary ser-
prise than sorne trainees realize, and supervision and sup- ting, you have rhe rnost ro learn, So, treat every interacrion,
port often come from many people on many differem wherher wirh an expert invesrigaror in biochernistry or a
levels. As a health care practirioner, you are accustorned ro research assistant charged with data collection, as a valuable
working in care tearns rhat might indude physicians, phar- leaming opportuniry,
macists, nurses and physiorherapisrs. Many of the tearn- In addirion, when selecting a supervisor you will need to
work ski lis you have developed in dinical settings will be seek a champion who is willing ro work wirh you and ro be
helpful in the research setting. Of COlme, you will need to an advocare for your project. Many supervisors have a
© 2011 The Royal College 01 Physicians and Surgeons 01 Cana da 19

The Research Guide: A primer for residents, other health (are trainees, and practitioners
number of trainees that rhey are working with and many environrnents for informal learning and orher potencial
orher diverse responsibilities, so they might not devore as opportunities to engage with other researchers and research
.- much time and energy as they would like ro promoting

your work, alrhough they are srill proficient in rhe supervi-
trainees: they are among your most valuable classroorns.
Make an effort to develop carnaraderie wirh colleagues and
sory role. Consequendy, your "charnpion" might turn out ro foster peer-Iearning opporruniries. A discussion of how
ro be your program direcror, a research mentor, someone you and your colleagues might nerwork and share compe-
with content experrise, someone who can provide practical tencies stemming frorn previous work or research experi-
support wirh ethics applications and approvals, or someone en ce is beyond the scope of this chapter; see chapter 4 for a
who can assist with participant recruitrnent. discussion of research nerworks and their benefits.
Being the small fish in a big and Practical tips for managing your
new pond supervision
Ir is irnporrant ro realize rhat much of your research train- Once you have a grasp of your topic area and your
ing will take place in informal and unstructured settings: research question, have secured a comrnirmenr from your
this is where you willlearn the rnost. You will not be called research supervisor, and have idenrified )'our supervisory
inro your supervisor's affice or required to sir through long team, rake sorne rime ro consider a little "rearn thcory."
lectures on how ro calibra te the data acquisition systern ro Yes, theoretical and pracrical research lirerarure provides
record beat-by-beat blood pressure, or how ro systernatical- an evidence-informed approach ro research and che
Iy wri te rhe discussion section of your paper. research team process! The lirerature on tea m dynamics
Ir is likely that your supervisor will facilitate group learn- and organizational psychology dernonsrrates rhe irnpor-
ing activities, such as attending presenrarions by experts tance of identi~'ing a "compelling goal" for the tea m as
and journal club meetings, A journal club meeting, in well as clear roles and responsibilities for all involved in
which a relevant papel" is discussed in detall wirh particular the projecr. I To get a project done, rhose involved need ro
focus on the merhods, can be a fmtastic setting for Iearn- believe that ir is interesting and importanr and willlead ro
ing. The coritent might be dircctly relevant ro rhe work that rhe achievement of a particular goal or aspiration. As the
yOLlneed ro do, such as estimating the sample size needed one with the vested interest in and resporisibiliry for rour
for a certain study. Even topics thar, on the surface, have no project, you need ro instill in your tearn members a sense
relevance ro your current project can spark new ideas for of rhe irnportance of the project and its rnulriple benefirs
refining your research question or analysing your data. for all involved. Ir can help ro poim out the direcr benefits
Especially valuable learning venues are research rounds or of rhe project ro your team and supervisor. However, ir is
meetings where ream members present data for critica] unlikely that you will have any authoriry over tea m mern-
analysis and consrructive criticismo These offer a great bers and rhe delivery of rheir componems of the projecr.
opportuniry ro develop critical rhinking skills and ro prac- This is why esrablishing explicit roles and resporisibiliries
tise rhe evaluation of rescarch resul ts: skills that will certain- is irnportanr.
1)' come in handy when ir is rime for you ro evaluare your Successful research projecrs take rearnwork, and rearn-
own srudy and irs results, or when you are reviewing pub- work requires coordinarion. Ir will be your responsibiliry ro
lishecl research ro inform your clinical practice. orchesrrare this coordination for your research projecr. One
Your supervisor mighr offer a number of informal useful guide from the corporate world is the RACI rnan-
learning opportuniries, such as ruroring you on preparing agement tool, which can be used ro idemify, and ro pre-
a research protocol for an erhics applicarion during a lunch vent conFusion about, roles and responsibiliries during me
break ar a research conlerence. Orher informal learning course of a project. 2 Where there is no confusion abour
opportunities mighr arise when )'OU are given a clesk in a roles and responsibilities, conliict is less likely. The elernenrs
room wirh orher trainees, where you mighr fincl rhat con- rhat make up the acronym are:
versations on dara interpreration sprour up from time ro
time, providing an opporruniry ro passively and actively • Responsible. The person who has responsibiliry fOí
gain insights applicable ro your own work. Seek out rhese getting rhe work done or a decision rnade. Ideally,
20 © 20 i 1 The Royal College 01 Physicians and Surgeons oí Cenada

rhis responsibility is assigned ro one person (e.g., the This oes nor counr rhe cost of the investigators' salaries,
research trainee), which are paid rhrough other sources, or rhe value of "in
• Accountable. The person who is accountable for rhe kind" equipmenr and infrasrrucrure cosrs that suppon
correct and rhorough complerion of the task. 1his many studies. 1ajor rnulri-cenrre drug studies powered to
muse be one person (e.g., research supervisor). 1he obrain sraristically significant results for irnporrant clinical
"responsible" person is accountable to and has his or outcornes may require budgets of over $1 billion. 1hese
her work approved by rhe person in this role. projecrs ofren have a rnulti-million dollar budget just for
• Consulted. 1he people who provide information their indusrry sponsors ca purchase reprines to disrribute ro
for the project are often subject experts (e.g., other physicians as pan of the post-study publicity!
research project co-supervisors/co-investigators). Here, Knowing about the funding structures of a research pro-
there is two-way communication. gram is imporranr (see also ch. 19). 1his informarion will
• Informed. 1hese are the people who are affected by help you appreciate rhe resources that your supervisor dedi-
rhe ourcorne and need to be kept informed about rhe cares to you and your project. Further, ir will give you per-
progress of the project. Here, rhe communication is specrive on the demands placed on your supervisor and
one-way, supervisory tearn to continue their research work as well as
the cornpetitive nature of the research environment in the
1hus, applying che RACr framework, you willlikely be pursuit of funding.
"responsible" for the entire exercise, your supervisor will
likely be "accountable" for the project, and rhe rest of che Research is hard work-as with most
supervisory team will provide "consultation." Your pro- things, proficiency takes time and
gram adrninistrarors will need ca be kept "informed" of practice!
your progress and cornpletion of the project. You will also
have to "iníorrn" your friends thar you can't go out because Our expectations invariably colour our experiences. Prorn
you are deeply engaged with the literature (or, more likely, the outside, research may seem like somerhing you can eas-
are stuck trying to figure our how to get your referencing ily rake in stride. 1he fact thar you have made ir rhis far in a
software to work because no one else on the tearn knows professional health sciences training program means that
how to use it). you have had many positive experiences. As a healrh trainee
1he RACr tool will beco me irnportant when you need or practitioner, you may receive positive feedback many
ca rnake decisions relevant to your research. You will ofren times a day. However, be warned: the world of research has
get conflicring opinions when you ask differene people the many dauming aspects thar may come as an unwelcome
same question (e.g., should you use rhe full Mini-Mental surprise. 1hree specific aspects of research should be con-
State Exam, or rhe abbreviared version, as a measure of cog- sidered as you contemplare the hard work that goes into
nirive impairment in your study?). Wirh RACr you can research: grams, publications and rime frame. Firsr, even
clarify rhe roles of each team member: so me have a consul- the most successful researchers fail in half (or more) of rheir
rative role, while orhers have more leverage in the decisions attempts to obtain comperirive grane funding. For exarn-
that will be made. Always make sure rhar you, your supervi- ple, in 2010-2011, the Canadian Institures for Health
sor, and your tearn members are clear about their roles from Research funded only 23% oE rhe applications received in
the outset of your work together, befare any decisions need rhe open operating grant cornpetirion." Gram applications
to be made. take a lot of rime to pur together and, given their low suc-
cess rate, a great deal of rime and effon can be invested in
funding proposals before a research project is up and roll-
Putting your research into context
ing. Second, it's nor easy to get published. Journals ofren
Ir is irnportant to know that rhe research projecrs reponed reject papers outright, or require many revisions even for
in majar medical journals, or even rhe mid-level journals, preliminary acceprance. It should be noted that researchers
ofren have budgets that exceed $1 million. 1he average who never have their papers rejected are not sending thern
5-year operaring grant awarded by crHR (the Canadian
Insritutes of Healrh Research) exceeds $750,000 in total.
b Seevvwwcihr-irsc.gc.ca/e/42857 .htrnl

ro sufficiently comperitive journals. Third, ar rhe senior lev- sorne clinicians working in research have not eaughr on r(
els, a major srudy such as a cohorr srudy or a randomized rhe imporrance of rearnwork, especially when addressiru
conrrol!ed erial ofren rakes five or more years ro complete.
·- Thus, healrh rrainees who enrer inro a relarionship wirh a
cornplex
approaches.
healrh problems
Consequenrly,
thar require interdisciplinar-
their researeh endeavours wil
research supervisor musr be prepared for a differenr pace be limired in scope and application.
and a delayed sense of grarificarion compared wirh clinical
work. Even a healrh trainee's research project wil! take
Conclusion
rnonrhs rarher rhan weeks ro complete. An excellenr draft
manuscripr will srill need dozens of correcrions (which can So rhat brings us back ro where we began: You need helpl
be considered negarive feedback by those who are rarely Qualiry supervision, from an individual or team, is a key
corrected) and may receive no specific plaudirs. factor for success. You are unlikely to perform well in
In short, research is hard. Bur our warnings are offered research as a soloisr. However, we fear that rhe widely
wirh the best ofinrenrions ro improve your research experi- accepred term, supervisor, emphasizes the wrong elemenrs
ence. Sorne senior healrh pracririoners find rhemselves of the relationship by srressing "oversight" aboye assistance,
looking fOI "quick wins" and "short-cuts" ro research suc- suppOrt, guidance, growth, col!aborarion and teamwork.
cess wirhout rruly raking rime ro Íearn rhe skills that are We hope rhar we have provided sorne insighr into the
required. In Outliers: 7he Story ofSuccess, Malcolm Gladwell wonderful world of research, The airn of rhis chaprer was ro
highlighrs research showing rhar excellence across a range of al!ow you to consider rhe importanee of research supervi-
activiries such as music, sport and academies required sion and ro encourage you ro consider your goals for your
10000 hours of rraining.31his was more irnportant rhan a project and researeh experience. Further, we wanred to
myriad oHacrors such as generic predisposirion and child- al!ow you ro place your research projecr in conrexr, rhus
hood environmenr. Inrerestingly, this rime requiremenr can making realisrie choices abour how you wil! pursue your
be applied ro research as well. And, clearly, you don'r have research projcer. Good luckl •
that many hours for your inirial research project! Furrher,
CASE POSTSCRIPT
Using what she learned from this chapter and the advice of others, the resident consulted the department website and
senior residents to determine which faculty members might be potentially suitable supervisors. After meeting with four
promising candidate supervisors, she chose one of them based on his research interests, methodological expertise and
success in getting research grants and in being published. Further, feedback from other trainees who had worked with
him on other occasions confirmed her impression that his working and interpersonal style would suit hers. Not only was
her research project a success, but after her residency program she continued to col/aborate with her research supervisor
on several successful projects. Now, as a new faculty member, she has the opportunity to serve on a supervisory team for
residents as they conduct their first research projects.
REFERENCES
1. Hackman JR. Leading teams. setting the stage for great performances. Boston (MA): Harvard Business School Press; 2002.
2. Value Based Management.net. RACI model. Available at: wvvw.valuebasedmanagement.net/methods_raci.html
3. Gladvvell M. Out/iers. the story of success. London & Nevv York: A/len Lane; 2008.
22 © 20 11 The Royal College of Physicians and Surqeons o' cc;-~ 4

~
~
•
EXERCISES
1. Consider
•.
these questions
What are your priorities for your research project: pursuing a research topie that you are really interested in, or
--
,..
finding a superior research environment
future research endeavours?
where you can gain valuable skills and experience that you can apply to
-
•••
• Do you have good knowledge of the work of the research groups within your program, department, faculty and
university? Do you have a concept of potential supervisor's approach to research and research supervision 7 Do
you have a network of people who might be able to give you insight into how these groups function and into
where, considering your priorities, you might best fit in 7
8 Do you have the necessary ski lis to work collaboratively in a research team? Should you seek training to improve
your collaborative skills?
2. Seek out one or more of the following through the Faculty of Graduate Studies or Student Development Centre at your
university:
• graduate student supervision guidelines
• codes of behaviour for trainees and research supervisors
• technical/scientific writing courses
© 2011 The Royal Coliege 01 Physicians and Surgeons 01 Canada 23

SUMMARY CHECKUST
o
.- Consider personal priorities: Is it your goal to acquire content knowledge
and methodology?
in a particular topic area, or in research ski lis
O Do some research: Scan your department for a supervisor who fits with your priorities.
O Ramp up your résumé, curriculum vitae and negotiating skills, as they may be required in your request for supervision.
O Formulate a manageable and interesting research question.
O Match your learning needs, expectations and priorities with the skills, interests, resources and traits of your supervisor.
O Look for a supervisor or supervisory team with content knowledge, experience in specific methodological approaches
and the research process, a history of conducting research, a history of successfully supervising students, and a track
record of research success.
O Consider your potential supervisor's access to resources: laboratories, office space, research tools or equipment, data-
bases, and study participants/recruitment options.
O Most importantly, try to seek supervision where you will have access to a diverse team of skilled experts who are inter-
ested and able to help you!
O Find a supervisor/supervisory team who pays attention to deadlines and timelines.
O Get comfortable with self-managed, self-directed learning: time management and project management skills are key
to successful research projects.
O Establish norms with regard to communication, authorship, expectations and conflict resolution.
O Look for the parallels in health care practice and the research process.
O Seek a champion for your project-someone who is willing to put in the time to help it along and to advocate for its
importance. This could be almost anyone-from a supervisor or a team member to a manager or fellow trainee.
O Be open to dynamic learning at every opportunity and from a variety of people.
O Use the RACI tool to manage supervision and your research project.
O Take it ail in stride and put your research into context. Don't expect to cure cancer overnight and recognize the con-
straints of time and funding common to research.
O Remember that failure is part of the research process. You are treading on new territory: embrace setbacks as part of
the experience.
O Have fun:
24 © 2011 The Royal College oí Physicians and Surgeons 01 Canada

4
,-.
Research teams and networks
---
Terry P. Klassen, MD, MSc, FRCPC
ILLUSTRATIVE CASE
In anticipation of the Fellows Research Day at the annual meeting of the Pediatric Emergency Research of Canada (PERC)
research network, an Emergency Medicine resident is encouraged to present a research idea he has been developing
about how the Ottawa Ankle Rules might perform in the context of pediatric careo Although he is rather daunted by
the prospect of presenting to a large group of colleagues and research experts an idea that is still at a preliminary stage,
he decides to prepare a presentation. He also wonders about the value of joining this research network as a trainee.
Specifically, he is unsure whether he is sufficiently qualified to join and whether the required investment of time and effort
will pay off.
• This chapter will examine the role chances rhat research findings will be applied to health care
of research teams and nerworks in fostering and promoting practice and decision-making.
research and illustrate how they can provide a fertile envi- This chapter will explore (bese beneíits by means of a
ronrnent in which trainees and other novice health concrete example, describing how che Pediatric Emergency
researchers can develop knowledge and skills. The last Research of Canada (PERC) nerwork has advanced che
decade has wimessed a blossoming of research nerworks knowledge base for pediatric emergency medicine and has
and teams, enabled in part by national and provincial had a positive impact on healrh outcornes among children
research bodies thar direct funding specifically ro tearn- or in need of emergency careo
nerwork-based research.!:? One great advantage of the
ceam-based approacb is rhe capaciry ir affords ro recruit
CHAPTER OBJECTIVES
large sample groups over relatively short periods of time,
chus achieving the statistical power necessary ro confident- After reading this chapter, you should be able to
Iy address irnportant issues faced by parienrs and heaith • describe research networks and teams and their value
care providers. Sorne of rhese issues are sufuciendy corn- in today's research world, using Pediatric Emergency
plex rhar only a research tea m or nerwork will have at irs Medicine as an example;
disposal the array of scientific disciplines, perspectives and • describe a research network and how it functions,
approaches rhar are needed ro solve che problem. More- outlining the benefits and challenges it presents; and
over, firrn connections berween research nerworks and • discuss how health care trainees and others new to
health care environmems can provide an ideal ccnrext for health research can benefit from working in such an
knowledge translation (see ch. 30), maximizing rhe environment.
-- __
. . -----------~
KEY TERMS
Iterative loop Research environment
Knowledge translation Research training
tearch disciplines

The PERC network PERC's work in croup serves ro illustrate how this
In Canadian emergency deparrrnents in me mid 1990s, there work. Croup is a relarively common respiratory diseas
was little clarity with regard ro derermining which children children, wirh a peak incidence in 1 ro 2 year 01ds.9 •
·- who presenred wirh a minor head injury involving loss of con- known rhat mosr cases are caused by parainfluenza virus
sciousness should receive a compured romography (Cf) scan thar its incidence peaks in alternare years. The main par
of rhe head. Concerned clinicians and researchers from pediat- physiological abnormality is subglorric edema and narn
ric emergency departments across me country gathered ro dis- ing of the trachea'? (steps 1 and 2 in Fig 4.1). Debate ra:
cuss approaches ro this problem. From mis efforr arose PERC, in rhe 1970s as ro che effecriveness of glucocorricoids
whose Iirst srudy demonsrrared that mere was subsrancial vari- reducing this swelling and achieving a clinical benerit
ability-as great as four-fold-across nine pediarric emergen- mera-analysis by Kairys and colleagues'? in 1989 dern:
cy deparrrnents in me frequency with which CT scans were srrated that, for croup patients admirred ro hospital, tre
ordered for children with the same type of minor head injury? ment with glucocorricoids would result in a signific
Led by Dr. Martin Osmond, and with funding from me decrease in hospital admissions and reduce the probabil
Canadian Institutes ofHealrh Research (CIHR), members of of intubation (step 3). This then spurred research in em
rhe PERC Head Injury 5rudy Group and others developed a gency deparrments ro examine rhe effecriveness of glucoo
decision rule ro guide clinicians that was eventually published ticoids in this contexto A series of randomized conrroll
in 2010.6 trials (RCTs) demonsrrated a decrease in hospiral adrnissii
The theoretical model for this research prograrn is based rates and more rapid clinical improvement wirh glucocor
on the "measurernenr iterative loop" described by Tugwell coid therapy" (step 3). A study of the "real world" effecr
and colleagues in 1985/ which provides a framework for glucoconicoids demonsrrared a significanr decrease in he
assembling, acring upon and evaluaring health inlormation pi tal admissions for croup over a 14-year period, which w
ro reduce the burden of illness.t" The logical sequence for coincident wi rh the publication of the evidence of the effe
research described by the loop includes the following sreps: riveness of glucocorticoids Ior croup'? (step 4). An econom
evaluarion, included as pan of an RC1~ dernonstrated rh:
• quantifying rhe burden of illness cases rreated wirh placebo cosr $93, as compared with $7
• identif)ring causarive facrors for those treated with dexamethasone, and had berter ou
• idenrifying gaps in evidence rhrough sysremaric comes with respecr ro rhe resolurion of croup symptoms an
reviews and evaluaring rhe efficacy of treatrnent lower rares of hospi ral admission 13 (step 5). Research is cu
oprions rhrough randomized controlled trials renrly under way ro examine the Imowledge translarion po
• derermining rreatrnent effecriveness in the field rion of rhe 100pl4 (steps 6-10).
• evaluating the cost-eflectiveness of inrerventions To succeed as a nerwork, PERC has had ro bring rogerl
• idenrif)ring barriers ro transfer and uptake of er collaborarors from a variery of disciplines. Ir has als
informarion by end users focused on having researchers ar al] levels participare, fror
• conducting systernatic reviews ro identity oprimal ways trainees all rhe way ro very senior, accornplished researcl
for knowledge rranslation ro reduce the gap berween ers. To be successful in the clinical research loop, ir hs
clinical pracrice and research engaged clinicians, statisticians, epidemiologisrs and healr
• disseminaring research findings from clinical and econornists. To be successful in knowledge translation,
knowledge rranslation research has formed a srrong parmership wirh leaders in this fielc
• allowing for the natura] diffusion of results in the "real ensuring that rhe tearn includes qualitative researchers, co§
world" nirive psychologists and medical sociologisrs. In our cluste
• assessing "real world" ourcomes RCT examining three methods for ensuring how crou
guidelines are besr incorporated into clinical pracrice, ir wa
The PERC ream mcdified the original iterative loop into noriced that responses varied among the various healrh car
a double loop, highlighring rwo areas rhar we believe ro be providers in the different hospirals across Alberra wir
crirical ro rhe optimization of healrh outcornes (Fig. 4.1): respecr ro adopting rhe evidence on rreatrnent for crou
clinical research (evaluating therapeutic interventions), and (unpublished data). A qualitarive researcher, as pan of rh
knowledge rranslarion (implementarion and uprake of team, conducred focus groups ro elucidare rhe underlyin
proven interventions). reasons for the variarion in response.
26 © 2011 The Royal College 01 Physicians and S rgeons o: C2'a:

4 Research teams and networks
Figure 4.1: A paradigm for research: the iterativ€ figure-eight. KT = nox .edge translation: SRs = systematic
reviews; RCTs= randomized controlled trials. Reproduced from Hartling et aL< by permission of John Wiley and Sonso
6
BARRIERS BURDE OF ILLNESS
.: Assess barriers to knowledge Determine health status using ~ \..
r transler/uptake health status indieators ~
7
EFFECTIVENESS
Identily gaps in KT evidenee
~
I AETIOLOGY ~R CAUSATION
Identily and assesspossible
me,"," r~9"
through SRs;evaluate KT
cluster ""
causes 01 burden 01 illness
\3
EFFICACY
KNOWLEDGE CLlNICAL
Identily gaps in evidenee
TRANSLATION RESEARCH
through SRs;evaluate treatment
DISSEMINATION LOOP LOOP
Widespread dissemination oPtionS}hrOUgh RCTs
of di '''.'\' " find inqs

4
COMMUNITY EFFECTIVENESS
Assess benelitlharm ratio 01
DIFFUSION potentially leasible interventions
Natural dilfusion 01 researeh and estimate reduetion 01 burden
\
in the "real world" 10
(non-research setting) EVALUATlON EFFIC¡ENCY ~
<, ~:s~~~~uteomes Determine relationships
between costs and elfects / /
.~Iworld"
01 treatment options /
~
PERC has believed so srrongly in creating the next gen- Conclusion

eratiori of researchers rhar, from irs first lormal narional This chaprer, rhrough the example of the PERC research
meeting, it has held a FelIow's Research Day. FelIows in nerwork, has explored the potential role of research net-
pediatric emergency medicine from across Canada present works and tearns in health research training. 5uch net-
rheir research projecrs at all srages, from idea-generarion to works offer a positive environmenr in which trainees and
cornplered projecrs. In a positive environrnenr, rheir peers orhers can acquire research skills and observe role models.
and senior researchers offer constructive feedback ro Research nerworks provide exposure ro a variery of
improve their work. Wirh a broad array of disciplinary research disciplines and methodologies, insighr into how
backgrounds, ir offers a very stimulating and supporrive nerworks operare, and pracrical experience in collaboraring
environment. The trainees also attend the portion of the with peers and more senior researchers. Additionally, if rhe
annual meeting where staff researchers present their work, resident decides on research as a career oprion, early experi-
giving thern a strong sense of rhe research environment ence working with a nerwork provides a valuable
and possibilities for the íuture. FelIows and trainees who foundation, forging links with a communiry rhar can play
have graduared from rhe program now arrend as junior an imponant pan in his or her future developrnenr in
staff members. healrh research. •
CASE POSTSCRIPT
The resident's presentation was well-received, and he was encouraged to carry out a systematic review on the pediatric use of
the Ottawa Ankle Rules. Being part of a network, he had access to a statistician, an epidemiologist and a librarian, and was
able to complete the systematic review and have it published in Academic Emergency Medicine. 15 It was also selected as one
of the articles relevant to InfoPOEMs (which provides short reviews of articles relevant to primary carel and was rated as very
relevant to 89% of those accessing this POEM.16

REFERENCES
1. Babl F,Borland M, Ngo P,Acworth J, Krieser D, Pandit S, et al. Paediatric Research in Emergency Departments International
Collaborative (PREDICT):first steps towards the development of an Australian and New Zealand research network. Emerg Med
.- Austra/as.2006;18(2):143-7
2. Walker DM, Tolentino VR, Teach SJ.Trends and challenges in international pediatric emergency medicine. Curr Opin Pediatr.
2007; 19(3):247-52.
3. Pediatric Emergency Care Applied Research Network. The Pediatric Emergency Care Applied Research Network (PECARN):
rationale, development, and first steps. Acad Emerg Med. 2003;10(6):661-8.
4. Hartling L, Scott-Findlay S, Johnson D, Osmond M, PlintA, Grimshaw J, et al. Bridging the gap between clinical research and
knowledge translation in pediatric emergency medicine. Acad Emerg Med. 2007; 14(11):968-77.
5. Klassen Tp, Reed MH, StielllG, Nijssen-Jordon C, Tenenbein M, Joubert G, et al. Variation in utilization of computed tomograph¡
scanning for the investigation of minor head trauma in children: a Canadian experience. Acad Emerg Med. 2000;7(7):739-44.
6. Osmond MH, Klassen Tp,Wells GA, Correll R, Jarvis A, Joubert G, et al. CATCH: a clinical decision rule for use of computed
tomography in children with mild head injury. CMAJ. 2010;182(4)341-8
7. Tugwell P,Bennett KJ, Sackett DL, Haynes RB. The measurement iterative loop: a framework for the critical appraisal of need,
benefits and costs of health interventions. J Chronic Dis. 1985;38(4):339-51.
8. Scott S, Hartling L, Grimshaw J, Johnson D, Osmond M, Plint A, et al. Improving outcomes for ill and injured children in
emergency departments: protocol for a program in pediatric emergency medicine and knowledge translation science.
/mp/ement Sci. 2009 Sep 22;4:60.
9. Denny FW, Murphy TF,Clyde WA Jr, Collier AM, Henderson FW Croup: an 11-year study in a pediatric practice. Pediatrics.
1983;71 (6):871-6.
10. Kairys SW, Olmstead EM, O'Connor GT. Steroid treatment of laryngotracheitis: a meta-analysis of the evidence from randomized
trials. Pediatrics. 1989;83(5):683-93.
11. Russe!1KF,Liang Y, O'Gorman K, Johnson DW, Klassen TP.Glucocorticoids for croup. Cochrane Database Syst Rev.
2011;(1):CD001955
12. Segal AO, Crighton EJ, Moineddin R, Marndani M, Upshur RE. Croup hospitalizations in Ontario: a 14-year time-series analysis.
Pediatrics.2005;116(1):51-5.
13. Bjornson CL, Klassen Tp, Williamson J, Brant R, Milton C, Plint A, et al. A randomized trial of a single dose of oral
dexamethasone for rnild croup. N Eng/ J Med. 2004;351 (13) 1306-13.
14. Johnson DW, Craig W, Brant R, Mitton C, Svenson L, Klassen TP.A cluster randornized controlled trial comparing three rnethods
of disseminating practice guidelines for children with croup./mp/ement Sci. 2006 Apr 28;1 :10.
15. Dowling S, Spooner CH, Liang Y, Dryden DM, Friesen C, Klassen Tp,Wright RB. Accuracy of Ottawa Ankle Rules to exclude
fractures of the ankle and midfoot in children: a meta-analysis. Acad Emerg Med. 2009; 16(4) 277-7.
16. Canadian Medical Association. Ottawa ankle rules accurate for children aged 6 years and older. Dai/y /nfoPOEMs. Available from:
wwwcma.ca/index.cfrn?ci_id=50728&la_id= 1&gmAction=/infoPoems/displayPoem. do&poemld= 110622.
SUMMARY CHECKLlST
o Describe how a research network could benefit your research.

O Describe how a research network could help you develop as a researcher.
O List potential research networks to join. If you are unsure, find an experienced researcher who can advise you.

5
Looking for an academic mentor
--
".
David L. Sackett, OC, MD, FRSC, FRCP
--
15 THI5 YOU?
Although some of your colleagues see the "Resident Research Project" as a hurdle to be jumped (crawledr) over, you
see it as a leap into your chosen career of academic clinician-investigator. You'd really like to divide your time and energy
between being a master dinician and a groundbreaking researcher. Moreover, although your research supervisor is
excellent for your resident research project, you want to link up with someone who would take a long-term interest in you
and your career. If this is you, read on. If it isn't, stop reading this chapter and get on with the other bits of this guide that
will get you over that "Resident Research Project" hurdle.
• What will determine your success

as an academic clinician-investigaror? lf you warit ro
• Faet: Burnout affects up to a third of physicians, is more
common among women, has a lousy prognosis and is
become a top-Hight, srudy-designing, gram-gening, study- prevented by control over workload, setting limits, having
finishing, Iirst-authoring, prernaturely prornored, tenure- a mentor and having adequate administrative supporr."
holding, award-winning, highly respected and widely • Fact: In a recent survey, 41 % of the Canadian residency
recruited academic clinician-investigator, you won'r ger program direcrors who responded agreed or strongly
there on your brains, imaginarion and energy alone. You agreed that there was a need for more formal mentor-
will need three more rhings: good academic mentoring.l-' ship within the training prograrn.?
effecrive prioriry-setting and rigorous time management.
What a good academic mentor

• Fact: Previous academic performance accoums for
provides
only 6% of the variance in posrgraduare medical
cornpetency' For newcomers (such as graduare students, research fellows, or
• Fact: Canadian obstetrics and gynecology fellows with new faculry), academic memoring provides three mings. Pirsr,
rnentors were more rhan rwice as likely ro receive ir provides resources without obligations. Second, ir provides
academic prornorion." opportunities without demands. Third, ir provides advice and
• Fact: Graduates ofU.S. internal medicine research protection. I hope it's therefore obvious thar an academic
fellowship programs who had "iníluential rnenrors" men tor is not rhe same thing as a research supervisor.
during their training were 5 times as likely ro publish at By resources, I mean that a really good men tor would sup-
least one paper and 3 times as likely to be PIs on a plement rhe resources rhat mighr be provided in the short
funded research grant.5 terrn by your research supervisor and provide you with:
• Fact: A full half ofU.S. psychiatric residents given • longer-terrn space ro work;
menrors, research courses and an academic sripend • productiviry-enhancing equipmem;
went on ro pursue research careers." • free phorocopy, email and Internet access;
• Faet: Alrhough gender-based inequalities exist in • occasional secrerarial suppon;
running households and raising children, women are as • money ro go ro courses and meetings;
successful as men in having their U.S. National • salary supplernenrs if your fellowship doesn't provide
Instirures of Health research grants approved and for necessities and simple graces; and
renewed.? • bridge-funding of your research unri] you get your first
grant.
© 2011 The Royal College oí Physicians and Surgeons 01 Canada 29

---_~'!'I'Y".;=".- •• -
In sorne departrnents, a11 or mosr of rhese resources are íour ski lIs are central ro your developrnent as ;
provided by rhe chair, and in orhers, none. In eirher serring, independenr invesrigaror. Wirhom rhern, you'
.- your menror should "wheel and deal" unril rhe resources

are in place. You should be spared borh rhe rime and rhe
master only the merhods that are required for
"given" projecr. Like the kid who received a sh
humiliarion oEbegging for rhese resources on your own. new binhday hammer, you'll risk spending the
By opportunities ar rhe beginner's level, 1 mean rhe of your career looking ar ever-Íess-irnportanr n
sysremaric examinarion of everyrhing rhar crosses your rhump with your same old, limired ser of ski lls
rnentor's desk for irs porenrial contriburion ro your 2. The opporruniry ro carry out duplicare, blind (
scienrific developmenr and academic advancement. of course, confidenrial) refereeing of manuscrir
and grants. The comparison of rhese cririques r
l. The opportuniry ro join one of your rnentor's only sharpens your crirical appraisal skills, but ;
ongoing research projecrs. This can provide more permirs you ro see your rnenror's refereeing sryl
than jusr "hands-on" pracrical experience in rhe and forces you ro develop your own.
applicarion of your graduare course content. You 3. The opporruniry ro accompany your mentor ro
can also learn how ro creare and funcrion as a meerings of erhics and granr review committees
member of a collaborarive team, and ro devclop learn firsr-hand how rhese groups funcrion.
skills in research managemenr. 4. The opportuniry, as soon as your comperency
Taking on a piece of your rnenror's projecr ro permirs, ro join your mentor in responding ro rl
run, analyze, present and publish is a rwo-edged invirarions from prominenr, refereed journals ro
sword. On the one hand, ir provides an excellenr wrire editorials, commenraries or essays. Nor on
opportuniry ro go beyond the classroom and will rhe joint review and synrhesis of the relevan
develop your pracrical skills in dara managemenr evidence be highly educarional. Ir also provides
and analysis. Moreover, ir gives you rhe opportuniry you the opportuniry ro learn how ro wrire wirh
ro srart ro learn how ro combine "science and clariry and sryle (see ch. 29). Finally, ir adds an
showbiz" in presenring your resulrs and writing for important publicarion ro your CV As soon as yc
publicarion. Finally, your curriculum virae (CV) conrriburion warranrs, you should beco me rhe le
will benefir. aurhor of such pieces. 1he ultimare objecrive is ~
On rhe orher hand, being given a projecr by your you ro become rhe sole aurhor (a11rhe sooner if
mentor can be harrntul. The grearesr risk here is your menror casrs a wide shadow).
thar your menror mighr "give" you a pre-designed One nore of caurion abour invired chaprers fo
sub-study or research projecr and encourage you books: unless the book is a very presrigious one,
ro use ir as your major (e.g., thesis) learning focus. amhoring a chapter in ir adds litrle or no weighr
Alrhough ofren done wirh rhe besr intention, your cv
accepring this "gifr" is bad for you. This is beca use 5. The opporruniry ro rake over sorne of your mente
raking on a pre-designed projecr robs you of the invirarions and learn how ro give "boilerplare"
opportunity ro develop your mosr importanr lectures (especially at nice venues and for generol
research skills. First, you'll lose the opportuniry honoraria) .
ro learn how ro recognize and define a problem G. Your inclusion in rhe social as well as academic
in human biology or clinical careo Second, you'11 evenrs thar comprise the visir of famous colleagt
lose rhe opponuniry of learning how ro convert from other insrirurions should become automari
rhar problem-recognirion inro a quesrion that is 7. 1he opporruniry ro go as part of a group ro
borh im porram and answerable (see ch. G). Third, scientific meerings, especialIy annual garherings
you'lIlose rhe opporruniry ro learn how ro select of the research clan. 1his has several advanrages.
the most appropriare srudy architecture ro answer Firsr, ir gives you the chance ro meet and hear rl:
your quesrion (see ch. 9). Fourrh, you'lllose rhe old fans in your field. Second, ir allows you ro
opponuniry ro identify and overcome rhe dozens of rneet and debate wirh the other newcomers who
"rhrears ro validiry" that occur in any srudy. These will become your fmure colleagues. Third, you
30 © 2011 The Royal College of Physicians and Surgeons of Can.

5 Looking for an academic mentor
can compare your impressions and new ideas with are ofren besr rnet by specific additional mentoring around
your mentor while they are fresh, in a relaxed and rhese issues from a woman.
congenial atrnosphere. \Vhen listening to you son through a job offer, ir is
8. The opportuniry to observe, model and discuss imporranr for your mentor ro help you recognize the crucial
teaching strategies and tacrics in borh clinical and
classroom siruations. When you are invired ro join
difference berween "wanring to be wanred for" and "want-
ing tO do" a presrigious academic post. You'd be crazy not to -
your rnenror's clinical tearn, you can srudy how feel elated at "being wanred for" any prestigious job, regard-
they employ different teaching srrategies and tacrics less of wherher it matched your career objectives and aca-
as they move from rhe post-take/morning repon, demic strengths. However, an "actively listening" mentor
to rhe daily review round, to the clinical ski lis can help you decide whether you really "wanr to do" the
session, to grand rounds. With time, you should work involved in that post. Ir is here that rhey may help you
take over rhese sessions and receive feedback about realize that the pOSt is ill-rnarched ro your interests, priori-
your performance. The same sequence should be ries, career stage, cornpetencies, or temperamem.
followed in teaching courses and leading seminars By protection, I mean insulating you from needless aca-
in research merhods. demic buffeting and from the bad behaviour of other aca-
demics. Because science advances though the vigorous
Ir is important that these opportunities are offered withour debate of ideas, designs, data and conclusions, you should
coercion and accepred without resentrnent. Crucially, they get used ro having yours subjected to keen and critica] scru-
must never involve rhe off-Ioading of odious tasks wirh lit- tiny, For the same reason, you needn't be tossed in al rhe
rle or no academic content from overburdened rnenrors to deep end. Thus, for exarnple, you should rehearse formal
the beholden mento red. presentations of your research in front of your mentor (and
By advice, 1 mean providing frequent, unhurried and whoever else is around). They can challenge your every
safe opportunities for you ro think your way through borh staternent and slide in a relaxed and supporrive setting. As a
your academic and social developrnenr. Topics here include result (especially in these days of PowerPoint), you can
your choiees of graduate courses, the methodological chal- revise your presenration and rehearse your responses to the
lenges in your research projects, rhe pros and cons of work- likely questions rhat will be asked abour it. The objecrive
ing wirh a panicular set of collaborators, and how to here is to face rhe toughest, rnosr critical q uestions abour
balance your career with the rest of your life. For exarnple, your work for the first time at a rehearsal among friends,
sorne mentors refuse ro discuss academic issues at such ses- not following its formal presenrarion among rivals and
sions until they have gone through a checklist of irerns strangers.
encompassing personal and family health, relationships, Similarly, your mentor can help you recognize the real
finances, and the like. Their advice should take the form of objectives of the critical letrers to the editor that follow your
"active listening," should focus on your developrnenr as an first publication of your work. Most of rhern are attemp(s
independenr thinker, and should eschew commands and ro show off (the "peacock phenomenon"), ro protect rurf
aurhorirarian pronouncements. and ro win at rheroric, rather rhan ro promote undersrand-
As long as gender-based inequalities exist in running ing. When serious scientists have questions abour a paper,
households and raising children, mentors rnust be knowl- they write to its authors, not to the editor. Your menrors
edgeable and effective in addressing and advising around can also help you learn how to wrire responses rhat repeat
the special problerns that face women in academic careers.'? your main message, answer substantive questions (if any),
Alrhough only 20% of female academics in one study stat- and ignore rhe rawdry slurs that your detractors attempt to
ed rhat it was importanr to have a mentor of rhe same gen- pass off as harmless wit,
der, it is imperarivc thar all women pursuing academic Finally, disputes berween senior investigarors ofren are
careers have easy access ro discussing and receiving fought over the corpses of rheir graduare srudenrs. This
informed, ernparhic advice about issues such as timing means yOLl. Your mentor must interven e swiftly and
rheir pregnancies, parental leave, time-out, part-tirne decisively whenever rhey derect such attacks on you,
appointrnents, sharing and delegating household tasks, and including especially those relared to your sex, gender, race,
the like. \\lhen the principal mentor is aman, these needs handicap, or orientation. The intention of your menror's

• _ 'OT""--" ,,- ..,.-, I -..., _ • •
- .
The Research Guide: A primer for residents, other hea/th care trainees, and practitioners
rapid retaliarion needn'r be ro overcome your atracker's competir ion is common, and you should seek help
underlying prejudice or jealousy. Ir should merely make rhe from your chair or program director if rhis happens to
m_ repercussions of picking on you so unpleasant for him rhat you (I devore lots of rime ro trying to resolve such
he never tries it again. If it wasn't already pan of your core coníiicrs before they destroy friendships and damage
training, a study of rhe elassic paper on "how to swim with careers) .
sharks" should be part of this exercise. 11
3. Your academic mentor should not directly control
your academic appointment or base salary. Such
How should you find an academic controls inrerfere wirh the free and open exchange of
mentor? ideas, priorities, aspirations and criticisms. For
example, you may find ir difllcult to rurn down al1
In a recent survey of young invesrigarors awarded rheir firsr irrelevant, time-consuming task offered by your
career awards from the Alberta Heritage Foundarion for academic mentor when rhey also control yOLlfsalary.
Medical Research;'? the preferred roure was for rhe host 4. Your academic mentor rnust like mentoring and be
departrnenr ro provide a Iist of potemial rnentors and let willing to devore the time and energy required ro do ir
rhe newcomers meet rhern informally and identiíy the one well. This ineludes a willingness ro explore and solve
rhat best marched rheir needs and aspirarions. As empha- borh the routine and rhe extraordinary scienrific and
sized earlier, women must have ready access to additional personal challenges thar arise when they take on chis
mentoring around the issues of women in academia, from responsibiliry.
anorher source if necessary. 5. Sorne institutions still lack policies for sropping che
tenure elock for childbirrh and caring for a young
What should you look for when picking an child, or for "re-entry" rights and discounted "résurné
acadernic mentor?
gaps." Your academic mentor should be informed
Your academic mentor should possess five crucial prerequi- aboue these, and should fighr for them when rhey are
sires: lacking.
6. Finally, your acadcrnic mentor must periodically seek
1. Your academic mentor has to be a competent invesri- feedhack from you abour how well rhey are perforrn-
garor. Although rnost wil! be clinicians, this needn'r be ing. They rnust periodicaJly evaluate rheir own
the case. Some of the rnost successful academic performance, decide wherher they rernain the besr
elinicians 1 know (ineluding me) were mentored by person to mentor you (and, if not, help you find a
biostatisricians. more suitable mentor), and identify ways ro improve 4
2. Your academic mentor rnusr not only have achieved their mentoring skills.
4
academic success rhernselves, but must treat you
accordingly. That is, your academic mentor rnust feel Space does not perrnit a discussion of prioriry-seuing and 4
secure enough rhat rhey are nor only comfortable time-management, the other main dererrninanrs of a suc- 4
raking a back seat to you in marrers of aurhorship and cessful career in academic medicine, in rhis chaprer. Ho\\'- 4
recognition. 1hey must actively pursue this secondary ever, 1 have recently discussed rhern in the Clinician- Trialisr 4
role. Everything fails if your academic mentor com- Rounds series in rhe journal Clinical Trials.13•14
petes with you for recognition. Unfortunately, such
Acknowledgement
This chapter draws in part from: Sackett DL. On the determinants of academic success as a clinician-scien ist.
Clin /nvest Med 2001 ;24(2):94-1 OO. The kind permission of the Canadian Society for Clinical !nvestigation is
gratefully acknowledged
32 © 2011 The Royal College of Physicians 2'0 S "<;¡eo"'S 'C¿--=::¿ ,

5 Looking for an academic mentor
REFERENCES
1. Sambunjak D, Straus SE, Marusié A. Mentoring in academic medicine: a systernetic review. JAMA 2006;296(9): 1103-15.
2. Sambunjak D. Straus SE. Marusié A. A systematic review of qualitative research oí the meaning and characteristics of mentoring
in academic medicine. J Gen Int Med. 2010;25:72-8.
,.•
3. Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMI •
2002; 324(7343):952-7 •
•
4. Wise MR, Shapiro H, Bodley J, Pittini R, McKay D, Willan A, et al. Factors affecting academic promotion in obstetrics and
gynaecology in Canada. J Obstet Gynaecol Can. 2004;26(2) 127-36.
5. Steiner JF,Lanphear BP,Curtis P,Vu KO. Indicators of early research productivity among primary care fellows. J Gen Intern Med.
2002; 17(11 ):845-51.
6. Kunik ME, Hudson S, Schubert B, et al. Growing our own: a regional approach to encourage psychiatric residents to enter
research. Acad Psych. 2008;32:236-40.
7. Ceci SJ,Williams WM. Understanding current causes of women's underrepresentation in science. Proc Na ti Acad Sci.
2011; 108(8) 3157-62
8. Spickard A Jr, Gabbe SG, Christensen JF.Mid-career burnout in generalist and specialist physicians. JAMA 2002;288(12): 1447-50
9. Donovan A, Donovan J. Mentorship in postgraduate training programmes: views of Canadian programme directors. Med Educ.
2009;43(2): 155-8.
10 Mason MA, Goulden M. Do babies matter: the effect of family formation on the life long careers of women. Academe.
2002;88(6):21-7.
11. Johns RJ.How to swim with sharks: the advanced course. Trans Assoc Am Physicians. 1975;88:44-54.
12. Straus SE,Chatur F,Taylor M. Issuesin the mentor-mentee relationship in academic medicine: a qualitative study. Acad Med.
2009;84(1): 135-9.
13. Sackett DL. Clinician-trialist rounds 3. Priority setting for academic success. Clin Tria/s. 2011 ;8(2) 235-7.
14. Sackett DL. Clinician-trialist rounds: 2. Time-management of your clinical practice and teaching. Clin Tria/s. 2011 ;8(1) 112-4.
EXERClSE
Track down the Assistant Professor in your clinical department who is considered their brightest potential star and in whose
footsteps youd most like to tread. Over drinks and dinner (read: beer and pizza-you pay), get their suggestions on the
most important stuff you should do right now in preparing for your future academic career.
SUMMARY CHECKLlST
o List the roles you would like an academic mentor to play for you.
O List the characteristics you would like to find in an academic mentor.
O List possible candidates to be your academic mentor.
O Meet with these candidates. Select one and see if he or she will agree to mentor you.
O Clarifyexpectations.

The Research Guide A primer for residents, other health care trainees, and practitioners
.-
34 © 2011 The Royal College of Physicians ano Surgeons 'c~-.c:;¿

6
Conceiving and formulating the research question
Christian Vaillancourt, MD, MSc, FRCPC, CSPQ
•
ILLUSTRATIVE CASE
•
It is the beginning of a new academic year, and a cohort of Emergency Medicine residents are being welcomed into the •
•
•
program. Their program director discusses the CanMEDS Roles and reviews a list of objectives for the residency program,
which include a requirement to complete a research project. One resident has never been involved in a research project
befo re and is quite anxious about finding a topic.
--
•
• AII activities associated with a experience has guided your choice. Or perhaps something
project are anchored to a research question. Formulating particular wirhin a field of practice has captured your ínter-
that quesrion is arguably rhe most imporrant and the most est, drawing you to a more speciíic area such as pre-hospital
difhculr step in any research projecr. It can help ro begin emergency medical care, rraumarology, intervention radi-
wirh a general research idea, bur as rhe project rakes shape ology, neonatal care or rhe healrh of seniors. As in healrh
your initial idea will need [O be refined into a specific and care pracrice, so in research: it is irnportant to work in an
c1early defined research quesrion, Research questions are area rhar you are passionate abour. Developing a research
often revised severa] times during che planning stages of a project and bringing ir ro fruirion requires a sustained
projecr as input from the research tearn is gathered, feasibil- effort and is much easier to accomplish when you are moti-
iry assessmenrs are conducted, and the key elernenrs of the vared by a keen interest.
study are taken into account. Who is the target population?
What is the intervention? How should groups be corn- What topics are you passionate about?
pared? What outcome measures and what srudy design

should be used? This chapter wil! help you find a research
idea and transform ir into a specific and "operarional"
research question,
Formulating a research idea

Where to start? CHAPTER OBJECTIVES
After reading this chapter, you should be able to:

• With a broad topic. We all have a reason for doing • develop a strategy to find a research idea
what we do in our professional sphere. Whether your train- • transform a research idea into a research question
ing has led you to emergency medicine, surgery, radiology, • understand the importance of a properly constructed
pediatrics or public healrh, perhaps something in your life research question
Confirmatory study Knowledge translation Research idea

FINERcriteria Operational research question Research question
Hypothesis generation Parameter estimation
Hypothesis testing PICOT question

• With a healrh condition,

infrequently, we encounter
procedure,
extraordinary
or tool. Nor
parienrs or cir-
Where to look?
• Researchers, educarors and colleagues. Perhaps rl
cumstances thar make an impression on us or spark an best way ro get started is ro identify the researchers an
interest in nnding the best way to approach a problem. You educarors in your group. Most suecessful researchers deve
may have been srruck by the processes of care involved in op a niche or an are a of experrise as one research q uestio
attempting to resuscitate a patient in cardiac arrest, or by leads to another, Many researchers have more researc
the insertion of a chesr tu be using a minirnally invasive questions than time ro answer rhern, and you may nnl
~ technique, or by a colleague's suggestion to use ultrasonog- points of inrersection berween your inreresrs and theirs
e
.- raphy to look for a pneurnothorax. Many dinical experi- Such opportuniries might on occasion lead you to
e ences can give rise to questions and a desire to learn more. research collaborator, supervisor or mentor ourside you
c: own neld or discipline.
ca
- What was "cool" about the last interesting patient for
whom you provided care?
List the research projects in your field of interest that are
in progress in your department and the department
faculty members who are most active in research.
• With a research methodology. Sometimes a research

interest srems from a desire to learn how to go abour • Reading texrbooks and artides. You might be sur-
addressing quesrions of a cerrain kind, in which case the prised to learn that much of what we pracrise or take for
particular research topic mighr be less important ro you granted in health care is based on lirtle or no scienrific evi-
rhan rhe merhodology or study design used to approach ir. dence. As you review speciíic rextbook chapters, pay arten-
Perhaps you are inreresred in working with an experienced tion to areas of controversy or knowledge gaps nored by rhe
investigaror [Q beco me familiar with a particular research authors. Seek out and read any references cited, bur bear in
rnerhod, or perhaps you have idenrified a specific research mind that most textbooks are at leasr five years Out of date
skill thar yOl! want to hone for a future research project you by rhe time they are published. Any exploration of a con-
have in mind. This interest in rnethod mighr lead you, for troversial topic will also need ro include the rnost recenrly
example, to learn how ro develop a dinical decision rule, or published articles available.
ro perform a good chart review, conduct a survey, or carry Reviewing and discussing reeenr journal arricles with your
out a systematic review. colleagues can be a great opportunity [Q generare a research
idea. For example, you might find mar a panicular anide raises
15 there a specific research method you would like to an interesting idea but uses a Bawed method ro address ir.
learn about and apply? Another article might describe an exemplary research rnethod
mar you could apply to a differenr research quesrion of your
own. Many artides generate hypotheses or draw condusions
that should be conhrrned in further srudies.
List gaps in knowledge or evidence that you ha e

identified in your recent readings.
36 © 2011 The Royal College of Physicians and SL-gEC"'S o' ra.;-.¿:;:¿

6 Ca c"'¡ -ng and formulating the research questian
• Meerings and abstracts. There is perhaps no berrer • Graming agencies. If your goal is ro obrain funding
place ro meer researchers and be exposed ro the work rhey from a granring agency, carefully review the crireria and
do rhan a scien tihc meering. Mosr research presenrarions requiremenrs of their funding applicarions. Narional gral1r-
include a rationale and clear objecrives for the work reponing agencies often pL!( funds aside for the srudy of specific
ed, and many give a sense of rhe quesrions that remain health conditions. Alrhough the ropic may be broad (e.g.,
unanswered. If you have nor yer had a chance ro attend a diabetes in Firsr Narions cornmunities), ir encourages
scienrific meering or are unable ro in the near furure, you researchers ro follow a specific line of research.
can browse instead rhrough published absrracrs in confer-
ence proceedings 01' associarion journals. Are there granting agencies for your field of interest that
might be able to provide funding for your project? What
are their timelines far funding applications?
Find out where and when the next national scientific
meeting in your area of interest will take place.
-
Practical considerations
Refining your research idea

• Frequency of disease or test use. Because rhe rirneline
to complete a research project during a health rraining pro- The answers you have jorted down ro the quesrions aboye
gram is ofren very shorr, it mighr be wise ro choose a have likely pointed the way ro sorne research possibiliries.
research idea rhat involves a healrh condirion or test that is
commonly encounrered or used, panicularly if you plan ro Review your answers and write down the first three
recruit parricipants prospecrively. Common condirions or research ideas that come to mind. You might start with
"1 wonder if ... tr
tests can be idenrified intuitively from your clinical prac-
rice, or by a review of sratisrics collecred by your deparr-
menr. 1. _
What conditions or tests do you most commonly see or 2. _

use in your practice?
3. _
Need help I List at least three researchers you could

approach to supervise, mentor or collaborate with you.
1. _
2. _
3. _

A FINER idea From research idea to research

Now ir's rime ro start refining your inicial concepr. Consid-
question
er the FINER criteria. Your research idea needs ro be: If, aíter this examination of your research idea, you sril
Feasible believe ir is worth pursuing, then you wil! need ro formu-
Interesting lare a specific research question. A research idea is yoUl
Novel srarring point: ir is less specific and less operational rhan e
research quesrion. A research guestion, on the other hand,
el Ethical
._c: Relevant is relarively narrow and needs ro be clearly defined. Ir is rhe

research quesrion that will enable you ro determine what
e Ir is not always easy for inexperienced researchers ro deter- study design ro use.
e mine whether ir isfeasible ro find rhe answer ro a panicular
ra research idea. Many things have ro be taken inro account, What kind of question is that?
-
e, including the study design rhat is best suired ro approach
rhe problem, your abiliry to recruit a sufficient number of
In determining
irnportant
rhe approach ro your research project, it's
ro know what kind of question you are asking.
parricipanrs, how ro go about measuring outcomes, the As you formulate your question, consider which of the f01-
availability of adequate technical expertise, and the time lowing caregories best describes your inguiry.
and other resources rhat you and your research team, if you
can join one, will have at your disposal. Input from an Pararneter estimation for a health condition 01' diagnos-
experienced research supervisor, mentor or collaboraror tic test. Sornetimes we observe that a health condition is
will be important to assessing and enabling the srudy's fea- present in a given patient popularion, such as depression
sibiliry. among elderly patienrs seeking care in rhe emergency
You will be spending a significanr amount of time work- departrnent, but we lack good informarion on its actual
ing on your research projecr. 1he quesrion you choose ro prevalence. Or, we mighr need ro determine rhe diagnostic
answer has ro be interesting ro your peers and, most irnpor- accuracy or orher characcerisrics of new healrh rechnolo-
tantly, to you! gies, such as the abiliry of ulrrasonography ro derecr a pneu-
PIease-not anorher study on rhe beneíirs of buffering morhorax al: the bedside.
lidocaine to decrease pain during subcuraneous infiltrarion!
Try to find a novel idea. Be irnaginative. WiU your study Hypothesis generation. There are many disease processes
confirm or refure previous study findings? Will it extend for which our undersranding is lirnired and for which we I
previous findings ro a new popularion? Will ir provide new would like ro formulare hyporheses. Studies rhat look [or
findings? associarions berween an exposurc variable and an ourcorne
Your research project needs to respecr the ethical con- are designed to generare new hyporheses rather rhan ro dern-
cepts of auronomy, integriry, justice, beneficence and non- onstrate causaliry. For example, you could explore wherher
maleficence. It must nor pose unacceprable risks to srudy rhere are any associarions berween nutritional facrors during
participants or invade thcir privacy (see also ch. 18). pregnancy and rhe incidence of aurisrn in children.
The effect of various diers on the skin tone of worms may
be of very Iitrle relevance ro you, but it is exrrernely releuant Hypothesis testing. Some research questions are formu-
ro some fishermen. Will rhe findings of your study add to lated ro test a hypothesis as rigorously as is feasible. rudies
scienrihc knowledge? Are rhey likely ro inBuence clinical based on rhese guesrions need ro llave sufhcient power (i.e.,
practice, health policy, or both? Will they be relevant in a suflicienr sample size) and an appropriare design ro esrab-
guiding fmure research directions? Not all research ideas lish causa!iry (e.g., a random ized comrolled trial). The
wil! pass [he "So whar?" test, which is why you were asked exarnple of hyporhesis generarion given above poinzs ro
ro write down rhree possibiliries. some of the challenges posed by rhis rype of quesrion. How
would you test rhe hyporhesis rhar virarnin O in- ce Í.:_
pregnancy influences the incidence of au ism? One _

ble research guesrion mighr be: "Do childre bos
women randomly assigned ro receive close oervisioa
38 © 2011 The Royal Colleqe o ?hys¡~rs a..--c- _~C$::=:::ra:c

6 'ng and formulating the research question
ensure they consume high doses of vitamin D during preg- To get me bese answer, formulare the best PICOT ques-
nancy have a lower incidence of autism compared ro those tion )"ou can:
whose mothers are not given specihc follow-up regarding Population (or People, or Patients)
vitarnin D imake?" Intervention (if applicable)
Control (or Comparison)
Confirrnatory study. You thought you had found the per- Outcome
fect research topic, but when you performed a quick elec- Timeframe
rronic search of rhe Íiterarure you discovered that several
papers on your topic had already been published. Don'r be Here's an example of abad question: "Is anti-coagulation -
discouraged: a study can always be replicated in a bigger or beneficial in patients with atrial Iibrillation?" Here's why.
berrer way! Published studies may have Haws, or may have The PICOT ingredients of your research question are the
been conducted with a very different patient population same essenrial ingredients of a properly designed research -,
than the one of interest ro you. Besides, new knowledge is prorocol. Although these wil! be covered in derail in chap-
unlikely ro be readily adopted in practice without a nurn- ter 17, you can already appreciare the lirnirarions of the
ber of confirmarory studies. One exception ... did 1 men- question we just asked: Ir fails ro take the fol!owing ele-
tion the excess of studies on buffered lidocaine? ments inro account.
Knowledge translation. Even if an exposure or interven- • What type of anticoagulation?

tion has been tested and coníirmed in various srudy popu- • What rype ofbeneht?
lations and in various relevant environments and settings, • In which speciíic patienr population?
there are still no guarantees that stakeholders wil! put this • In acute or chronic atrial fibril!ation?
knowledge ro use. For example, we know rhar cardiopul- • What method wil! be used to test the hyporhesis?
monary resuscitation (CPR) can achieve a rwo- or three- • What is the time frame of interest?
fold improvement in survival alter out-of-hospital cardiac
arrest, yet fewer than 30% of victims receive CPR from a Now compare the previous question with rhe following
bystander before emergency medical services arrive. Or, ro one. Which is better? "Do patients over the age of75 with
rerurn ro our earlier exarnple: Is buffered lidocaine still not arrial íibrillation for longer than 48 hours who are ran-
being used routinely in your institurion, despire al! the sup- domly assigned ro receive coumadin have a lower l-year
porring evidence? risk of embolic cerebrovascular accident compared wirh
those randomly assigned to receive aspirin or placebo?"
Because it is so precise, rhis question allows you ro focus
Making your research question
on exacdy the information you are trying ro obrain and
operational
al!ows others ro clearly understand what your research is
Of course, many rypes of research question are possible, meant ro achieve. \'(Iho are these "others"? They are:
and the categories given aboye are not exhaustive. The
im portant thing is ro understand what type of quescion you • institutional ethics review boards
are asking and then ro rnake ir "operational." An opera- • funding agencies
tional research question should: • col!eagues
• journal editors
• speciíy rhe population ro be studied (who?) • patients and lay people
• specify rhe intervention, exposure or behaviour under • consumers of your research results
study (whati')
• specify the timeframe (when?) Naw try to farmulate yau awn research question.
• speciíy rhe setting (where?)

• reflect a rheorerical rationale (why?)
• speciíy the methodology (how) and primary ourcorne
• cornrnunicate the hypothesis

~~'Yl!~~-\~ • v _ •• ' • .'
----_------------------------------ ------------------
The Research Guide: A primer forresidents, other health care trainees, and practitioners
Use what you think is the FINER of the three research ideas
Conclusion
you had earlier. Make sure to include all of the PICOT ele- Formularing a research quesrion can be a dauming task,
ments. And remember: Your research question is a work in especially for rhose new to research. The objective of this
progress and will likely be modified as you read the other chapter was ro provide you with a strategy ro find and
chapters in this guide and meet with your research supervisor. develop a research idea and with rhe tools to transform that
idea into an operational research question. A properly con-
structed research question will make a world of difference
in your abiliry to design an effective study llsing rhe
research methodology best suited to your task. •
-
CASE POSTSCRIPT
The Emergency Medicine resident was introduced to an excellent, established researcher from the Department of Family
Medicine. They both shared an interest in sports medicine, and collaborated on a chart review project evaluating muscle
necrosis in patients with acute compartment syndrome. This resident took a liking to research, decided to complete a
research fellowship at the end of his residency, and wrote the chapter you just read!
Haynes RB. Forming research questions. In: Haynes RB, Sacket DL, Guyatt GH, Tugwell P,editors. Clinica! epidemio!ogy: How to do
c!inica! practice research. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2006:3-14.
Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Oesigning clinica! research: an epidemio!ogic approach. 3rd ed.
Philadelphia Lippincott Williams & Wilkins; 2007. Chapter 2, Conceiving the research question; p. 17-26.
Smith KM. Building upon existing evidence to shape future research endeavors. Am J Hea!th Syst Pharm. 2008;65(18): 1767-74.
Stone pw. Popping the (PICO) question in research and evidence-based practice. App! Nurs Res. 2002; 15(3): 197-8.
Thabane L, Thomas T, Ye C, Paul J. Posing the research question: not so simple. Can J Anaesth. 2009;56(1):71- 9.
• The resources listed aboye can help budding researchers develop cornprehensive and well-forrnulated research questions.
SUMMARY CHECKLlST
o Off the top of your head. list some problems or areas of interest that you might want to research.
O List some potential research topics from recent clinical cases or situations.
O List some researchers or colleagues with whom you might want to work. What are they researching?
O List so me research methods or approaches that interest you.
O List some potential research ideas from recent texts or papers you have read or presentations you have heard.
O List any granting agencies that might support work on your topics of potential interest.
O Looking at your list, select one or two ideas to develop into a question. Refine these using the FINERcriteria.
O Operationalize your question using the PICOT framework.
O Share your idea with colleagues and your supervisor. 15 it ready to use 7
40 © 2011 The Royal College of Physicians and Surgeons oí Cenada I

7
Searching the literature
Lorie A. Kloda, MLlS, PhD (candidate)
ILLUSTRATIVE CASE
A resident in Otolaryngology is interested in designing a study to investigate the effectiveness of ventilation tu bes in
reducing the incidence of recurrent ear infections in young children. In a quick scan of the literature, she locates a
systematic review in the Cochrane Library. The review concludes that, although the insertion of ventilation
been shown to be effective in reducing the rate of infections at six months after surgery, many of the studies conducted
tu bes has -
in this area are flawed and more research is needed. The resident realizes that an overview of the available evidence
will be crucial in convincing a granting agency to support her research, but she has never conducted a comprehensive
literature search before.
• In an age when any health sciences therapy, or other patient groups, such as older children or
researcher with an Internet corinection can access a wealth even adulrs. Once rhe scope of rhe informarion being
of inforrnarion quickly and freely, rhe art of searching rhe sought is defined, sources where relevant literature can be
lirerature is ofren overlooked. A literature search is an located are selected and searched, using srraregies
importanr early srep in rhe research process. Ir enables rhe appropriate ro each source. The results of these searches are
researcher ro become familiar with the published evidence typically managed using specialized software, and the
in an area, to refine his or her research quesrion, and ro jus- records are reviewed by rhe researcher for relevance. The
rify the research project irselfby idemifying Iimitarions and relevanr iterns are rhen retrieved and integrated inro a
gaps in rhe current state of knowledge. Nevertheless, narrative literature review that can be incorporared inro a
researchers ofren underesrirnate rhe rime and thought grant proposal or a manuscript.
required to conduct a proper literarure search and rhen ere-
ate a rcview ro suppon and jusrify a research projecr. 111e
CHAPTER OBJECTIVES
suppon of a qualified Hbrarian-typically, with a rnaster's
degree in Iibrary and information swdies-who has expe- After reading this chapter, you should be able to:
rience searching the biomedical and allied healrh literarure • describe the various steps involved in conducting a
will be essemial ro the qualiry and comprehensiveness of literature search;
your literature search. • select appropriate sources to search;
The lirerarurc search consists of several sreps. The first is • create search strategies for the retrieval of relevant
ro idemify the informarion needed, which is oíten broader literature;
rhan the research topic. In our case exarnple, the research • save search strategies and results;
quesrion concerns one type of intervenrion for otitis media • document a literature search; and
in young children, bur rhe Íiterature search could be • describe the professionallibrarian's role in supporting
expanded ro include other approaches, such as anribiotic literature searches, particularly for systematic reviews.
KEY TERMS
Bibliographic databases Embase Literature review Scopus
BIOSIS Google Scholar MEDLlNE Snowball searching
Citation indexes Grey literature MeSH Subject headings
C itation software Hand searching Pearl-growing Web of Science
Cochrane Library Librarian PubMed

Selecting sources and psychology. A selecrion of other darabases rhat may

Most medical researchers start their líterarure searches with wonh considering is provided in Table 7.1.
MEDLINE, the premier biomedical database provided You mighr also want ro conducr searches in databa:
free of charge (through the PubMed interface) by the U.S. outside biomedicine and allied healrh. For examp
National Library ofMedicine. Although MEDLINE is an management and compurer science darabases are releva
excellent source, indexing journal publications back ro for informarics-relared research, and educarional databas
1950, it is far from the only one. Several databases should are useful for psychosocial research. Many specializ.
be searched ro increase the yield of records retrieved and ro darabases are available as well, such as Aurismfrara" ar
.- reduce geographic bias. AIso, beca use differenr darabases POPLlNE.b Consulr a librarian for assisrance in selecrir
c: use differenr indexing pracrices, searching several will help and gaining access ro appropriare darabases. Various vendo
e ro compensare for any skewing of resulrs thar mighr be ere- offer access ro licensed darabases, and your institutio
-e,
ro ated by a reliance on only one indexing sysrcm.
Bibliographic databases
library's websire is an excellent gareway ro these resources.
Citation databases
Bibliographie databas es contain records of publicarions In addirion ro bibliographic darabases, citation indexe
such as journal anides, abstracts, conference proceedings, such as Web of Science and SciVerse Scopus- can be use
books and book chaprers, theses, and other materials. Sorne as pan of a complete literature search by helping you t
darabases, such as MEDLINE, are lirnited to journal arti- idenrify journals, books, conference proceedings and othe
eles, while orhers indude all rypes of publications. The con- sources where rhe references of interese ro you have alread
tents of thesc darabases ofren do not indude complere been cired. In this way, sraning with one or more key refer
("full rexr") anides. The darabase records contain several ences on a topic, you can enhance your lirerature review b
fields conraining descriptive information about each publi- tracking citations ro those refcrences across scienrific an:
carion, such as title, authorship: the source journal, book, academic disciplines, porentially discovering relate:
or conference; and volume, issue number and page range. research in unexpected areas. Web ofScience and Scopus an
Bibliographic récords also commonly indude fields for an subscription based and should be available through you
abstraer and for keywords or subject headings. Fields can be institutions library. For a more complete picture of citation:
searched separately or in combination. ro a publication of interest, you can also use Google Sehol
In addirion to MEDLINE, darabases ro consider ar, which is freely available rhrough the Internet.'
searching for biomedical topics indude BIOSIS, which
contains references ro biomedical and life sciences
Designing a search strategy
research, and Embase, which complements MEDLINE
rhrough more comprehensive indexing of records pub- Subject searching
lished in Europe and Asia. For pharmacology research, When you search a bibliographic database, ir is imporrant te
Embase is a more derailed and comprehensive index rhan generare terrns that will yield relevant records and ro reduce
MEDLlNE, I and a search of Embase is recommended in me number ofirrelevant records retrieved. The PICOT rnne-
any case if rhe comprehensiveness of a MEDLINE search monic for formularing an "operational" research question
for a particular topic is in doubt.? The MEDLINE, BIO- described in chapter 6 of mis guide, can also help you frarne
SIS and Embase darabases are often available rhrough uni- your literarure search." In the case presenred, me question
versiry libraries. "What is rhe effectiveness of ventilation tubes in reducing me
To locare sysremaric reviews in health care, DARE incidence of recurrent ear infections in young children" can be
(Darabase of Absrracts ofReviews ofEffects), provided free broken down into rhe following componenrs:
of charge by the Centre for Reviews and Disscrnination in
rhe Unired Kingdom, as well as the Coehrane Library are
important sources. Searching first for a recent systernatic aSee WWIN.autism.org.uklautismdata
review can provide you with borh an overview of your b See WWIN.popline.orgl
research topic and a useful lisr of exisring publicarions. c See http://thomsonreuters.comlproducts_serviceslscience/science_

productsla-z/web_ol_science/ , www.info.sciverse.corn/scopus/ and
Depending on your topic, you may wish ro search databas-
http://scholar.google.ca/lar Web of Science (Thompson Reuters). ~~:;_5
es in allied healrh ca re fields, such as rehabilirarion, nursing (Elsevier) and Google Scholar, respectively.
42 © 2011 The Royal College of Physicians and Surqecns 'C¿~¿::.

7 Searching the literature
• Patient, Problem, Population: middle ear infections in In additio to searching subject headings, ir is also rec-
young children ommended mar yOL! search keywords (also called rext-
• Intervention: ventilation tubes words). earching keywords requires sorne creativiry:
• Comparison/Control: antibiotics therapy/no intervention because rhese are the words as rhey appear in a record's title,
• Outcome: reduction in recurrence abstraer and subject heading lisr, they can take rnany forrns.
• Time frame: 6 months after surgery
In your search, select the most imporranr concepts and

Brainsrorrn word variarions, alternarive spellings and syn-
onyms to increase rhe potencial
your darabase search. When appropriate,
for a match ro be found in
use the trunca-
,
-
search rhese separately, In this case, we can search the con-
cepts middle ear infections (P) and venrilation rubesíl).
tion fearure (rypically indicated by an asrerisk)
word srerns with various endings or rhe plural forrn.
(O search
----
Q;
- -
Ideally, subject headings, if available, can be used ro Be sure to combine search terrns using logical operators
rnaximize the search resulrs. MEDLINE's exrensive lisr of (AND, OR) and to record your search on paper so thar you
Medical Subjecr Headings, known as MeSH, is useful for can visualize the relation berween terms before rhey are
searching. Embase uses irs own list of headings, called ente red into rhe darabase. For example, if you want (O
Emrree. 1hese standardized headings are organized hierar- design a search suaregy in MEDLINE to retrieve as many
chically and provide the option of induding narrower records as possible that pertain ro the use of ventilation
headings in your search by using the "explode" fearure. For tubes for otitis media in children, the following strategy is
example, the MeSH for Middle Ear indudes several nar- possible:
rower headings:
Exp Otitis media (MeSH) Ventilation (MeSH)
['ar, Middle
OR OR
Ear Ossicles
Mastoiditis (MeSH) Tubulation (MeSH)
Eustachian Tube
Glomus Tympanicum OR OR
AND
Stapedius Tympanitis (MeSH) Grommet*
Tensor Tympani
OR OR
Searching the "exploded" MeSH "Ear, Middle" wil! Exp Ear, middle (MeSH) Tyrnpanostom*
yield records of articles rhar discuss rhe middle ear, as wel! as AND (infect* OR inflam*)
those thar discuss any of the narrower terms (e.g., Eusta-
chian rube). This will increase rhe number of records Afrer this search is conducred, ir could be combi ned
retrieved by your search and ensure rhat relevanr iterns are with rhe MeSH Middle ear venriliation using the logical
not inadvertenrly excluded. operator ORo
MEDLINE offers other relevant MeSH for the ropic, Once the search terrns are selecred and run against rhe
induding "Otitis Media" and "Middle Ear Ventilarion." darabase, orher oprions are available for refining rhe resulrs.
Searching rhese MeSH, and rhe rerrns subsumed within These "limits" restricr rhe search by publicarion year, lan-
rhern (using rhe "explode" feature, indicated with the guage, publicarion rype, study design, age group, sex and
abbreviarion "Exp") will rerrieve al! records in MEDLINE many other attributes. 1he limits thar are available vary by
indexed with rhose rerrns. As you search for the correct darabase. For our example search, limiting retrievals ro
MeSH in MEDLINE, remember thar rhe database offers publications that discuss infanrs (aged 12 to 23 monrhs)
"scope notes," or definirions of the subject headings. Read- OR preschool children (2 to 5 years) would be appropriate.
ing the scope nore for a panicular MeSH will help yOL! Keep in mind rhar MEDLINE and other darabases offer
decide whether ir is rhe correct choice and perhaps many fea tu res that you can rake advanrage of to increase or
lead yOL!to an alternative or an appropriare cornbinarion of reduce the nurnber of records retrieved. To learn abour
headings. these, consulr rhe user guides available for each darabase
(see Addirional Resources).
© 2011 rhe Royal College 01 Physicians and Surgeons 01 Canada 43

The Research Guide: A primer lor residents, other health care trainees, and practitioners
• Table 7.1: Bibliographic databases relevant to literature searches in the health sciences
Sponsoring
Database (overage agency or owner Access
AMED Over 152 000 records from approximately 600 Health Care Subscription based;
(Allied and journals in three separate subject areas. covering Information Service medical and
Complementary 1985 to the present: of the British Library university libraries
Medicine Database) • professions allied to medicine
• complementary medicine
.- • palliative care
c: Many of these journals are not included in other
e biomedical indexes. Journals are mainly European;
ca
-
most titles are in English. Search terms are based on
MeSH.
Q..
See www.bl.ukJreshelp/findhelpsubjecVscitectenv/
medicinehealth/amed/amed.html
810SIS BI0515 Previews. Indexeslife sciences and Thomson Scientific Subscription based;
biomedical research from over 5000 journals as medical and
well as non-journalliterature; up to 18 million university libraries
records from 1926 to the present.
Biological Abstracts. Index to the internationallife
sciences literature, covering over 4200 journals
from 100 countries; ccntains over 11.3 million
records from 1926 lo the present.
See http//thomsonreuters.com/contenVscience/
p~lf!BIOSIS_Fact5heet.pdf
------------------------- ----
CINAHL Indexes journals, books, chapter, theses and other EBSCOPublishing 5ubscription based;
(Curnulative lndex to publications covering nursmq and allied health care medical and
Nursing and Allied disciplines. Uses MeSH as well as CINAHL subject university libraries
Health Literature) headings.
See www.ebscohost.com/cinahl/
John Wiley & Sons, UK Subscription based, but

Cochrane Library A collection of six databases indexed using
Cochrane Reviews, free by arrangement in
MeSH:
CENTRALandthe some countries and
• Cochrane Database of Systematic Reviews
Methodology Register Canadian provinces.
(Cochrane Reviews)
are produced by
• Cochrane Central Register of Controlled Trials
rnember groups of the
(CENTRAL)
international,
• Cochrane Methodology Register
not-for-profit Cochrane
• Database of Abstracts of Reviews of Effects
Collaboration.DARE,
(DARE)
HTA and NHS EEDare
• Health Technology Assessment Database (HTA)
maintained by the
• NSf-i Economic Evaluation Database (NHS EED)
Centre for Reviews and
Dissemination,
See www.thecochranelibrary.com
University of York, UK
44 © 2011 The Royal College oí Physicians ano S:m;eors~' C;;:¿::G

7 Searching the /iterature
Table 7.1 continued

DARE Contains 15000 abstracts 01 systematic reviews, Centre for Reviews and Free access through the
(Database 01 Abstracts including over 6000 quality-assessed reviews and Dissemination, Internet
01 Review 01 Effects) details 01 all Cochrane reviews and protocols. Focused University 01 York, UK
on ellects 01 health interventions; includes reviews 01
interventions that are elearly health related;
interventions with the potential to afteet health;
adverse effects; diagnostic and prognostie studies; and .,
individual patient data.
-
Q
See www.crd.york.ac.uklcrdweb/
-
••••
--
-----------------------
Embase Contains 24 million records, indexing over 7500 Elsevier Subscription based; ••
••••
-
joumals in biomedicine and pharmacology, including medical and university
over 2000 biomedical ti ti es not covered in MEDLlNE. libraries
Also includes conlerence abstracts. Records are
indexed using Emtree subject headings.
See http://embase.com/inlo/what-is-embase
HaPI Contains over 145 000 record s Irom 1985 to the Behavioral Subscription based;
(Health and Psychosocial present relating to appraximately 15000 Measurement Database medical and university
Instruments) measurement instruments (e.g., questionnaires, tests, Services, Pittsburgh, PA libraries
checklists, rating scales, etc.) in health and
psychosocial sciences. The database does not provide
access to the instruments, but rather to inlormation
about them.
LlLACS Database with English, Portuguese and Spanish BIREME(Biblioteca Free access through the
(Latin American and interfaces covering more than 800 medical journals Regional de Medicia) Internet
Caribbean Literature on f rorn 19 countries in Latin America and the Caribbean, and PAHO (Pan
Health Sciences 01 which most are not indexed in MEDLlNE. Covers American Health
Database) human health sciences, including medicine, public Organization)
health, dentistry, nursing, veterinary medicine, sanitary
engineering, pharmacy and chemistry, biology,
nutrition, psychology, ecology and the environment.
See http://bases.bireme.br/cgi-bin/wxislind.exe/iah/
online/?lsisScript=iah/iah.
xis&base=LlLACS&lang=i&lorm=F
MEDLlNE Contains 18 million citations to the lite sciences US National Library 01 Free sccess
literature from approximately 5400 journals in 39 Medicine throughPubMED at
languages; Records are indexed using MeSH. www.ncbi.nlm.nih.gov/
01 citations added in 2008, 47% were to articles pubrned/ or the NLM
published in the United States and 92 % to articles Gateway at http://
published in English. Focuseson biomedicine and gateway.nlm.nih.gov
health, encompassing lile sciences, behavioural Subscription based
sciences, chemicalsciences, and bioengineering though medical and
relevant to clinicians, health researchers, and those university libraries, as
engaged in public health, health policy, or health care well as the Canadian
education. Medical Association
Seewww.nlm.nih.gov/pubsilaetsheetsimedline.html
------------------- ...---------------

Table 7.1 continued

PsyclNFO Abstracts and citations to the scholarly literature American Psychological Subscription based;
(mainly journals) in the psychological, social, Association medical and university
behavioural and health sciences. Coverage is mainly of librarles
journals. Approximately 2500 journals, mainly from
North America, are represented. Indexed using subject
headings from the Thesaurus of Psychologicallndex
en Terms.
e
.-
Seewww.apa.org/pubs/databases/psycinfo/index.aspx
e
e Pearl-growing
. Devising an effective search strategy is a daunting rask, and
Citation software
Software such as EndNote
for storing search results
or Ref\XIorks is useful for stor-
ttS
-
c.. asking for assistance from a librarian is strongly recom-
mended. To get starred, however, you can use the pearl-
ing, nnding, citing and sharing references. Free alrernarives
ro rhese produces are frequently appearing, Zotero and
growing rechnique, which is useful for subject searches, Mendeley" being rwo recent examples. Citation software
For this rechnique, begin with a known reference and use it has the capabiliry to nnd, link ro and atrach the Iull-rext
ro genera te more subjecr headings and keywords. In MED- publication (if ir is available online as a PD F [portable doc-
UNE, you could locate the record for the Cochrane review umem format] file, for exarnple), crcating a virtual library
mentioried in our case, for exarnple, and examine rhe riele, or personalized database of literature pertinenr ro your
abstract and MeSH terrns for possible search rerms. 1hese research inreresrs.
subject headings and keywords can rhen be used ro help If you don't already use citation software, you should
devise your own search straregy in the same database. consider several facrors in selecring rhe righr one. If you are
. Librarians use chis technique especially when their searches working as part of a tearn, determine what orhers are using,
retrieve few relevant resulrs. as coIlaborarion wiIl be hindered by incompatible software.
Similarly, not aIl software produces are available for aIl
operating sysrems. Sorne products exist only in the online
Storing searches and results environment and may not be compatible with all word
Saving search strategies and creating alerts processors. You may also want ro coman your library and
Many databases ailow the user ro save mulriple search his- informarion technology departmeni ro derermine whether
rories in a personal account. Saving your search strategy so they offer training and support for a particular producr.
rhar you can replicare 01' rnodify ir larer is strongly advised. To suppon literarure searehes and the writing of reviews,
To benefit from rhis fearure of rnost darabases, register for a citation software is used ro import references direcrly from
personal account ae the outset and save various iterations of the databases once each search is conducted. After each
your searches. You can share these later when consulting database has been searched, and the results added ro a cita-
with a librarian or colleague, and revisit rhern in cases tion software database, duplicares can auromatically be
where a topic is of ongoing interest. MEDLINE, for exarn- identiíied and removed. You can son the references alpha-
pie, allows you ro rerun a saved search ro retrieve references beticalIy or chronologically and review thern for relevance
thar were added ro the database since the last time your directly wirhin the software, rather (han online in the data-
search was runo An alternarive ro rerunning a saved search base. 1hese personalized darabases can be saved and shared
is ro ser up an alert in ordcr ro receive notiíications of new with others. For grant proposal and manuseripr prepan-
darabase records rhar match your criteria. Derails such as tion, collaborarors can use ciration software ro find and cit
search frequency and email recipiems can be enrered, and relevant literature and creare referenee lisrs, Ciration sorr-
new records matching your search will be sent ro you auto- ware is becoming increasingly useful not onlv for sy ma
maticaIly as rhe database is updated. reviews, but also for rhe management of references relevar
ro one's research and practice.
d Seewvvwzotero.org/ and www.mendeley.coml
46 © 2011 The Royal College oí Physicians ano S:rgec"S - ~2-2:E

7 Searching the /iterature
Documenting the search In any lirerarure review, you wil!likely engage in snow-
When you embark on a new research activiry, it's always a hall searching (also called reference harvesting). In the case
described, rhe residenr would likely use the Cochrane
good idea ro keep track of your work and of all the deci-
sions you make along the way. 1he literature search is no review as a srarting point for her literature search and scan
irs lisr of references ro identify potentially relevant publica-
exception. Wherher you are conducring the search yourself,
as pan of a tearn, or wirh the assisrance of a librarian, main- tions. The rourine scanning of reference lists is always
advised for those conducring sysrematic reviews, and is
rain records of the sources selected (and rejecred), your
helpful for any researcher wanting to gain familiariry with a
search srraregies and any supplemental search methods
ernployed. 1his informarion will be useful ro you and your research area.
collaborarors during rhe Iirerature review process and can
The role of the librarian
help ro inform your literature searches in the furure.
Throughout this chaprer, you have been encouraged ro con-
Systematic reviews sult or enlist the services of a librarian ro suppon the litera-
If you are conducring a systernatic review, you will be ture search process. Librarians serve many roles in the
research process, ineluding advising and offering rraining
required ro provide, in the merhods secrion of your pub-
lished repon, exrensive derails regarding your literature on sources ro search and on effecrive search srraregies, and
search. As a form of research study, sysremaric reviews providing access ro resources and software. A librarian with
require rigorous merhodology (see ch. 15). You will need ro experience in rhe healrh sciences and uaining in expen
become well acquainted with current PRISMAe guidelines searching can conducr literarure searches ro rneet specific
research needs. In our sample case, enlisring a librarian ro
for documenring and reporting on literarure searches for
conduct a thorough search befo re beginning rhe clinical tri-
sysremaric reviews and ro enlist the services of a librarian
al could prevent potencial errors and the needless duplica-
with experience conducring and documenting literarure
searches for sysremaric reviews. PRISMA requires the peer tion of research effortS.
For systematic reviews, librarians are rypically invited ro
review of che search srraregy ro ensure rhat ir meers strict
be members of the research tearn.t They can direcr you ro
qualiry criteria.' PRISMA also requires rhe use of a How
valida red search srrategics, also cal!ed "hedges," which are
diagram ro document rhe srages of the literature search and
designed ro optimize retrieval from certain darabases. Hedg-
the selection of ineluded srudies.
es exist for al! rypes of srudy designs, ineluding qualitative
approaches,l and for several darabases, rypically MED-
locating relevant literature: UNE, Embase and CINAHL. As a member of the research
supplemental methods team, the librarian may also manage orher aspects of the lit-
erature search, such as overseeing hand searching and grey
Depending 011 the scope of your research, you may wish ro
enhance your lirerature review by using sources orher than literature searching, and documenting the search process.
darabases. For sysremaric reviews, hand seardúng is oíten

conducted, such that seminal journals and conference Conclusion
proceedings are identihed by the research rearn and their rabies
of contenrs screened for relevam references. Grey lirerature This chapter provided an overview of the steps required to
searches can be conducted as wel!. Grey literature ineludes all successfully complete a literature search as the foundation of
a literature review. Whenever possible, consulr a librarian or
forms of literature thar fall outside the realrn of traditional
commercial publishing and are rherefore not indexed in your library's websire for guidance and suppon in this pro-
darabases. Grey literature ineludes, for instance, government cess. Stan by refining your informarion needs and selecring
and association repores, websites, brochures and theses. appropriare sources for searching. Develop search suaregies,
Various resources are available ro support searches information or use hedges ro retrieve relevant records from the darabases,
and consider citation searching and supplemenral srraregies.
in this format (see Additional Resources).
Finally, manage your references and document the lirerature
search process: your present and future research projecrs will
e The acronym PRISMA stands tor Prelerred Reporting Items lor Systematic
Reviews and Meta-Analysis. The most recent PRISMA checklist and benefi t as a result. 11
explanatory documents can be accessed at www.prisma-statement.org
© 2011 The Royal College 01 Physicians and Surgeons of Canada

47
CASE POSTSCRIPT
After consulting her librarian, the resident conducts a literature search in several health sciences databases, including
MEDLlNE, Embase, L1LACSand DARE. Using a combination of subject headings and keywords, and limiting her search to
research about children, she retrieves many references to relevant studies. In addition, a citation search in Scopus confirms
that no important papers were overlooked. The resident keeps a record of her search strategy for each database so that,
should her grant application be successful, she will be able to rerun the searches in order to update her literature review in
the course of preparing her manuscript.
·-
REFERENCES
1. Brown CM. The benefits of searching EMBASE versus MEDLlNE for pharmaceutical information. Online Inf Rev.1998;22(1 ):3-8.
2. Sampson M, Barrowman NJ, Moher D, Klassen TP,Pham B, Platt R, et al. Should meta-analysts search Embase in addition to
Medline? J C1inEpidemiol. 2003;56(10):943-55
3. Kloda LA. Evidence summary: Use GoogleScholar, Scopus and Web of Science for comprehensive citation tracking. Evid Based
Libr Inf Pract. 2007;2(3):87-90.
4. Richardson WS, Wilson MC, Nishikawa J, Hayward RS.The well-built clinical question: a key to evidence-based decisions. ACP )
C1ub.1995;123(3) A12-3.
5. McGowan J, Sampson M, Lefebvre e

An evidence-based checklist for the peer review of electronic search strategies. Evid
Based Libr Inf Pract. 2010;5(1): 149-54.
6. McGovvan J, Sampson M. Systematic reviews need systematic searchers.) Med Libr Assoc 2005;93(1):74-80.
7. Grant MJ. How does your searching grow? A survey of search preferences and the use of optimal search strategies in the
identification of qualitative research. Health Inf libt J. 2004 21 (1) 21-32.
Canadian Agency for Drugs and Technologies in Health (CADTH). Grey matters: a practical search tool for evidence-based
medicine. Ottawa: CADTH; 2009. Available from: www.cadth.ca/media/pdf/Grey-Matters_A-Practical-Search-Tool-for-Evidence-
Based-Medicine .doc
• A directory of sources to search for health care literature, including bibliographic databases and an extensive list of online
sources for grey literature.
Centre for Reviews and Dissemination. Finding studies for systematic reviews: a resource list for researchers. York (UK): University
of York; 2010. Available from: www.york.ac.uklinstlcrd/pdflFinding_studies_for_syste[1;atic_reviews.pdf
• A directory of sources to search for studies to support systematic reviews.
--o The InterTASC Information Specialists' Sub-Group Search Filter Resource. York (UK): University of York; n.d. Available from:
www.york.ac.uklinstlcrd/intertasc/index.htm
• This resource organizes research on optimal search strategies (hedges) by study design and other topics, linking directly to
original publications measuring the sensitivity and specificity of these.
---o Systematic reviews: CRD's guidance for undertaking reviews in health careo York (UK): University of York; 2009. Available
from: www.york.ac.uklinstlcrd/index_guidance.htm
• This online manual describes the process of creating a systematic review, giving step-by-step instructions.
PubMed Online Training. Available from: www.nlm.nih.gov/bsd/disted/pubmed.html
• This site links to the full PubMed tutorial as well as to quick tours that demonstrate how to search using eS, 50 e
searches, and export search results to citation software.
48 © 2011 The Royal College oí Physicians a 51..'

_ro-lS~~.a::2
7 Searching the literature
EXERClSES
• Identify one or two key articles relevant to your researeh question, either by as, ing knowledgeable colleagues or by doing
a PubMed search. Look at the subject headings assigned to the article(s) and read the abstraet(s) to identify potential
keywords. Use these terms to develop a search strategy that will retrieve relevant literature.
• Brainstorm databases that may eontain relevant references. Look at library websites for subject guides or contaet a health
•
sciences librarian for suggestions. Run a brief test seareh using a few keywords in eaeh database to determine whether it
•
contains relevant references.
• Conduct a citation search starting with a key publication related to your research topie in all three citation databases,
--
•
•
•
if your institution provides aecess: Web of Science, Scopus, and Google Scholar. Compare the retrieval from the three
databases. Which one provided the most useful results, and why?
•
-
SUMMARY CHECKLlST
o Describe the information you need to search for. Write it down.

O Select the sources you will use for your seareh.
O Create an effeetive search strategy.
O Seleet a high-value referenee that you have found, and use its terms to refine your search strategy.
O Diseuss your search with a professionallibrarian or information specialist. Refine your strategy again.
O Save your seareh appropriately.
O Create alerts for your topies of interest.
O If desired, supplement your strategy with hand searches and grey literature. Contact knowledgeable authors in the
field for their recommendations.
© 2011 The Royal eollege 01 Physicians and Surgeons 01 Canada 49

----------------------------- ------------------------ -
.-

8
Data: Variables and levels of measurement
.,
-
ILLUSTRATIVE CASE
--
--
A nurse practitioner student completing a placement at a Community Health Centre wants to determine what proportion
••
of people served by the centre received the recommended preventive services during the previous 2 years. After speaking
to his field supervisor and deciding to carry out a review using the centre's electronic health record (EHR) database, he
realizes he will need to determine which EHR data elements will be needed for the study.
• Health information about people • A generic assessmenr determines that a woman carries
and condirions is ofren ascerrained by counting, making the BRCA1 gene.
observations, and raking measurements. 1hese collected
values form a ser of data that can be quite varied, as rhe fol- When these counts, observations and measurements differ
lowing exarnples dernonstrate: from person ro person or from rime ro rime, rhey are
referred ro as variables. Variables are rhe building blocks 01'
• In prepararion for a srraregic planning retreat, rhe many research studies, especially rhose thar use quanrira-
executive director of a local cornmuniry health centre tive merhods. Statistical rools, described and discussed in
determines the counrry of birth and language spoken chaprers 24 and 25, help us berter undersrand and make
at home for each patient seen at rhe centre during rhe sense of rhe dara collecred wirh respecr ro rhese variables.
previous 2 years. She finds rhe rnost frequenr ro be However, rhe choice of srarisrical rechniques or tests used
China and Mandarin, respecrively. ro describe and analyze variables will depend on rhe rype of
• A woman with a 2-year hisrory ofhigh blood pressure dara being examined. In prepararion for rhe larer discus-
uses her home blood pressure cuff ro learn that her
blood pressure is 150/90 mm Hg.
CHAPTER OBJECTIVES
• On his firsr visit ro a primary care centre, aman
reporrs thar he has 3 adulr children. After reading this chapter, you should be able to:
• As pan of a hospital self-evaluarion, a healrh services • define data variable and level of measurement;
researcher develops and adminisrers a quesrionnaire • list, describe and recognize the three roles that data
that includes an irem asking each of 500 randomly variables play in research studies; and
selecred respondenrs ro use a five-poinr seale, ranging • list, describe and recognize the five common levels of
from "poor" ro "excellenr," ro rare rheir lasr clinical measurement used in research studies.
visir ar rhe hospiral.
KEY TERMS
Categorical data
Continuous data
Dichotomous
Discrete data
(binary) data Interval data
Level of measurement
Quantitative
Ratio data
data l
Co-variable Explanatory variable Nominal data Response variable
Data Exposure variable Ordinal data True zero point
Data variable Independent variable Outcome variable Variables
Dependent variable Indicator variable Predictor variable

The Research Guide: A primer for residents, other health care trairrees. and practitioners
sions in rhis guide on specitic research approaches and Whether a variable serves as an outcorne variable, pi
merhodologies (ch. 9), rhe collecrion and management of dicror variable or co-variable depends on the question th
dara (ch. 22), and techniques for analyzing data (chs 24 & is being asked. For exarnple, in a srudy examining the pe
25), this chapter discusses rhe rhree kinds of data variables sible effecr of serum cholesterol on heart disease, cholesrer
used in research studies as well as the five levels of measure- leve! is the predicror/exposure variable and heart disease
ment (also called rypes of data). 1his chaprer, however, does the ourcorne variable. However, in a srudy examining tl
not apply ro rhe kinds of data thar arise from qualirarive porential effect of dierary far on cholesterolleve!, dietary f
research studies, which are addressed in chaprer 16. is the predictor/exposure variable and cholesterol leve!
.- the outcome variable .
e Types of data variables

c: Levels of measurement (types of data
ro
-
e,
Three rypes of variables are used in research srudies: out-
come variables, predictor variables and ca-variables.
rype has a specihc purpose.
Each Data are generally
Categorical
either categorical or quantitativ.
data, as rhe name suggests, are observarior
or variables thar are c1assified or caregorized by means e
Outcome variables labe!s or orher descriptive terrns. In rhe examples liste
An outcome variable serves as an endpoint of inrerest. Also above, coumry of birth, language spoken at horne
referred ro as the response or, more rraditionally, rhe depen- presence of rhe BReA] gene, and the 5-poim scal
dent variable, (he value of an ourcorne variable is one that is (ranging from "peor" ro "excellenr") are caregorics
believed ro be affecred by the potential causes (measured by measures. Quantitative data are observations, count
prediccor variables, defined and discussed below) being and measures made by determining quantities o
investigated in rhe research. An example of an ourcorne vari- assigning numeric values ro the data. In the exarnple
able is the diagnosis oflung cancer among participants in a above, the blood pressure of 150/90 mm Hg and th
study examining rhe risk facrors for this discase. number of children are quantirative measures.
More derailed descriprions of caregorical and quanrita
Predíctor variables tive levels of measurement are presenred be!ow.
A predicror variable is one that is being measured ro derer-
mine whether ir affects rhe outcorne of interese. Also Categorical data
known as independent, exposure, indicator or expla.na- Caregorical data are data that fall into speciíied caregories
tory variables, rhese may be measured before the study Ar a caregorical leve! of measurement, some particula
begins (e.g., age, body mass index, sex) or may be assigned qualiry of the observarion is used ro classify ir into one anc
as pan of the srudy (e.g., ro receive the intervenrion or pla- only one of a series of caregories. Categorical data shoule
cebo in a randomized c1inical trial). An example of a pre- sarisfy rwo condirions: (1) the categories must be rnutuallj
dictor variable is rhe reponed leve! of physical activiry in a exclusive (a qualiry should fall into only one category); ane
study examining risk lactors for ischemic heart disease. (2) rhe categories rnust be collectively exhaustive (all possi
ble responses can be classified). 1here are three specifi:
Co-variables rypes of categorical data: nominal, dichotomous (a specia
Co-variables are add irional variables collected beca use kind of nominal data) and ordinal.
rhey are known or hyporhesized (Q be relevant ro the
srudy. These usually include characteristics thar are or Nominal data fall into categories that have no inheren
could be related ro the predicror and/or outcorne vari- order (i.e., the categories simply serve as names). Example,
ables béing srudied, and are therefore ofren of interest (Q of such categories are occupation, coumry of birth, Jan
assess and, when necessary, control for in the data analy- guage spoken at home, diagnosis and blood rype.
siso Examples of co-variables include factors such as age
and sex as well as potential competing risk facrors such as Dichotomous (oc bina.ry) data are a special kind of nomina
physical inacriviry, smoking history or past occupational data thar are frequently used in healrh research and fal! inrc
exposures. one of only rwo possible categories. Examples of che caregorie
52 © 2011 The Royal College 01 Physicians ano u'gec"lS e' 5-,,::

Da -a: Variables and levels of measurement
used ro describe dichotomous data are female/male, rreat- Such data are ofren collecred by counring, as in deterrnin-
menr/ control, diseased/ not diseased and alive/ dead. ing me number of teerh wirh cavities, rhe number of preg-
nancies, and rhe nurnber of children in a family. Although
Ordinal data. Like orher kinds of qualirarive data, ordinal these data, like ordinal data, can be rank-ordered, the rela-
dara fall into caregories. Unlike the caregories used to rion berween values can be readily dererrnined. For exarn-
.,
describe nominal data, rhose used ro describe ordinal data pie, a family with 2 children has exactly half as many as
exhibir an inherent ranking (e.g., from lowesr to highesr). anorher rhar has 4 children.
So me examples are rhe following:
-
----
Continuous data, as rhe name suggesrs, include a ful! Q
• levels of care: primary, secondary, terriary, quarernary range of evenly spaced and possible fracrional values: thar
• amounr of pain: none, mild, moderare, severe, is, no rnatter how close any rwo values are to one another,
excruciating
• level of health: excellent, very good, good, fair, poor
other values always exist berween rhern. Of course, wherher
values dose to one anorher can acrually be individual!y
---
derected usual!y depends on rhe precision of the measuring
The differences or "disrances" berween adjoining poinrs technique being used. However, the rneasurernenr and
on an ordinal scale are often either unknown or indererrni- recorcling of continuous variables may, in practice, be con-
nare. For exarnple, we do nor know how much berter fined to a limited number of points (often integers) on rhe
"excellent healrh" is rhan "very good healrh." Furrhermore, conrinuum if grearer detail is not warranted by the preci-
the differences berween the various poinrs on an ordinal sion or use (or both) of the measurement. For example,
scale are usually not equal; for example, rhe difference although a particular blood pressure reading may in fact be
berween "moderare pain" and "severe pain" may be greater 154.768923/90.325146 mmHg, ir would usual!y be
or smaller than the difference berween "rnild pain" and reponed simply as 155/90 mmHg. Orher examples of con-
"moderare pain." tinuous data include blood glucose (mg/ 100 mL), height (in
Alrhough numbers are sometimes used to identity inches or centirnetres), temperature (usually in degrees Fahr-
ordinal categories (e.g., cancer srages, rhe New York Hearr enheit or Celsius) and age (which can be measured on any of
Association Functional Classificarion), rhese numbers are several possible scales: in hours, days, weeks, rnonths or years).
simply ranked labels, and rhe difference berween each For continuous dara, rhe absolute difference berween
caregory is usually unequal or unknown. This facr, however, values can always be deterrnined by subrraction. However,
does not stop researchers from performing marhemarical relative measures involving rnultiplication and division
operations on ordinal data. Even for a small number of (e.g., x is rwice as big as y) can be performed only wirh con-
caregories, some researchers wil! compute means and rinuous data rhat have a true zero point. For exarnple,
standard deviations or add and subtract rhe values of beca use age has a true zero poinr (i.e., birth), we can deter-
caregories. This practice is rarely jusrified and is generally mine that a person who is 40 years of age is rwice as old as a
discouraged. However, ir is common and appropriate for 20-year-old. In contrast, temperature measured in degrees
rhe frequency of each nominal and ordinal caregory to be Fahrenheir or Celsius does not have arrue zero point, so ir
counted and for rhe applicable percentages to be calculared would be inaccurare ro state that 50" F is rwice as hor as
and reponed. 2YF. However, the Kelvin remperature scale does have a
true zero poinr. Conrinuous data with arrue zero poim are
Quantitative data called ratio data, whereas rhose without a true zero point.
Quanritarive data are rhose that can be described by are called interval data.
numerical quantities; rhey are observations based on counts Some variables can be reponed using more than one lev-
or measurernents. Two rypes of quantitarive clara will be el of measurement, depending 011 the preterence of rhe
presenred in furrher detail: diserete data and eontinuous researcher. For example, a researcher could repon a patient's
data. actual blood pressure (continuous), caregorize ir as low,
normal or high (ordinal), or label it simply as normal or
Diserete data are whole numbers; rheir values are presenr- abnormal (dichorornous).
ed only as imegers (i.e., they do nor include fractions).

Conclusion
Data are the building blocks of health research. 1he leve! of research studies: as outcorne variables, predicror v.
measurement (i.e., rype) of the data collected during the or ca-variables. Later chapters of this guide discu
course of a study determines the type of sratisrical analysis data variables are employed in various research m
rhat should be performed on those data. When data, col- (ch. 9), in the collection and management of data (e
Iected as counts, observations and measurernenrs, difFer and in the selection and conduct of applicable sta
en from person ro person or from time ro time, they are tools to summarize, describe and better understand
e
.- referred ro as variables. Variables play three distinct roles in data (chs 24 & 25). •
e
e
ca
-
e, Acknowledgement
The development of this chapter was informed by Harvey BJ, Ancker JS, Bairnsfather S, Bukowski JA, Hudson S, Lang TI
al. Statistics for medica/ writers and editors. Rockville (MD): American Medical Writers Association; 2009. Chapter 2, Tyr
of data; p. 5-11.
EXEROSES
(Answers are given on the following page.)
1. What level of measurement involves only two cateqories?

2. What level of measurement involves whole-number counts?
3. What level of measurement involves a set of category names with no inherent orderinq?
4. What level of measurement includes a full range of possible fractional values?
5. What level of measurernent involves a set of inherently ordered categories?
6. In a study of the association between physical activity and mental health, women were asked how many children
they hado What level of measurement is "number of children"? What types of variables are "number of children,
"physical activity" and "mental health"?
7. Survey respondents were asked to rate their current health on a 5-point scale ranging from "excellent" to "poor.
What level of measurement best describes this measure?
8. In the Women's Health Initiative Trial, women who were randomly assigned to receive hormone replacement ther
(HRT) had heart disease more frequently than women who did not receive HRT What level of measurement is the
variable" had heart disease" 7 What types of variables are" hormone replacement therapy" and "heart disease"?
9. A study was conducted to assess the association between age and the occurrence of osteoporosis What level of
data measurement is "aqe "? What types of variables are" osteoporosis" and "aqe" 7
1O. A study was conducted to assess the association between serum cholesterol concentration (in mg/dU and systolic
blood pressure (in mm Hg). What levels of measurement are "systolic blood pressure" and "cholesterol concentra
tion"?
11. Researchers extracted data from hospital records to explore whether the number of surgical procedures
ever completed by each surgeon is associated with the surgeon's rate of surgical complications. What level
of measurement is "number of surgical procedures ever completed" 7 What types of variables are" surgical
complications" and "number of surgical procedures"?

12. Researchers conducted a study to assess the effect of radiation exposure (in mSv) on the risk of lung cancer amon
a group of uranium miners. What level of measurement best describes "Iung cancer"? What types of variables an
"radiation exposure" and "Iung cáncer"?
54 © 2011 The Royal College oí Physicians and Surgeons 01 Ca

8 Data: Variables and levels of measurement
ANSWERS
1. Dichotomous (binary) data
2. Discrete data
3. Nominal data
4. Continuous data
5. Ordinal data
6. Discrete data; co-variable (number of children), predictor variable (physical activity) and outcome variable
-
(mental health)
7. Ordinal data
8. Dichotomous; exposure variable (hormone replacement therapy) and outcome variable (heart disease)
9. Continuous data; predictor variable (age) and outcome variable (osteoporosis)
10. Continuous (both)
11. Discrete data; predictor variable (number of surgical procedures) and outcome variable (surgical complications)
12. Dichotomous data; predictor variable (radiation exposure) and outcome variable (Iung cancer)
SUMMARY CHECKUST
o Thinking about your research project, list the types of variables involved.
O List the kinds of data (i.e r levels of measurement) involved in your research project
© 2011 The Royal College of Physicians and Surgeons al Canada 55

.-
c:
c:
ca
-
e,
56 © 2011 The Royal Coliege 01 Physicians and Surgeons oí Car

9
Research methods and design:
Bringing the research question to life
ILLUSTRATIVE CASE
Motivated by the recent experience of one of her patients, a Family Medicine resident wants to study the outcomes and
experiences, such as death, metastases, incontinence and impotence,
cancer (i.e., low-grade,
of men diagnosed with favourable-risk
localized tumours with a low PSA level) who are managed with active surveillance versus those
prostate
e
roVl
who are treated by radical prostatectomy. She arranges a meeting with the hospital's chief of Urology to discuss the
possibility of designing and carrying out a randomized trial or so me other kind of study to address this question. -
11 This chapter builds on the preceding applicable primary Outcome to be assessed (and, if appli-
-
rhree chapters-on developing a research question (ch. 6), cable, any secondary outcornes): and, finally, anyapplica-
conducting a literature review (ch. 7) and appIying identi- ble Time frames, such as rhe period of follow-up. Each of
f)'ing data variables (ch. 8)-each of which lays important rhe components specified in the research question should
groundwork to help you select a research design suitable be taken into account as you consider potencial research
for your study. To choose the best design for the purpose, designs.
you will need ro be familiar wirh rhe options available and The Family Medicine residenr in our illustrative case will
consider the advamages and disadvantages of each. It is need ro choose a research design rhar enables her ro idenrify
irnportant ro realize that no single design can answer all and/or recruit a sufhcienr nurnber of men diagnosed wirh
rypes oE questions, and so your choice will depend on, favourable-risk prostate cancer (parient popularion), some
among other rhings, rhe narure of your research question. of whom will be treated with radical prostatecromy (inrer-
This chaprer examines this array of potential study vention) while orhers are managed with "watchful wairing"
designs in detail, focusing on quantitative designs: that is,
those that involve counting, taking measurernents and/or
CHAPTER OBJECTIVES
categorizing observations, The merhods and conduct of
qualirative research are discussed in chapter 16. In addi- After reading this chapter, you should be able to:
tion, rhe research merhods used to design and conduct sys- • discuss the purpose, advantages and disadvantages of
tema tic reviews will not be discussed in this chapter, bur are descriptive, cross-sectional, case-control, cohort and
presenred in chapter 15. experimental study designs;
• discuss selection bias, information bias and confounding;
how they can affect a research study's results; and how
The research question
they can be addressed in the study's design, conduct
As discussed in chaprer 6, a well-formed research question and/or analysis;
shapes several components of the anticipared research proj- • choose an appropriate research design, given a research
ecr, which are often summarized using the PICOT .mne- question and context; and
monie: the Population, people and/or parienrs of interest • argue the relevance and importance of your proposed
ro be studied; any Intervention rhat is being assessed; any research project (i.e., answer the questions "So what?"
Comparison rhat will be made against rhe intervenrion, rhe and "Who will care?").

KEY TERMS
Allocation Ecologic studies Non-differential misclassification
Allocation concealment Effect modification Observational studies
Before-after (pre-post) studies Effect modifier Odds ratio
Bias Experimental studies Placebo
Blinded, blinding External validity Placebo effect
Case Generalizability Prevalence studies
Case-control studies Historie (retrospective) cohort studies Quasi-experimental studies
Case reports Incidence/surveillance studies Randomization
Case series Information bias Random error/variation
Cohort studies Interaction Randomized controlled trial (RCT)
Confounding Internal validity Relative risk
Control Interrupted time series analysis Risk difference
Cross-over studies Longitudinal surveys Selection bias
,,-
Cross-sectional studies Masked, masking Stepped wedge design
Descriptive studies Matching Stratified randomization
Differential misclassification Natural experiments Two-by-two (four-fold) table
"'-
I Ecologic fallacy N-of-1 studies
L _ ____
I
1
(comparison). Each group will be followed for a sufficient variables to be conducted using techniques such as strati-
period of time (time frarne) to determine each participant's ned tables and calculations such as ratio of proportions,
healrh ourcornes of interest (e.g., survival in good healrh, difference of means and correlaciono
developrncnt of progressive prostate cancer, development The insights gained through descriptive studies may also
oE treatrnent cornplícations, death as a result of prastate ídentify possible causarive agents-for exarnple, the role of
cancer, or death from another cause). chimney dust in the development of scrotal cancer.:'
Descriptive srudies can also be conducted to gain an inicial
understanding of observed occurrences, thus helping to
Research designs
identify possible explanations that can be tested using more
Potential research designs range from relatively simple and analytic, comparative, study designs. An initial repon of
straightforward descriptive or cross-sectional srudies to young rnen diagnosed. with Kaposi's sarcoma and. Pneumo-
cornplex experimental studies.' Research examiníng a gív- cystis pneumonia in rhe early 1980s is an example of a
en healrh issue ofren progresses from description, to expla- descriptive study." As is well known, this and other early
nation, ro predicrion, to control. descriptive studies led ro further studies that ulrirnately
resulted in the identificaticn ofHIV and AJOS.
Descriptive studies There are several types of descriprive studies.' Case
The most straightforward research design is rhe descriptive reports and case series describe the first instance or instanc-
study, which, as rhe narne suggests, enables investigators ro es of a novel illness (e.g., AJOS, SARS), test (e.g., cornputed
describe various aspects of a phenomenon. 1,2 Features such tomography, magnetic resonance imaging) or treatrnent
as who is affected, where they live, when they were affected, (e.g., laparoscopic surgery). Incidence/surveillance srudies
and any other characteristics of inrerest can be determined measure the occurrence of distases in a dehned population
and summarized. Individual (i.e., univariate) variables are (e.g., communicable diseases in Canada). Prevalen ce stud-
usually sumrnarized using various descriptive statisrics (see ies measure the presence of health indicarors (e.g., expo-
ch. 24), such as mean, median, percentage, range, standard sures, behaviours, diseases) in a defined population (e.g.,
deviation and interquarrile range. Oescriptive studies ofren people who repon being physically active, people with dia-
enable a preliminary analysis of rhe relationship berween betes in a Canadian province or territory).
58 © 2011 The Royal College of Physicians and Surgeons of Canada

9 Research methods and desígn
Cross-sectional studies Healrh Uf\"ey (CCHS). The dara collecred in rhese srudies
Cross-sectional studies use data garhered at one poinr in are ofren available ro researchers. For example, Naiman
rime ro measure potential exposures, outcornes and orher and colleagues used data frorn the CCHS ro examine rhe
variables of interese rogerher. These srudies can be impacr of public smoking bans on self-reported smoking
conducted in various ways, such as rhrough phone surveys, status and exposure ro second-hand smoke.?
self-adminisrered quesrionnaires or rhe examinarion of
exisring databases. However, beca use potencial exposures Case-control studies
and ourcornes are measured at rhe sanie rime, cross- The case-control study design is almosr as efficienr as
secrional studies are well suired for esrablishing prevalence descriprive and cross-secrional designs, while providing a
(of ourcomes and risk factors) but poorly suited for rhe more rigorous way ro assess porenrial cause-and-effecr rela-
idenrificarion of potential cause-and-effecr relarionships. tionships.!" The case-control design is one of three "analyt-
Like descriprive studies, they are useful in rhe formulario n ic" or "cornpararive" research approaches in which rwo or
of hyporheses that can be rested in more rigorous, analyric, more groups are formally compared ro determine whether
studies. there is evidente of an associarion berween rhe ourcorne (or
In cross-secrional srudies, individual variables are usually outcornes) of interest (e.g., occurrence 0[, or dearh resulring
summarized using various descriprive srarisrics (see ch. 24), from, cancel") and rhe exposure (or exposures) under study
such as mean, median, percentage, range, standard devia- (e.g., roxin exposure, level of physical activiry, screening
tion and interquartile range. In addirion,
berween variables can be derermined
rhe relarionship
using stratified rabies
mammogram). The other rwo analyric approaches-cohort
srudies and experimental srudies-will be discussed larer.
-
or calcularions such as the ratio of proportions, difference The firsr srep in designing a case-control srudy is ro iden-
of means, and correlations. riE)' a group of individuals (cases) who have rhe ourcorne of
Sorne cross-secrional srudies are carried out using popu- inrerest and a comparison (control) group selecred ro pro-
larion averages rather rhan data about individual people vide an estimare of the frequency of the exposure in the
(e.g., arnount of alcohol sold each year in each Canadian population from which rhe cases are drawn. In a case-con-
province, annual provincial rates of liver cirrhosis). These trol study, an outcome refers ro a pre-exisring condition or
are called ecologic studies. Alrhough rhis rype of study is ourcome in rhe case group, such as (1) having being diag-
useful for suggesring hyporheses ro be tesred in rigorous nosed with the disease under study, (2) having developed a
analytic studies, ecologic srudies cannor be used to draw complicarion of a cerrain disease (e.g., metastatic spread of
causal conclusions. This is because they do not make clear a cancer), or (3) having died from a cause under srudy (e.g.,
wherher the individual people who developed rhe ourcorne in a motor vehicle crash, due ro a cancer, erc.) As such, rhe
of inrerest were those who were acrually exposed ro rhe controls should be as similar as possible ro rhe cases, aside
reponed variable (i.e., rhey do not provide direcr cause- from not having the ourcorne of interest. Alrhough rhe
and-efFecr informarion). For example, rhe facr thar a certain number of conrrols is ofren quite similar ro rhe number of
geographic area has both high levels of environrnental tox- cases being srudied, if only a limired number of cases are
ins and high rates of cancer is not sufficient ro dernonsrrate availabl~ the power of a case-control srudy can be enhanced
that the roxins are the cause of rhe cancers. 1his rype of by selecring multiple conrrols per case. (However, selecring
inference is referred ro as rhe ecologic fallacy. more rhan la controls per case provides only marginal add-
Because cross-secrional and descriprive studies can usu- ed benefir.) Once rhe cases and controls have been idenn-
ally be complered relatively quickly and inexpensively, and fied, informarion about each group is soughr, especially
sornetimes wirh dara that are already available, these concerning the potential exposure or exposures of inrerest
approaches are ofren rhe firsr ro be used ro garher inirial (e.g., pasr imrnunizations, history of smoking). This pasr-
insighrs and ro determine wherher more rigorous, analyric, exposure informarion can come from a variery of possible
studies are warranred. sources, including healrh records, orher records (e.g., cell
Canada is internarional!y renowned for its longitudinal phone records, for a srudy on brain cancers or motor vehicle
surveys, such as rhe Narional Longirudinal Survey of Chil- collisions associared with cel! phone use; ernploymenr, for a
dren and Yourh (NLSCY), rhe Narional Population srudy on the healrh risks associared with cerrain occupa-
Healrh Survey (NPHS) and the Canadian Communiry tional exposures), participanr selí-reports and even repares

· -------- - ,
by spouses, orher family members or friends. If rhe expo- bave associared health issues they gene rally poorly reflee
sure is rruly associared wirh rhe ourcome of interest, rhen rhe cbaracrerisrics of rhe underlying population; for thi
rhe frequency of rhar exposure should be found [O differ reason, drawing conrrols from this patient group can leal
significandy berween the groups being srudied (e.g., a his- to biased resulrs.
rory of smoking being more frequenr among rhose wirh One strategy that is often used ro enhance the compara
lung cancer cases rhan among rhe conrrols, a record of biliry of cases and controls is matching rhe selected con
iníiuenza immunizarion being less frequenr among rhose rrols ro rbe cases on rhe basis of facrors rhat are rhoughr [(
who conrracred influenza). be imporram, such as age, sex, socioeconomic status, erc
The associarion berween the occurrence of the exposure One of the shorrcomings of matching is that, if roo man)
and case-control status (i.e., in those with and without the facrors are chosen, ir might not be possible ro hnd suitable
outcorne of inrerest) is often summarized with a rwo-by- controls for some of the cases, who then cannot be includec
two table (also known as a four-fold table) and quanrified in rhe study. Another disadvanrage is that the role played b)
using an odds ratio-the ratio of the frequency of exposure rhe matching facrors in causing the outcome of interest
versus non-exposure among cases and the frequency of cannor be explored because, by design, rhese facrors do no!
exposurc versus non-exposure among controls (i.e., NC .;- difFer berween the case and control groups.
B/D or AD/BC). The second key factor affecting rhe accuracy of a case-
control study is rhe derermination of past exposureís): rhis
,,-
Cases Controls
musr be done as precisely as possible and, in parricular, in a
Exposed A B manner that is comparable berween the case and control
Not exposed C D groups. The recall of pasr informarion by study panicipanrs
is a common so urce of systernaric error (bias) in case-con-
trol studies. For example, ir bas often been dernonstrared
For example, a case-control study of 100 people with rhat people wirh an illness, or who have a cbild born with
lung cancer (the cases) and 100 people without lung cancer congenital abnormalities, recal! past exposures to putative
(rhe conrrols) found the following results: causal agents more accurately than unaffecred (control)
parricipams.
Cases Controls To the degree thar case-control comparabiliry and the
Smokers 80 20 accurate dererrnination of past exposures are not achieved,
----------------
Non-smokers 20 80 any observed differences berween cases and conrrols may be
biased and, as a result, fail tó reflect real differences.
Because the outcomes of interest bave already occurred
The resulting odds ratio would be 16 (i.e., 80/20 -i- 20/80 or wben a case-control study is initiated, these studies can usu-
80 x 80/20 x 20). Case-control studies can often be ana- ally be completed relatívely quickly and inexpensively,
lyzed using logisric regression, which readily provides an requiring a relarively modest number of study parricipants.
odds ratio adjusted for any orher variables of interesr that They are parricularly well suited for rhe study of rare diseases
were also srudied (e.g., age, sex). (i.e., when the ourcome of interest is expected to be infre-
Ir should be evident, and wil! be discussed further below, quenr) and for formulating hypotheses that can be tested in
that rhe accuracy of case-control srudies is highly depen- furrher srudies using more rigorous ana1yric designs.
dent on rwo factors. The brst is rhar rhe cases and conrrols
are as comparable as possible (except, of course, that one Cohort studies
group has the disease/outcome of interest while rhe other Whereas case-control srudies sample people on (be basis of
does nor).G,7Conrrols can be identified by methods such as (he outcorne, cohort studies sarnple people on (he basis o
random selection from rhe same neighbourbood as rbe rheir exposure [O some putarive causal agem, such as
case, or by randorn-digir dialing in rbe same telephone smoking or asbestos. By their nature, cohorr studies
exchange, or from rhe examinarion of records in rhe same minimize one of the sborrcomings of case-control srudies.':
health sysrem. Alrhough bospiral conrrols are ofren consid- Specitically, to reduce che porential risk of rhe exposureís)
ered and somerimes used, beca use individuals in bospirals ofinterest being measured differemially berween cases and
60 © 2011 The Royal College 01 Physicians and Surgeo s o; Cc~:;oa

9 Research methods and design
Died Alive Total

controls, rhey are designed to derermined such exposures
prospecrively, befo re rhe occurrence of the outcorne of Smo. ers 9132 19453 28585
interese. However, this gene rally results in cohort studies Non-smokers 940 4915 5855
raking much longer to complete, and therefore being much 10 al 10072 24368 34440
more expensive, than case-control srudies, since more rime
is required for an adequate number of the parricipanrs to The observed risk of death was 319.5 per 1000 among
develop and be diagnosed with rhe outcorne or ourcornes rhe smokers and 160.5 per 1000 among the non-smokers.
of interese. Historie (retrospective) eohort studies, As a resulr, the relative risk of death among smokers corn-
however, are an exception to rhis, and are possible only pared with non-smokers during rhat 20-year period was
when detailed, hisrorical exposure records are available for l.99 (319.5/160.5). In other words, rhe study's results
a well-defined population, sueh as employees in a specihc suggested that smoking doubled the risk of dying. The
workplaee. However, both prospective and retrospective risk diflerence was 159.0 per 1000 (319.5-160.5), sug-
cohort studies almosr always require sample sizes much gesting that smoking is responsible for an excess of 159
larger than those needed for case-control srudies to ensure dearhs per 1000 among smokers compared with non-
rhat a sufficient number of participants will ultimarely smokers.
develop and be diagnosed with the outcome of interest, Despite their improved measurement of the
Although case-control studies, by definition, address exposure(s) of interest, cohort studies share with case-
only those instances in which the outcome of interest is control studies the challenge of the porenrial non-compa-
dichotomous (i.e., individuals either do or do not have rhe rabiliry of the groups being srudied. Like all of the srudy
outcorne of interese), cohort studies can be used to examine designs discussed so far, cohort srudies simply examine
a wider range of outcorne measures, such as continuous, dis- the porential resulrs of "naturally occurring" decisions
crete and ordinal values (e.g., mean serum cholesterol con- (e.g., a person decides ro smoke, a health care provider
centrations following a dietary program). Like case-control prescribes a treatment). This is why cohort, case-control,
srudies, rhe association berween rhe exposure and a dichoto- cross-sectional and descriptive studies are collectively
mous outcome of interest in a cohort study is ofren sumrna- referred ro as observational studies: the researeher is
rized with a rwo-by-rwo tableo However, because a cohort simply observing decisions and outcornes determined
srudy ineludes an entire srudy population, thus enabling outside of his or her control. 111e observarional nature of
incidence rates to be determined (which is not possible in cohort studies means thar rhe groups being compared
case-control srudies), the association berween the outcorne (e.g., those who smoke, rhose who are physically active)
of inrerest and any exposures of interest can be quantified by willlikely differ with respecr to many orher characreristics
the relative risk or rhe risk difference. The relative risk is the beyond those being srudied. As will be discussed in fur-
ratio of the frequency (or "incidence") of rhe outcorne ther derail below, this means that any observed diflerence
among those exposed and the frequency among the non- in an outcorne of inrerest (e.g., rhe occurrence of hearr
exposed (i.e., AJA+B .;. C/C+D). The risk differenee is a atracks) may also be rhe result of these orher, differing,
measure of rhe excess frequency of rhe outcorne of interesr "exposure" factors-both those thar are known and mea-
artributed ro rhe exposure (i.e., [AlA+B] - [C/C+D]). surable as well as those that are currenrly unknown and
therefore cannot be measured. In an efforr to account for
Outcomes No outcomes porential differences between the srudy groups, research-
Exposed A B ers often use techniques such as srrarihcatiori and regres-
Not exposed e D sion analysis, which enable study results to be calculated

in a manner that adjusts for any porential differences. Of
course, including any poreritially differing factor in such
For example, a cohort srudy of 34 440 British doctors an analysis requires thar the factor in quesrion has been
was conducted by Doll and Peto? to determine rhe health measured for al! study participanrs. However, even alter
effects of cigarette smoking. Afrer 20 years of follow-up, 10 adjusring for an)' known and measured Iactors, rhe groups
072 of the study parricipants had died. Deaths were dis- may still differ because of orher facrors that are eirher
tributed among smokers and non-srnokers as follows: unknown or were not measured.
© 2011 The Royal College 01 Physícians and Surgeons 01 Canada 61

The next section will discuss experimental studies, which

and making participanr assessrnenrs are unaware of whic
use research designs that are betrer able ro ensure rhe corn-
study group each participanr is in (referred ro as blindin
parability of the groups being srudied.
or masking). A1location concealment is used during tl
assignmem of participanrs ro srudy groups to ensure rh:
Experimental studies
rhe assignmenr is not manipulared, whereas blindin
Unlike in observarional srudies, in which participant or
occurs alter group assignment to reduce the risk that thos
healrh provider decisions and choices determine the stud-
involved in rhe study will know which study group parrici
ied exposure, in experimental studies each participanr's
pants are in, ideally to ensure rhar all parricipants are treat
exposure of interest is assigned according ro rhe research
ed in a comparable fashion. Blinding is ofren applieo
protocol specified by the researcher, rhus enhancing the
where possible, ro those in rhe cornparison group receivin:
comparability of the groups being studied.' In borh experi-
a placebo interveurion, which should be as similar as pos
mental and observational studies, rhe groups being corn-
sible (e.g., pilIs that look and taste rhe sarne) to rhe ínter
._
pared should, ideally, differ from one anorher only with
venrion under srudy. In this way, ir should be impossible re
respecr ro rhe exposure of interesr-and, the more that rhis
determine wherher a participanr is in the srudy group or th:
can be achieved, rhe stronger the evidence will be of any
comparison group unril rhe secured allocarion informatior
-_
re!arionship between the studied exposure and outcome.
is accessed ro "break the code." If this leve! ofblinding of al
The comparability of srudy groups is irnportant because ir
rhose involved in rhe srudy can be accomplished, then each
better enables any observed difference in the outcome mea-
participanr will be treated in a similar fashion (i.e., each will
sure to be attribured to the exposure under study. In con-
receive the usual standard of care, while one group also
trast, the degree ro which rhe study groups otherwise differ
receives the interven tion under srudy and the orher receives
perrnits alternative porential explanations for any observed
the placebo). Alrhough the phrase "double blinding" is
differences. In reality, the comparabiliry of srudy groups
olten used, those who are blinded (e.g., participants, rhose
can be achieved only through a randomized experimental
providing care, rhose making assessrnenrs) should be explic-
study design in which participanrs are randomly assigned
itly identified in rhe srudy protocol and resulting report.
ro the study groups (e.g., a ra.ndomized controlled erial).
The association between the exposure and outcome of
Wirh this design each group (if sufficiently large) will, on
interest in an experimental srudy such as a randomized con-
average, have a similar distribution of a11factors, measured
trolled tríal (RCT) is summarized and quanrified in exactly
and unmeasured, associared wirh the outcorne of interest.
the same rnanner as in a cohort study, This should make
Because of rhe resulring comparability of rhe study groups,
sense, given rhat RCTs are cohort srudies with rhe unique
any between-group difference in the occurrence of the out-
fearure of the exposure/intervention being derermincd
come can be more conhdently artribured to the exposure according to the researcher's srudy prorocol.
(e.g., treatment) being srudied.
Recently, invesrigarors have begun ro use a stepped
As a resulr, experimental studíes that use non-random
wedge design in which the intervenrion is rolled our in a
approaches to assign participanrs to srudy groups are, like
sraged fashion ro a11parricipants, such that rhose receiving
observarional srudies, at risk of assembling groups rhat differ
rhe intervenrion in later srages serve as the comparison for
from one another wírh respecr ro factors other than the expo-
rhose who began ro receive ir in earlier srages. 12-14 The
sure under srudy. It should also be noted mar, even with rhe
implernenrarion of rhe intervention is ofren carried out in
use of rigorous randomization rechniques (and, especially, "clusters" of participanrs, in which case the order in which
in re!ative!y small clinical trials), srudy groups can differ from
each cluster receives the intervenrion is ofren deterrnined
one anorher simply by chanceo As a result, investigarors ofren by a randomizarion process.
use stratified randomization, so thar participanrs are evenly
allocared ro rhe srudy groups, thus ensuring rhat rhey do not
difler according ro any importanr co-factors. Other research designs
However, this comparability is best achieved when par- Alrhough rhe research designs discussed aboye (and surn-
ricipams are randomly assigned la ro study groups, when rhe
marized in Table 9.1) represenr those that are frequenrly
sequence of group allocation can be successfully concealed
used in healrh studies, many other designs can be, and have
from rhose involved in the srudy' 1 and, wheie feasible and
been, used. Alrhough ir is beyond rhe scope of this chaprer ro
applicable, when the panicipanrs and rhose providing care
discuss all possible designs, a few are rnenrioned briefiy here I
62 © 2011 The Royal College 01 Physicians and Surgeons oí Caneca

to help you determine whether rhey might be applicable w -of-1 studies21-23 are a special kind of before-after
your research projecr and warrant furrher exploration. srudy in t\VO\Vays: (1) only a single individual is studied and
(2) rhar individual serves as his or her own control. These
Quasi-experiments srudies are often used ro help guide trearment decisions by
In sorne circumstances, alternative experimental srudy comparing the effecrs experienced by a parient when he or
approaches have been used. These include quasi-experi- she receives, in a randomized fashion, rhe active rreatrnent
mental studies in which parricipanrs are allocared to the or the placebo. Of course, such studies are possible only
study groups using non-random merhods (e.g., according under specifi.c circumstances, such as recurrent illness (e.g.,
ro birth date or hospital number) or the group assignment asthrna, headaches) and when the active rrearrnent given
is simply alternated as each participant is recruited. (Even will "wash out" over a reasonably shorr period afrer it is
worse, in sorne insrances, is the pracrice of assembling study stopped, Authors have also considered the feasibility of
groups as "grab" Of "convenience" samples.) However, combining the resulrs of rnulriple N-of-l trials." Alrhough
because these merhods do not assign individuals ro study rhese studies are largely focused on informing treatrnent t
groups in a truly random fashion, rhe comparability of the decisions, they also provide a valuable introduction ro many 11
resulring groups may be compromised.P'"? of rhe merhods necessary for research.
-•..
\J
Natural experiments
Another variam is a natural experiment in which naturally
occurring circumstances result in one group of individuals
Interrupted
rupted
time series analyses
Although similar in approach ro before-after srudies, in ter-
time series analyses usually involve mulriplemea-
-
•••
being exposed while others are no(.18.19This design is probably sures borh befo re and after the imervemion being assessed. 25
more accurarely considered a cohort srudy in which "nature" These mulriple data points provide a much more complete
has determined exposure. However, because rhe exposure is understanding of any temporal changes leading up ro and
naturally determined, the groups being compared may be following the intervention of interesr, Alrhough these stud-
more comparable than is usually the case in cohort studies. ies can be used ro collect data concerning individual peo-
pie, they are otren used to assess rhe cffecrs of
Before-after studies (also known as pre-post population-wide policies and practice, such as the applica-
studies) tion of clinical guidelines,26 the enactrnenr of ami-smoking
As the name suggests, in before-after studies measures are laws5.27 or seat belt legislation,2S and the irnpacr of PUD-
made of research participams before and after sorne in ter- lished research on prescribing patterns.I?
vention or event (e.g., leve! of physical fitness leve! before
and after participation in a physical acriviry program).20 Cross-over studies
Because the participams serve as their own conrrols, many As in before-after srudies, participams in cross-over stud-
orher relevam facrors remain consta m, and so the study's ies also serve as rheir own controls.30-34 However, in cross-
ability ro derect an effect is strengthened. However, it has over studies it is also possible for rhe intervenrion ro be
been shown rhat simply being included in a research study adrninistered ro participanrs more than once and for rhose
can change people's behaviour, and so it is not always clear administrations ro be deterrnined in a blinded fashion and
wherher any observed change is due ro rhe intervention using random allocation. 1his design is often used to assess
under study or ro sorne orher change morivated conscious- the eAlcacy of drugs. Of course, rhis design requires an
ly or unconsciously among study parricipants. Also, it is intervention whose effects readily subside during a "wash-
also possible for parricipants w experience a placebo effect out" period before the cross-over, and that the health issue
simply because rhey have been started on a treatrnent, even being studied will recur aíter thar periodo Ir would rhere-
if rhat rreatment has !lO eAlcacy. As a result, many research- fore be inappropriare ro use this design ro study an inrer-
ers will also include a parallel comparison study group thar venrion, such as an educational program, rhat is expected
receives an "inert" control inrervention (e.g., a series of ro have long-lasting effects. A cross-over design would be
educational sessions, weekly newsletters or a placebo rrear- suitable, for exarnple, ro assess the eAlcacy of a new drug for
ment) ro help assess what role, if any, rhe experience of the trearrnent of tension headaches.
being srudied might have played in rhe observed results,

• Table 9.1 A comparison of study designs
5tudy design Characteristics Advantages Disadvantages
Descriptive Enables the characteristics 01 Usually efficient (quick and Provides description only
a group to be summarized small) Unable to determine causal
Data may be readily available relationships
and plentilul
Provides initial insights
Cross-sectional "Survey" 01 a single Usually efficient (quick and Unable to determine causal
"population" small) relationships
Data frorn a single point in Data may be readily available Exposure and outcomes
time measured at same time
Provides population
prevalence estimates
Case-control Comparison 01 two groups Usually ellicient (quick and Potential lor biased exposure
·- (according to outcome status) small and retrospecive (i.e.,

the outcomes 01 interest have
measurrnent
a_ Exposure occurs "naturally"

already occurred)
Potential conlounding
result 01 group diffrences
as a
Weak design lor deterrnnunq

causal relationships
Cohort Comparison 01 two or more Exposure measured prior to Usually large and lengthy
groups (according to outcome outcome Costly
status) lVIoderately stong design to Potential conlounding and
Exposure occurs "naturally" detemine causal relationship biases as a result of group
differences
-------_-
Experimental Comparison 01two or more Most rigorous research design Usually large and lengthy
groups (according to intervention Comparability 01 study groups Costly
assignment)
Strongest design for determining
causal relationships
Other study factors

panrs can and should be included in any given study:
Severa! general design issues apply ro a!l research srudies. however, the selection of rhis group will dictare rhe degree
These indude rhe context within which rhe study is carried ro which rhe srudy and its results rnight be genera!ized.
out and potenrial facrors rhat can compromise the accuracy (This is also referred ro as the external validity of rhe srudy.)
of a study's results. These facrors are discussed briefly below. For exarnple, if a researcher wishes ro determine the pro-
portion of parients wirh diabetes who have had an ophthal-
Study context and generalizability mologic examination in the previous 24 monrhs, rhe choice
As you design your study, consider rhose ro whom the of how participants are idemified and recruired-s-wherher
resulrs wil! appIy. Research srudies are alrnost always carried from a diabetes specialry dinic, a prirnary care pracrice Oí a
out tO gain insights and draw conclusions that are applica- province-wide diabetes registry--wiII dictare che applica-
bIe beyond the individuaIs who participa te in the study. biliry-e-rhat is, the generalizability-of the srudy's resul s.
The study population indicated in the research question
defines rhe broader contexr of your research: this is the pop- Study error and internal validity
ulation from which yom study sample shouId be drawn, When a research srudy is carried out, ir is hoped mar i
and ro which your resulrs should be applicable. For practi- results will reflect rhe "rruth," such that the observed effea:s •
ca! and ethical reasons, only a lirnired number of partici- are "real." However, the results of a given srudy can o - ~
64 © 2011 The Royal College oí Physicia s and Sur -:¡' (¿-a;¿ ~
4
rimes be explained by facrors other than truth, 1he firsr of (Q selecr the conrrol group from rhe same population in
rhese is randorn error/variation. 1his arises when, by which cases occur+? Alrhough the selecrion of conrrols is
chance, the individuals selecred for the study are not repre- srraighrforward in some circumstances, ir is not always clear
sentarive of the group being srudied. As a result, the study's whar kind of conrrols should be used: neighbourhood,
resulrs do not reflecr the trurh, bur differ because a non-rep- friend, associare, relative or hospital. However, withour the
resenrative group of participanrs was selecred by chanceo Ir selecrion of a suirably comparable control group, rhe srudy's
should be nored, however, that chance can affecr a study's results will be sysremarically biased and provide an inaccu-
resulrs in eirher direcrion, being equally likely ro lead ro rare estímate of rhe true relarionship berween the
overestimation or ro underestirnarion. As discussed in chap- exposureís) and ourcorneís) being srudied, In facr, as dis-
ter 25, statisrical tests and rechniques are used ro determine cussed aboye, even experimental study designs can be
rhe probabiliry rhar randorn variarion (i.e., bad luck) mighr affecred by selecrion bias, parricularly if rhe rnerhods used
be responsible for the observed srudy results. for randornization'? and alloearion concealrnen t!' are not
As suggesred aboye, facrors such as non-comparable sound. Of COLme,as discussed earlier, even when an appro-
study groups and inaccurare measures of exposures and/or priare randornizarion rechnique is used, groups are still ar
outcornes can alter a study's results so that rhey do nor reflecr
rhe true relarionship berween the exposure(s) and
risk of differing sizably fram one anorher sirnply by chanceo
Selecrion bias also comes inro play in derermining who gets -
outcornets) being srudied. 1hese are examples of orher fac- into a study. 1hose who volunreer (in sorne srudies, this can
---
rors, such as bias and coníounding." rhar can distort a
study's resulrs. 1he potenrial for bias and orher errors should
be taken inro account as you select a srudy design, for each
be as few as 20% of those approached)
eral population, jeopardizing
validiry of the srudy. 1his is a problem
differ from the gen-
ro sorne degree the externa]
for al! research
-
•••
approach differs in its susceptibiliry ro these problems. In designs that enrol parricipanrs.
addition, unlike rhe random effects of chance, bias and con- Information bias arises as a result ofhow information is
founding are systernatic errors and, as such, can arise beca use collected about the participanrs in a srudy. 1his can occur at
of weaknesses in a study's design and/or conduce, resulring any juncrure from inirial recruitrnent ro final follow-up
in a predicrable over- or underesrimarion of rhe effect being where che measurement of the study groups might di ffer,
assessed. Internal validity is rhe terrn used ro describe rhe but ir is particularly likely when rhe exposure and/or out-
soundness or rigour with which a study is designed and con- come of interest are measured. For example, differential
ducred. Ir is also an indicaror of the accuracy of the study's misdassification/informarion can result when a quesrion-
results (i.e., how well rhey approxirnate the "trurh"). naire does nor accurately collecr soughr-aíter informaríon or
1here is ofren a rrade-off berween internal and external when srudy participants are nor followed for a sufficiendy
validiry. 1he more righ tly conrrolled rhe selecrion of parrici- long period for the outcornes of interest ro be observed. Ir
panrs for a srudy (e.g., with long lists ofinclusion and exclu- can also arise if those who are providing and/or collecting
sion crireria) and the delivery of rhe intervenrion, rhe parricipanr information are aware of which group a parrici-
greater the internal validiry will be, but ar rhe cost of rhe panc is in. As discussed aboye, one way ro reduce rhis risk is
generalizabiliry of rhe results (exrernal validiry). ro mask (blind) any applicable individuals (e.g., the partici-
pants themselves, practitioners providing care, those making
Study biases rhe measurernenrs or adjudicaring ourcornes). Ifblinding is
Biases are facrors rhar, arising from rhe design and/or con- accomplished successfully, then rhe measuremerir of infor-
ducr of a study, cause the results ro diverge from rhe rrurh, marión should be achieved in a comparable rnanncr, inde-
Although many different rypes of bias have been pendenr ofwhich group a parricipanr is in."
described.t':" rhey gene rally faIl into rwo main caregories: Selecrion and information biases are of particular con-
selecrion bias and informarion bias. cern in cornparative srudies: if the groups are aflecred dif-
Selection bias arises when the groups being compared ferenrially, their comparison can porenrially be distorted.
differ as a resulr of how srudy participanrs are chosen. As Furrher, rhe groups compared in non-randomized experi-
discussed aboye, in observarional srudies this can arise when mental and observarional srudies may differ from one
rhe groups being srudied are selecred in a way rhar fails ro another with respect ro facrors orher rhan the exposure
ensure rhat they are comparable. This poses a particular (e.g., rrearrnent) being studied. As a result, special care rnust
challenge in case-control srudies, where, ideally, one strives be taken in rhe design, conduce, analysis and inrerpretarion

of these kinds of studies. However, ir should be noted that, other hand, if confounding is not present, then the leve! o
even if the measurements are made comparably among rhe risk will remain unchanged when the potential contoundei
study groups, there might still be sorne level of inaccuracy. is controlled foro
Although such non-differential misclassifications/mea-
surements will not resulr in a favouring of one group over Effect modification
che other, they do bias the study's result toward che null, Stratiíied analyses are also used ro explore wherher a factor
That is, che resulting "randorn error" causes che result might be an effect modifier. As the name suggests, such a
observed by the study to underestimate che "rrue" resulto factor rnodifies the effect of another factor. For example,
studies have shown that rhe effect of asbestos exposure is
Confounding magnified in those who also smoke. This is why the risk of
Confounding occurs when the apparent effecc of rhe expo- deve!oping mesorheliorna, a cancer unique!y associated
sure under study is the result of irs association with anorher with asbestos exposure, is markedly greater among rhose
causative (confounding) factor. To be a confounder, a factor wirh a hisrory of smoking. This would becorne apparent in
must be associated wirh rhe exposure, must be an indepen- an analysis stratified by smoking because the leve! of risk
.- denr deterrninant of the outcorne, and must not be an (e.g., odds ratio or relative risk) of mesotheliorna would be
intermediare scep in rhe causal pathway. For exarnple, greater in the smoking as compared with rhe non-smoking
although a study might find an increased risk of ischemic subgroup. Effecc modihcarion is sometimes referred to as
heart disease associated with coffee drinking, it is possible interaction, meaning thar facrors interact with one another
that this apparent association is confounded by sorne other to creare a greater effect than either would cause on irs own.
risk factor for ischemic heart disease, such as smoking,
which is also associated with coffee drinking. The presence
Conclusion
and effecc of potential confounders can be deterrnined by
assessing the level of risk of the exposure under study while A wide range of approaches is available for you to choose
conrrolling for any potential confounding factor. This is from as you design your research srudy, However, your
accomplished by carrying out a stratified analysis that choice of srudy design should be guided by how much is
adjusts for the arnount of such a confounding variable known about the subject being studied and by the advan-
(smoking), which enables rhe effect of rhe other variable tages and disadvantages associated with each possible
(coffee drinking) to be determined independently of rhe designo Although this chapter has provided a general over-
potential confounder's effecr. If confounding is present, the view of the various issues that should be considered in
level of risk associared with che exposure under study will choosing a study design, che seven chapters that follow
be reduced (i.e., rhe amount of risk attributable to the con- each address a particular type of study in furrher derail. •
founding factor has been removed/controlled for). On rhe
CASE POSTSCRIPT
Although the resident and her supervisor recognize that several research designs, including a randomized clinical trial,
could be used to address her research question, to better inform their planning they choose to begin with a preliminary
descriptive study to better understand the characteristics and experiences of men diagnosed with favourable-risk prostate
cancer who are managed at their medical centre. The findings of this descriptive study allow them to better determine '[re
quality of the data available from the hospital's electronic health records database. Guided by these preliminary result ,
they then design and complete a retrospective cohort study comparing the outcomes of the men managed wi acti e
surveillance versus those attained by men treated with radical prostatectomy.
66 © 2011 The Royal College 01 Physicia s a.,d S ;~ror:s::' ¿-.a:G

REFERENCES
1. Grimes DA, Schulz KF.An overview of clinical research the lay OT :he land. Lancet. 2002;359(9300):57-61.
2. Grimes DA, Schulz KF.Descriptive studies: what they can and cannot do. iencet. 2002;359(9301):145-9.
3. Waldron HA. A brief history of scrotal cancer. Br J Ind Med. 1983;40(4):390-401.
4. Centers for Disease Control. Kaposi's sarcoma and Pneumocystis pneumonia among homosexual men-New York City and
California MMWR Morb Mortal Wkly Rep. 1981 ;30(25)305-8.
5. Naiman AB, Glazier RH, Moineddin R. Is there an impact of public smoking bans on self-reported smoking status and exposure to
secondhand smoke? BMC Public Health. 2011 Mar 3; 11 146.
6. Schulz KF,Grimes DA. Case-control studies: research in reverse. Lancet. 2002;359(9304):431-4.
7. Grimes DA, Schulz KF.Compared to what? Finding controls for case-control studies. Lancet. 2005;365(9468): 1429-33.
8. Grimes DA, Schulz KF.Cohort studies: marching towards outcomes. Lancet. 2002;359(9303):341-5.
9. 0011 R, Peto R. Mortality in relation to smoking: 20 years' observations on male British doctors. Br Med J. 1976;2(6051): 1525-36.
10. Schulz KF,Grimes DA. Generation of allocation sequences in randomized trials: chance, not choice. Lancet. 2002;359(9305):515-9.
11. Schulz KF,Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359(9306):614-8
12. Mdege NO, Man MS, Taylor Nee Brown CA, Torgerson OJ. Systematic review of stepped wedge cluster randomized trials shows
that design is particularly used to evaluate interventions during routine implementation. J C1in Epidemiol. 2011 Mar 15. [Epub
ahead of print].
13. Brown CA, Lilford RJ.The stepped wedge trial design: a systematic review. BMC Med Res Methodol. 2006 Nov 8;6:54.
--
14. Hussey MA, Hughes JP Design and analysis of stepped wedge cluster randomized trials. Contemp C1in Trials. 2007;28(2): 182-91.
15. Morgan GA, Gliner JA, Harmon RJ.Quasi-experimental designs. J Am Acad Child Adolesc Psychiatry 2000;39(6) 794-6
16. Johnston MV, Ottenbacher KJ, Reichardt CS. Strong quasi-experimental designs for research on the effectiveness of rehabilitation.
Am J Phys Med Rehabil. 1995;74(5) 383-92
17. Shadish WR, Cook TO, Campbell OT. Experimental and Quasi-Experimental Oesigns for Generalized Causallnference. Boston
(MA) Houghton Mifflin Company; 2002.
18. Petticrew M. Commentary: Sinners, preachers and natural experiments. Int J Epidemiol. 2011 ;40(2):454-6
19. Petticrew M, Cummins S, Ferrell C, Findlay A, Higgins C, Hoy C, et al. Natural experiments: an underused tool for public health?
Public Health. 2005;119(9) 751-7.
20. Anonymous. A primer on before-after studies: evaluating a report of a "successful" intervention. Eff C1in Pract. 2002;5(2):100-1.
21. Jaeschke R, Cook O, Sackett OL The potential role of single-patient randomized controlled trials (N-of-l RCTs)in clinical practice.
J Am Board Fam Pract. 1992;5(2) 227-9
22. Guyatt GH, Heyting A, Jaeschke R, Keller J, Adachi JO, Roberts RS.N of 1 randomized trials for investigating new drugs. Control
C1in Trials. 1990; 11(2):88-1 00
23. Guyatt GH, Keller JL, Jaeschke R, Rosenbloom O, Adachi JO, Newhouse MT. The n-of-l randomized controlled trial: clinical
usefulness. Our three-year experience. Ann Intern Med. 1990; 112(4):293-9.
24. Zucker OR, Ruthazer R, Schmid CH. Individual (N-of-l) trials can be combined to give population comparative treatment effect
estimates: methodologic considerations. J C1inEpidemial. 2010;63(12): 1312-23
25. Matowe LK, Leister CA, Crivera C, Korth-Bradley JM. Interrupted time series analysis in clinical research. Ann Pharmacother.
2003;37(7-8): 1110-6.
26. England E. How interrupted time series analysis can evaluate guideline implementation. Pharm l. 2005;275:344-7.
27. Naiman A, Glazier RH, Moineddin R. Association of anti-smoking legislation with rates of hospital admission for cardiovascular
and respiratory conditions. CMAl. 2010; 182(8) 761-7.
28. Masten S. The effects of changing to primary enforcement on daytime and nighttime seatbelt use. Traffic Safety Facts: Research
Note, OOT HS 810 743. Washington (OC): National Highway Traffic Safety Adminitration; March 2007. Available from: www.
nhtsa.gov/people/injury/research!rSF/HS81 0743/810743 .html

The Research Guide: A primer for residents, other health care rainees, and practitioners
29. Tu K, Mamdani MM, Jacka RM, Forde NJ, Rothwell DM, Tu JV.The striking effect of the Heart Outcomes Prevention Evaluation
(HOPE)on ramipril prescribing in Ontario. CMAJ. 2003;168(5) 553-7
30. Jackson PR,Yeo Ww. Cross-over trials-a commentary. Eur J e/in Pharmacol. 1997;52(2): 155-8.
31. Jackson PR,Yeo ww. Cross over trials. Br J e/in Pharmacol. 1996;42(3):401-3.
32. Jones B, Donev AN. Modelling and design of cross-over trials. Stat Med. 1996; 15(13): 1435-46.
33. Grimes DA, Schulz KF.Bias and causal associations in observational research. Lancet. 2002;359(9302):248-52.
34. Sackett DL Bias in analytic research. J Chronic Ois. 1979;32(1-2):51-63.
35. Lang TA, Secic M. In: Lang TA, Secic M. How to report statistics in medicine: annotated guidelines for authors, editors and
reviewers. 2nd ed. Philadelphia (PA): American College of Physicians; 2006. Appendix 5: Sources of error, confounding, and bia:
in biomedical research: p. 449-57.
36. Schulz KF,Grimes DA. Blinding in randomised trials hiding who got what. Lancet. 2002;359(9307):696-700.
Dennis ML, Perl HI, Huebner RB, McLellan Al Twenty-five strategies for improving the design, implementation and analysis of health
,,-
services research related to alcohol and other drug abuse treatment. Addiction. 2000;95 Suppl 3:S281-308.
• An interesting and reasonably extensive discussion of issues to consider to strengthen the design and conduct of health
research.
Fletcher RH, Fletcher Sw. C1inical epidemiology: the essentials. 4th ed. New York Lippincott Williams & Wilkins; 2005.
• A ve¡'y readable and informative text describing the various asoects of clinical epidemiology and research.
Hulley S8, Cummings SR,Browner WS, Grady DG, Newman TB. Oesigning c!inical research. an epidemiologic approach. 3rd ed.
Philadelphia Lippincott Vviiliarns & Wilkins; 2007.
• A readable and informative text discussing the various steps in designing and conducting clinical research.
Streiner DL, Norman GR. POQ epidemiology. 3rd ed. Shelton (CT): People's Medical Publishing House; 2009.
• A concise, informative and very readable "dassic" text providing an overview of epidemiology principies and methods from the
"Pretty Darn Quick" series.
Grimes and Schulz Lancet series
• This is a multi-article series published in The Lancet. Each article provides an informative overview of an aspect of clinical
research and research designo
Grimes DA, Schulz KF.An overview of clinical research: the lay of the land. Lancet. 2002;359(9300):57-61.
Grimes DA, Schulz KF.Descriptive studies: what they can and cannot do. Lancet. 2002;359(9301): 145-9.
Grimes DA, Schulz KF.Bias and causal associations in observational research. Lancet. 2002;359(9302):248-52.
Grimes DA, Schulz KF.Cohort studies: marching towards outcomes. Lancet. 2002;359(9303):341-5.
Schulz KF,Grimes DA. Case-control studies: research in reverse. Lancet. 2002;359(9304):431-4.
68 © 2011 The Royal College of Physicians and Surqso o: r¡;:-.a:¿

Schulz KF, Grimes DA. Generation of allocation sequences in randomized tria.s: nance, not choice. Lancet. 2002;359(9305):515-19.
Schulz KF, Grimes DA. Allocation concealment in randomised trials: defending against deciphering. Lancet. 2002;359(9306):614-8.
Schulz KF, Grimes DA. Blinding in randomised trials hiding who got what. Lancet. 2002;359(9307) 696-700
Schulz KF, Grimes DA. Sample size slippages in randomised trials: exclusions and the lost and wayward. Lancet. 2002;359(9308):781-5.
Schulz KF, Grimes DA. Unequal group sizes in randomised trials: guarding against guessing. Lancet. 2002;359(9310):966-70.
Grimes DA. Uncertainty. Lancet. 2002;360(9341): 1242.
Schulz KF, Grimes DA. Sample size calculations in randomised trials: mandatory and mystical. Lancet. 2005;365(9467): 1348-53.
Grimes DA, Schulz KF. Compared to what? Finding controls for case-control studies. Lancet. 2005;365(9468): 1429-33.
Grimes DA, Schulz KF. Refining clinical diagnosis with likelihood ratios. Lancet. 2005;365(9469) 1500-5.
----
Schulz KF, Grimes DA. Multiplicity in randomised trials I endpoints and treatments. Lancet. 2005;365(9470):1591-95. -
Schulz KF, Grimes DA. Multiplicity in randomised trials 11: subgroup and interim analyses. Lancet. 2005;365(9471) 1657-61.
Grimes DA, Hubacher D, Nanda K, Schulz KF, Moher D, Altman DG. The Good Clinical Practice guideline: a bronze standard for
clinical research. Lancet. 2005;366(9480) 172-4.
Grimes DA, Schulz KF. Uses and abuses of screening tests. Lancet. 2002;359(9309):881-84. Erratum in: Lancet.
2008;371 (9629): 1998.
SUMMARY CHECKLlST
o Consider your research question and the methods found in this chapter. Which design or designs would be best to
choose for your study?
O Reviewing your design, consider how you will address generalizability and internal validity. Consider potential sources
of error and bias. Refine your design
O Discuss your selection with your research supervisor(s) and/or a methodologist. Refine your design.

.. -
I
4
•
•
•
•
•
70 © 2011 The Royal College oí Physicians and S'Jf_~
•
t
10
Developing and assessing health measurement
items and scales
David L. Streiner, PhD, CPsych
ILLUSTRATlVE CASE
A resident in pediatrics is working in a pediatric epilepsy clinic and realizes that effective management of these patients
and their families involves more than simply controlling seizures, beca use the impact of epilepsy on quality of life must also
be taken into account He does a literature search and finds several scales that try to measure qua lit y of life, but he is not
sure how to choose among them. He'd like to know what features he should look for in deciding which, if any, of these
instruments to select
•••••
• This chapter outlines what you rhings beco mes more cornplicared when we rry ro assess
should know abour evaluating scales used ro assess various attribures such as moods, feelings, arritudes, beliefs or orher
aspects of health and behaviour, such as quality of life, interna] states that we can't observe direcdy. How can we be
mood and pain. Ir is aimed borh at people who need (Q sure that a scale is really measuring a person's qualiry oflife,
select an instrument ro use in a research project, and at rather than, say, depression or pain, or thar a test of a
consumers of research who wanr ro know whether a scale patienr's knowledge about rhe management of his condi-
used (Q garher data is a good one. The chapter will review tion is in fact an accurate reflection of what he needs (Q
how scale iterns are developed and how ro determine know?

whether a scale as a whole is reliable and valid for the use ro These rypes of attributes, which are not direcdy observ-
which ir is put. The chaprer also provides a launching point able, are general!y referred ro as hypothetica1 constructs'
for rhose who want ro collaborare wirh orhers in develop- (or latent variables). That is, we do not see anxiery or inrcl-
ing new scales. ligence per se; al! we can observe are rheir ourward manifes-
Ir is relatively easy ro measure physical attribures, such as
heighr, blood pressure, creatinine levels, and so íorth, Mrer
CHAPTER OBJECTIVES
al!, if we stand a person on a scale there is no doubr that we
are measuring his or her weight, and the company that After reading this chapter, you should be able to
manufactures a uiglyceride test guaramees rhar rriglyceride • describe how scales are constructed;
levels are in facr what their test rneasures, and nor sorne- • discuss the properties of a scale, such as reliability and
rhing else. Whar we require of rhese measurements is thar validity; and
rhey be accurare, reproducible and feasible. However, • evaluate existing scales.
KEY TERMS
Construct validity Inter-rater reliability Psychometrics
-l
Criterion validation Intra-class correlation coefficient Reliability
Cronbach's a Latent variables "Satisficing"
End-aversion bias Negative predictive value Social desirability bias
Hypothetical constructs Pearson correlation coefficient Test-retest reliability
Internal consistency Positive predictive value Validity
-.-_._--------------------------------------- _-
.---_ ..

rations, For example, if we see that a patient is pessimisric pain-then rhe effects of pain on rhe patient's life woulc
abour rhe furure, rhat her sleep is disturbed, that she has losr not be covered. Conversely, if the developer sees pain in ;
interest in doing things that she used to enjoy, has decreased more holisric way and believes that its efiecrs colour mos
libido, and has been having choughts of suicide, we hypoth- aspects of daily life, then quesrions about its impact on nor-
esize rhat all of these are relared and arise from an underly- mal activities would be included.
ing state, 01' construct, we call depression. Similarly, we view For you, as a reader, user or developer of a scale, this mar-
a large vocabulary, knowledge about areas ourside one's spe- ter of defining rhe construct has three implications. First, it
cialry, skill at problem-solving, and speed in solving puzzles means that you muse have a clear idea of the construct in
as refl.ecrions of something we call intelligence. Again, note mind before you begin. In your conceptualization of the
that we do not see the depression or intelligence direcdy, construcr, what should and should not be included? Ir will
but only phenomena that rhey are hyporhesized ro affect. be of great help to you later on if you pause to write down
Manyareas that we want to measure in healrh-s-pain, qual- these areas for inclusion and exclusion, and in as much
iry of life, mood, satisfaction with care, activities of daily detail as possible.
living, caregiver burden, and so on-are in fact hypotheti- Tbe second implication is rhar the scale's author should
cal constructs. do rhe same rhing in the arricle rhar introduces the tool to
Over the past century, researchers prirnarily in rhe fields the world. Tbe result of these rwo steps-having a clear idea
.. - of psychology
techniques
and educarion have developed
that allow us to develop scales to rap-rhat
an array of
is,
of rhe construcr as ir applies to your research or clinical pur-
poses, and understanding how rhe construct is applied in a
measure or assess-these constructs relatively accurately. To given scale-might imrnediarely eliminare sorne scales
appreciate rhese rnethods, though, requires cerrain skills frorn your considerarion. For example, if your model holds
along with a particular vocabulary. In rhis chapter, we will rhat qualiry oflife is a purely subjective phenomenon, and
review those techniques, albeit in a non-rechnical way. By that people who are severely impaired can still enjoya high
che end, you should be able ro critique anides that present quality of life,? then you would not consider using a proxy
new scales. However, just as you wouldn't perform an scale, such as rhe EQ- 5D, 3 which rates only what people do
appendecrorny until youve had proper training and have 01' seem ro experience.
had an expen looking over your shoulder for a sufficient The third irnplication is that, if you are contemplating
nurnber of cases, developing a new scale is not something developing your own scale, then an explicit starernent of
you should uy on your own. With thar said, let's go through what should be included and excluded is mandarory and
rhe sreps required to bring a new scale to life so rhat you'Il should be reported in your first paper abour your new scale.
know what to lcok for. Aírer you have a model, either explicidy (ideal) or
implicitly (less than ideal), it's time ro begin drafring iterns
for the scale. In realiry, few scales are developed de novo;
Developing the items
rhey borrow from existing scales ro varying degrees. It's not
The fact thar hyporhetical constructs are not direcdy easy ro write good items, and rhere are a lirnired number of
observable has implications for whar items do or do not ways ro ask if a person is feeling sad, for insrance, or experi-
appear on a scale. It's obvious that a pain scale, for example, encing pain. So, "borrowing" items from a number of scales
should measure intensiry, duration and frequency, but ro construcr a new one is quite common.
what else can and should ir cover? Should a pain scale also Ar times, however, one must srart from scratch beca use
tap the degree to which rhe pain inrerferes with work or lei- no adequate instrument exists. One example is me
sure activities? Similarly, should a sede of aggressive behav- CHEQOL-25,4 which assesses, from their own perspecrive,
iour in patienrs with dementia tap only episodes of a the qualiry of life of children with epilepsy, We began with
parienr striking someone else, 01' should ir be broadened ro various qualirarive research merhods, such as focus groups
indude verbal ourbursrs, spitting, and so on? To use a with rhe children and rheir parems, ro elicir whar me chil-
phrase rhat wiIl recur ofien in this chapter, it all depends. dren rhought was important." and, based on ryped rran-
Whar ir depends on is rhe test developer's rheory or under- scripts of the numerous sessions, exrracred Iive or SLX broad
sranding of rhe construct. If "pain," to rhe scale's aurhor, domains (rhe number varied berween me child' an par-
refers only to the subjective experience-the sensation of enr's version). The team rhen wrore a number o ire 2[
72 © 2011 The Royal College 01 Physicians and Su'<;eo-s e' :::;;:-¿:¿

7O Deve/oping a das essing hea/th measurement items and sea/es
we felt rapped each domain-far more than we would 1he man}' kinds ofbias rhat can influence how a person
eventually use, for reasons we'll discuss below-and werit answers an irern are far toO numerous for all to be rnen-
back ro the original parricipanrs ro ask: (1) Do rhe irerns tioned here, but a more complete lisr is provided by 5treiner
seem ro rap the dornains? (2) Do any items appear irrele- and orrnan." The mosr common biases are social desir-
vanr? (3) Did we miss any important ones? and (4) Does ability bias, end-aversion bias and "satisficing." People
rhe wording seem appropriate? want orhers ro rhink well of rhern, and ofren give responses
rhar are socially desirable-"I floss my teeth three rimes a
day," "1 never drink ro excess," or "1 always rake all of my
Checking the items
medicarions"-somerimes knowing rhat rhey are disrorring
Whenever you write new iterns, or use a scale in a popula- rhe trurh a bit, and sornetime unaware rhar they are doing
tion in which ir hasn't been rried befo re, ir is irnportant ro so. In eirher case, you can'r trust rhe responses to be accurare
do so me "cognirive inrerviewing'" (1' 127-9) wirh a sarnple of reflecrions of the person's actual behaviour. As rhe name
respondenrs. Anywhere from 5 ro 10 people are asked, with
respecr ro a given itern, "Tell me how you arrived ar your
irnplies, end-aversion
avoid the extremes
bias refers ro people's tendency
of scales (usually labelled "Always,"
to
e
n:
answer," or "Rephrase rhe irern in your own words." 1he
answers may alert the developer to difficulries thar respon-
"Never," "Extrernely," and so on) because rhey can rhink of
an exceprion. The effecr of rhis is to reduce the number of -
v.
dents might have in undersranding the wording or intent of
the item. Experrs in the field are also asked to rate the rele-
vanee of each itern to the construct, and rhose iterns deemed
response options rhat are acrually used, which makes rhe
test less reliable (more about reliabiliry larer). "5arisficing"
means giving an answer that sarishes the minimum
--
-.¡¡
to have low relevance are eliminated. Finally, it's very use fui demands of the quesrion-giving a response-but one thar
to construct a marrix, assigning a row to each itern and a isn't the besr reflection of rhe person's arritude or stare, or
column to each rherne you identiíied earlier as irnportant. that isn't even accurare. This ofren happens when the scale is
You should then be able ro match each itern with one of the roo long, or rhe person finds rhe irern roo difficulr ro under-
rhemes, and every therne should have a nurnber of iterns stand and opts for the first or lasr response alrernative in a
associared wirh ir.6 If you can't, then you should either reject list or finds so me orher easy way out.
the scale (if you were rhinking of using ir) or go back ro rhe Once iterns have been screened with these biases in
drawing board (ifyou are developing a scale). rnind, rhe test developer should be lefr wirh far more irerns
Nor every scale rhar has been developed has been devel- rhan are necessary or feasible to indude. The next rask is ro
oped wel!. So, wherher you are planning to use someone weed out iterns that are redundant, don't pertorm adequare-
else's scale or to write your own, you must be aware of what ly, or don't fir nicely into one of the domains rhat were pos-
makes an itern good or bad. Beware rhe "double-barrelled" tulared ar the outset, Wherher you're evaluaring how the
ítem that rolls rwo quesrions into one. For exarnple, an irern test developers did this or, more crucially, rhinking of devel-
on a scale of sarisfacrion with care may be written as, "My oping a scale yourself, the besr advice is, "Don't rry rhis at
doctor was prompr and courreous." The "and" in rhe sen- horne." You can read more about the techniques rhar are
rence is a dead give-away thar rhis is a double-barrelled ques- used for these purposes, such as facror analysis" and itern
tion. How should rhe patienr answer if rhe doctor was response theory," but these are jobs for experrs in rhe fields
prompr but rude, or courteous bur late? The problem is rhat of statistics and, especially, psychornetrics.
different people will arrive at an answer differently: sorne will
answer "False" because borh pares aren'r true, while others
Reliability
will answer "True" because at least one pan is, and you'll nev-
er know which respondents favoured which logic. Orher Once the iterns have been chosen and a scoring scheme
double-barrelled questions are more subrly so, especially in developed, ir's rime ro determine wherher rhe scale as a
scales of healrh-related qualiry of life. Take the item, "My whole performs properly. This "proper performance" has
arthritis gets in the way of my enjoymem of sporrs." Does rwo componems: reliabiliry and validiry. Over the years, a
"False" mean that rhe arthritis doesn'r ger in the way, or that number of orher rerrns have crepr inro the Íiterature (espe-
rhe person never enjoyed SPOrts ro begin wirh, and so the ciaUy in medicine) as supposed synonyms of reliabiliry and
arthritis has no efiect? Again, you'll have no idea. validiry, such as "reprcducibiliry," "precision," "accuracy,"

"stabiliry," and so on. Later, we'll see why none of rhese is studenr, ir may be that one has inreracted with rhe student
satisfactory:" for now, suflice ir to say rhat you should srick more rhan the orher, or in a different setting. Thus, it can be
wirh rhe original terrns, "reliabiliry" and "validiry." We'lI difíicult to differentiate unreliabiliry from unique knowl-
discuss reliabiliry in this section, and validiry in the next. edge. But, even when rhe latter is at play, ir imposes limita-
Ar rhe simplest level, reliability refers to rhe srabiliry of a rions on how a scale can be used and who can complete ir.
scale's scores. This can-and should-be assessed in a nurn- For insrance, if parents' reporrs differ from those of teach-
ber of ways. If rhe scale is self-adrninisrered (i.e., rhe person ers, then either one or the other rnust be used exclusively,
fills ir in himself or herself), then we are interested in test- or, if both are used, rhe srudy must recognize thar they
retest reliability. Thar is, if the person compleres the scale reflect different-and complementary-views of rhe child.
in one way today, will he or she complete it rhe same way at The most common way to measure resr-retest and inter-
a later time (assuming rhat whatever ir is we're measuring rater reliabiliry is to correlare rwo sets of scores with a Pear- I
hasn't changed)? The rationale is that, if rhe respondent son correlation coefíicienr.? However, a berter statistic to
hasn't changed but his or her score on the scale does, then use is the intra-class correlation coefficient (1CC), G
we can't be coníidenr rhat we're getting accurate results because the latter "penalizes" rhe raters if one is consistendy
eirher time; rhis would be akin to trying to measure a piece higher (or lower) than rhe other, or if they systernarically
of wood wirh a rubber ruler-we'd always get a different repon higher (or lower) scores the second rime they rate a
answer. Usually, rhe interval berween rhe first and second scale. For exarnple, if rater A consistently evaluated resi- I
"'- adrninisrrarions of the scale is lOto 15 days. If it were any denrs one point higher than did rarer B, the Pearson corre-
shorrer, then we'd have to be concerned that the person is lation would be 1.0, and rhe ICC would be somewhat I
simply recalling what he or she put down che firsr time and lower. Both the Pearson correlarion and the ICC can range
is repeating it, rather than responding to the questions from O to 1, wirh O reflecting no reliabiliry and 1 showing
again; if the inrerval were too much longer, then the proba- perfect reliabiliry, A rough rule of thumb is that correlations
biliry would increase that rhe atrribure itself will have below 0.70 are unacceptable, while those in the 0.70s are
changed. Needless to say, rhe nature of what the scale is fine for the early srages of research, those in the 0.80s are
measuring affects the interval. Sorne constructs, such as acceptable for more matute research areas, and correlations
pain, ma)' change over shorter periods of time; orhers, like over 0.90 are required for scales used for clinical or deci-
introversión, change slowly if at all, ancl so the test-retese sion-making purposes abour individuals."
interval can be adjusted to account for this. A rhird index of reliabiliry is sornewhat difFerenr from
Sorne scales are filled out not by the person being assessed these rwo and looks ar the interna] consistency of rhe
but by someone else. For example, supervisors evaluare scale--rhar is, the degree ro which rhe irems are correlared
their studenrs, parents act as surrogates for young children: wirh one anorher. The rationale is thar most scales are
older children are often surrogates for elderly, cognitively designed to rap one attribute, such as depression, pain, arti-
irnpaired parents; psychiarrisrs or nurses may assess the tudes toward end-of-lífe care, and so forrh. Consequently,
mood or the rhought processes of patients who do not all of the ii:ems should relate to this one construcr: rhar is,
appear to be able to accurately evaluare their own status; the scale should be unidimensional. If the iterns aren't cor-
and so on. Here, we are inrerested in the agreement between related (i.e., if internal consisrency is low), then we can't be •
rwo or more raters, or the inter-rater reliability. Again, rhc sure exacdy whar the scale is measuring, because ir's likely
rarionale is faidy self-evident: if rwo observers don't agree rapping a number of differenr areas. Sorne scales are •
about what rhey've seen, then we likely can'r trust either of designed ro be mulrifacrorial-rhac is, ro cap difFerenr
them. Inter-rater reliabiliry is relarively easy to evaluare in aspects of a person or an attribute. For exarnple, a "sarisfac- ti
research settings, where both observers can be trained, if tion wirh care" scale may be subdivided to measure sarisfac- 1
ncccssary, ro use rhe scalc, and both have equivalent intor- rion wirh physicians and rhe nurses (who are usually rar d •
marion about the person being assessed. However, inter- quite positively), wait rimes (always too lona, che food ,
rarer reliabiliry can be difíiculr ro achieve in practice. If a (always bad), and rhe faciliry (parking Spots are always roo
morher and farher don't fill out a scale about rheir child in a
similar way, does that mean that the scale is unreliable, or
few and too expensive). In chis case, we would be interested
in rhe internal consistency of each of che individual sub-
•
that, for exarnple, each parent sees the child in different cir- scales, rather rhan of the scale as a whole.
cumstances? Similarly, if rwo supervisors disagree about a
74 © 2011 The Royal College of Physicians and Surgeo o' C,,--==c 4

•
7O Deve/oping an . assessing hea/th measurement items and sea/es
er component LO reliabiliry, and that is the degree ro which

Even so, rhe notion that a scale should be unidimension-
rhe scale can spread people out on a continuum. Despite
al is a bit slippery, because sorne constructs are more horno-
Thomas Jefferson's declararion that "AlI men are crea red
geneous than others, and that homogeneiry depends on
equal," scales are based on a different premise, which is rhat
how finely grained we wanr the scale to be. For insrance, if
no rwo people are equal, and it's the job of the scale ro dif-
we want a scale ro sirnply detecr whether a patient in a fam-
ferentiare among rhern. This simple staternent leads to
ily physician's office is depressed, it is sufficiem ro rhink of
sorne seemingly counrer-intuitive conclusions. Imagine
depression as a single, unirary coristruct and ro use a scale
that we are evaluating 20 trainees using a rypical three-
rhar gives us one rotal score. However, if we were working
point scale ofUnsatisfactory / Satisfacrory / Superior. Now,
in a clinic that specializes in depression, we may want a
when's rhe last time you heard of anyone raring a trainee as
scale rhat looks at different aspects of depression: rhe affec-
"Unsatisfacrory"? (Even Jack rhe Ripper would likely get a
tive component, the behavioural component, the cognitive
Satisfacrory for technical skills.) In custornary practice it's
component, and so fonh. In this case, we would want rhe
interna! consisrency of the entire scale to be lower rhan for
not an option, which means rhat "Satisfacrory" is now the e
~ro
the family doctor's screening tool.
Also be aware that sorne scales are not meant to be inter-
signal to rhe trainee rhat he or she had betrer watch out-
and even this is rarely used. 'TIlLlS, virrually everybody is _.
VI
nally consistent, much like checklists of signs or symptoms
rated as "Superior." So, in rared al! the trainees as Superior c..c
that may or may not occur together.10 A prime example of
and so did you, our inter-rater agTeement would be 100%.
Similarly, if we rated thern again rwo weeks later, we'd likely
::s
-,
this is the five-item Apgar scale, 11where ir is quite conceiv-
give the rrainees equally high marks, and the test-retest
able for a neonare ro score low on one itern (e.g., muscle
agreement would rherefore be 100%. However, the reliabil-
tone) but not on the other four. In rhis case, we would not
ity of our scale would be O, because there is no variabiliry
bother to look at internal consistency. In medical circles,
among rhe people we're raring! Consequently, rerrns such as
developing this rype of scale is sometimes referred to as
"agreemenr" and "reproducibiliry" are misleading because
"clinimetrics,"12 but, for a number of reasons, 1 recommend
they do not capture this aspect of reliabiliry: the abiliry of
avoiding this rerrn.P:"
the scale ro repeatedly discriminate among people.
The rnost widely used sratistic for measuring internal
The fact that rhe variabiliry of scores affects rel iabiliry
consistency is Cronbach's a (alpha),'? whose value, like
has a second implication, which is that reliability is not a
that of rhe Pearson conelation and rhe ICC, can range
fixed property ola test. 1hat is, ir's wrong ro talk abour the
from O ro l. Cronbach's a is probably the most widely used
reliabiliry of a test, as if it were a fixed and immutable quan-
index of reliabiliry, because the scale needs to be given only
tiry, (Indeed, many journals in rhe field of psychometrics-
once ro calcula te ir, as opposed to rwo or more times, as in
scale development-auromatical!y rejecr any article
rhe case ofinter-rater or rest-retest reliabiliry. Unfortunate-
submission that refers ro reliabilit)' as a propen)' of rhe
ly, the populariry of this test is not matched by its useful-
test.) Rather, reliabiliry is an interaction among rhe scale
ness. The major problem is rhat rwo facrors inBuence rhe
and the group being assessed and the circurnstances. An
magnitude of a: che internal consisrency of the scale (which
insrrument that may be reliable when used with one group
is good), and rhe number of irerns (which is bad). \'V'hat
of people may be much less reliable when used with a group
rhis means is rhar if a scale has more than 15 or so irerns, a
of people who have a rnuch smaller range of scores. What
is guarameed ro be high even if rhe irerns are not internally
this means for you as a user of scales (or of research papers
consisrenr." So, don't believe rhat a long scale (> 15 irerns)
that repon data gathcred by means of scales) is that )'ou
is reliable if the only cvidcncc reponed is coefficient a; rely
should never accept at face value rhe author's staternent
on orher forms of reliabiliry, or orher indices of interna]
that "the scale has been shown to be reliable." You have ro
consistency, such as rhe average itern-total correlarion (how
ask wherher rhe variabiliry in scores that you expect to see
much each itern cGrrelares with the total score) and rhe aver-
in your parient population is the same as that of the author's
age inrer-item corrclation (how much the iterns correlare
group. Ifyou expect rhat the variabilirywill be less (because,
with each orher).
for instance, you're dealing with inpatients or outpatients
If this were the exrent of reliabiliry, then terrns such as
only, rarher than with borh, as in rhe paper), rhen the reli-
"reproducibiliry" and "consistency" would seem to be legit-
abiliry in your group may be lower-even much lower+-
imare synonyms insolar as they capture the degree ro which
scores are replicable across time or rarers. But chere's anoth- rhan in rhe original article.
75
© 2011 The Royal College 01 Physícíans and Surgeons 01 Canada
Validity wrongly identify a healthy person as being ill. Consequenr

Assuming rhat the scale has shown adequate reliabiliry with ly, we are inrerested in rwo propertíes of diagnostic tests: the
a group of people similar to rhe one you intend to study, we positive predictive value (PPV), which is the proportior
are ready for the next step in determining whether the scale of positive test rcsults that are actually from people whc
is performing properly: that is, ro assess its validity, The have the disorder being rested for; and rhe negative predico
conceptualizarion of validiry has changed over the past few tive value (NPV), which is rhe abiliry of the test to rule OUI
decades. If you read a textbook from rhe 1960s (or recent a disorder when ir is absent. When the prevalence of the
anides by people who haven'r kept up with the field) , you'll disorder is high, rhe PPV is often very good while rhe NPV
find definirions along these lines: "Validiry rells us if the is poor. Conversely, when the prevalence is low (which is
scale is measuring whar we think ir is." That is, validiry (like the more common situation), the NPV is usually quité
reliabiliry) was seen as a properry of the instrument thar, excellent while the PPV is poor, resulting in a high propor-
once established, was esrablished for good. These days, tion offalse positives."
validiry is defined as "the degree of confidence we can place In both of these exarnples, it is not the test that is valid or
on the inferences we make about people based on rheir invalid, bur the use ro which it is put, which depends on the
scores from that scale.?" (p 251) In other words, the focus is circumstances (example 1) and the population (example 2).
not on the scale per se, bur rather on the scores produced by As with reliabiliry, rhe leading psychometric journals will
rhe scale. This is nor simply a semantic difference. Rather, it reject any anide whose aurhors daim, wirhour making any
"'- ernphasizes the fact that, again in parallel to the conceptu- such qualificarion, that they have esrablished the validiry of
alization of reliabiliry, validiry is an interaction among the a test.
scale, the people completing ir, and rhe circumstances There's a second way in which the conceprualizarion of
under which ir is filled out. Let's look at sorne examples of validity has changed over the years. Older texrbooks (and,
how validity is context-dependent. again, recent anides by people oblivious to developments
Many personality assessrnent tools, such as the Minne- now over 40 ycars old) spoke of different types of validiry,
sota Multiphasic Personaliry Inventory-Z (MI\lfPI-2), 17 More specitically, they rcferred ID the "three es" of content,
have sub-scales that determine respondents' "test-taking criterion and ccnstruct validiry. Each of these was oíten bro-
artitude": were they under- or overstating the extent of their ken down inro subrypes, and batrles raged as to wherher ro
clifficulties; were they being defensive about aclmirring cal! a given study an cxample of say, "discrirninant validiry"
frt.ults;were they presenting themselves in an unrealistically or "concurrenr criterion validiry." Now, though, because
positive light; and so forth. Few people would endorse an validiry is concerned with the interpreration and meaning
irern such as, "1 have never rold a lie," for example, unless of test scores, everything is seen as a form of construcr valid-
rhey were trying to creare a positive image of thernselves. ity. 'We can talk abour criterio n validation (rhis is ene
\X/lien used to assess hospital inpatients or those seeking merhod of establishing construct validity) , but nor crireri-
hclp from a health care provider, elevations on these sub- on validí ry, as if i t were a different animal from other t)'pes
scales are seen as pathological and as reBecting attitudes of validiry resting.
rhat can get in rhe way of successful treatment. However, ro So, whar do we mean by construct validiry? Let's go back
take one example, when the MMPI-2 is adrninisrered to to the beginning: most of what we are trying ro measure are
people who are in the midst of custody and access disputes hyporherical construcrs, or "consrructs" for short. Nor only
with an ex-spouse, high seo res are inrerpreted in a much do we hypothesize that the construct exists, but we also
more positive light: rhe person should be trying to mini- have theories about rhe construct irself. These needn't be
mize his or her faults, and attempting to portray hirn- or grand rheories thar rival rhose of evolurion or relariviry in
himself as psychologically healrhy. Indeed, if the person's rheir scope and level of development, but rather a series of
score were nor elevared, we'd be concerned about his or hcr hyporheses abour whar might iníiucnce rhe coristrucr, and
appreciation of social norms and mores. how the construct relates to other phenomena. For exarn-
A second example is the well-known fact thar the abiliry pie, our "theory" about pain would lead ro hypotheses such
of diagnostic tests (O rule in or rule out disorders is depen- as: (A) patients in an arthritis c1inic should score higher on a
denr on rhe prevalence of the disorder in the population pain scale than those arrending a dinic for dermatitis; (B)
being tested." Diagnostic tests can (and always do) make scores on a pain scale should decrease afrer an analgesic is
(WO rypes of errors: they can miss true cases, and they can administered; (e) parients who repon chronic pain should
76 © 2011 The Royal College 01 Physicians and Surgeons o' Car¿aa

7O Deve/oping ano assessing hea/th measurement items and sea/es
also score higher on a depression scale than people who are direcrly. Consequendy, unlike lab tests or rulers, there is no
nor in chronic pain; and (D) scores on a pain scale should perfecr gold standard against which we can evaluare rhem.
be unrelated ro intelligence. For rhis reason, rerms like "precisión" or "accuracy" should
None of these hyporheses is world-shattering or would be reserved for lab tests, and are imprecise and inaccurare
lead ro a Nobel prize in physiology or medicine, bur rhey do when used with rhe eypes of scales we've been discussing.
point ro ways ro determine whether the scale is performing Use "validiry"
as ir should. Ir's worrh noting thar srudies of validiry are
much more variable in terms of their design and analysis
Utility
rhan are reliabiliry studies. For reliabiliry, we simply adrnin-
ister rhe scale once (for interna] consistency) or rwice (for The Iinal arrribure you should be interested in is rhe uriliry,
inter-rater and rest-reresr reliabiliry), and calculare the or usabiliry, of rhe scale: rhat is, how feasible is ir for you ro
appropriate sratistic. 1he design and analysis of validiry use in your serring? 1he Ersr consideration is copyright.
srudies, on che other hand, are dicrated by rhe quesrion. For
hypothesis A, for example, we would give the scale ro rwo
There are many scales f10ating around in hospital and uni-
versiry sertings rhat really shouldnt be. If a scale is to be used
e
·ro
groups of people-those
dermarology clinic-and
in an arthritis clinic and rhose in a
compare the scores using a t test.
Wich the second hypochesis, we would adrninister che scale
for research purposes, you can ofren ger permission from the
developer ro use it free of charge, alrhough some. authors
request rhat you ler thern know of any publications that
-
Vl
ro rhe same people, before and alter analgesic medication,

and use a paired t test (or, if we were being fancy, we would
have a placebo group, and then use a "group by time" analy-
emanare from the research (a reasonable request). When ir is
ro be used for clinical purposes, though, it is neirher ethical
nor legal to sirnply photocopy rhe scale for your own use.
-
sis of variance). Hypotheses (C) and (D) are correlational in The next aspecc ro look at is how rhe scale is to be corn-
nature, and so we would have the participanrs EII out rwo plered. Self-adminiscered scales require, in mosr inscances, a
quescionnaires. We hypothesize a priori in (C) thar there Grade 6 reading level in English (the level of the average
should be a modera te correlation berween pain and depres- high school graduate; asad comrnenrary on our educational
sion, and in (D) thar there should be no correlacion berween sysrern), and so might not be pracrical ro use in cercain
pain and intelligence. groups, such as recent immigrants. Nor would a self-adrnin-
If the results obcained are as hypothesized, then we will istered test be practical in a fracture clinic, where many
know that our theory is valid and rhar the scale has been parients have rheir arms in a case. Other aspects to consider: .
validared ro some degree in ibis population. However, if the Is the scale self-explanarory, or will the respondenr need ro
results are nor as we expecred, rhen (1) rhe rheory is righc be rold how to complete ir; if the latrer, are there sraff avail-
but the scale isn't working as expecred, (2) the scale is Ene able to do rhis? If people are ro EU ir out while waicing for
but the rheory is wrong; or (3) both the theory and the scale their appointrnenr, where exactly will this be done? Is rhere
are Bawed. 1he problem in this latter situation is rhar we space available? Is there a place where rhey can wrice? Is it
won't know which of che rhree explanarions is rhe right one. quiet and free from distractions? 1hese seemingly small
You should also note that there is no end ro the hypoth- things can pose irnportant obsracles in the real world.
eses thar can be generated and tested using a given scale. In If the scale is ro be used by one person ro rare another,
chis regard, validiry testing is never finished. Another factor somewhat different questions need ro be answered. In
rhat contribures ro rhe never-ending rask of validarion is many cases, those administering a scale will need some
rhat, even when a scale is developed for one group or wich degree of rraining to know what the response options mean;
one purpose in mind, researchers will always be interesred if rhis is rhe case, it will require resources in the form of
in expanding its reach, so that ir can be used wirh different sorneone who knows rhe scale and can do rhe training, rime
patienr groups or under differenr circumsrances. Because, for che ratee, ro receive thc craining, possibly someone ro
as we've saíd, validiry is not an inherent properey of a scale, acr out different behaviours ro check on the rarers' reliabili-
but rarher depends on rhe scale and the targer group and ey, and so on.
che assessment si ruation, we have ro re-eval uare validi ey The next consideration is rime. You should allow at least
whenever any of rhese facrors change. 30 seconds per irern, and so even a 20-item scale can take
As mentioned at the beginning of rhis chaprer, rhe con- sorne respondenrs about 10 minutes ro complete. This is
structs rhat rhese scales are trying ro measure cannot be seen rarely an issue in a research contexr, where parricipanrs

come in just for the study, But, if you are using patients and evaluation. In 2010, rhe [ournal 01 Psychosomatic Research
asking thern to complete other scales for a validity study, began a series of articles devored ro critiquing scales in a
rhe battery of tests can seem to thern to be an assault and number of areas such as depression, anxiery, quality of life
battery. and apathy." If rhe scale you want ro use is not in any of
Finally, how is the scale scored? For most, this is simply a those resources, the Summary Checklisr at the end of the
matrer of adding up the number of items that have been chapter will guide you rhrough the process of evaluation.
endorsed, or rhe scores for each oprion. Other scales, pri-
marily those used in personality assessment, may have corn-
Conclusion
plicated scoring algorithms and can be scored only by
computer. 1his can limit rheir pracricabiliry, depending on Once we leave the realm of direct physical measuremenr
the resources available for your research. and try ro measure phenomena such as attirudes, behav-
iours, feeling states and quality of life, we rnust use scales
that tap rhese constructs. Developing a good scale-one in
Finding and evaluating sea les
which rhe items are clear, unambiguous and free from bias,
.- Fortunately, ir's not necessary to dig into the often arcan e and that as a whole is reliable and valid for the popularion
literarure ro find out if a scale is ready for prime time. Two under study-is a time-consuming task, Reliability implies
=- excellent compendia of scales are listed in rhe Additional

Resources section at the end of this chapter. 1hey repro-
thar, if the person hasn'r changed,
response under various circumstances
we will get che same
(differenr time, dif-
duce many frequently used scales (but sce the caveat in rhe ferent rarers) and that rhe scale is mcasuring only one con-
preceding section before you blithely photocopy the pages) strucr, Validiry implies that we can place sorne meaning on
and, more imporrantly, critique rheir psychometric proper- the score and use it ro tell us something about the respon- ,
ties and utiliry, One of the appendices in Health Measure- dento Finally, ro be useful, any scale must be feasible ro use
ment Scales (also listed in Additionaí Resources) directs you wirhin rhe constraints of the assessment situation. • e
ro other, less widely used scales and, in sorne cases, ro their
REFERENCES
1. Cronbach U, Meehl PE.Construct validity in psychological tests. Psychal Bull. 1955;52(4):281-302.
2. Albrecht GL, Devlieger PJ.The disability paradox: high quality of life against all odds. Sac Sci Med. 1999;48:977-88.
3. Tamim H, McCusker J. Dendukuri N. Proxy reporting of quality of life using the EQ-5D. Med Careo2002;40:(12): 1186-95.
4. Ronen GM, Streiner DL, Rosenbaum P; Canadian Pediatric Epilepsy Network. Health-related quality of life in children with
epilepsy: development and validation of self-report and parent-proxy measures. Epilepsia. 2003;44(4):598-612.
5. Ronen GM, Rosenbaum p. Law M, Streiner DL. Health-related quality of life in childhood disorders: a modified focus group
technique to involve children. Qual Life Res. 2001;10(1)71-9.
6. Streiner DL, Norman GR. Health measurement seales: a practical guide to their develapment and use. 4th ed. Oxford: Oxford
University Press; 2008.
7. Norman GR, Streiner DL. Biastatistics.· the bare essentials. 3rd ed. Shelton (CT): BC Decker; 2007.
8. Streiner DL, Norman GR. "Precision" and "accuracy": two terms that are neither. J Clin Epidemial. 2006;59(4):327-30.
9. Nunnally lC, Bernstein IH. Psychometric theary. 3rd ed. New York: McGraw-Hill; 1994.
10. Streiner DL. Being inconsistent about consistency: when coefficient alpha does and doesn't matter. J Pers Assess.
2003;80(3):217-22.
11. Apgar V. A proposal for a new method of evaluation of the newborn infant. Curr ResAnesth Analg. 1953;32(4):260-7.
•
78 © 2011 The Royal College of Physicians ano Surqeors c= ara::¿

4
7O Deveiopim; ero cssessing hea/th measurement items and sea/es
12. Feinstein AR. Clinimetrics. New Haven (CT): Yale University Press; 19 7.
13. Streiner DL. Clinirnetrics vs. psychornetrics an unnecessary distin tion. J Clin Epidemiol. 2003;56(12): 1142-45.
14. Streiner DL. Test development: two-sided coin or one-sided M6bius strip I J Clin Epidemiol. 2003;56(12): 1148-9.
15. Cronbach LJ. Coefficient alpha and the internal structure oí tests. Psychometrika. 1951; 16(3):297-334.
16. Cortina JM. What is coefficient alpha? an examination of theory and applications. J Appl Psychol. 1993;78(1 ):98-1 04.
17. Butcher JN, Dahlstrom WG, Graham JR, Tellegen A, Kaemmer B. The Minnesota Multiphasic Personality Inventory-2 (MMPI-2).
manual for administration and scoring. Minneapolis (MN): University of Minnesota Press; 1989.
18. Meehl PE, Rosen A. Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychol Bull.
1955;52(3) 194-216.
19. Streiner DL. Diagnosing tests using and misusing diagnostic screening tests. J Pers Assess. 2003;81 (3):209-19.
20. Keszei AP, Novak M, Streiner DL. Introduction to health rneasurement scales. J Psychosom Res. 2010;68(4):319-23.
•.
Fischer J, Corcoran K. Measures for clinical practice and research: A sourcebook. Vol. 7. Couples, families and children. 4th ed.
New York: Oxford University Press, 2006.
-
•••
•••
"'"
__ - Measures for clinical practice and research: A sourcebook. Vol. 2. Adults. 4th ed. New York: Oxford University Press, 2006. ""
• These two volurnes are similar to McDowell's book but cover a rnuch wider scope, including marital functioning, mood and
development.
McDowelll. Measuring health: A guide to rating scales and questionnaires. 3rd ed. Oxford: Oxford University Press; 2006.
• An excellent compendium of existing scales in areas such as physical disability and handicap, social health and anxiety. The
. actual scales are presented and critiqued.
Streiner DL. A checklist for evaluating the usefulness of rating scales. Can J Psychiatry. 1993;38(2): 140-8
• This article provides a brief "cheat sheet" for evaluating existing scales without going into all of the rnessy details
Streiner DL, Norman GR. Health measurement scales.·a practical guide to their development and use. 4th ed. Oxford: Oxford
University Press; 2008.
• This book, as the subtitle states, guides the reader through the steps necessary to develop a scale. It can also be used by
people who are interested in appraising scales developed by others and defines the terrns riecessary to understand articles
that describe scales. One appendix tells readers where they can find existing scales in a wide variety of areas.
© 2011 The Royal College of Physicians and Surgeons 01 Canada 79

SUMMARY CHECKLlST
1. Background
o Do the authors explain the theoretical rationale for the scale?

O Do they state how this scale differs from other scales tapping the same or similar constructs?
2. The items
O Do all of the items seem to relate to the construct?

O Are there aspects of the construct that aren't tapped?
O Do any of the items seem vague or ambiguous?
O Are technical terms used (e.g., "shock," "myocardium")?
O Are any items double-barrelled? (Look for "ANOs" and "ORs.")
.- O Is the reading level appropriate for your study population?
3. Reliability
o Is the internal consistency > .807

Id Does the test -retest reliability rneet commonly accepted norms (> .70 for research in new areas; > .80 for research
in established areas; > .90 for clinical use)?
D !f appropriate. does the inter-rater reliability meet these same standards (> .70 for new research areas; > .80 for
established research areas; > .90 for dinical use)?
D Have these been establishcd in a population similar to the one you are studyinq?
4. Validity
o Have enough validity studies been done so that you're confident about what the scores refiect?
O Has validity been established in a population similar to one you are studying?
5. Utility
o 15 the test copyriqhted? If so, do you have permission to use it?

O How long does it take to complete?
O Are instructions or training required and, if so, do you have the resources for this?
O How easy is it to score?
80 © 2011 The Royal College 01 Physicians ano s.....~J"S = :3-a:;¿

11
Surveys
Susan J. Bondy, BA, MSc, PhD, FACE
ILLUSTRATIVE CASE 1
An Endrocrinology resident is interested in the experiences of primary care physicians in managing older patients with
diabetes. She obtains a mailing list from the College of Family Physicians of Canada from which to draw a simple random
sample of physicians and designs a survey following the method of a popular guidebook.
ILLUSTRATIVE CASE 2
An Obstetrics and Gynecology resident wishes to determine and describe the rates of adverse outcomes of labour and
delivery in all five of the health region's labour and delivery centres, which vary somewhat in the volume of deliveries they -
manage. Here the population of interest is composed not of people but of episodes of careo Because the residerit hao
planned in advance to report data for each hospital, she oversamples from the smallest hospital to achieve a minirrium
sample size that will be useful. AII deliveries that occur during a defined period are eligible for the study. At each hospital,
--
-
a random sample of 500 deliveries is drawn, and the charts are reviewed for the outcomes of interest. Outcome rates are
reported for each hospital separately and are presented for the whole region using sampling weights to account for the
oversampling in the smaller centres.
The term "survey" is used in many ways. and an outcorne of interese, these data can be derived from
In the conrext of research, some people use rhe word when- pre-existing large health survcys or gathered by orher
ever a questionnaire is involved, although this is not neces-
sarily accurate. Not all research that uses questionnaires
CHAPTER OBJECTIVES
requires a representative sarnple, as surveys do, and not all
surveys involve questionnaires. Moreover, as the sccond After reading this chapter, you should be able to
case example illustrares, not all surveys are surveys of people. • discuss the design elements that characterize survey
Represenrarive surveys can be used ro describe any popula- studies
tion of people or other entities, such as cases or procedures. • describe the importance of correct popu!ation sampling
Ir is irnporranr also ro distinguish berween surveys and methods in survey research
orher forms of cross-sectional research. Representative • locate additional resources to guide the development 01
surveys are often done ro estimare rhe prevalence of disease a survey project
or orher characreristics in a population. Many researchers • describe the implications of different sampling methods
also use cross-sectional data (including existing survey for the analysis of survey data
data) ro test hypotheses about, for example, differences • better understand how to use and interpret existing
berween groups, or the relationshi p berween an exposure survey-based research
I KEY TERMS
Clustered sampling Finite population correction
-----_.
Random sampling Sampling weights

"---1
I
Complex samples Generalizability Representative surveys Simple random sample
Confidence intervals Margin of error Representativity Statistical precision
Cross-sectional data Measurement errors Response rate Stratified sampling
Design effect Minimizing error and bias Sample size Survey data analysis
Estimates Point estimate Sampling errors Survey mode
External validity Precision of the estimate Sampling frame Variance

specifically desigoed means. Srudies such as rhese, whose • Minimizing rneasurement errors requires rhe use
primary objecrive is ro test a hyporhesis, are more correcrly of reliable and valid measures. For guidance on
described as cross-secrional analyric srudies, or case-control developing measurernent scales and quesrionnaire
studies in sorne iosraoces. The sampliog desigo thar is most irems, see chapter 10.
cosr-effecrive in resriog a hyporhesis is usually very different
from rhe desigo rhat would be used ro describe a naturally Statistical precision is an importanr attribu te in a
occurring popularion, as in a represenrative surve)'. For rhe study designo In surveys, the precisíon of the estimate
purposes of this chapter, then, a "survey" iovolves: or, in other words, rhe degree of confidence one ma)' ha
in the estimare obrained-is usually reponed using con
• rhe unbiased selection of individuals ro study from a dence intervals or by preseoring each statistic along wi
dehned population of like individuals; its margin of error. A 95% confidence inrerval, for exar
• rhe measurernent of one or more characteristics or ple, is a range of values rhat has a 95% likelihood of co
artributes for each member of the sample (e.g., rhe rainiog the true value of the variable rhat has bee
presence or absence of a panicular coodition); and estirnared (see also chs 24 & 25). Margin of error
• data analysis ro estimare the distribution of rhat reponed as a value following a ± signo For example, for
characrerisric in rhe population of interest (i.e., rhe finding such as "40% of respondenrs agreed," one coul
popularioo from which the sarnple is drawn). repon a 95% confidence interval of, say, 37% ro 43% o
equivaleody,state rhat the 40% estimate was accurate t
within ± 3%, 19 rimes out of 20. The statisrical merhoc
What makes a survey valid?
used to calculate confidence inrervals and margins of erre
111e methodological strength of a survey is determined by for survey results are somewhat different frorn rhose use,
rhe degrec ro which ir rninirnizes error and bias and for orher research designs discussed in this guide. Diff<:~
achieves sratistical prt'cision. ent merhods aod software may be required; these ar
Iv.LLnlfYjjziJJ.g e1'1'01' and bias means providing statistical introdueed briefly later in rhis chapter,
estimares that authentically describe the population of
interest. To achieve rhis airn, the researcher must avoid both
Designing a relatlvely simple survey
sampling errors and rneasurernent errors, as follows:
1. Defining who (01' what) you want to describe. 111.
• Sampling errors are minirnized by ensuring thar firsr, crucial step in designing a survey is ro define th.
rhe sarnple obtained is truly representarive of rhe population you wish ro srudy. In doing so, consider )'ou
popularion being described. Surveys are held ro a higher survey's anticipared generalizability (or externaI validi
standard for popularion represenrativeness than are ty): that is, the future and disrant popularions ro whon
most other research designs. For exarnple, in a clinical you expect your resul rs ro be applicable. This shoulc
efficacy tri al, externa! genera!izabiliry can be achieved underpin the survey's inclusion and exclusion crireria
by L1singexplicir and quite narrow inclusion criteria, which define the people or rhings that make up thar wid-
and random sampling is rarely used to select eligible er population. The researcher in our first case exarnple i:
parienrs. The findings are generalizable, but only ro interested in diabetes careo In formularing her researcl:
patienrs with those same characreristics. In contrast, the question (see ch. 6), she decided rhat she wanted ro study
people in the popularion being described in a survey rhe physicians who provided this careo Her design would
may be quite heterogeneous (e.g., young and old, have been very differenr had she wanred ro study che
sick and well). If rhe survey sample is represenrative, experiences of diaberic parienrs regardless of who cared
rhen rhe sample should indude che same mix uf for rhern. In the second case exarnple, "deliveries" were
characterisrics as the underlying popularion, and in rhe sampled, which is not the same as sampliog individual
same proporrions (a propert)' called representivity). morhers (who mighr have more rhan one delivery wirhin
Random sampling is an irnporrant means ro obtain a study period) or individual babies. The sample thar is
mis represenrarivity Sample size is also important. The used, and the method ro obrain ir, rnust be marched ro
larger a randorn sample is, the betrer the odds that it the rarger populariori exacdy as den ned in rhe research
will be represenrarive. quesrion.
82 © 2011 The Royal College of Physicians and Surgeons oí Canaca

77 Surveys
The second case also illusrrares rhat one study can address sarnpling irames should be evaluared for coverage gaps
rnultiple, related populatioris simulraneously (one for each (who is left Out) and other errors (e.g., old addresses). Prob-
of the hve hospirals). Researchers ofren plan in advance [O lems wirh your sampling Irame can resulr in biased samples
repon statistics for rnultiple subgroups. \Y/e will return [O rhar do nor represem rhe intended target popularion well.
rhis case exarnple later wirh respect ro rhe irnplications for For one example, in rrying [Q survey a specific professional
sample size and the approach [O data analysis presented by group, one mighr have ro rely on lists from professional
che use of oversampling and strarification. However, we organizarions wirh voluntary mernbership. This can resulr
might pause here [O note a diflerenr but relared problem in a biased sarnple, beca use rhose who are wiIling ro join
presented by chis case. Although rnulriple births (e.g., (and [Q pay rhe membership dues) may be systematically
twins, rriplers) are less common rhan singleton birrhs, rheir clifferem from rhose who are not. Old address lisrs may sys-
high cornplication rates can gready infiuence reponed out- rernarically exclude new graduares and new pracrices. To
come rates for che entire study sample. Thus, although give anorher exarnple, having ro rely on sampling based on
including sorne multiple births would be essenrial ro make
the survey generalizable ro all deliveries, a simple random
relephone numbers may inclucle roo few young people who
use only cellphones or orhers who have joined no-call lisrs.
e
ro
sarnple (see scep 5, below) of all deliveries would include
relatively few multiple births and would rherelore have lit-
These facrors can creare sorne bias in rhe clemographic
characteristics of the sample.
.."
de sraristical precision in describing these births separately. The terrn survey mode refers ro the rnerhod by which
A report speciíic ro multiples would have ro wait for a sepa- recruitrnent and data collection are conducred. Modes
rate or much larger study. Ir is also irnportant ro note rhar with the longest hisrory ancl rhe best-developed literarure
col!ecting data on exceptional cases that do nor meer a sur- are face-ro-face household surveys, relephone surveys and
vey's inclusion criteria and wiIl not be used in the analysis is rnail surveys. The mode will be dererrnined by consider-
borh cosrly and unethical, arions such as the contact informarion available ro rhe
researcher (e.g., mailing addresses) and rhe measures rhat
2. Defining what you want to measure. Many good need [O be raken [O obtain the relevanr inforrnarion (e.g.,
handbooks on survey design recommend that surveys be blood work cannot be obrained by relephone, alrhough
construcred on the basis of a breakdown of rhe specific recruitmenr by telephone might be used). The cheapesr
rypes of informarion rhar you wanr ro be able ro reporr. mode is rarely rhe besr. For example, Internet surveys have
Ideally, rhis would involve mapping out detailed mock proliíerated even though rhey are frequendy ignorecl by
resulrs rables rhat show every population-based point esti- porenrial respondents. Simple-ro-Iollow bur well-
mate ro be presenred (e.g., means, proporrions) overaIl and researched guides exist for al! rhese modes (see Addirional
for each planned subgroup. You wiIl need ro define how Resourees). These guides explain how [O recruit partici-
every staristic wiIl be calculared and who will be included panrs, maximize response rares, track samples and present
in ir. For exarnple, in our survey of physicians, one ques- data. Household, telephone and large clinical surveys are
tion might ask abour rheir satisfaction with certain services all besr carried out by acadernically orienred professional
ro which they can refer parients. Some respondenrs wiIl say services on contraer.
rhis quesrion is "nor applicable" because they do not use However, surveys conducted in health care settings are
these services; you will have [O decide how ro han die these not addressed in most general survey books, and rhey
responses, which wiIl have the effecr of reducing rhe sample present a number of concerns and chaIlenges unique ro
size for specific questions and srarisrical measures. their setring. Many clinician researchers have rried ro
conducr a survey alongside the provision of routine care,
3. Seleeting sampling frames and survey modes. A using existing staff, only [O find that the response rares
sampling frame provides a rneans [Q enumerare a popula- rhey obrain are very low and thar rhe resulring sarnplc is
tion and draw a random sarnple from ir. Sampling frames Bawed by selecrion bias. For example, clinical sraff
are ofren lisrs, but they can be any means [Q creare a census conducring a survey mighr be more Iikely ro approach
or regisrer of rhe individuals in your population of interest. patienrs rhey know are affected by rhe problem being
This can include ereating maps of where people are found srudied, or rhose rhey perceive ro be rnost likely ro
and rnethods ro keep future patienrs in sequence and draw participare. A rigorous study plan should be developecl,
a fair sample of thern. For scientific rigour, al] porenrial and staff dedicared [O rhe study should be ern ployed.

----~- -_-
---
The Research Guide: A primer for residents. other health care trainees, and practitioners
Derailed records musr be kepr of those who were eligible our quesrionnaires as a whole and ro pre-resr rhem. In b:
for rhe survey, were approached, and participared, 1,2 all of rhe derailed analysis plan described in sreps 1 ro 3 is use.
which requires a lor of effort. The research plan should be generare a lisr of key measures ro be addressed by rhe sur
reviewed for scienrific merir (by someone wirh survey irems. Always start by esrablishing a clear definition of e
expertise, and not only clinical knowledge in the area), concepr you wanr to repon on, and review rhe methodol
and approval should be soughr frorn rhe appropriare ical literature regarding how each of rhese is besr measur
Research Erhics Board. Cherry-picking inreresting questions from other peop
Parienrs must feel free ro decline to participare in a sur- work can be a dangerous exercise. First, ir can lead you (
vey. Approaching a parienr wirh a requesr to participare in track and make your questionnaire long and burdensor
research can be inadvertently coercive if the patient fears Previously used quesrions should be examined critically,
rhar non-participation mighr jeopardize his or her careo see wherher rheir re!iabiliry and validiry were previou
Poor response rates or unreliable responses, on the orher assessed. Ir is common for bad quesrions ro be used ag;
hand, can resulr if patienrs lack confidence that rhe infor- and again, even in major public healrh surveys. Bad qu
._ marion rhey provide will be held securely and in confi-

dence. A good pracrice ro facilitare informed consenr for
rions include rhose rhar are confusing, leading (that is,
gesr an answer), impossible ro answer wirhin rhe contrair
SL
participarion in a survey is to give advance norice of an of rhe quesrionnaire (e.g., rhey require a qualifier such as '
upcoming survey by phone or mail, and by someone who depends"), or are difhculr to analyze (e.g., in rhe case
is not a healrh care provider. Again, conrracring a proles- questions that ask rhe respondent ro provide a ranking ord
sional service ro do or supervise rhe work can head off pat- for a ser of choicesl.v" If you need to creare new quesrior
ents' concerns about the impact of survey participation or such as new measures for acritudes (or perceived "irnpo
non-participa tia n on their careo tance"), seek out expert tips on designing questions ar
Anonymous drop-box surveys are often used in patient response options Irorn the Íiterature. 'X/riting good que
waiting-roorns using self-explanatory consenr materials tions for surveys is a specialized skill.
and quesrionnaires. Such materials should, of course, llave Guidebooks and expert consultants also provide usefi
Research Ethics Board approval. However, wirh respect to advice on optimal lengths for survey insrrumenrs, wir
representariveness, anonymous drop-box questionnaires respect ro borh rime required and number of irems. In ger
are a very weak survey designo Response rares are Iow and, eral, surveys are besr kept short. However, if rhe survey is (
worse, their response rares are ofren impossible ro estimare direct interest ro rhe respondenrs (e.g., asking patienrs aboi
because rhe toral number oE eligible individuals (i.e., rhe rheir specific condirion, or physicians about issues thar al
denominaror) cannor be defined. Nor can rhe researcher be imporranr ro thern), they will usually sray morivared for
cerrain who filled Out rhe questionnaires (or, for thar mar- longer rime. The same guidebooks on survey design also prc
ter, who mighr have done so repeatedly). Such convenience vide tips on making rhe quesrionnaire (or web-based form:
samples rarely resulr in generalizable or publishable results, etc.) as appealing as possible. These guides can indude ver
alrhough their findings can be oflocal interest, specific advice on considerarions such as the choice of colou
For externally funded research, convenience samples paper qualiry, rypefaces and rhe use of graphics.
are rarely defensible, alrhough there are circumstances for Ideally, full-scale pre-tesring (see ch. 21) will indud
which no sampling frame exisrs, such as research involv- practice sraristical analysis and presentation of resulr:
ing homeless, displaced or hard-to-identify populations. Many problems can be avoided by pilot-testing a surve
Sampling rnerhods for such populations, such as "snow- design in rhis way.
ball'" sampling and sampling based on locarion, are
described in the literarure (see Additional Resources). 5. Creating the sampling designo The most basic surve
design is the simple random sample (SRS). This desigi
4. Translating the survey objectives into a questionnaire requires:
or interview. This aspecr of survey design is ourlined in
chapter 10 and is described in derail in various specialized • that all members of rhe so urce population can be
rexrs lisred under Addirional Resources. Such texts provide identihed and have an equal chance of being selected,
examples of good and bad quesrions and describe how to lay and that oversamp!ing is not used;
84 © 2011 The Royal College 01 Physicians and Surgeons 01 Cenad,

77 Surveys
• rhat elernents" are sampled independently (e.g., not in or dus ered sampling and weighting will also affecr the
clusters), and margin of error of the calculated estimares, Unless rhese
• rhe assumption that one is sampling wirh replacernenr" aspectS o rhe survey design are taken into account in the
from a very large target population and that no sratistical analyses, the resulring estimares may be biased
Iinire population correcrion (FPC) will be used. and the confidence intervals or margins of error will be
(The concept of rhe FPC is described later, under reporred as narrower than rhey should be.
Determining rhe Sample Size). Clusrered sampling, in particular, can easily be
rnisapplied. One mistake is ro analyze clustered data as
In rnediurn- ro large-scale surveys, SRS designs, srrictly though people had been sampled individual!y and, as a
defined, are rareo Complex sarnples are used ro accommo- resulr, ro report an incorrect margin of error. In designing a
date subgroup comparisons and rnultiple research goals survey with clusrers (or primary sampling units, PSUs, in
and can make rhe overall project far more cost-effecrive. survey methods jargon), rwo common mistakes should be
Cornplex sarnples can involve any of rhe following ele- avoided. The firsr is ro use what arnounrs ro a convenience
rnents: sample of groups (for practical reasons), and rhe second is
• stratiíied sampling, in which random samples are

to indude toO few groups. ]USt as no study of3 ro 10 people,
chosen non-randornly,
rhe so urce population,
could provide a reliable estimare of
no sarnple of 10 families, 3
-
drawn separately from each subgroup of interest 01'
(e.g., each hospital in our second case exarnple).

Oversampling from smaller groups is often necessary
treatment centres, chosen by convenience could make for a
represenrarive survey sample. Even when the selection of
-
to achieve a minimum sample size per group. groups is randorn, it is common to underestimate rhe
However, unequal probabilities of selection for number of groups required. Standardized methods for
differem subgroups can rnake rhe overall sample non- surveys in public healrh recommend that no fewer than 30
representarive: ro 50 PSUs be sampled at random from a popularion ser of
• sampling weights, which are developed ro compensa te PSUs many times larger.3-G People doing small surveys
for oversampling and insrances of non-response and might stick ro an SRS design, or else sample everyone and
are applied to the analysis ro restore representativeness: treat this as an SRS. For any other design, an experienced
• clustered sampling, by which people are sampled in statistician should be enlisted to assist in creating rhe
groups (e.g., one samples srudents by drawing a large sampling design and analysis plan. Again, rhe need for
random sample of whole classes, or samples patients starisrical assistance would indude even the faidy simple
by selecting at random a large nurnber of dinics) as a design described in rhe second case example, which would
way ro reduce field costs. require weighted and srratified analyses for the region as a
whole, necessitating the use of specialized software (see
Novice researchers need ro be aware of the implications Additional Resources).
of all of these design features, each of which will complicare
the analysis. For example, rhe researcher in our second case 6. Determining the sample size for a simple survey.
example draws a random sample srratified by hospital. Esti- Your dererrninarion of the sarnple size for your survey will
mates describing the entire health region (i.e., across all srarr with the plan you devised in step 1. The minimum
hospirals) will thus need ro be based on strarified and sample size needed ro achieve the desired level of precision
weighted calculations, as opposed ro simple ones. Srratihed (for each estimare) can be obtained using various sarnple-
size calculators, induding sorne that are available free
oriline.' If you are planning ro conduce secondary analyses
a In survey jargon. e/ements are the individuals 01 interest. the single unit
and hyporhesis tests, you will need ro calculare (he mini-
oí analysis lar the study.
b In sampling with replacement, individuals that are sampled are mum sample size ro obrain appropriate staristical power for
"returned" to the original sample, such that the sample size and each of these tests or comparisons.
the probability lar each individual 01 being sampled remain constant
throughout the sampling process. This means that individuals have some
chance 01 being sampled more than once in the same study. This is rarely
done in health surveys, but the assumption is considered to be met
where the target population is very large (such as a million or more) e For an example, see www.OpenEpi.com

Note thar the required sample size refers [Q the final ro know ro turn off this oprion in the software )'OU are u
number of complete responses used [Q calculare each statistic (Ir is common for the FPC ro be a defaulr setting.) Usin
of interese. l11is is not the number of questionnaires mailed, FPC can resulr in falsely narrow confidence intervals
or even all of the cornplered interviews before subgroups are findings of staristical significance where none exisrs. }
considered. To determine rhe size of the initial sample or peer reviewers will reject the use of the FPC in he
mailing, and the required budget, the target sample size is research. The bottorn line is rhat small samples prac
rnultiplied by inBation factors that account for errors in the wide margins of error. There is no quick fix..
sampling frame and rhe expected non-response rateo One
should even estimare rhe percentages of "not applicable" Conducting the survey
and "don'r know" responses for each quesrion. For example, Developing the field merhods for the conduce of your s
one might ask how satisfied a patient was with a specific vey is as irnportant as any other aspecr of rhe designo Sr,
service, but perliaps only 40% of the sample have used that by-step advice on survey recruirment and fieldwork
service and are thus abJe [O contribure data on satisfaction. avai!able for mail, telephone and other kinds of surveys (:
Spreadsheets are usually used [O estimare how a simple ran- Addirional Resources). Major considerations in planni
,,- dom sarnple will be naturally disrribured over subgroups of the fieldwork methods are: (1) crearing interest in rhe stu.
interese. For example, if you hope [O report pregnancy out- ro maximize parricipation rates: and (2) quality assurance.
comes by age group of the mother, you will need ro guess Dillrnan's social exchange theory" provides a basis for ti
the percemage of reen mothers that you will have in your merhods he oudines ro motivare people ro participa te i
sample. This process oE determining how large a sample will surveys. How the survey is advertised in advance and ni:
need LO be ro adequately represent al! subgroups usually presented LO porential respondents can infiuence participa
increases rhe sarnple size targets, and COSE, or else suppresses tion rates, Advancc announcements can generate interes
the researcher's apperire for subgroup analyses. and help convince people that the survey is importanr an:
Finally, if yOl! are planning to use a complex sampling legitimare (and not, say, unsolicited marketing). Cover let
design, )'ou will need LO inflare the target sample size to cor- rers and announcernents should be pilot-tested ro ensure
rect íor the impacr rhar your sampling design will have on rhat rhe language is effective in communicating rhe value oí
the margin of error ofestirnares. The degree to which statis- the research. With respecr ro approaching peoplc, the best
rical precision would be decreased (or the required sample advice is ro personalize communications as much as possi-
size increased) is referred ro as che design effect and can be ble, for example by using personally addressed envelopes,
diflicult ro estimare without the guidance of a statistician. real stamps and real ink signarures on cover letters. Proles-
sional looking, appealing and easy-to-cornplete material,
Commems on small samples from small target increase response rates, too.
populations Follow-up reminders are essential ro achieve a reasonable
Ir is not unusual in health research ro find rhat rhe total pop- response rare, alrhough more rhan 3 ro 4 follow-ups will
ularion of interese is small, comprising jusr a few hundred rarely result in much additional return. Again, handbooks
patients or an even sma!!er number of healrh professionals. on survey research give advice on how many follow-ups ro
In such cases, rhe best approach is generally ro do no sarn- use and at what intervals. Keeping the reminders fresh,
pling and ask ever)'one ro rake pan, treating the resulting appealing and personal is good advice. Research has also
sample as an SRS. The finite population correction (FPC) shown that srnall monetary incentives in crease response
should not be used wirh sarnples of rhis kind. The FPC is an rates by a small but irnportant amount.v"
advanced concepr, but an important one ro raise even in this Even for small surveys, your fieldwork will require
inrroductory guide.111e FPC reíiects the fact that the results very careful record-keeping ro manage mailing lisrs,
from a sarnple of, say, 90 out of 100 porenrial parricipanrs ensure rhat pcople who have already parricipated are
are not likely [O be much difíerenr from the results from all thanked and are not sent further reminders, and ro ensure
j OO. Because the use of the FPC correction in estimating proper handling of informed conserit and the appropri-
sample sizes or calculating Pvalues and confidence intervals ate storage of cornplered questionnaires and data files
is very rarely appropriate in health rescarch," you wi!! need
audits, which are used only within an organization and whose results
d Using the FPC can make sense in circumstances such as quality-assurance would never be applied to any external group or place
86 © 2011 The Royal College of Physicians and Surgeons oí Cana:~

77 Surveys
containing personal and health information. Every per- toO-ea5!- rnarker for porential selection bias is a low response
son in the initial sarnple musr be tracked and given a final rateo In realirv, 10\V response rates (ofren below 50%) are,
deterrninarion wirh respect ro who parricipared, who with rare exceprioris, ro be expecred, even when best prac-
refused, and who turned out ro be ineligible. There are rices ro maximize response rates have been followed." There
detailed standard definitions for al! of rhese outcornes, is simply no hard and [ast rule as to what response rare is
and rhese nurnbers are used ro estimare the response rare acceprable, as opposed ro being too low Por rhe survey
in ways that are scientifically acceptable.? Depending on resulrs ro be valido An acceprable response rate depends on
rhe scale and cornplexiry of the study, training and super- how much the sarnple remains represenrative of rhe target
vision of staff arid monitoring of data qualiry need ro be population specihcally with respecr ro the behaviours or
considered. TI1is can include data quality checks, repeat- characrerisrics being reponed on, and this wil! vary from
ing a sample of inrerviews, and orher methods for qualiry study to srudy. The staristical issues of selection bias (miss-
assurancc. ing people) and response bias (missing and wrong answers)
are discussed in detail in rigorous survey rnerhods books
Survey data analysis (see Additional Resources). Ar a mínimum, most research-
The analysis of survey data has rhree main components: ers wil! describe rhe demographic and other characteristics
reponing a correct response rate; calcularing rhe estimates of rhe sample and compare these with the corresponding
without bias; and calculating the margin of error or vari-
ance, as appropriare to rhe sampling designo
characterisrics in rhe popularion rhey wanr ro describe.
Where to get more information

_.
Response-tate deíinitions and forrnulae are given in
rnost survey guides and starerncnrs of standard survey prac- Many books and guides have been published on survey
tices (see Addirional Resources). To repon response rates rnerhods, and it is irnporranr ro selecr those rhat are appro-
correcdy, you wil! need the detailed notes you kepr while priare to healrh research and focus on methodological
you were administering the survey on the number of people rigour. Among resources aimed at non-expens, some have
who were approached, refused, parricipared, and rurned thorough and easy-to-follow discussions on questionnaire
out ro be ineligible. design and data collection. The Total Survey Design and
Poinr estimares (the actual means or proportions) are Tailored Survey Design series by Don Dillrnan, Por exarnple,
often correcred Por the bias created by complex sampling are well-researched, srep-by-srep guides to the most corn-
designs using sampling weights. Some researchers apply mon survey modes. Health professionals should choose
additional correcrions Por non-response patterns and Prom rhe more academic varianrs of such guides.3.7.8Al the
known differences berween rhe sample and rhe actual other end of rhe spectrum are texts that provide derailed,
population. technical presentations of rhe starisrical aspects of survey
As soon as weighting or clustered sampling is involved, design and analysis. These are imporrant resources for peo-
special sofrware is needed ro calculare the correct conh- pie working with data Prom existing large surveys, or plan-
dence intervals (and any P values calculated for subgroup ning a large survey, but their starisrical conrent can be
comparisons and orher tests). Many statistical packages challenging and they do not always apply ro smaller srudies
now include procedures designed specifically Por complex or clinical research specitically,
samples; examples include Epi Info, Srara, SAS, the SPSS The ideal resources are rhose written Por healrh profes-
Complex Samples module, SUUDAN and Wesvar. Within sionals and the sraff of governmem health agencies. They use
rhc rnulti-purpose statistics packages (Epi Info, SPSS, SAS language targcred ro non-staristicians, but includc rhe level
and Srata), only the commands explicirly inrended Por of detail needed for rigorous research, along with material
complex sarnples should be used. Again, surveys using on ethical considerations, privacy and confidentialiry.
srnall sarnples present a chal1enge, as the merhcds rhar are Exam ples ranging from brief guides ro full texrs appropriare
easiest ro use assume large samples. TI1e analysis plan should for public healrh staff and other health care professionals
be considered at the design stage, and always in consulta- inreresred in conducting surveys are listed among the ínter-
rion wirh a suirably experienced sratistician, rnediate-level rexts in the Addirional Resources secrion.
TI1e final and important step in doing a survey is ro eval-
uare the net irnpact of selection and measurernent errors. A

The ResearehGuide: A primer for residents, other health eare trainees, and practitioners
Conclusion
Representative sarnple surveys can be a powerful tool require close atrention to many technical considerati.
when used ro describe actual healrh-relarcd outcomes and including che principles of sampling and selection 1:
experiences in a known population and [O introduce measuremenr theory and statisrics, and require SllPI
sratistical estimares to this evidence. Surveys are a popular from a biosrarisrician well versed in survey methodolc
research rnerhod, and the estimares they produce are of Involvernenr in a large- or srnall-scale survey project of
interest to a wide variery of audiences. Surveys also have a valuable opportuniry and challenge for a resea
disrinct requirements for methodological rigour and trainee. •
REFERENCES
l. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting
~ randomized trials: explanation and elaboration. Ann Intern Mee/. 2001; 134(8) 663-94.
e
"'-.
2. American Association for Public Opinion Research (AAPOR) Standard definitions: final dispositions of case codes and outcom
e tetes for surveys. Lenexa (KS): AAPOR, 2006. Available from: www.aapororg/Standard_Definitions/3049.htm
en
»"""'"
3. Statistics Cariada. Survey methods and practices Cat no. 12-S87-X. Ottawa: Statistics Canada; 2003. Available from: www.
statcan.gc.ca/pub/12-S87-x/12-S87-x2003001-eng.pdf
\n 4. United Nations Department of Economics and Social Affairs, Statistics Division. Designing household survey samples: practica
OJ guidelines. Studies in Methods Series F,no. 98. New York: United Nations; 2008.
O s. Hoshaw-vvocdward S. Description ane! comparison of ttie metllods of cluster sampling and lot qua lity essuretv:e sampling to
assessimmunization coverage. Geneva: \!\Imld Health Organization; 2001. Available from: www.who.intlvaccines-documents/
DocsPDF01/wwwS92.pdf
6. Frerichs RR, Shaheen íVIA. Small-community-based surveys. Annu Rev Public Health. 2001 ;22:231-47.
7. Dillman DA, Smyth JD, Melani Christian L. Internet; mail ane! mixee!-moe!e surveys: tbe tailoree! design methoc/. 3rd ed.
Hoboken (NJ) John Wiley; 2009.
8 . .l'lday LA, Cornelius U. Designing and conducting Ilealth surveys. a comprehensive guide 3rd ed. New York: Jossey-Bass; 2006
Brief guides to the basics of survey research for health researchers
Abramson J, Abramson ZH. Research metllods in community medicine.· surveys, epidemiological research, programme evaluatíon,
clinical trials. 6th ed. Hoboken (NJ): John Wiley, 2008.
e This popular book is useful with respect to the goals of a community health survey, but provides little information on
sampling methods.
Salant P,Dillman D.A. How to conduct your own survey Hoboken (NJ): John Wiley; 1994.
e This concise paperback guide contains easy-to-follow discussions of questionnaire design and data collection but does not
address proper sampling methods.
University of North Carolina Center for Publie Health Preparedness A guide to sampling for community health assessments and
other projects. Chapel Hill (NC): The Center; n.d. Available from http//cphp.sph.unc.edu/PHRSTS/
Two-stage cluster sampling.· general guidance for use in public health assessments. Chapel Hill (NC): The Center; n.d.
Available frorn: http//cphp.sph.unc.edu/PHRSTS/
88 © 2011 The Royal eollege al Physicians and Surgeons al Canads

77 5urveys
• The above brief guides provide advice on the development o' CO,:n l. sarnples in the form of easy-to-reed newsletters.
Intermediate to advanced resources on surveys and secondary analysis of existing surveys for health researchers
Aday LA, Cornelius LJ. Oesigning and conducting hea/th surveys: a comprehensive guide. 3rd ed. New York: Jossey-Bass; 2006.
• This popular textbook for first-Ievel courses on surveys in public health and other disciplines provides a broad introduction to
all key tapics of survey planning, conduct and analysis and does not require advanced statistical knowledge.
Dillman DA, Smyth JD, Melani Christian L. Internet, mai/ and mixed-mode surveys. the tai/ored design method. 3rd ed. Hoboken
(NJ) John Wiley; 2009.
• This textbook provides a broad introduction to all key topics of survey planning, conduct and analysis, with specific detail on
several survey modes. It does not require advanced statistical knowledge.
Groves RM, Fowler FJJr, Couper MP, Lepkowski JM, Singer E, Tourangeau R. Survey methodoloqv. 2nd ed. Hoboken (NJ): John
Wiley; 2009.
• This textboak provides a broad introduction to all key topics of survey planning, conduct and analysis and does not require
advanced statistical knowledge.
Statistics Canada. Survey methods and practices. Cat. no. 12-587-X. Ottawa: Statistics Canada; 2003. Available from: www, .
statcan.gc.ca/pub/12-587-x/12-587-x2003001-eng.pdf.
-,
United Nations. Department of Economics and Social Affairs, Statistics Division. Oesigning househo/d survey semples: practica/
guide/ines. Studies in Methods Series F, no. 98. New York: United Nations; 2005. Available from: http://unstats.un.org/unsd/ .
demog raph ie/sources/surveys/Handbook23J LI neOS.pdf
United Nations. Department of Economics and Social Affairs, Statistics Division. Household sample surveys in developing and
transition countries. Studies in Methods Series F,no. 96. New York: United Nations; 2005. Available from: http://unstats.un.org/
unsd/hhsurveys/pdf/Household_surveys.pdf
• These classic overall guidebooks are written for applied health researchers and government employees who use survey research.
The United Nations volumes are the ultimate guidebooks for population surveys in public and global health. The chapters on
methods and sampling are excellent guides to learning about intermediate and advanced aspects of sampling design without
advanced statistics training. The Statistics Canada book has good material on questionnaire design and fieldwork and is also
exceptionally good for its relevance to the Canadian context, including with respect to information on ethics and confidentiality.
Specialized resources
American Association for Public Opinion Research (AAPOR). Standard deiinitions: final dispositions of case codes and outcome
retes for surveys. Lenexa (KS): AAPOR; 2006. Available: www.aapor.org/Standard_Definitions/3049.htm
• This technical reference describes professional standards for survey methods. These standards are widely are used in the
private sector as well as by major government and university-based survey research institutes.
8rackertz N. Who is hard to reach and why? ISRWorking PaperoMelbourne, Australia: Swinburne University of Technology lnstitute
for Social Research 2007. Available from: www.sisr.netlpublications/0701 brackertz.pdf
Faugier J, Sargeant M. Sampling liara to reach populations. J Adv Nurs. 1997;26(4) 790-7
• These resources provide advice on special methods to survey hard-to-reach populations, such as "snowball" sampling and
sampling based on location.
McDowelll, Newell e Measuring heeitr: a guide to rating scales and questionnaires. 2nd ed. Oxford: Oxford University Press;
2006.
• This book focuses on the development of self-report measures related lo health for use in surveys and any research using
self-report measures.

Texts on technical and statistical aspects of survey design and secondary analysis of surveys
Groves RM, Dillman DA, Eltinge JL, Little RJA. Survey nonresponse. New York: John Wiley; 2002.
Kom EL, Graubard BI. Analysis of health sutveys. Hoboken (NJ): John Wiley; 1999.
Levy PS, Lemeshow S. Sampling of populations: methods and applications. New York: Wiley; 2010.
Lohr SL. Sampling: design and analysis. 2nd ed. New York: Cengage Learning, 2009.
Lehtonen R, Pahkinen E. Practica! methods for design and analysis of complex surveys. 2nd ed. Hoboken (NJ):John Wiley; 2004.
SUMMARY CHECKLlST
For planning a survey:

O Define the population you want to describe in formal terms.
O Define the key measures of your survey and review how each is best measured.
O Design and pre-test your questionnaire (or data collection materials).
O Identify and critique possible survey sampling frames and data collection rnodes.
O Design field-test methods, including quality control measures.
C! Determine what margin of error is acceptable for each estimate.
O Define random sampling procedures.
O Determine sarnple size necded and the budget this will require.
For conducting a survey:

O Obtain ethical approvals (including for pre-testing, if required).
O Hire and train staff as required.
O Announce and promote the survey.
O Conduct random sarnpl'nq as per plan.
O Invite participants and record the outcome of all invitations.
O Obtain informed consent.
O Complete data collection according to the study plan.
O Prepare statistical analyses that describe the population of interest, with honest margins of error,
and that reflect the complexity of the sampling designo
90 © 2011 The Royal College 01 Physicians and Surgeons of Canaca

12
Medical record review studies
Andrew Worster, MD, fV1Sc,CCFP(EM), FCFP
ILLUSTRATIVE CASE
A second-year resident in Emergency Medicine sees a patient with a swollen and tender lower leg. The patient had been
seen in the emergency department just two days befo re for the same problem. At that time, the patienthad a D-dimer
test to look for evidence of deep venous thrombosis (DVT) The result was negative, and the patient was discharged .:
The resident is aware that this test can be used to rule out DVT in patients with a low likelihood - i.e , a low pretest
probability (PTP) - of the disease. There is no record of venous thromboembolism (VTE) risk factors or a formal PTP
calculation on the patient's chart from his previous visit. The resident wonders if the PTPwas taken into consideration
before the D-dimer test was ordered and, more generally, how often PTPis documented. The answer to this qúestion may
have significant implications regarding the quality of care provided by emergency physicians to patients with suspected
VTE. The resident decides to conduct a study to answer her research question: "15 PTPassessment of patients who present
to the emergency department with suspected VTE routinely documented befo re D-dimer testing 7"
11 This chapter provides an introduction order a D-dimer test. However, this is not feasible frorn a
ro medical record review (MRR) srudies, ourlining how resource perspective and would be subject ro rhe Haw-
they are conducred, and why. Like all research merhods, thorne effecr: rhat is, once study participanrs rcalize rhey
MRR srudies serve a particular purpose, allowing the are being observed, rhey willlikely change their behaviou r.
researcher ro obtain data ro answer certain rypes of ques- Insread, rhe resident and her supervisor decide rhar rhe best
tions. In our case exarnple, the resident discusses with her way ro approach her research quesrion is through a review
supervisor the possible study designs she could use ro
answer her research quesrion. One merhod that they con-
CHAPTER OBJECTIVES
sider is a survey of emergency departrnent physicians and
residents. Borh the residenr and her supervisor recognize After reading this chapter, you should be able to
that results from such a survey would be subject ro pro fes- • list the advantages and limitations of medical record
sional desirabiliry bias (i.e., respondenrs providing answers review studies
rhar rhey rhink would make thern look good, rarher than • describe how cases are selected
describing what actually occurs). Anorher rnerhod they • discuss how the sample size is calculated
consider is a prospective cohort study, in which the resident • describe the sampling method
would observe s,aff physicians and fellow residenrs in the • discuss how lo optimize the data quality
emergency departrnenr and question thern each time they • discuss the ethical issues applicable to this type of study
¡-_o . . _
KEY TERMS
Bias Data quality Sample size
Case selection Imputation Sampling
Chart review Simple random sampling
Inter-observer agreement
Cohen's kappa Missing values
© 2011 The Royal Coilege of Physicians and Surgeons 01 Canada 91

of emergency departmenr patient records=-rhar is, ro carry quesrion. Presenting complainrs and discharge diagnoses
out what is commonly called a chart review. However, the are not necessarily valid case-idenrificarion crireria unless
terrn "chart" is ver)' speciíic and do es not encompass al! rhey are integral ro the research question, as in rhe following
available sources of parient medical informarion, rnuch of example: "Is pretesr probabiliry assessmenr of ED patienrs
which is now recorded in aggregate form on elecrronic wirh a presenting complaint of leg swelling documenred
darabases. For this reason, rhese srudies are best described befo re D-dimer tesring?" The study population in rhe resi-
as medical record review (MRR) studies, a phrase thar cov- dent's research question is emergency patienrs with suspect-
ers an)' study using pre-recorded, patient-focused data, ed venous thromboernbolisrn (VTE) who undergo
such as physician and nursing notes, ambulance cal! D-dimer tesring. Because "suspected VTE" is neirher a pre-
reports, diagnostic tests, and so forth.1,2 Ir should be noted senting complaint nor a discharge diagnosis, the best way ro
thar various study designs can use MRR for the purpose of accurarely identify this popularion wirhour reviewing the
data collection: rhese include certain observational meth- rhousands of cases seen during rhe study period is ro lisr all
odologies such as case-series and case-control srudies. of rhc subjecrs who had D-dimer tests during their erner-
gency department visir. The lisr will also contain, as we wiJl
Advantages and limitations of medical see, many ineligible cases, bur ir should be possible ro elimi-
record review studies nare rhese cases using exclusion criteria established a priori.
1he obvious advantage ro conducting an MRR is that the

. data havc already been collecred. This can save considerable Sample size calculatlon
time and resources. This rnerhod also makes it possible LO The next question, then, is "How many cases are needed
address research questions that would be difficult ro answer for rhe study?" That is, how many subjects are required ro
prospectively such as che effects of harmful or beneficial ensure that sufficiendy precise estimares of the study mea-
exposures, the occurrence (lf rare events, and panerns of sures of interest can be calculated?" As for any quanrirative
disease or behaviour over prolonged pericds, ro narne a few study, the method used ro calcularé sarnple size for MR.-R.
exarnples.' However; despite rhe convenience, rhe use of studies is derermined by rhe sraristical test ro be used in rhe
pre-collected data can be a disadvantage, and researchers analysis of the primary ourcome. For the residenr's ques-
must undersrand the porenrial pitfalls befare embarking tion, the sample size calculation is based on the desired reli-
011this kind of srudy. Typically, the data collecred in medí- abiliry of the primary ourcornc (i.e., rhe proportion of
cal rccords are not meanr for research purposes and are not charts wirh docurnentarion of rhe pre-rest probability). The
as valid or reliable as data whose collection is prospectively resident decides that rhe reliability of her final resulrs
planned.? Clinical findings, such as wherher a parient has a should have a confidence interval of 95%, which means
heart murrnur 01' sorne other physical condition, are not rhat there is a 95% probabiliry rhar the actual proportion
consisrcnrly agreed upon or documemed. Of course, sorne of charts that inelude docurnentarion of che PTP is wirhin
types of information found in medical records, such as her predefined range of values.' Using an online calcularor,"
demograph ic data and test results, are less open ro inrerpre- she is able ro determine how f!1any cases she needs ro review
ration and hence more reliable. The absencc of sorne data is ro achieve this precision. Because of a variery of faciors, I
also a common prob!em ami one that presents a major such as the porenrial for error in ordering the test, not all of
thrcar to the validiry of MRR study resulrs. The data sarn- the parients who had D-dimer tests during their emergency
pie in MRRs can also be biased by an absence ofblinding departrnent visir wil! be eligible. 1he resident undersrands I
and of randornizat ion. 1hese issues, along with potential rhe need (O selecr a grcater nurnber of cases than was derer-
solutions, are presented in the fol!owing sections. rnined by her sample size calcularion and estimares thar
abour 5% will be incligible. The realiry is rhar she can'
Case seJection
know with any degree of certainry whar proponion of cases
Because "cases" of interesr in an MRR srudy have already wil! be found ineligible, bur at leasr she has ami ipared 1
occurred, rhe firsr challt:nge Ior the researcher is ro idenrify problem and cornpensared for the probable shorrfalJ.
those that are eligible for inclusion in the srudy The merhod
used ro make this selecrion will depend on the research
aSee, for example, http://statpages.org/iiPower
92 © 2011 The Royal e ol!ege 01 Physicians aná Surgoo-:s e' Ca;-c::¿

12 Medica! record review studies
Sampling method
ers ro extraer che daca, and thus avoid bias Irorn her 'own
Now rhar rhe resident has a lisr of all potentiall)' eligible opinions. he can furrher optimize the qualiry of her srudy
cases, she needs ro pick out the required number of cases dara by applying several daca exrraction srraregies. A
from a list ofhundreds. This is referred ro as sampling (see description of rhe strategies used ro avoid bias in data
also ch. 11). A common method of sampling is ro select all exrracrion is a requirement for publicarion in many peer-
of the consecutive cases within a given time frame. This is reviewed journals." Some of rhe key straregies described in
an acceptable approach if rhe period is long enough ro the lirerarure are as follows:
account for seasonal variations or orher temporal changes
thar are relevant ro rhe research quesrion. Bear in mind rhar • Train rhe daca exrractors.'!""
sampling consecurive cases over a long period can resulr in • Keep the data extractors blind ro the study hyporhesis
rhe inclusion ofmore cases rhan are required." For example, or, in rhis case, rhe scudy objecrive."
imagine rhat your sarnple size calcularion has shown rhar • Establish, a priori, unambiguous definicions for the
your study requires 100 cases, and you see more rhan 100
porenrially eligible cases each month. You would need ro
srudy variables (e.g., PTP) and clear inclusion and
exclusion crireria.':'!
e
'::"(0
. -v.
collecr daca only for a period long enough ro rneet che sarn-
ple size requiremenc. However, if che clinical event in ques-
cion is subjecr ro seasonal variarion, such as trauma or
• Creare a study-specihc compucer daca form (e.g., using
Microsofc Access, Microsofc Corp., Redrnond, WA).14
• Advise rhe data extractors ar che beginning rhat rheir
.\0--
pneumonia, you would need ro collect cases for a longei work will be checked for accuracy. 12
:J
period of time (e.g., an entire year) ro avoid tirne-period • Check the reliabiliry of rhe abstracted data in randorn
seleccion bias. This could mean collecring 12 times che sarnples."
number of cases needed Ior your final sample size. An alter- • Escablish unambiguous rules regarding rhe
native ro consecucive case sampling, and che best non-con- management of missing or conflicring data."
secutive sampling rnerhod for our case vignette, is simple
randorn sampling. Simple random sampling provides an Missing inforrnation can lead ro non-response bias in
equal opporrunity (probabiliry) for each eligible case ro be che resulrs if che cases are sirnply deleced from che analysis.
selecced withour bias and is best achieved by using a ran- The irnpact of chis will depend on che proporrion of cases
dorn number generaror ro identify rhe records for selec- involved. One rnethod of managing missing daca and/or
. tion.? Random number generarors are available online." missing cases is irnputation. This rnerhcd subsrirures sorne
Orher sampling rnethods, such as incidental selecrion value for a missing value. For example, one could choose ro
(selecrion of the rnost easily accessible cases) and systernatic subsrirute an average value of the variable in quescion for
selection (selecrion of every "nth" case), are more suscepti- che missing value. Mulciple irnputarion, which is rypically .
ble ro bias and are besr avoided. reserved for large darabases, is the mulriple substiturion of
missing valuesY Once all missing values have been irnput-
Optimizing data quality ed, rhe dataser can be analysed using standard srarisrical
tests. There are no resrrictions on rhe quantiry uf data thar
General!y speaking, each medical record is composed of can be subsrirured, bur rhe greacer che proportion of irnpur-
differenr inrerprerarions of differenr scenarios, ofcen by dif- ed values, the less valid the results. Anocher rnethod of
ferenc observers.ó'? Moreover, rhe Iree-rexr formar corn- managing missing values is by sensitiviry analysis, in which
monly used in parienrs' medical charts can be difficulr ro the missing values are replaced with "bese-case" and "worst-
inrerpret. These shortcomings can creare a larger-rhan- case" outcornes or values. An advantage of sensiriviry analy-
anticipared arnount of erroneous and missing daca. There- sis is rhar ir can dernonstrare the robuscness of the resulrs
fore, (he onus is on rhe researcher ro dcrnonsrrare rhar rhe undcr difierenr circurnstances.
data have been exrracred reliably and in an unbiased rnan- \'\Ihen rwo or more dara exrracrors arrive at differenr ver-
ner." The first step the resident should rake is ro recruir orh- sions of rhe same event, the researcher is faced with conflict-
ing data. These discrepancies are ofren best resolved rhrough
consensus and, like che nurnber of cases for which data are
b See,for example, wwwrandom.org/integersl missing, should be reponed in the resulrs along with che

93
outcome(s) of their resolution, However, the readers of the erhical approval (see also ch. 18). In sorne cases, however,
study wil! also need ro know ro what extent the data extrae- insrirutional review boards provide blanket approval for
tors agreed in general. Cohen's kappa (K) is the most corn- qualiry improvemenr research deerned ro presenr little risk
monly reponed measure of inter-observer agreement for ro me parienrs whose data are used, and so me medical jour-
MRR studies. Ir is interpreted as rhe extent of agreemem nals will accept this practice, depending on the nature of rhe
achieved compared with rhe rotal amount of agreemenr pos- srudy. Regardless of rhe requirement for erhical approval,
sible beyond chance agreemenr and is reponed as a value every effon should be made ro ensure rhar patienrs are nor
fram -1 (perfecr disagreement) ro 1 (perfecr agreemenr) .18 identiíiable in the presentation of rheir data.
As wirh many formulas, Cohen's K has its limirarions; in
some cases, orher tests mighr be more appropriate.
Conclusion
Alrhough MRR srudies are not easy ro conduct, they can

Ethical issues
ofren be cornpleted in Iess rime and at less expense rhan
The parient healrh informarion rhat the residenr intends ro rnost prospective studies. MRR studies are recognized as a
view, abstraer and analyze is confidenrial. As such, she is valid and valuable research design rnerhod in che medical
required to abide by al! ofher hospital regularions regarding literature, and recommendations for conducting and
rhe privacy of patient informarion. She will most líkely need reporting thern have been published. Residents who have
ro apply ro her local Research Erhics Board for approval to taken the rime to undersrand the !imitarions and rnerhcd-
conduct her srudy, and as a rule medical journals will nor ological challenges ofMRRs have published their studies in
consider for publication any study that did not receive prior high-impact journals. •
CASE POSTSCRIPT
The resident submits c) study protocol to her institution's Research Ethics Board and obtains approval to proceed. With
support from hel' supervisor and other content experts in the fieid, she develops a data collection form and trains two medical
students to revievv the relevant patient records. Working with a final sample of 100 charts, as determined by her sample size
caiculation, she finds that adequate documentation of PTPwas provided for only 10% of all patients with suspected DVT.
Hsr findings are presented at a research meeting and prompt the creation of a standardized form for inclusion in selected
charts that encourages clinicians to record the elements of a clinical probability assessment for DVT.
REFERENCES
1. Worster A, Haines T Medical record review studies: an overview. Israeli J Ireume, Intensive Cere Emerg Med. 2002,2(2):21-6.
2. Worster A, Haines T Advanced statistics: medical record review (MRR) studies. Acad Emerg Med. 2004; 11: 187-92.
3. Burnum JF.The misinformation era: the fal! of the medical record. Ann intem Med. 1989; 110(6) 482-84
4. Smith C, Mensah A. Mal S, Worster A. 15 pretest probability assessment on emergency department patients with suspected
venous thromboemboiism documented befare SimpliRED D-dimer testing? CJEM. 2008; 10(6):519-23.
5. Last JM. A dictionary of epidemiology. 3rd ed. New York (NY): Qxford University Press; 1995.
6. Cummings SR, Newman TB, Hul!ey SB. Designing a cohort study. In: Hulley SB, Cummings SR, Browner WS, Grady D, Hearst
N, Newman TB. Oesigning elinical research: an epidemiologic approach. 3rd ed. Philadelphia (PA): Wolters Kluwer/Lippincott,
Williams & Wilkins; 2001 p. 101.
7. Worster A, Bledsoe RD, eleve P,Fernandes CM, Upadhye S, Eva K. Reassessingthe methods of medical record review studies in
emergency medicine research. Ann Emerg Med. 2005;45(4):448-51.

72 Medica! record review studies
8. Gilbert EH, Lowenstein SR,Kozoil-McLain J, Barta DC, Steiner J. Cha . re. 2.'."5 In emergency medicine research: Where are
the rnethods? Ann Emerg Med. 1996;27(3)305-8
9. l.emer EB,Zachariah BS,White LJ.Conducting retrospective emergency medical services research. Prehosp Emerg Careo
2002;6(2 Suppl):S48-51.
10. Rangel SJ,KelseyJ, Colby CE, Anderson J, Moss RL.Development of a quality assessmentscale for retrospective clinical
studies in pediatric surgery. J Pediatr Surg. 2003;38(3) 390-6
11. Horowitz RI,Yu EC.Assessingthe reliability of epidemiologic data obtained from medical records. J Chron Dis.
1984;37(11) 825-31.
12. Reid JB. Reliability assessmentof observation data a possible methodological problem. Child Dev. 1970;41(4) 1143-50.
13. Allison JJ,Wall TC, Spettell CM, Calhoun J, Fargason CA Jr, Kobylinski RW,et al. The art and science of chart review. Jt Comm
J Quallmprov. 2000;26(3) 115-36.
14. Banks NJ. Designing medical record abstraction forms. Int J Qual Health Careo 1998; 10(2):163-7
15. Beard CM, Yunginger JW, Reed CE, O'Connell EJ,Silverstein MD. Interobserver variability in medical record review: an
epidemiological study of asthma. J Clin Epidemiol. 1992;45(9) 1013-20.
16. Wu L, Ashton CM. Chart review. A need for reappraisal. Eval Health Prof. 1997;20(2): 146-63.
17. Schafer JL. Analysis of incomplete multivariate data. Monographs on Statistics and Applied Probability 72. Boca Raton (FL):
Chapman & Hall/CRC; 1997.
18. Cohen JA. A coefficient of agreement for normal scales. Educ Psychol Measure. 1960;20(1 ):37-46. -
SUMMARY CHECKUST
o Carefully consider the limitations and advantages of an MRR study before you start. Specifically, anticipate the data gaps
you may encounter and their implications for the validity of your results.
O Devise a reliable method for identifying potentially eligible cases for screening.
O Frame your inclusion and exclusion criteria carefully to remove any ambiguity about case selection.
O Remember that unbiased sampling methods and appropriate sample size calculations are as important in MRR studies as
they are in other forms of research.
O Take the necessary steps to optimize the quality of data collection: this willlikely involve properly training those who will
be reviewing the patient records, verifying data quality and reproducibility, and preventing bias by blinding the data
abstractors to the study hypothesis.
O Set out the groundwork for successful chart review by establishing clear definitions for what needs to be collected and
designing easy-to-use paper or electronic data collection forms.
O Remember that medical records contain privileged patient information and that research based on such récords requires
the approval of a Research Ethics Board both to safeguard the integrity of the work and to ensure that it will be eligible
for consideration by a peer-reviewed journal.

._

13
Adrninistrative database research
Carl van Walraven, MD, MSc, FRCPC
ILLUSTRATIVE CASE
A second-year Community Medicine resident wonders if there is an association between circulation of influenza virus
and rates of hospital admissions for pneumonia, chronic lung disease and congestive heart failure. As a first step, he
approaches faculty members with experience and expertise in accessing and analyzing the relevant provincial health
databases.
11 Administrative data are created in any portien of the system. These groups may be of any size,
rnultiple serrings wirhin the healrh systern (e.g., in primary
care practices, arnbularory clinics, hospital deparrments, o
ranging from the parienrs of a walk-in clinic ro a country's
entire population. Administrative darabases can also be
-
nursing homes, and healrh authorities at the regional, pro- used ro define patient cohorts and determine whether
vincial, federal and internacional level). Consider a parient patients encountered particular exposures and had specific
who presems ro the emergency deparrrnent of a universiry outcornes. These capabilities give ADR the potential ro
reaching hospital wirh shortness of brearh. Her derno- answer imporrant quesrioris regarding associations and
glaphic information and rime of arrival are recorded in rhe outcomes in health careo
parient registration sysrern, her chest radiograph repon is However, ir is difficult ro define ADR purely in terms of
recorded in the hospital's diagnosric imaging systern, and the types of data it uses. Typically, ADR uses data collected
rhe dose and rime of all medicarions she receives are record- to support the adrninistration of healrh care by capruring
ce! in the hospital's pharmacy system. Finally, the identiry
of the physician who assessed the patient and made the
CHAPTER OBJECTIVES
final diagnosis is recorded in the Narional Arnbularory
Care Reporting Sysrern." 1hus, panicular aspects of each After reading this chapter, you should be able to:
patienr's cncounrer are recorded in differem datasets. • describe what administrative database research is and is
In adrninistrativc database research (ADR) , these not
hea1th datasets are linked by researchers [Q creare a surn- • identify research questions suitable for administrative
mar)' of each patienr's encounter wirh various aspecrs of the database research
health care sysrern. This allows researchers [Q systematically • describe the basic steps necessary to design and conduct
srudy the interacrions rhat groups of individuals have with an administrative database research project
• discuss the strengths and limitations of administrative
database research
a See http://secure.cihi.ca/cihiweb/dispPage.jspicw_page=services_nacrs_e
,---------------------------------------.----_.
KEY TERMS
Administrative data Data-sharing agreement Electronic medical record (EMR)--J
Database Electronic health record (EHR) Health datasets
--------0-.- . _ . ._.
__. _
© 2011 The Royal College oí Physicians and Surgeons 01 Canada 97

demographic inforrnarion and calculating rates of service which could involve rhe investigator obtaining additionai
use, volume of procedures, diagnostic rates, and so forth. rraining, enlisting suirable collaborarors or hiring expefl
There is sorne uncertainry as ro whether research that uses consultants-ADR is relatively inexpensive.
data in electronic medical records (EMRs) and electronic
health records (EHRs) constitutes administrative health ADR is relatively quick. In addirion ro being expensive,
research in that sense. On the orher hand, sorne data corn- data collection can be very time-consuming. Since the data
ponems of EMRs and EHR~, such as diagnostic or proce- required for ADR already exist, delays related ro data col-
dural codes, are identical to those rypically found in lection are avoided. Although time is required ro examine
adrninistrative databases. Other components, such as labo- and clean rhe data used in an ADR, rhis rakes much less
rarory test results or radiology repores, would be considered time than actually collecting the data.
"gold standard" data; sorne investigators using such data (as
in a medical record review) would not consider themselves ADR is often population-based. A frequent Iimirarion of
to be doing ADR. For rhese reasons, a standard deíinition many research projecrs is the limited or unknown general-
ofADR does not exist. izabiliry-the externa] validity-of the results. This limita-
111c type of data used in ADR demands that several par- tion arises from rhe facr that most research is conducted on
ticular steps be taken ro plan a research project. Firsr, you a sarnple, which can be a very small or non-randorn extract
musr determine wherher rhe adrninisrrarive data rhat you of the population rhat i5 purportedly being studied. ADR
have in mind can actually answer your question. This can avoid rhe limiration of poor generalizabiliry if the data-
requires discussion with someone who has an in-depth bases that are used actually capture an entire, identiíiable
knowledge of rhose data. Factors ro be considered in this study population. For exarnple, the Omarío Drug Benefirs
step include data quality, darabase completeness and data- database can be used ro conduct population-based research
base duration (i.e., does the database go back far enough in regarding out-ot-hospital medication use arnong Ontari-
rime to answer your question?). Second, you must ensure ans aged 65 years and overo This is because covered medica-
thar you, or anorher rnernber oíyour research team, will be tions dispensed ro all of Ontario's seniors are captured in
able ro access the required data. If you cannot access the this database. Being able ro conduct population-based
data, your project will be impossible. Last, you will need research makes adrninistrarive databases especially relevanr
someone on your research tearn with sophisticared proro researchers in population health and ro policy-rnakers.
gramming skills and an intimare knowledge of the data- Of course, if rhe databases being ernployed are relatively
base. Manipularing large darasets in ro an accurate analytical "focused"-such as a database covering a hospital's erner-
dataser requires extensive programming skills and an appre- gency department-e-the generalizability would be limired
ciation of rhe intricacies of each dataset used in the analysis. ro those individuals who received care there.
Without such a person, rhe projectwill be impossible.
ADR makes you independent. It is nor uncommon for a
research project ro fail because a member of rhe tearn do es
Advantages of administrative database
not live up to his or her commitments. ADR can liberare
research
you from rhis porential problem. With a few exceptions (see
The several advantages that rnake ADR an attractive next section), rhere are very few barriers in ADR thar stand
approach ro many research quesrions can be described as berween you and a completed project-assuming rhat you
follows. can formulare a good research question and crirically review
rhe relevant literature, as discussed in chapters 6 and 7,
ADR is relatively inexpensive. The bulk of most research respectively. Minimizing the number of people upon whorn
hudgets is spel1t on data collecrion, which requires dedicar- you rely is cssenrial ro a succcssíul research projecr and
ed sraff and a great deal of time. Because administrative career (especially when you are just starting out).
databases already exisr, data collection and irs associated
costS can be avoided (assuming rhat no supplernentary data ADR analyses can be repeated to monitor interventions.
collecrion is required). With rhe proviso that darabase and One of the most satisíying aspects of ADR is rhe fact rhar ir
specialized biosrarisrical experrise are often required- draws upon databases that are cominually updared. This is

13 Administrative database research
especially advantageous when one wants ro monitor che healrh r or in order (O creare classifications or groupings.
eflecrs of inrerventions that have been introduced ro address l": umerous sreps occur berween rrue diagnosis and rhe
problems identified through ADR. If the original analysis assignmem of acode, and error is possible at any one of
rhat idenrified the problem to be addressed by rhe new rhese steps. The published literature is replete with exarn-
intervention has been recorded in a computer program, the pies of codes being eirher insensitive or non-specihc for a
effect of the intervention can be immediately-and repeat- particular diagnosis or procedure rhat it supposedly repre-
edly-determined by simply rerunning the analysis. senrs, As a resulr, the inclusion of so me proportion of inac-
curare codes will yield resulrs that differ gready from those
based on rhe actual enriry represented by che codeo
Potential disadvantages of administrative
The problem of coding error is amplified by the large
database research
numbers of records rhar are screened for the presence or
The several notable potencial weaknesses of ADR can be absence of rhe code of interest. The proporrion of rows in
ourlined as follows. an adrninistrative dataset with a particular code can be very
low. Trouble can arise when a disease with a ve~ylow preva-
Data inaccessibility. Whether they cover a practice, lence (i.e., a low pretest probabiliry) is combined with an
department, insritution or enrire province, adrninistrative irnperfect test (such as a diagnostic code being used in :he
databases conrain personally sensitive information. There- administrative database research study). 111e probabiliry
fore, data custcdians (i.e., the organization responsible for that someone who is assigned a given code truly has the.dis- -,
the dara's securiry) will be appropriately risk-averse when ir ease ir represents can be calculated by multiplying·the pre-
comes ro providing researchers with access. Frequendy, test odds of the disease by rhe posirive likelihood ratio of the
databases can be accessed only in a controlled manner, on- codeo When this calculation is made in cases where the con-
site at the organizarion responsible for rhem, and only aíter dition is rare or the code is inaccurare, the probabiliry rhat
permission has been formally requesred and approved. people with the code rruly have the condirion it is supposed
Accessing data wirhin such organizations is usually feasible ro represent can be less than 50%.1
only if one is sponsored by or col!aborares with a researcher Because of the issue of coding inaccuracy, an ADR thar
within rhat organizarion. However, administrative data are requires a code ro define the cohort, exposure or outcornes
generared ar al! hospitals and are usually much more avail- should be preceded by a srudy to determine the accuracy of
able for ene-time research projecrs. Such projecrs require the code as wel! as the frequency of the condirion in ques-
approval from the local research erhics board and should rion.! Both coding inaccuracy and the rariry of a condirion
involve a formal data-sharing agreement berween rhe will dramatically increase the risk ofbias in ADR.
researcher and rhe hospital.
Poor data detail. When clinical researchers srart uSlI1g
Poor or unknown data gua1ity. Although data qualiry is a administrative darabases for research, they are frequenrly
porenrial concern in al! rypes of research, ir is a particularly srruck by rheir lack of clinical detail. Even basic, commonly
familiar rherne in ADR. Adminisrrative databases are usu- used clinical information (such as the patienr's viral signs)
ally crea red for a variery of reasons unrelated (O rhe conducr are nowhere (O be found in rypical adrninistrarive databas-
of research (i.e., for adminisuative or clinical purposes). As es. This lack of clinical derail can severely limir the rype of
a resulr, the prioriries placed on information processing are research quesrions that can be answered with ADR. Two
very difFerent from rhose required for ADR. Therefore, rhe rnerhods are cornmonly used ro make up for chis weakness:
accuracy of so me adminisrrative dara required for a panicu- supplementing the information in the administrative data-
lar research question may be poor if those data are not base wirh data collected through sorne other rneans (e.g., a
irnportant for rhe main purposes oF rhe darabase and, as charr review); and linking pure adrninisrrative data with
such, are not collecred or maintained wirh the artention data from high-qualiry clinical data reposirories, such as
required for high-qualiry data. elecrronic medical records, which conrain much more
Many adminisrrative darabases record diagnoses and derailed clinical information.
procedures using codeso These codes are assigned by health
records analysts after they have reviewed the applicable The requirement for computer programming and statis-
© 2011 The Royal College 01 Physicians and Surgeons of Canada 99

The Research Guide: A primer for residents, other health eare trainees, and practitioners
tical expertise, Many steps willlikely be required to modify ror or ro acquire sufficient project funding to hire a con:
administrative dara into a suitable analyrical dataser for ranr with the necessary biostatistical expertise.
your research projecr. A firm knowledge of a programrning A final "limitarion" of ADR arises from rhe fact t.
language rhat allows you ro do this (e.g., SAS, SQL) is nec- adminisrrarive datasers enable large sample sizes and,
essary ro conduct ADR. Many people are scared off by the that reason, the results of rnost inferential analyses will
thoughr oflearning how ro programo lf at all possible, rhese sraristically significant. As a result, ir is essemial ti
feelings should be repressed. Learning how to program does researchers pay more attenrion ro the c1inical importar
more than make independent ADR possible. Ir is a skill that and practical significance of the differences seen berwe
will assist you in all research fields, since rhe vasr majoriry of populations as opposed ro rhe sratistical significance of su'
all research projecrs will have an analyrical daraser, Being differences.
able ro examine and analyze that daraser (which you will be
able ro do afrer learning how ro program) is key ro knowing
Conclusion
your data and rruly understanding your projecr. Many
books and courses are available ro reach you how ro pro- Administrative database research holds many importar
.- gram common Janguages. (Two examples are given in Addi- advantages, especially for junior researchers, provide
iional Resources.) rhat appropriare support is available from one or mor
" ..... A high leve! of sraristical knowledge and expertise is ofren

required. ADR may involve repeated dara measures that are
suitable senior researchers.
keep rhe notable limitations
However, ir is irnportant
of ADR in mind when yOI
t,
often collinear and, as such, require special analytic tech- consider using rhis merhodology ro answer your res ea re!
niques. It is otten not feasible for individual researchers ro quesrion. •
learn these, which means that you will either need to seek
out a biosratistician to join the research ream as a collabora-
CASE POSTSCRIPT
Io examine the association between circulation of influenza virus and rates of hospital admissions for pneumonia, chronic
lung disease and congestive heart failure, the resident identified, approached and collaborated with several established
health services researchers who enabled his access to the provincial influenza surveillance and hospital discharge databases.
With this assistance, particularly that of a biostatistician with expertise with time-series analyses, the study was successfully
completed and published" in a widely respected peer-reviewed journal.
REFERENCES
1. van Walraven C, Bennett C, Forster AJ. Administrative database research infrequently used validated diagnostic or procedural
codesoJ C1inEpidemiol. 2011;64(10):1054-59
2. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin
Epidemiol. 2005;58(4):323-37.
3. Upshur RE, Knight K, Goel V Time-series analysis of the relation between influenza virus and hospital adrnissions of the elderly
in Ontario, Canada, for pneumonia, chronic lung disease, and congestive heart failure. Am J Epidemiol. 1999; 149(1 ):85-92.
Cody R. Learning SAS by example. a programmer's guide. Cary (NC): SAS Press;2007.
Delwiche LD, Slaughter SJ. The little SAS book. a primer. 4th ed. Cary (NC): SAS Press;2008.
80th of these references provide an excellent introduction to the SAS programming language and applica ion.
100 © 2011 The Royal College of Physicians and Surgeo s o' Ca-~
13 Administrative database research
EXERClSE
List some research questions that may be suitable for administrative database research. List some that would not be,
SUMMARY CHECKLlST
o Select a question that you might use administrative data to address.

O List the advantages and disadvantages of this approach for your specific question.
t:n:
-
V
-
•••

11_
,,_
102 © 2011 The Royal College of Physicians and Surgeo s o; C¿~¿:¿

14
Health professions education research
Jason R. Frank, MD, MA(Ed), FRCPC
ILLUSTRATIVE CASE
A 2nd-year Emergency Medicine resident knows that she will soon need to start a research project as a requirement for
her programo Although most of her colleagues are doing medical record reviews of clinical problems, she is not particularly
interested in following suit. What is really on her mind is her concern about the radiology curriculum in her residency
programo She feels more at ease reading musculoskeletal
whether
and chest films than brain CT scans, for example, and wonders
other residents share her concern that the program has gaps that need to be addressed. However, she has no
_.
idea how she would explore this question or even if it is an acceptable topic for scholarly research.
.......
11 The daily lives of front line health professionals. Everyrhing from admissions decisions ro
care professionals provide arnple evidence of rhe successes reaching merhods, curriculum topics, assessrnenr tools,
and chal!enges of healrh professions educarion (HPE). In program evaluarions and more are pan of this subject area.
cvery clinical encounter, we wirness the abilities of other The discipline is in faet mulri-dimensional, and its aspecrs
healrh ca re professionals; somerimes, we contemplare rhe are pervasive in rhe work of al! rhose involved wirh rhe vari-
limirs of our own comperence. HPE is a borh a field of ous aspecrs ofhealrh careo
scholarship and an enrerprise focused on rhe preparation of
healrh care professionals to meet rhe needs of rhe patienrs
and communiries rhey serve. 1he spectrum ofhealth pro-
CHAPTER OBJECTIVES
fessions education spans undergraduare rraining (e.g.,
MD, DDSc and BScN programs), posrgraduare (e.g., resi- After reading this chapter, you should be able lo:
dency) educarion leading ro certificar ion and practice, fur- • define health professions education (HPE)
ther advanced rranung (e.g. fellowships), lifelong • define HPE research
conrinuing professional developrnent, and faculry develop- • describe four broad categories of HPE research
ment in academic health science cenrres (see Fig. 14.1). As • describe the typical areas of interest of those involved in
a scholarly field and an enrerprise, HPE explores the most HPE research
c:ffective curricula and sysrerns ro ensure that patienrs and a describe typical rnethods used in HPE research
socieries are besr served by comperenr and caring health
r----~ .._---.- .. -----_. __ .. -_--

KEY TERMS
Assessment Kirkpatrick's four-Ievel framework
Cohen's d Needs assessment
Competencies Program evaluation
lnstructional methods Spectrum of health professions education
_._j

The Research Guide: A primer for residents. other health care trainees. and practitioners
Figure 14.1
The spectrum of health professions education
Undergraduate health Graduate health Fellowship education Continuing Faculty development

professions education professions education professional
(residency or GME) Advanced training development (CPD) Preparation.
Formal programs leading ",.,. in focused areas enhancement, retention
10 an initial health Formal proqrams Lilelong learning and renewal 01 Iaculty
prolessions degree leading to certilication in practice lar academic careers
(e.g. MD, DDSc, BScN) ancl practice
Hea!th professions education research is a legitimate • rhe idenrificarion and characterization of

domain of scholarship, ro which numerous journals," texrs competencies needed by a group of health care
(see Additional Reading) and careers are devored, If HPE professionals in a given setting or conrext
plays a critica] role in preparing healrh care professionals to • group consensus processes ro creare staternents on
.• - serve sociery, then HPE research provides insights and evi-

dence to guide that education.
introducrion
111is chaprer provides an
to the field and an overview of the kinds of
prioriry curriculum
• rhe identification
clernenrs
of gaps berween idenrihed prioriry
abilities of professionals and rheir current abilities in a
HPE research studies you might be interested in contribut- popularion (needs assessment)
ing too All of rhe principles and essenrial steps involved in • rhe characterization of exisring or innovarive curricula
any research study, and all of the major merhodologies or assessrnent sysrems.
described in rhis guide, are relevant ro HPE research siud-
ies, However, che subject areas ofHPE and the contexrs in These srudies are essenrial ro ensure rhar health profes
which they are explored make HPE research unique.l " sions education is focused on rhe priorities that are mos
relevant to the needs of the parienrs and popularions servec
by gradu3.tes. Wirhour rhis body of work, currículum (;011.·
Categorizing health prof'essions
tent is otten left up to each individual reacher or prograffi
educatíon research studies
director in an ad hoc rnanncr, Neufeld and colleagues call
Whar kinds of studies are addressed by health protessions this kind of research Iocusing on "dcmand-side medical
education research? Both quantitarive and qualitative educarion.'?
methods beiong here, as does research focused on eirher an
underlying theory or a pracrical everyelay problern."
Although there are many ways of concepwalizing research How should we teach, lea m or assess?
in this fidd,5J, one simple way is to categorize HPE research This caregory deals with instructional methods or rcch-
inro four arcas according ro che broad questions being niques used ro assess a healrh professional's competence.
asked (see Table 14.1). These four areas of questioning are Akin ro case reports in the clinicalliterature, srudies in rhis
outlined in rhe following sections. area are ofeen descriptions of curricula, rools or sysrems,
including:
What to teach, learn or assess?
The studies in rhis caregory answer questions about whar • descriprions of existing or innovative curricula
belongs in the curriculum of a health care professional. e descriptions of reaching or learning rnerhods
\Xlhat should be taught? What should be learned? Whar • descriptions of assessmenr rools and processes
should be assessed? These quesrions can be thoughr of as • hisrorical srudies on health professions education
dcaling wirh "conrent issues" 01' "inpurs." Conternporary
research in this area is con cerned with, among other things: These kinds of studies may be declining in rhe HPE
research lirerature, giving way to increasing ernphasis
on rhe next caregory, namely srudies rhar evaluare
a E.g., Academic Medicine, Medical Educetio». Medical Ieochcr, Advances
in neettt. Sciences Education, Gerontology & Gerietrics Education,
objective and subjecrive ourcomes relared ro curricular
Journal of Dental Education, International Joumal of Nursing Educetion approaches.v+
Scbolership, and Joumal of Nursing Education.
104 © 2011 The RoyalCollege01 Physiciansand Surgeo s o: Ca-,,:;¿

• L! Health professions education research
Table 14.1: Categories of health professions education (HPE) research studies
Category Areas of interest Methods
What to teach, learn, or assess? • Needs assessments (perceived. observed, • Surveys

societal, epidemiologic, institutional) • Delphi and other consensus methods
These studies seek to describe or identify
what should be taught, learned or • Group consensus s atements
assessed by a given group of health • Descriptions of curricula
professionals in a given context.
• Descriptions of assessment strategies
---------------------------- ---------- -----------
How to teach, learn, or assess? • Descriptions of curricula • Educational case studies
These studies seek to describe, clarify, • Teaching/learning techniques • Historical studies

and identify instructional or assessment • Teaching/learning innovations
methods for the health professions.
• Assessment techniques / tools
• Assessment innovations
t
• History of some aspect of HPE
n
What works? • Program evaluations • Controlled educational trials ..,
y
le
These studies explore the outcomes of
curricula, programs and systems in HPE_
• Validation of assessment tools
• Effectiveness of teaching approaches
• Goal-based evaluations
• Responsive evaluations ---
• Descriptions of outcomes
• History of some aspect of HPE
or effects • Expert reviews
• Focus groups and other qualitative
methods
-
• Cohort studies
• Case-control studies
• Historical studies
Why does it work? • Theory/concept building • Controlled educational trials
These studies clarify why a given element • Theory/concept testing • Cohort studies
of HPE does or does not contribute to • History of some aspect of HPE • Case-control studies
a certain impact or outcomes. They
• Focus groups and other qualitative
contribute to our understanding of methods
underlying theories or concepts related
• Historical studies
to HPE.
What works? Program evaluations use an enormous variery of rnerh-

Increasingly, scholars in healrh professions educarion have odologies. For any study, the selecrion of rhe rnost appro-
been focusing rheir atrenrion on characrerizing rhe eflec- priate merhod should be derermined by rhe narure of the
riveness of elemenrs of thc HPE enrerprise. This is consid- quesrion being asked. For exarnple, "goal-based" or pro-
ered ro be pan of the cornmirment ro be accountable ro the gram-orienred evaluarions compare the actual ourcornes of
populations served by the health care professions.' These a program design wirh irs inrended ourcomes. "Responsivo"
srudies ofren rry ro solve pragmaric problems in rhe field or parricipanr-orienred evaluarions focus on rhe inrended
and represenrs pcrhaps rhe majoriry oE conrcmporary pub- or unintcnded effccrs of a program as pcrceived by srakc-
lished papers. This is the domain of program evaluations holders. So-called "connoisseur" evaluarions rely on reviews
oE all kinds, using qualitative or guantirarive rnerhods, or by experrs in a field.9 Epidemiological approaches are also
borh. In this caregory one can find srudies on: relevanr herc, including conrrolled rrials, systernaric
reviews, cohorr srudies and case-control srudies. These
• rhe effecriveness or irnpacr of exisring or innovarive approaches ro healrh professions educarion research are
curricula sornetimes considered rhe most robust, given their Eocus on
• the effectiveness of reaching or learning merhods outcornes and rheir use of methods thar are similar ro rhose
• rhe efl-ecriveness of assessmem tools and processes applied in the clinical research lirerature. Regardless, HPE
• hisrorical perspecrives on health proíessions educarion research is cerrainly a broad and varied dornain of acriviry

\
-"'r:.;;;;.. '
' •.• ~ ~I;'It- l. "
that is essential ro informing the advancement of health rule ofHPE reporrs. Ir is essential ro conduct a thor
professions educarion as a field and rhe structure and func- ough lirerature review ro ensure that you are aware o
tion ofHPE as an enterprise. what has already been done and what theories, con
cepts or hyporheses have already been explored. Ma
Why does it work? jor search databases are the same as rhose discussed il
The final category includes studies that ofren use rnerhods chaprer 7, with the addirion of ERIC.d Consultatior
similar ro those used ro address the question "Whar with a research mentor who is familiar with this lit
works?" bur focus instead on what Cook and colleagues call erature and enlisring the help of a librarian can sav:
"clarificarion."? These studies are usetul in informing the you rnonths oE searching.
development of conceptual frameworks wirhin health pro-
fessions education, making predicrions trom theories 3. Carefully frame an appropriate health professions
applied ro HPE, and resring hypotheses in HPE.5.10 A wide education research question. To be effective, yOUl
variery of srudies and merhods can be found here. Never- research question should be tightly focused and c1ear-
rheless, this category is ofren neglected by HPE scholars, a ly srare the relevant population, variables and conrext
facr rhat is decried by sorne authors." 11-13 of rhe srudy (see also ch. 6).
4. The research question usually dictates the rnost

Conducting a health professions
appropriate methods, The merhods you selected for
education research project
your srudy should be practical, and may need ro in-
Doing research in health proíessions education involves the volve both quantitarive and qualitarive approaches.
same fundamental steps and principies described elsewhere Because it is dirficulr to control variables in educa-
in rhis guide. However, context is especially critica] ro HPE rional research, randornized controlled rrials are rela-
research: more than is rhe case, for exarnple, with research tively inEreguent in rhis field.
011 physiological phenornena, rhe validiry, reliability and
generalizability of HPE studies are affectcd by the settings 'o Consider carefully the population of interesr. Bias
and the population studied. By way ot emphasis, the fol- can easily be introduced at this step in any study.
lowing list outlines some irnporraut elernents ro consider in Which population of health care professionals is rnost
conducting research relared ro HPE. 14.; 5 appropriate fOI your question? Which aspect of rhe
specrrum of healrh professions education is relevanr?
1. Question the curricula around you. Does rhe teach- Is your sample size adequare ro enable you ro make
ing, learning and assessmenr in your professional en- generalizable conclusions abour rhe population?
vironment work well? Whar elernents could be bet-
ter? Are rhe rnost appropriate topics being raughr? 6. Select outcomes carefully. These should not only
How do you know wherher the graduares of a given match the goals of rhe study (e.g., a wrirren test for a
program are cornperenr? Viewing your own environ- srudy of change in knowledge) bur, wherever possible,
ment rhrough these lenses can help identify impor- should rry ro connect an intervention ro "higher or-
rant problerns ro explore in a scholarly fashion. der" outcomes such as change in behaviour. Kirkpat-
rick's four-Ievel framework is often cited as a guide
2. Review the relevant health professions education ro selecring outcornes. Ot rhese levels, Kirkparrick
research literature. There are rhousands of published suggesrs thar levell-participant reacrion or satisfac-
srudies going back more rhan a century related ro rion-is the most basic and most commonly mea-
health professions educarion. These include papels sured, Beyond this are three more rypes of outcornes:
published in major general medicine jourrials" and in change in knowledge, change in behaviour, and irn-
HPE journals.' There is also an exrensive grey litera-
Evaluation in the Health Professions, Medical Educaríon, /.eC'CE -s;r-",-

b E.g.. JAMA. Journal of Generallnternal Medicine, the New England and Teaching and Learning in Medicine.
Journal of Medicine and The Lancet. d ERIC--the Education ResourcesInformaiion Cen er-is a'l 0'1' -e::;--=-
e E.g.. Academic Medicine. Advances ir¡ Health Science Education, library of education researchand information; see wW .. eP .e-::.,,~.
106 © 2011 The Royal College of Physicians and Surqecrs e: ::::",-a::c

í4 Health professions education research
pacr. For health professions educarion research, rhe review, Others are very strict in considering any HPE
latrer can be thought of as patienr- or populaticn- research srudy rhar involves human subjecrs ro be
level ourcornes (e.g., improved diabetes control)." in need of ful! review. Regardless of che local REB's
perspecrive, you must fol!ow standard procedures tO
7. Consider validating your instruments. Alrhough prorec[ your subjecrs from any potencial harm arising
many instruments already exist (O measure heal rh from their parricipation in the study. You should also
professions educarion research outcornes of relevance, bear in mind rhar many HPE journals wi!! req uire
few are validared, This limi tation gready weakens evidence of ethics approval.
some srudies, Can you use a pre-existing rool for your
popularion and context? If you have ro creare a new 10. Be cautious in interpreting your data. No study is
one, how can you reassure a peer reviewer of its valid- perfect, and so be circurnspecr in considering what
iry and reliabiliry? your data reaHy show. What are the limirations of
your rnerhods? Whar do you rhink is generalizable
8. Carefully consider effect srze. Data analysis in beyond your serring, corirext or population? How can
health professions education research srudies is nor your study reinforce or refute accepted beliefs, or gen-
always focused on P values. Staristical significance erate new hypotheses and concepts? What does your
is often less imporrant than educational irnportance srudy contribute ro rhe field?
and irnpact. Your srudy prorocol should define a pri-
ori what constitures a meaningful difference. Where
Conclusión .
rhe literature does nor provide a benchrnark, starr
with careful reasoning and, if possible, expert censen- Research in health professions educarion is a dynamic, rap-
sus. Cohen's d is a sratisrical rneasure that is sorne- idly growing field of scholarly endeavour that informs and
times used for rhe analysis of educacional impact.'? explores al! aspects of preparing healrh professionals for
practice. [t is a legitimate area of inquiry, even for novice
9. Be mindful of differing perspectives on the ethics researchers, and its results can inform improvernenrs ro
of health professions education research. Some myriad aspects of the education and training of healrh care
Research Erhics Boards (REB) do not consider HPE professionals. •
program evaluarions ro need a certificare of ethical
CASE POSTSCRIPT
Fortunately for the resident, her next shift is with a clinician-educator faculty member. After discussing the resident's research
interest with her, the faculty member agrees to serve as her supervisor on a project that will be designed to explore radiology
teaching in the EM programo After further discussion with her supervisor, and after reading relevant chapters of this guide,
the resident settles on conducting a national needs assessment survey of program directors and residents in Emergency
Medicine. Her conclusions are presented during her department's research day and form the basis for recommendations that
are adopted by the residency training committee the next year.
REFERENCES
1. Norman G. Research in medical education three decades of progress. BMI 2002;324(7353) 1560-2.
2. Harden RM, Grant J, Buckley G, Hart IR. BEMEGuide no.I: Best Evidence Medical Education. Med Teach. 1999;21 (6):553-62
3. Dauphinee WD, Woocl-Dauphinee S. The need for evidence in medical education. Acad Med. 2004;79(10):925-30.
4. Albert M. Understanding the debate on medical eclucation research: a sociological perspective. Acad Med. 2004;79(10):948-54
5. Cook DA, Bordage G, Schmidt HG. Description, justification and clarification: a framework for classifying the purposes of
research in medical education. Med Educ. 2008;42(2) 128-33.

6. Regehr G. Trends in medical education research. Acad Med. 2004;79(10):939-47.
7. Neufeld VR, Maudsley RF,Picring RJ,Walters BC, Tunbull JM, Spasoff RA, et al. 1993. Demand-side medical education: educatin
future physicians for Ontario. CMAl 1993; 148(9): 1471-7.
8. Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79(10):955-60.
9. Fitzpatrick JL, SandersJR,Worthen BR. Program eveluetion: alternative approaches and practical guidelines. 4th ea. Boston:
Pearson/ Allyn and Bacon; 2011.
10. Bordage G. Conceptual frameworks to illuminate and magnify Med Educ. 2009;43:312-9.
11. Regehr G. lt's NOT rocket science: rethinking our metaphors for research in health professions education. Med Educ.
2010;44(1) 31-9.
12. Bunniss S, Kelly DR. 2010. Researchparadigms in medical education research. Med Educ. 2010:44(4):358-66
13. Gibbs T Durning S, Van Der Vleuten e

Theories in medical education: towards creating a union between educational practice
and researchteaching. Med Teach. 2011 ;33:183-7.
0"\ 14. Shea JA, Arnold L, Mann KV A RIMEperspective on the quality and relevanceof current and future medical education research.
.c:
- Acad Med. 2004;79(10):931-8 .
e 15. Cook DA, Bowen JL, Gerrity MS, Kalet AL, Kogan JR,Spickard A, et al. Proposed standards tor medical education submissions to
the Journal of General Internal Medicine. J Gen Intern Med. 2008;23(7):908-13.
C'i
.-
~
16. Kirkpatrick DL, Kirkpatrick JD. Evaluating training proqrems: the four levels. San Francisco: Berrett-Koehler; 2006.
OJ 17. I(enny DA Statistics for the social and behavioral sciences. Boston: Little Brown; 1987.
O
EXERClSES
1. Brainstorrn sorne questions or problems you perceive frorn your experiences in health professions education.
Are any of them worthy of an HPE research study?
2. Select one of the problems you identified in the exercise above. Do any published studies examine this problem?
If so, what populations and contexts do they involve?
3. Select any study from one of the major health professions education journals mentioned in this chapter.
Critique its methods. Do you agree with its conclusions?
SUMMARY CHECKLlST
o Oiscussyour interest in health professions education research with a qualified education research mentor.
O Identify sorne important problerns to explore in an HPE research study
O Select the category of HPEresearch study
O Review the HPEresearch literature.
O Carefully frame an appropriate HPEresearch question.
O Select methods that are feasible in your context
O Define the population of interest
O Carefully select outcomes.
O Define and describe the Kirkpatrick levels involved.
O Consider validating any instruments.
O Carefully consider effect size.
O Carefully design your analysis.
O Submit your proposal for REBreview, as appropriate.
O Interpret your results in the context of the HPEresearch literature.
108 © 2011 The Royal College oí Physicians and Surgeons 01 Canada

15
Systematic reviews
Thomas A. Lang, MA
ILLUSTRATIVE CASE
For her master's thesis, a nursing student is considering a research project that will explore the effects of nurse staffing
levels on patient outcomes in acute-care hospital settings. Her advisor suggests that she conduct a systematic review on
the topie. Because she is unfamiliar with how to carry out a systematic review, her supervisor suggests several resources
(including this guide) for her to review before their next meeting.
-,
• A systematic review is a planned, In contrast, narrative reviews are usually prepared by
comprehensive, and reproducible summary of research experts in the field who draw only on rheir personal read-
results on a specihed topic.' Systemaric reviews are con- ing, experience and judgment ro summarize the topic.
ducred according ro a written prorocol, developed in Thus, a narrative review is neither sysremaric nor reproduc-
advance, that carefully defines the research quesrion to be ible; difierenr experts may cire differeut evidence for differ-
addressed, rhe desired characteristics of rhe research needed ent reasons, have different experiences, and reach different
ro answer the quesrion, and the derails of rhe search strate- condusions. Narrative reviews also rarely allow the resulrs
gies that will be used to find articles reporting rhis research. ofindividual studies ro be pooled or synthesized statisrically
The protocol also specifies criteria for selecting the anides in any valid fashion.
ro be reviewed from among rhose identified in the search,
the specific variables for which dara will be absrracted from
each anide, and the methods used ro reduce the risk that
CHAPTER OBJECTIVES
these selection processes will bias the srudy. Finally, the
resulrs are compiled into evidence rabies so that rhey can be After reading this chapter, you should be able to
interpreted in the context of al! similar studies. In some cir- • list at least three ways in which systematic reviews differ
cumstances, the numerical resulrs can be pooled statistical- from narrative reviews
ly in what is called a meta-analysis. The planned and • list three uses of systematic reviews
sysrematic nature of rhese reviews helps to reduce bias, and • describe the steps in conducting a systematic review
because their results are in tended ro be reproducible, the • identify at least three sources of information about
interna] validiry of rhese reviews can be verihed. systematic reviews
KEY TERMS
Ancestry searches Grey literature MOOSE Statement Realist reviews
Clinical trial registries Hand searches Narrative reviews Risk of bias table
Cochrane Collaboration Heterogeneity Pearl-growing Scoping reviews
Data abstraction Inter-rater agreement PRISMA Statement Selection bias
Evidence tables Intra-rater agreement Publication bias Sensitivity analyses
Forest plot Meta-analysis Rapid reviews Summary of findings table
--------- -------------------------------

Sorne vananons on sysrernatic reviews are becoming • summarize a large and cornplex body of literature on
more common. Rapid reviews are abbreviared systernaric topic;
reviews designed ro be cornplered in a few weeks ro a few • clarify the strengths and weaknesses of srudies on a tapie
rnonths, as opposed ro rhe 6 ro 12 monrhs generally • assess rhe consisrency of results across studies;
reguired ro conducr a rypical full sysrernaric review. Time is • identify and attempt to explain rhe reasons for
saved by limiring the scope of the Iiterature search, having a conRicting reports in rhe literature;
single invesrigaror selecr srudies and abstraer data, not eval- • document the need for funher study; and
uating sources for bias, and so on. These reviews can be use- • collect rhe data needed ro plan large c1inical trials, sucl
fuI for answering cenain rypes of c1inical and policy as rhe expected variance of the outcorne of interesr,
q uesrions, but rhey do not replace full sysrernatic reviews.? rypical patient accrual rates, sources of bias, and so on.
Scoping reviews assess the general characteristics of a
problern. They are a form of "reconnaissance" or planning Meta-analyses based on systematic reviews can also be
done in advance of a full srudy ro help inform its designo useful to:
Scoping reviews may or may not be sysrernatic, They often
.- indude a wider range of study designs or sources than a • provide a quantitative estimare of a treatment effect;
comparable sysremaric review and are less concerned with • improve rhe precision of an estimated treatrnent effecr;
.-
V')
the merhodological
combining
qualiry
thern into a single result.
of the induded
Instead,
srudies or wirh
rhey are con-
• detect smaller trearrnent
reported in individual
effecrs rhan have been
studies, and
QJ cerned wirh what rhese srudies can contribure ro furure • investigate variations in treatment effects rhrough
el research efforts, such as informarion

Realisr reviews are essentially
about ranges or trends."
systernatic reviews of
subgroup (stratified) analyses.
qualitative data írorn a variery of sources, induding inter-

The Cochrane Collaboration
views, observarions and published research. Their purpose
is LO iden tifY, test or refine possible explanations about what Perhaps rhe single most important resource for health-relat-
inrerventions or programs work, in what respects, for ed systernatic reviews is rhe Cochrane Collaboration."
whorn, how, and in which circumstances. Traditional sys- Founded in 1993 and named atter rhe Brirish epiderniolo-
ternario reviews might [OCUS on, say, a program's effective- gisr Archie Cochrane, rhe Cochrane Collaboration is an
ness ..whercas realist reviews focus on why the program is or international not-íor-profir organizarion rhar suppons evi-
is nor effective. The goal is ro provide a detailed, realistic dence-based pracrice by producing, disseminating and
and practical understanding of complex activities that can updaring systernatic reviews and meta-analyses of health
be applied in planning and implemenring prograrns." care inrerventions. Through irs website and regional centres,
Systcmatic reviews demand the same scientihc rigour the Collaboration offers training and resources for conducr-
reguired ofall healrh research. Because they are rhe product ing and reponing systernaric reviews" and provides informa-
of a specific research methodology, they follow many of rhe tion for researchers, clinicians, policy-rnakers and patienrs.
same principies of inquiry, and involve design and execu- Ir also maintains rhe Cochrane Darabase of Sysremaric
tion issues thar are similar ro rhose of other research merh- Reviews, which archives systernatic reviews, meta-analyses
ods. In addirion, because they do not necessarily require a and their associated prorocols. Similar darabases are main-
well-funded research infrastructure and extensive research tained by the Universiry of Adelaide's ]oanna Briggs
experience, they can often be undertaken by c1inicians early Insriture," which archives sysremaric reviews and mera-
in rheir careers," although the amounr of time, expertise analyses on healrh ca re pracrices and outcomes, and by me
ami work rhar a good systernaric review enrails should not Campbell Collaboration Library of Sysrernaric Review ,e
be underestirnatcd. which archives sysremaric reviews and rnera-analyses in [he
fields of education, crime, jusrice and social welfare.
The value of systematic reviews

aSee www.cochrane.orq
Sysremaric revicws have several purposes.' For example,
b Seewww.joannabriggs.edu.au/aboutlhome.php
rhey can be conducred ro: e Seewvvw.campbellcollaboration.org/
110 © 2011 The RoyalCollege 01 Physiciansand Surgeonso' a -33

15 Systematic reviews
Conducting a systematic review

pro ocols are being published with rhe sysremaric reviews or
Sysremaric reviews, by definirion, are secondary, rerrospec- are orherwise made available ro researchers who wanr ro fur-
rive srudies of original research. They should, like al! srud- rher eval ate or replicare rhe review.
ies, be rigorously designed and conducred. As are orher
research designs, they are subjecr ro error, confounding and Sample selection: the search strategy
bias," al! of which need ro be minimized rhrough careful The "sarnple" for a sysrematic review-which, ideally, will
planning and arrention ro detail. The sreps in conducring a be a "census" of al! eligible anides of inreresr-usual!y COI1-
systernaric review are described brieRy below. More detailed sisrs of published anides from rhe indexed scienrific litera-
informarion on all aspecrs of conducring systemaric reviews rure. Orher rypes of dara can and ofren should be included.
is available frorn rhe Cochrane Handbook" Somerimes referred ro as grey Iiterature, these sources
indude unpublished dissertations, technical repores and
The research question conference proceedings that may or may nor have been
Systematic reviews, like other research rnethods, require a
restable hyporhesis or a focused research question. The
peer reviewed. The protocol for sample selection specihes
how rhe anides are ro be identified and the crireria rhey
t:rE
hyporhesis or research question usually has several parrs-
the PICOT components described in other chaprers of this
musr meet (O be induded in final sample for review.
A healrh sciences librarian should creare, or at Ieast assisr -
V
---
guide (see especially ch. G): a Parienr (or population) with a with, the search straregy (by identifying the index terrns
problem or diagnosis; the Intervenrion,
nostic test to be evaluated; a Cornparator
ment, placebo, standard rrearrnent);
exposure, or diag-
(e.g., no treat-
an Ourcorne, and a
and Boolean operarors rhat determine
be combined)
how the rerms wil!
and idemify rhe darabases ro be used in the
literarure search (see ch. 7). Although MEDLINE is rhe
--
.le
Time period of interest. Sornetirnes rhere is also a serring of largest database of published anides in the health sciences,
interese (e.g., inrensive care units, Japan, submarines). For a thorough review wil! include several orhers, such as such
example: "In varsiry high-school footbal! players (rhe ser- as Excerpta Medicas biomedical and pharmacological data-
ring and population), is arhletic raping or srrapping (rhe base, Embase; rhe Cumulative Index ro Nursing and Allied
inrervention) more effecrive at prevenring injury or re-inju- Healrh Literarure (CINAHL); and rhe Web ofKnowledge,
ry of the ankle (the ourcorne) than not taping (the compar- an academic cirarion indexing and search service rhar
aror), given advances in taping technology in rhe past 5 includes rhousands of scientific journals, parenrs and con-
years (rhe time Irame)?" ference proceedings. Other straregies include hand search-
lhe scope and even rhe feasibiliry of conducring a sys- es (i.e., looking page by page rhrough key prinr or online
ternatic review on a hyporhesis can often be assessed by journals ro find relevant anides), ancestry searches (i.e.,
determining how many and what rypes of studies are avail- examining the reference Iisrs in selected anides ro find orh-
able for review. If the literature on the quesrion is vasr, rhe er relevant studies), and pearl-growing, in which rhe key
quesrion can be narrowed ro make rhe review more man- words or index rerms (rhe "pearls") of a source anide are
ageable. If rhe lirerature is sparse-usually an irnportanr used ro locate similar anides ("growing the pearl").
finding in itself-rhe quesrion can be changed or rhe proj- The search srraregy needs ro be rigorous ro minimize
ecr abandoned befo re more resources are spent. selection bias, or bias caused by a discrepancy berween rhe
ser of al! eligible srudies and those rhar are acrually idenrified
The research protocol as eligible. Srudies can be missed if the search srrategy is
Once rhe hyporhesis has been derermined, a prorocol for Bawed, if not al! relevanr darabases are searched, if srudies
conducring the review rnust be wrirren. The protocol should published in some languages are nor inciuded, or if eligible
idemify in derail the search srraregíes (O be used (O find arti- srudies are not identified because rhey are indexed
eles, rhe eligibiliry criteria for selecring which articles ro incorrectly or inconsisrenrly. To minirnize che porential for
review, rhe caregories of data ro be abstracted from thern, and publication bias-bias creared by rhe facr rhar posirive
rhe process by which differences berween reviewers in anide srudies are more likely ro be published rhan negarive ones-
seleerion and dara abstraction will be addressed. Increasingly, rhe search strategy also ofren needs ro indude the grey
especially on journal websites and in registries, such as rhose lirerature. For example, 45% ro 65% of meering abstracrs
managed by the Cochrane and Campbell col!aborarions, are not followed by rhe ful! reporrs of the srudies they

\ ---------
represent." Many of rhese srudies may nor have been 4. Possible bias caused by rhe incomplere reponing o
published because their resulrs were not statistically ourcorrie data
significanr, and so failing ro indude rhem in a review can 5. Possible bias caused by the selecrive reponing of Out-
bias rhe resulrs. For rhe same reason, clinical tri al registries comes
should be searched ro determine wherher the resulrs of 6. Other issues thar pose a threat ro the validiry of the
regisrered trials have been published as anricipared. study
In addirion ro addressing the hypothesis, induded studies
are usually resrricred ro cerrain srudy designs (e.g., only ran- An example is how the "blinding" domain mighr be
domized controlled trials), study characterisrics (e.g., only assessed (Table 15.1). Invesrigarors describe specifically
studies with at least 1 year oHollow-up), dare of publicarion how each study addressed each dornain (Table 15.2); devel-
(e.g., wirhin me pasr 5 years), popularions enrolled (e.g., op specific crireria for judging whether rhe likelihood of
patients with dememia), imervemions used (e.g., antide- bias is high, low, or uncerrain (Table 15.3); and rhen assign
pressants), speciíic endpoinrs or outcomes (e.g., pain relief, a judgment.
survival), or logisrical realiries (e.g., rhe rime, cost and accu- These assessmenrs can then be used ro conduce sensi-
raey of rranslating anides published in orher languages). tivity analyses, in which individual aspecrs of che review
Studies included in reviews need ro be assessed for quali- are assessed ro determine how much each affecrs the
ty. This assessment used ro be based on any of several more- results. If the resulrs change markedly when srudies with,
or-Iess objective scales or checklists, bur these are no longer say, a higher risk of bias are induded in the review, rhe
recommended. The current rhinking from the Cochrane results are said ro be "sensitive" ro srudy quality, and the
Collaboration is that a subjecrive assessmenr of the risk of condusions of rhe review can be inrerpreted, presenred
bias is more realisric and usefu!. This assessment is recorded and even stratiíied accordingly.
in a risk of bias table, which documents rhe results and The included studies comprise [he sample and are lisred
allows 0[l1e1'5 LO verify the results, G in the published review, if space perrnits, or pub!ished
Yo creare a risk of bias table, invesrigarors evaluare six online. Á Iist of excluded articles and rhe reasons for their
"dornains" ofa study: exdusion is ofren published as well (sometimes as an
online-only appendix) or are otherwise made available ro
l. How the randornization sequen ce was generared (rhe any researcher who mighr want to replicare or validare the
source of random numbers) review. To determine whether identified anides rneet the
2. How the allocarion sequence (the order in which pa- study's inclusion crireria, each anide is ofren assessed
tients are ro be assigned ro treatrnent groups) was kept independenrly by rwo reviewers. Differences in choices
secret from invesrigarors ("allocarion concealment") are usually resolved by consensus or wirh the aid of a rhird
3. How blinding was accomplished parry.
11 Table 15.1: A sample description of the blinding domain in an assessment of the risk of bias
Domain Description Question to be answered
Blinding of participants Describe measures used to blind study Was knowledge of group assignrnent adequately prevented
participants and personnel from knowing the during the study?
group assignment of each participant.
Describe any efforts to determine whether

blinding was effective.
112 © 2011 The Royal College 01 Physicians and Surgeons oí Car-ada

15 5ystematic reviews
• Table 15.2: A sample summary descriptions and comments for assessingthe risk of bias in blinding
Domain Description
Blinding of participants "The placebo lozenge was identical to the active treatrnsnt. save that the active ingredient had been
rernoved. "
Cornrnent: Blinding was clearly atternpted
Blinding of participants "Patients were blinded to group assignrnent."
Cornrnent: Side effects were pronounced enough to elirninate any effectiveness of blinding
----_._-_ .._._----_.~-_._-----_._---_._._ ..
11 Table 15.3: Sample criteria for determining the degree of possible bias in the blinding domain
Risk of bias Criteria for assessing the risk of bias
High Blinding is possible and desirable but not reported
Low
Unclear
Blinding is described in detail and its effectiveness
Blinding is clairned but details are not provided; other

is confirrned at the end of the study
research by these authors has used blinding successfully.

-
Data collection
When the anides ro be included in rhe review have been iden- form of inrerpretarion than is possible in rraditional narra-
tified, the specihed data are absrracred fmm eaeh of thern, usu- rive reviews. Under sorne circumsranees, rhe nurnerical
ally by (WO or more trained abstracrors working independently results of the individual srudies can be pooled statistically
with a standardized and pre-tested data formo A third rater is in a meta-analysis ro help furrher interpret the resulrs. If so,
otren neeessary ro resolve any differenees in the values abstraer- rhe results are ofren presented in a characreristic forest plot
ed by rile prirnary rarers, As a resulr, inter-rater and intra-rat- thar shows the estimates and confidence imervals for the
er agreement is usually assessed ro identiíy any possible bias individual studies as well as rhe po oled estimare and irs
ereared by differenees in judgment. eonfidence interval (Fig 15.1).
Data abstraction is nor neeessary straighrforward. For In addirion, rhe variarion among studies is ofren assessed
exarnple, alrhough the size of the treatmenr and conrrol staristically with measures such as Cochran's Q (or
groups may be readily apparent, rhe proportion of males Cochran's chi-square) test or J2, ro determine wherher ir is
and females or rhe nurnber of parients actually completing within rhe range expecred by chance or wherher ir indicares
cerrain phases of the tri al might have ro be cornpured from heterogeneity among rhe study resulrs: thar is, wherher rhe
orher numbers. Also, key data are often missing and canuor observed variation among study results is greater than
be obtained from rhe study authors. expected by chance.? For example, J2 ranges berween 0%
and 100%, wirh lower values indiearing !ess hererogeneiry.
Data analysis and interpretation Results found ro be signifieantly hererogeneous usually
Once absrracted, rhe data are organized into one or more warrant funher exarninarion, ofren rhrough planned sub-
evidenee tables. Typically, evidence rables in sysrematic group analyses, in an efforr ro idenrify porential explana-
reviews may show more qualitative data (Table. 15.4), rions, When heterogeneiry is high among srudies, rheir
whereas rabies in mera-analyses may show more quanrira- results are usually not combined. In any event, systernatic
tive data (Table. 15.5). Once organized into evidence reviews and rnera-analyses require competenr statistical
rables, the data can be interprered in rhe conrext of data surporr, which, as in all research, should be sought befo re
frorn similar srudies, providing a more evidence-based the research is planned in detail.
© 2011 The Royal College al Physicians and Surgeons al Canada 113

• Table 15.4: An evidence table showing qualitative data from a systematic review of the effects
of nurse staffing on institutional financial outcomes
Internal validity External validity
Age ofdata
Design Hospitals (n)
Study Duration (mo) Nurse units (n) Clinical gradet
(year) Potential bias* Patients (n) Effects on institutional financial outcomes Statistical grade
Collins Prospective 1989 Changing from team nursing to all RN staffing, mean h/d
1981 10 of nursing sick leave dropped from 1.24 to 0.48 h/d
High (P=002). p.35
Changing from a tearn model of nursing to all RN

staffing, mean h/d of overtime dropped from 0.79 to
0.39 h/d (P not reporteo). p. 35
I3rett Prospective 1981 A 17% reduction in RN minutes/patient/shift (16 min,
.,_ 1993 9
Moderate
3
from 96 to 80 min), after moving to team nursing,
lowered average acuity-adjusted costs per patient
by $8, $13 and $88 on the 3 nursing units (P < 0.04) p. 39
day
Harrison Prospective 1985 Nursing model (skill mix) had no significant effect 011
1983 10 total cost/patient day. The mean cost was $22.12 on the O
Low Primary Nursing unit (100% Ri'J),$21.59 on the Modular Nursing
3 O
unit (50% RN/50% LPI\J),and $20.19 on the Team Nursing unit
(50% RNI25% LPNI25% aides) (Pnot reported) p. 180
., Potential bias W3S graded as moderate unless the presence or absence 01 a design or analytic ieature seerned to make the study more or less subject to bias.
t Three investigators independently graded the clínical importance 01 each findinq. Differences were resolved in discussion. O = the linding wao not considered te be
clinícally important; ) = the finding may or may not be clinicaily important; 1 = the linding \Nas considered to be clinically important.
t Statistical grades: O = The Pvalue was > 0.05, was not reported, or the results were describec! as not being statistrcally siqnificant.
1 = The Pvalue was < 0.05, or the results were described as being statistically significan\.
RN ~ registe red nurse: LPN = licensed practical nurse
114 © 2011 The Royal College 01 Physicians and Surgeons o' a~

• Table 15.5: An evidence table showing qualitative data from a systematic review of the effects of nurse
staffing on institutional financial outcomes
Control group Treatment group

5tudy (year) Difference
Notes n Drop-outs Event rate, % (n/n) n Drop-outs Event rate, % (n/n) between rates, %
Early cancer
Aaron (1989)
Isolated disease 74 14 8 (5/61) 78 8 17 (11/63) -9
--~._--------- -----
Carty (1990)
Isolated disease 23 3 10(2/20) 27 2 12 (3/25) -2
-- _- - ~- _._---
Edwards(1990)
Comorbid disease 82 6 7 (5/76) 79 4 8 (6/75) -1
e
ro
Advanced cancer
-
~
~
FeJlow(1991)
=:¡
Comorbid disease
Halleron (1992)
31 9 9 (2/22) 31 6 4 (1/25) 5
--
:l
Isolated disease 55 7 2 (1/48) 51 4 4 (2/47) -2 lO
Figure 15.1: A fictitious example of a forest plot showing the point estimates (the closed squares) and the 95%
confidence intervals (the horizontal lines) for individual studies in a meta-analysis. Smaller studies have vvider confidence
intervals (indicating less precise estimates), and larger ones have narrovver intervals (indicating more precise estimates).
The pooled estimate (the open diamond) refiects the pooled sample sizes and thus has the narrovvest confidence interval.
The area of each square is proportional to the number of events in the study. A risk ratio of 1 means that the risk of the
treatment group is the same as that of the control group. Values less than 1 indicate a reduced risk in the treatment group,
and values greater than 1 indicate an increased risk. When the outcome is expressed as a risk ratio, confidence intervals
that cross 1 indicate that that result vvas not statistically significant at the 0.05 level. The confidence interval for the pooled
estimate shovvn does not cross 1, indicating a statistically significant overall result
Favours Favours
5tudy -d.-- treatment placebo -~;.-
Odds ratio 95% el Weight, %
6 rett, 20016
1·-------_-+----· 0.59 0.09 to 3.63 7.3
Rusty, 2001
Shelly, 20028
7
-T---------r-
I •
0.39
0.14
0.07 to 2.05
0.08 to 0.25
7.7
38.2
,-- ¡
Collins, 20049
I ------¡-------- 1.25 0.47 to 3.26 14.1
Sander, 2004'0 ------- ---&-- --:---_. 0.31 0.05 to 1.80 9.1

Patrice, 2004" ---·--1 0.46 0.20 to 1.07 23.6
Pooled O 0.32 0.23 to 0.46
0.01 0.1 10 100
Odcls ratio and 95% CI
© 2011 The Royal College 01 Physicians and Surceons 01 Canada 115

The Research Guide: A primer for residents. other health care trainees, and practitioners
Conclusión
The review can also inelude a summary of findings
table, which presents the main findings of a review as well For c1inicians focused on patient care, conducring systern-
as indications of rhe qualiry of evidence, the magnitude of aric reviews can provide an in-depth understanding of a
rhe treatment effect and the sum of available data on the topic thar will supporr c1inical decision-making. For
rnain outcornes." In addirion, the Grades ofRecommenda- researchers, conducting sysrernatic reviews can provide an
tion, Assessment, Developrnent, and Evaluation (GRADE) in-deprh undersranding of rhe currenr knowledge about a
ranking system can be used to indicate rhe general qualiry topic and can be an important firsr step in planning origi-
of rhe findings in a systernaric review or rnera-analysis." nal research studies. Systematic reviews will make borh of
chese groups of health professionals familiar with the litera-
Publication of the results ture on the topic: who is publishing in this area; what end-
1he final stage of any research is publicarion. The PRISMA points are commonly assessed, what sources of error,
Statement'? presents guidelines for reporring systemaric confounding and bias need ro be considered; and what
reviews and meta-analyses of randomized controlled trials, results tend to be common. Finally, systematic reviews can
and the MOOSE Staternent!' presents guidelines for be relatively inexpensive to conducr and can provide even
.- reporring systernatic reviews and rnera-analyses of observa- new investigators the opporrunity ro make irnporranr COI1-
tional studies. Many journals require thar these guidelines tributions ro rhe lirerature. •
be followed as a coridirion of publicarion.
Ce.SE POS:'SCRIPT
VVorking with a medicallibrarian, the student desiqr.s a strategy for searchinq the literature. The search identifies 2897 titles.
After screening the abstracts of these articles, the student decides to retrieve 490, 43 of which meet the inclusion criteria she
and her supervisor have identified. She then abstracts data on the same variables from these articles, organizes the data into
a series of evidence tables, and compares the results of all 43 studies. Examining the evidence tables, she notes that all studies
had adjusted for patient case rnix and nursing skill mix 5he finds some evidence that richer nurse staffing is associated with
lower failure-to-rescue rates, lower inpatient mortality rates, and shorter hospital stays. Further, total nursing hours and skill
mix do appear to affect some important patient outcomes.
REFERENCES
1. Lang TA, Secic M, editors. How to repon statistics in medicine. 2nd ed, Philadelphia (PA): American College of Physicians; 2006.
2. Watt A. Cameron A, Sturm L, Lathlean T, Babidge W, Blamey S, et al. Rapid versus full systematic reviews: validity in clinical
practice 7 ANZ J Surg. 2008;78(11): 1037-40.
3. Rumrill PD, Fitzgerald SM, Merchant WR. Using scoping literature reviews as a means of understanding and interpreting exis ing
literature. Work. 2010;35(3) 399-404.
4. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review-a new method of systematic review designed for complex policy
interventions. J Hmlth Serv Res Poliey 2005; 1O Suppl 1:21-34.
5. Lang T Systematic reviews as research assignments for training physicians. Aead Med. 2004;79: 1067-72.
6. Higgins JPT,Green S, editors. Coehrane /-Iandbook ior Systematie Reviews of In terventions Version 5.1.0 [upda ed areh 20
The Cochrane Collaboration. 2011. Available from: www.cochrane-handbook.orq
7. Olsen O, Middleton P,Ezzo J, Getzsche pe, Hadhazy V, Herxheimer A, et al. Quality of Coehrane reviews: assessmen of so pe
írom 1998. BMI 2001 ;323(7317) 829-32
116 © 2011 The Royal College of Physicians aoa S_'S"C~5 ¿-a=¿

8. Scherer RW, Langenberg P,von Elm E. Full publication of results ini: a Iy resented in abstracts. Cochrane Database Syst
Rev. 2007 Apr 18;(2):MR000005
9. Higgins Jp,Thompson SG, Deeks JJ,Altman DG. Measuring inconsistency in meta-analyses BMI 2003;327(7414) 557-60
10. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMAGroup. Preferred repo ing items for systematic reviews and meta-analyses:
the PRISMA Statement. PLoS Med. 2009;6(7) el 000097
11. Stroup DF,Berlin JA, Morton Se, Olkin 1, Williamson GD, Rennie D, et al.: Meta-analysis of Observational Studies in Epidemiology
(MOOSE) Group. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA 2000;283(15) 2008-12.
Barza M, Trikalinos TA, Lau J. Statistical considerations in meta-analysis. Infect oi5 Clin North Am. 2009;23(2): 195-21 o.
Callcut RA, Branson RD. How to read a review paper. Respir Care. 2009;54(10):1379-85.
• A good overview regarding completing and critiquing "review papers."
Centre for Reviews and Dissemination. Systematic reviews: CRo's guidance for undertaking reviews in health care. York (UK):
University of York; 2009. Available from: wvvw.york.ac.uklinstlcrd/pdf/Systematic_Reviews.pdf
Crumley ET,Wiebe N, Cramer K, Klassen Tp, Hartling L. Which resources should be used to identify RCT/CCTsfor systematic reviews: a
systematic review. BMC Med Res Methodol. 2005 Aug 10;524.
Egger M, Davey Smith G, Altman D, editors. Systematic reviews in hea/th care: meta-ana/ysis in context. London: BMJ Books, 1995.
• A dated but excellent source of information on systematic reviews.
Equator Network: Enhancing the QUAlity and Transparency Of health Research. Available from: www.equator-networkorg/
• This site has links to guidelines for reporting studies using any of several research designs.
Higgins J, Thompson S, Deeks J, Altman D. Statistical heterogeneity in systematic reviews of clinical trials a critical appraisal of
guidelines and practice J Hea/th Serv Res Policy 2002;7(1 ):51-61.
Hunt M. How science takes stock: the story of meta-analysis. [\]ew York: Russell Sage Foundation; 1997.
• A good history of systematic ,eviews and their application in several areas.
loannidis JPA, Lau J. The impact of high-risk patients on the population risk and on the population treatment effect. J Clin Epidemiol.
1997;50 1089-98.
loannidis Jp,Lau J. Pooling research results: benefits and limitations of meta-analysis. Jt Comm J Qua/lmprov. 1999;25(9):462-9.
loannidis JP,Lau J. State of the evicJence:current status and prospects of meta-analysis in infectious diseases. Clin Infect Dis.
1999;29(5) 1178-85.
Jüni P,Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMI
2001 ;323(7303):42-6
Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282(11): 1054-
60.
Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med 2003;96(3): 118-21.
Manchikanti L, Datta S, Smith HS, Hirsch JA. Evidence-based medicine, systematic reviews, and guidelines in interventional pain
management: part 6. Systematic reviews and meta-analyses of observational studies. Pain Physician. 2009; 12(5):819-50.
Margaliot Z, Chung KC. Systematic reviews a primer tor plastic surgery research. P/ast Reconstr Surg. 2007; 120(7) 1834-41
Sagoo GS, Little J, Higgins JPT Systematic reviews of genetic association studies. PLoS Med 2009;6(3):el000028.
Simon SD. Stetisticel evidence in medica/ triels: What do the data rea/ly te// us? Oxford: Oxford University Press;2006.
• See especially chapter 5, on systematic overviews and meta-analysis.

.Sweet M, Moynihan R. fmproving population health: the uses of systematic reviews. New York: Milbank Memoriai Fund and Centers
for Disease Control and Prevention; 2007.
Treadwell JR,Tregear SJ,Reston JT,Turkelson CM. A system for rating the stability and strength of medical evidence. BMC Med Res
Methodol. 2006 Oct 19;6 52.
Wallace BC, Schmid CH, LalJ J, Trikalinos TA. Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data. BM(
Med Res Methodol. 2009 Oec 4;9:80.
Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009; 104(5): 1086-92.
Zwahlen M, Renehan A, Egger M. Meta-analysis in medical research: potentials and limitations. UrolOncal. 2008;26(3):320-9.
EXERCISE: ASSESSMENT QUESTlONS
1. What is the difference between a narrative review and a systematic review?
iIl •••••••
'" 2. What is the difference between a systematic review and a meta-analysis?

(V
O 3. What are ihe PRISf\M\ and MOOSE staternents?
4. How is quality assessment of studies used in él systematic review?
5. What is the Cochrane Collaboration?
SUMMARY CHECKlIST
o Thinking about your research topic, list the ways in which a systematic review would be useful.
O Do the same for a narrative review.
O Using the P!COT frarnework, select a specific question to explore in a systematic review.
O Vvhat kind of question will you choose? \/Vhy?
O \/Vrite a detailed systematic review research protocol, including your search strategy and sources. Refine it with the help
of a librarían or methodologist.
O Pilot your search. Refine again.
O Conduct your search.
O Carefully rabel the records you find, using a system to track each one.
O Screen your initial results for relevance, foiiowing your a priori criteria for further in-depth review. Flag results that are
relevant background material for your later write-up of the project.
O Screen your included results in more detail and keep those that meet all your criteria.
O Analyze your data according to your plan.
O Report your findings, keeping in mind published guidelines for the reporting of systematic reviews.
118 © 2011 The Royal Colleqe of Physicians and Surqeons c" 2-¿r_¿
16
An introduction to qualitative research
June C. Carroll, MD, CCFP
Fiona Alice Miller, MA, PhD
ILLUSTRATIVE CASE
A resident notices that the staff physicians in her clinic take differing approaches to the management of diabetes, and
that not all follow the most recent guidelines of the Canadian Diabetes Association. She would like to understand how
physicians feel about clinical practice guidelines and how they decide whether or not to use them. It seems to her that the
best way to approach these questions would be through a qualitative study, but she has no experience with this kind of
research. She would like to know more about qualitative methodology and how it can be applied.
_.
• The first and most important step social processes as rhey are, rather rhan under experimental
in undertaking a research project is ro formulare a research conditions. Qualirarive researeh gathers data from and abour
question. Wharever rhar quesrion may be, ir should inspire people using inrerviews, focus groups, observation (participanr
your curiosiry and inrellecrual passion, for you will be and non-parricipant), document analysis and relared tech-
expending a grear deal of rime and energy in rhe effon ro niques." Qualirative research is defined by the use of an induc-
answer ir. Ir is also irnportant ro derermine what rnerhodol- rive approach ro analysis (alrhough deductivo srrategies can
ogy offers the best approach ro your quesrion. This chapter also be used), drawing on rnulriple or rieh accounrs ro describe
will help you derermine whether a qualitarive approach is a phenomenon or ro genera te rheories or hypotheses abour ir.
suited ro your research projecr.
CHAPTER OBJECTlVES
What is qualitative research?
Qualirative research helps us understand social phenomena, • list the types of research questions most appropriate for
including beliefs, behaviours, pracrices and interactions.' qualitative research;
Whereas quanritarive research arremprs ro measure phenorne- • identify ways to conduct a qualitative study, either as an
na, ro idenrify quanriry and size, or ro invesrigare rhe cause- independent project or in tandem with a quantitative
and-effecr relationships berween measured phenomena, study;
qualirative research examines rhe "\V'hat?" "Why?" and • describe qualitative methodologies and methods,
"How?" of rhe social world: the meanings people malee of rheir together with their advantages and disadvantages;
experiences, and che narure and social signinc::mce of their • identify approaches to qualitative data analysis; and
actions.' Ir is a naturalistic approach rhar srudies people and • list techniques to ensure credibility in qualitative research.
KEY TERMS
Case studies Grounded theory Redundancy
Document analysis Immersion/crystallization Reflexivity
Ethnography In-depth interviews Sampling
Field observation Participant observation Writing as a method of inquiry
Focus groups Qualitative description
© 2011 lhe Royal College of Physicians and Surgeons of Canada 119

The Research Guide: A primer for residents, other health care trainees. and practitioners
When should you use a qualitative

consrructing knowledge. However, for rhe purposes of thi
approach?
chaprer we rake the pragmaric view that qualirative rnerh
In deciding wherher a qualirative merhodology offers [he ods can generate useful and transferable knowledge abou
best approach ro your research quesrion, consider rhe fol- me social world, borh alone and in combinarion wirh quan
lowing: titative approaches, and that this can be done wirhour sus-
tained philosophical attention ro rhe narure of knowledgé
• Qualirarive research can be used ro explore phenomena and realiry' However, for those inreresred in a fuller díscus-
and generare rheories about rhem, or ro describe sion of rhe rheorerical underpinnings of qualitative
or evaluare initiatives such as policies, programs or research, rhe work by Creene," Cohen and Crabrree," Mays
organizarional developrnents, and Pope," Glanz and colleagues" and Teddlie and Tashak-
• Qualirarive research can be used ro supplemem kori? provides a gooe! starring-poinr.
quantitarive work, ro bring derail ro or illuminare
findings, or ro help explain unexpecred or confusing Qualitative research methodologies
resulrs (e.g., srudies with negarive resulrs, low response Once you have decided that qualirarive research is the best
,,- rates or high drop-our rates). way ro answer your research question, ir is important ro
• Qualirarive approaches are ofren rhe best choice for choose which speciíic approach ro use. Differem rnerhod-
.,_ the study of "social Qf: psychol_ogical phenomena ological frameworks are useful for answering different
such as behaviours, morivations, perceptions and sorrs of questions ' and wilI guide the particular rnethods
expectations.": 1heyare used when social context is you use in data collection and analysis. Consider how dif-
seen as irnportant . ferenr types of quesrions are addressed by differenr quali-
• Qualitative research is sometimes undertaken in rative approaches:
advance of a quantirarive srudy ro explore the
meaning of phenornena, determine how rile issues lo \.\1hat 15 the culture ofthis group? This lype of ques-
under srudy are perceived, identify useful outcome tion is oíten addressed through the research methodol-
measures, genera te hypotheses, suggesr the existence ogy called ethnography, which involves "studies of rhe
- of confounding facrors, or develop relevant and values, beliefs and practices of groups."' The merhcd
answerable survey questions. typically used is participant observation, by which
rhe researcher is present in rhe phenomenon being
Philosophical orientation studied and records his or her observations in derailed
field notes. As an example, this merhodology might
The rise of qualirative research merhodologies has been be used ro undersrand the way of Jife of Iirst-year resi-
accompaniee! by a grear de al of discussion and debate about dents in a particular specialry, The goal would be ro
the nature of the knowledge they generare. Quantitative understand the behaviours and beliefs thar characrerize
researchers have sornetirnes been skeprical of the value of the culture of rhe group. The analysis moves beyond
qualirative research, while some qualitarive researchers descriprion ro "reveal or explain social parterns or ob-
have argued thar their research adopts a perspective toward served conducr."' This rnerhod is quite time-inrensive,
knowledge thar is incommensurare with quantitative as ir typically involves "lengrhy parriciparion or irn-
merhods, rendering mixed-merhod approaches impossi- mersion in the everyday life of a chosen serring."IO
ble." The polarization of rhese views has diminished over 2. What is going 00 here? Quesrions of this type are
lime, and healrh services and c1inical researchers are sornetimes examined through case studies, which
increasingly artuned ro the conrributions rhar qualitarive examine complex phenomena in conrexr. A "case"
approaches can make. Neverrheless, debates continue might be an individual, a program, an organizarion.
about the extent to which qualitarive research requires a a community or region, or an event, By providing a
specific philosophical orientarion and a commirmenr ro a holistic and detailed look at single or mulriple cases,
specific rheorerical frarnework." these studies help ro explain how contextua] facrors '
AH researchcrs can bcnefir from exploring the philoso- shape rhe phenomena being srudied. Case srudies
phy of science and rhe panicular role of rheir own work in typically use mulriple dara sources, including inrer-
120 © 2011 The Royal College of Physicians and Surgeons oí Ca~=ca

An introduction to qualitative research
views, observarion and documenrs, and are ofren • Focus groups are often used as a means of data
presenred as a narrative description. They can also in- collecrion if che research quesrion is likely ro evoke
corporate q uantitative data, 11.12
Case srudies are corn- a range of views and rhe researcher wants ro elicit
monly used in evaluative or policy analysis research, inrera tion, debate and rich discussion among
examples would be a study of rhe introducrion of a participants about the research question. Because
new funding model for primary care (e.g., espira- of their more public nature, focus groups provide
rion) that explores staff expectations and experiences, insight inro shared meanings and beliefs but can
change processes, and influences on implernenrarion rnake ir difficulr ro delve into individual or conrrary
and ourcomes. \1} experiences. Focus groups require a facilitaror skilled
3. What does this experience mean? Various qualira- in managing group dynamics.
tive approaches (e.g., grounded rheory, phenomenol- • Document analysis can indude the exarnination
ogy, erhnomerhodology) seek ro explore how people of records of organizarions (e.g., meeting minutes,
experience the world and the meaning they make of annual reports), individuals (diaries, blogs), or
ir. An example of rhis rype of study might be an anal- groups (newsletters, newspapers, websites). These
ysis of prenatal screening rhat aims ro explore how can be used ro help explain the context for a case or
women understand screening, what ir rneans ro thern the culture of a group, or ro elicit feedback from an
and how ir affecrs the experience and social process of
pregnancy or motherhood. Interviews or focus groups
individual about her or his experiences (e.g., using an
educational brochure ro elicit reactions from wornen
_.
are rypical techniques used wirhin this approach. ro rheir experience of prenatal screening). Researchers
rnust think through rheir reasons for selecting certain
documents and not others, and how rhey should read
Qualitative methods: Which is and inrerpret their content in light of what is known
appropriate? of rheir origin or hisrory.
The specific methods you use for data collection and analy-
sis in a qualirative study should be appropriate ro rhe
Qualitative methods: Collecting data
research quesrion and your particular methodologic frame-
Field observation
work. Data may derive frorn observarion, in-deprh inter-
views, focus groups, documenrs, or a cornbination rhereof The researcher engaged in field observation spends time in
a "natural" setting-that is, a setting in which rhe phenorn-
• Field observation is used to "record social phenomena ena under study actually take place. By doing so, rhe
direcdy and prospecrively."!" The observer can researcher can experience, explore and seek ro represem the
be a parricipant in the phenomenon under srudy sociallife and social processes of that setting. Further, rhe
(panicipanr-observer) or a non-participant, In either duration and frequency of the researcher's on-site preseílce
case, the researcher spends time in the environmem can fosrer the developrnent of a respecrful relarionship and
under study and records observations and interactions rappon wirh people in that setting. Data are collecred
in field notes. rhrough extensive field notes written on a regular basis ro

• In-depth interviews may be chosen when rhe capture phenomena as they occur or shordy thereafcer. In
research question requires detailed probing of feelings ethnographic studies, researchers interview people in their
or behaviours, or when the researcher is exploring sening; alrhough rhese intcrviews share fearures with quali-
panicularly sensirive experiences that informams may tative interviewing more generally, the ongoing coritact
be more willing ro discuss privarely rhan in a more and emerging rappon characteristic of the ethnographic
open setting. For example, individual interviews might approach give che researcher an opportuniry to conducr
be used ro understand why physicians don't follow repeated interviews that enable a rich "exchange of views."15
cenain practice guidelines. Individual interviews can
Interviews and focus groups
also be used for practica] reasons: for example, key .
inforrnanrs can be interviewed in their own offices ar a ln-depth interviews are usually conducted only once per indi-
rime convenienr ro rhern. vidual and lasr from 30 minutes ro several hours." They rypi-
© 2011 The Royal College 01 Physicians and Surgeons of Canada

121
cally involve one respondenr, although pairs or small groups Interviews and focus groups are typically recorded on
might be inrerviewed in cases where rhis is more comfonable audio rape, so rhat discussions can be rranscribed in fui!. A
for che respondenrs (e.g., a married couple, a key informant good recording systern is therefore essential, and a back-up
with a helper). Ir is imponanr for me interviewer ro rapidly system is advised: technical failure has resulted in rhe loss of
develop a rappon wirh me imerviewee. Researchers who many a researcher's data. In addirion, researchers should
choose this approach would benefit from reading abour tech- make notes on imponant points, rhoughrs, and the mood
niques ro achieve the various stages of rappon. 16--20 01' tone of responses, as well as on non-verbal behaviour, as
Focus groups ideally comprise 6 ro 8 people; groups with these field notes can complernent the analysis.
fewer than 5 participanrs rend not to generare enough In interviews or focus group sessions, it is important to
interacrion, and groups of more than 10 will usually not consider parricipants' needs and convenience; artenrion
give each panicipam enough rime to speak. Answering a should be given to practical considerations such as parking,
research quesrion will usually reguire 4 ro 6 focus groups, child care, an accessible locarion, refreshrnenrs, and reirn-
and a minimum of 4 is usually needed ro reach redundancy bursement for time, expenses, ete.
or saturarion (see definirion in section on sampling). Focus
groups usually last from 1 ro 2 hours. Sampling
Alrhough participants for qualitative research can be
Conducting an intervíew/focus group. Interview and obrained rhrough random sampling, ir is more common ro
focus group discussions are typically "semi-struccured," use "purposive sampling"-rhar is, to recruit parricipants
meaning that rhey are loose!y organized around a series of frorn groups whose views or experiences rhe researcher
questions that define an area to be explored. Quesrions anticipares will be relevant to the phenomenon under
should be open-ended ro encourage discussion; guestions srudy.I,ZI,2ZPurposive sampling often involves "maximllm
that can be answered with a "yes" or "no" are to be avoided. variation sampling," which aims ro maximize rhe range of
Questions should also be sensitive and clear, and should types of informants by relevant crireria (e.g., age, gender,
nor inadvertently suggest an expected answer, Ir is irnpor- race, socioeconomic status, urban or rurai location, organi-
tanr that the interviewer/facilitaror enable participants to zarional affiliarion, occupation, social role, etc). Ir might
go below the surface in their cammentary and explore the also be imporrant ro sample for "extreme" cases, "typical"
phenomenon in derail. The inrerviewer/faciliraror should or "ideal" cases, or "negative" cases that are rhe "exception
probe rhe participanrs' meaning, rather than taking ir at to the rule."22 Sampling strategies depend 011 the research
face value. Questions such as "Tell me more about ... " or question and will somerimes need revision as rhe research '
"Can you tell me what you mean by ... " can be helpful ro progresses. Theorerical sampling involves efforrs to pur- i
richly explore rhe topic. The interviewer/facilitator should posefully recruit respondents who can address issues the
try nor ro ralk very much, focusing insread on facilitaring researcher identifies in rhe data collecred ro that poinr, ro
the conversarion of others. Ir is important ro ensure that explore thern in more detail, or ro confirm ar deny their
everyone has a chance ro speak on a topic, ro be accepring relevan ce.
of divergent opinions, and ro not assume agreement. An alternarive ro purposive sampling is convenience
Researchers should develop a guide lisring relevant ques- sampling, in which rhe pool of potential participanrs is so
tions and probes ro help organize the interview or focus small or so difficult ro aCCeSSthat the researcher recruirs
group. In some siruarions, these guides can be circulared those who are available. This is ofren rhe case for "key infor-
before the session, for exarnple ro enable key informants to manrs"-rhat is, individuals who playa significant role or
garher relevant materials that can better inform the discus- have special knowledge about rhe phenomenon under
sion. Open discussion during an interview 01' focus group ses- study; such as key policy-rnakers, program direcrors and
sion is conducive ro eliciring answers ro questions abour orhers." In these cases, a "snowball" sampling srrategy may
behaviour or experiences, opinions, beliefs, and feelings, but ir be used, wherein future porential respondents are idenrified 1
is less suited ro obtaining answers ro questions about factual by existing parricipanrs. Whatever sampling rechnique you
rnatters or demographics. Such questions are oíten included choose, ir is critica] ro jusrify your choice and ro providc a ~
in a short quesrionnaire adminisrered ro participants either full description of your sampling srraregy in rhe methods
before or arter the inrerview or fOcus group session. section of your research repon.
i22 © 2011 The Royal College 01 Physicians and Surgeons 01 Ca,aoa ~
•
-6 n introduction to qualitative research
In contrast (O quantitarive methodology, there is ofren Quali- ive data analysis generally involves inducrive rea-
no predererrnined sample size in qualitative research. Data soning ro identify patterns, rhernes and categories in rhe
collection and analysis usually occur iteratively, such thar data. \'(1irh an inductive approach, findings emerge from
rhe analysis of the initial interviews, focus groups or field rhe interaction berween the researcher and the data, rather
notes informs subsequent data collection efforcs. Research- than being deduced using an existing framework. Although
ers read and analyze data as they are generated, consider rhe inductive strategies are rypical, qualitative researchers may
findings, and highlight topics rhar need clarification or also use deductive approaches ro test theories or hyporhe-
questioning in funher depth.23 1his ongoing analysis ses, or a mix of inducrive and deductivo strategies, in which
informs subsequent data collection-that is, both the sarn- sensitizing concepts or theories are used alongside more
pling and the questions being asked.!" Interview or focus open explorarions of rhe data ro search for new patterns or
group guides are often rnodified as a result of this process. thernatic interprerations.
In addition, sampling srrategies oíten become more theo- 1he key analytic su-ategy is classification (O identify pat-
retical. Sampling s(Ops when there is redundancy on core terns, categories 01' rhernes and code rhe data accordingly. o
issues-that is, no new trends or thernes are being uncov- 1he process is iterative: initial classification schemes are ro
ered and a thorough understanding of the phenomenon
under study has been reached. 1his is called "saturation." As
refined through ongoing review and analysis of the data.
_.
-VI
\.C
a rule of rhurnb, Guest and colleagues" suggest that G ro 12
interviews will be needed when the sample comprises a rela-
Common features of qualitative
In line with the work of Miles and Huberrnan,"
analysis
rhebasic _.
:J
tively homogeneous group. rasks of qualitative analysis can be categorized as
follows:
Analyzing qualitative data • assigning codes (O field notes or transcripts;

Data, data preparation and data organization • writing any rernarks in margins;
Data for a qualitative study rypically consist of audio- or • identifying similar phrases, thernes, patterns and
videorapes of interviews or focus groups, transcriprs of differences;
those tapes, field notes made by rhe researcher, documents • taking these comrnonalities and difierences and thernes
collected by the research tearn, and notes raken during thc back into the field for Iurther data collection,
ongoing analysis meetings. • gradually constructing thernes or generalizations thar
To aid analysis, interviews and focus group sessions describe the patterns in rhe data; and
should be transcribed word for word, with wide margins for • comparing rhese generalizations against construcrs or
coding, one line space between each speaker, individual theories.
speakers identified by code such as letrers or numbers, lines
and pages nurnbered, and rhe date and place of the inter- 1he various approaches used in qualitarive data analysis
view or focus group cleady noted. reflect different rheoretical assumptions. 1hey vary by rhe
Some researchers use computer software programs such degree (O which the researcher seeks ro make inferences
as Erhnograph (Qualis Research, Colorado Springs, Colo.) about rhe data: for exarnple, low-inference interprerations
or NVivo (QSR Internarional, Cambridge, Mass.) to assist stay close ro the data, while high-inference inrerprerations
with data management. 1hese programs do not perform seek ro generate wider conclusions, rypically through the
rhe analysis but help with data organization and basic cod- development of conceptual frameworks or theories. Ana-
ing and retrieval of data. Sections of text can be groupecl lytic approaches also differ by the extent ro which the
under ceded categories 01' rhernes for ease of review or researcher draws on a pre-existing ser of ideas, concepts 01'
cross-referenci ng. hyporheses in rhe data analysis: purely inductive approach-

es seek ro limit rheir reliance on pre-structured analyrics,
Data analysis while rnixed inductive/deductive approaches use pre-struc-
Broadly speaking, qualitative data analysis involves effons tured thernes or rheories as sensitizing concepts or hyporh-
ro synthesize and make sense of a volume of qualirative eses ro facilitate rhe analytic process_ A further difference is
material ro reveal "core consistencies and meanings."5 the degree ro which rhe analysis is formalized through an

• Table 16.1: Examples of qualitative research papers

Citation Method Comment
Lau et al32 Focus groups Good example of iterative process of
Patients' adherence to osteoporosis methodology and analysis. Provides a good
therapy: exploring the perceptions of description of steps taken to ensure rigour
postmenopausal women (minimization of researcher bias, and
triangulation); good demonstration of analytical
method (development of code book; use of
qualitative data retrieval computer program);
and good presentation of findings (factors
influencing adherence and adherence strategies
are mapped onto medication-taking process).
Lee et al33 Interviews Provides a good description of technique used

Exploring family physician stress: for purposive and maximum-variation sampling;
heipful strategies good analysis ano discussion using systems
,,-
theory; good use of figures to illustrate findings.
Pottie et al34 Exploratory focus qroups Good example of iterative process: issues
Integrating pharmacists into family related to collaborative practice with
practice teams: physicians' perspectives pharmacists that emerged from the focus group
on collaborative care analysis were used to inform the development
of a semi-structured interview guide for
individual interviews
5chaufel et af.35 Observational study based Good example of analysis based on audiotaped
"So you think 1'11
survive?": a qualitative on dialogue between transcripts of dialogue
study about doctor-patient dialogues patients with serious heart
preceding high-risk cardiac surgery or disease and their physicians
intervention
Vikis et al36 In-depth interviews Provides a good example of an interview guide.

Teaehing and learning in the operating The interplay among thematie eategories is well
room is a two-way street: resident illustrated in a figure.
perceptions
Wilkes et al." In-depth interviews A good example of qualitative data analysis.

Patient experienee of infertility Data are categorized into emerging themes
management in primary care: an in-depth using constant comparison including
interview study • open coding (Iabelling)
• axial coding (categorization)
• selective eoding (eore eategorieslthemes)
Includes the development of eoneep diagrarr
describing findings.
126 © 2011 The Royal eollege oí Physicians a~!l S_-~-:s:::<::'::2=c

í6 An introduction to qua/itative research
ro determine wherher ir was appropriate for rhe • Discussion

research quesrion? Are the lirnitarions of your study discussed?
Were the srudy methods described in enough Is your research described in sufficienr derail thar
derail rhar a trained researcher coulcl be confidenr rhe reader can judge how your findings might be
that the steps in the data collection and analysis applied ro orher settings?
process were rrue ro rhe metbodology and that rhe
findings were sound and supporrable?
Do you explain any iterative processes used in Conclusion
collecting and analyzing dara? Qualitarive research is a valuable approach that can help us
Do you explain how you addressed researcher bias come ro a berter understanding of experiences, behaviours,
and used techniques of reflexiviry? cultures, phenomena and events. Ir is important ro conduce
Have you included a srarernenr giving details of qualitarive research with the same attention ro merhod-
Ethics Review Board approval for your study? ological qualiry as is expecred in quanritative research.
• Findings Well-execured qualirative srudies can reward borh research-
- Have you justiíied the thernes and interpretarions ers and readers with rich derail and fresh insighrs ro guide
presenred in your analysis and supporred thern their understanding, inform pracrice and policy develop-
with data (i.e., appropriare quotations)?
Have you explained how your findings relate ro
menr, and suppon
and quanritative
and guide funher rigorous qualitarive
research. •
-
your hyporhcsis, ro existing theorerical frameworks
or ro theory developmenr?
REFERENCES
1. Wright JG, McKeever P.Qualitative research: its role in clinical research. Ann R Co/l Physicians Surg Can. 2000;33:275-80.
2. Pope C, Mays N. Qualitative methods in health research, editors. In: Pope C, Mays N, editors. Qualitative research in health
careo Oxford (UK) Blackwell Publishing; 2006. p.l-ll.
3. Kuper A, Reeves S, Levinson W. An introduction to reading and appraising qualitative research. BMJ. 2008;337:a288.
4. Greene Jc. 15 mixed methods social inquiry a distinctive methodology? J Mix Methods Res. 2008;2(1 ):7-22.
5. Patton MQ. Qualitative research & evaluation methods. 3rd ed. Thousand Oaks (CA): Sage Publications; 2002.
6. Cohen DJ, Crabtree BF.Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam
Med. 2008;6(4) 331-9
7. Mays N, Pope C. Qualitative research in health care: assessing quality in qualitative research. BMI 2000;320(7226)50-2
8. Glanz K, Rimer BK, Viswanath K, editors. Health behavior and health education: theory, research, and practice. 4th ed. San
Francisco (CA): Jossey-Bass;2008.
9. Teddlie C, Tashakkori A. Major issues and controversies in the use of mixed methods in the social and behavioural sciences. In:
Teddlie C, Tashakkori A, editors. Handbook of mixed methods in social and behavioral research. Thousand Oaks (CA): Sage
Publications; 2003. p. 3-50.
10. Pope C. Conducting ethnography in medical settings. Med Educ. 2005;39(12) 1180-7.
11. Byrne D, Ragin CC, editors. The SAGE handbook of case-based methods. Thousand Oaks (CA): Sage Publications; 2009.
12. Yin RK. Case study reseercti: design and methods. 4th ed. Thousand Oaks (CA) Sage Publications; 2009.
13. Stake RE. The art of case study research. Thousand Oaks (CA): Sage Publications; 1995.
14. Giacomini MK, Cook DJ. Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of
the study valid? Evidence-Based Medicine Working Group. JAMA 2000;284(3) 357-62.
15. Heyl BS. Ethnographic interviewing. In: Atkinson P,Coffet A, Delamont S, Lofland J, Lofland L, editors. Handbook of
ethnography. London: Sage Publications; 2001. p. 367-83.
~ 2011 The Royal College 01 Physicians and Surgeons 01 Canada 127

16. Dicicco-Bloom B, Crabtree BF.The qualitative research interview Med Educ. 2006;40(4):314-21 .
17. Briggs CL. Learning how to ask: A sociolinguisitic appraisal of the role of the interview in social science research. Cambridge
(UK): Cambridge University Press; 1986.
18. Miller WL, Crabtree BF.Depth interviewing. In: Crabtree BF,Miller WL, editors. Doing qualitative research. 2nd ed. Thousand
Oaks (CA): Saqe Publications; 1999. p. 89-107.
19. Rubin HJ, Rubin IS. Listening, hearing and sharing social experiences. In: Qualitative interviewing: the art of hearing data.
Thousand Oaks (CA): Sage Publications; 2005. p.1-18.
20. Spradley J. The ethnographic interview. New York: Holt, Rinehart & Winston; 1979.
21. Barbour RS. Making sense of focus groups. Med Educ. 2005;39(7):742-50.
22. Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. BMJ. 2008;337:al 035.
23. Pope C. Ziebland S, Mays N. Qualitative research in health care. Analysing qualitative data. BMI 2000;320(7227):114-6.
24. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field
en Methods.2006;18(1):59-82
t:
.- 25 . Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. Thousand Oaks (CA): Sage Publications; 1994.
e 26. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334-40.
en
,,-
27. Glaser BG, Strauss AL. The discovery of grounded theory: strategies far qualitative research. Chicago Aldine Publishing; 1967.
\:n 28. Lingard L, Albert M, Levinson W Grounded theory, mixed methods, and action research. BMJ. 2008;337:a567.
<V 29. Creswell Jw. Qualitative inquiry and research design.· choosing among five approaches. 2nd ed. Thousand Oaks (CA): Sage
el Publications; 2007.
30. Richardson L, St. Pierre EA Writing A method of inquiry. In: Denzin NK, YS Lincoln YS, editors. The 5AGE handbooi< oi
qualitative research. 3rd ed. Thousand Oaks (CA): Sage Publications; 2005. p. 959-75.
31. Morse JM, Barrett M, Mayan M, Olson K, Spiers J. 2002. Verification strateqies for establishing reliability and validity in
qualitative research. Int J Qual Methods. 2002; 1(2): 1-19.
32. Lau E, PapaiOél!1Il0UA, Dolovich L,Adachi J, Sawka AM, Nair N, et al. Patients' adherence to osteoporosis therapy: explorinq
the perceptions of postmenopausal women. Can Fam Physician. 2008;54(3):394-402.
33. Lee FJ,Brown JB, Stewart M. Exploring family physician stress: helpful strategies. Can Fam Physician. 2009;55(3):288-9 el-6.
34. Pottie K, Farrell B, Haydt S, Dolovich L, Sellors C, Kennie N, et al. Integrating pharmacists into family practice teams: physicians'
perspectives on collaborative care. Can Fam Physician. 2008;54(12): 1714-17 .e5
35. Schauíei MA, Nordrehaug JE, Malterud K. "So you think 1'11 survive?": a qualitative study about doctor-patient dialogues
preceding high-risk cardiac surgery or intervention. Heert. 2009;95(15): 1245-9.
36. Vikis EA, Mihalynuk TV, Pratt DD, Sidhu RS.Teaching and learning in the operating room is a two-way street: resident
perceptions. Am J 5urg. 2008; 195(5) 594-8
37. Wilkes S, Hall N, Crosland A, Murdoch A, Rubin G. Patient experience of infertility management in primary care: an in-depth
interview study. Fam Pract. 2009;26(4):309-16.
Atkinson P,Coffey A, Delamont S, Lofland J, Lofland L, editors. Handbook of ethnography. London: Sage Publications; 2001.
• This is a comprehensive reference work for ethnographic research across the social sciences, including both introductory
overviews and detailed critical essays.
Braun V, Clarke V Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101.
• This article provides clear guidelines for those wanting to start a thematic analysis or to conduct one in a more deliberare él
rigorous way.
128 © 2011 The Royal College of Physicians and Surqe ns z Ce ¿:;¿

5 An introduction to qua/itative research
Crabtree BF,Miller WL. Ooing qua/dative research. 2nd ed. Thousand Oaks . ): 50 Je P blications: 1999.
• This introductory textbook offers an excellent overview of qualitati '2 resear n methods along with chapters on strategies for
data collection and analysis.
Denzin NK, Lincoln YS, editors. The SAGE handbook of qua/itative research. 3rd ed. Thousand Oaks (CA): Sage Publications; 2005.
• This text is considered an essential reference-perhaps the definitive reference-for qualitative research.
Giacomini MK, Cook DJ. Users' guides to the medicalliterature XXIII. Qualitative research in health care A Are the results of the study
vahd? Evidence-Based Medicine Working Group. JAMA 2000;284(3)357-62
• This paper describes how to assessthe validity of qualitative research reports, including assessment of sampling, methods and
comprehensiveness of data collection, analysis and findings.
Glesne e Becoming quslitetive reseercbets: an introduction. 4th ed. Englewood Cliffs (NJ): Prentice Hall; 2010.
• This text is a very informative and readable guide to the design and conduct of qualitative research.
Kuper A, ReevesS, Levinson W An introduction to reading and appraising qualitative research. BMI 2008;337:a288.
• This excellent introduction to qualitative research includes a list of definitions of key terms used in qualitative research and a
brief outline of theoretical approaches to qualitative research. _8
Liamputtong P,Ezzy D. 2005. Qua/itative research methods. 2nd ed. Oxford (UK): Oxford University Press;2005.
• This text is a very readable guide to the design and conduct of qualitative research, with a particular focus on health.
Mays N, Pope e Qualitative research in health care. Assessing qua lity in qualitative research. BMI 2000;320(7226):50-2.
• This article gives a detailed description of ways to improve validity in qualitative research and includes a list of questions to pose
in evaluating a qualitative study
Miles MB, Huberman AM Qua/itative data ana/ysis: an expanded sourcebook. Thousand Oaks (CA): Sage Publications; 1994.
• This practical sourcebook on qualitative research provides detailed information on qualitative data analysis.
Schwandt TA The Sage dictionary of qua/itative inquiry. 3rd ed. Thousand Oaks (CA) Sage Publications; 2007.
• As its title suggests, a clictionary/reference of qualitative research.
Wright JG, McKeever P.Qualitative research: its role in clinical research. Ann R ColI Physicians Surg Can. 2000;33:275-80.
• This is a good general article on the role of qualitative research, clifferences from quantitative research, and a brief overview of
qualitative methods.
EXERClSES
1. Describe the difference between qualitative and quantitative methodology.
2. Reviewa qualitative research paper and see if you can list the steps taken to ensure quality of the data and the analysis.

The Research Guicle: A primer for residents, ather health care trainees, and practitianers
SUMMARY CHECKLlST
o Considering your research questian, list ways in which a qualitative appraach wauld be useful-or nat so useful.
O Select a qualitative method relevant to yaur research questian.
O Write a draft protocol defining all elements, including the target population, sampling, data collection, data
organization and analysis.
O Consider any ethical issues arising from your pratocol.
O Review your draft protocol with a qualitative methodologist. Revise.
O Write up your results, being mindful of published guidelines for qualitative analyses.
el
.t:
.,...
e
en
,. lIiOa
'"OOJ
130 © 2011 The Royal College of Physicians and Surqeons c..~

17
Writing a research protocol
Vicky Tagalakis, MD, MSc
ILLUSTRATIVE CASE
During the two years you have spent as an Internal Medicine resident at an ambulatory clinic, you have observed the
results of different approaches to blood pressure monitoring. Yciu form a hypothesis that patients whose blood pressure is
managed with the use of a 24-hour ambulatory blood pressure monitor are more likely to achieve target blood pressure
than patients whose blood pressure is managed by office-based manual sphygmomanometry alone. An extensive
search of the literature reveals that very little research has been done on this topic. Your supervisor, an active researcher
in hypertension and cardiovascular disease, is enthusiastic about your idea and proposes that you conduct a study to
investigate it. Having formulated your research question, reviewed the medical literature, discussed your research question
and proposed a study plan with your supervisor, you are now preparing to write a research protocol so that you can apply
for a grant to fund your study.
A research protocol is a detailed field, or ro address rhe issues rhar will inevitably come ro
plan that wiIl help you ro formalize and operationalize your Iight during rhe wriring process as you focus on the details _.
research ideas, map out how the study will be carried out, of your prorocol.
acr as a reference ro ensure all members of rhe research
ream adhere ro the merhods ourlined, and provide refer- Organize a tearn of collaborators and advisors early, The
erice poinrs for moniroring the study's progress and evalu- input, feedback and advice thar your tearn can provide
ating irs outcomes. Proficiency in prorocol writing is an during rhe prepararion and review of your wrirren protocol
acquired skill. Alrhough sorne people are naturally better at are invaluable. They can help you refine your srudy ques-
it than others, anyone can beneíit from the simple tips tion and objectives, advise you on rnerhods, and provide
described in this chaprer. you wirh rools and resources. Once the prorocol is written,
rhey can provide feedback on rhe content, giving panicular
Start writing early, You should begin (O write your proto-
col at least four mcnrhs befo re rhe granr submission dead-
CHAPTER OBJECTIVES
line. This will give you time ro search rhe medical lirerarurc,
seek advice, and revise your research quesrion and srudy After reading this chapter, you should be able to
plan. Leaving the writing process ro rhe last few weeks • describe the major components of a research protocol
before the deadline won't leave you cnough time ro obtain • apply useful tips for writing a good research protocol
and act upon input from collaborarors and experts in rhe • develop good research protocol writing practices
KEY TERMS
Ascertainment bias Intervention bias Referral bias
Co-intervention bias Measurement bias Selection bias
Contamination
Information
Instrument
bias
bias
bias Non-respondent
Outcome variables
Recall bias
bias SMART
Withdrawal bias J
_-
artention ro clarity and coherence. Their input should Once you have formulated your research quesrion
address four key quesrions: searchecl the lirerarure, sought input and feedback from
your collaborarors and supervisor, devised a plan for your
• .Is rhe research question sufficienrly clear and refined ro research srudy, looked at examples of orher prorocols, and
serve as an anchor for the research endeavour? referred ro the granring agency's websire for guide!ines and
• Are the proposed study design and merhodology instructions, you are ready ro starr wriring the protocol.
adequare ro answcr rhe research question(s) and
achieve the study objecrive(s)?
The components of a research protocol
• AJe the design and merhodology feasible (e.g., is the
sample size adequate)? Although requiremenrs vary from one granring agency ro
• Is rhere sufficienr detail and instrucrion ro ensure that another, research prorocols are mosr commonly srructured
rhe study is replicable? as follows:
If the answer is yes ro all four questions, then you have • projecr title
succeeded in deve!oping a well-rhoughr--our and well-writ- • projecr surnmary (up ro ene page)
ten prorocol rhar rhe reviewers wil! appreciare. • staternenr of the problem (up ro one page)
• research proposal (about 75% of the allorred pages)
Get the writing style right, When you apply ro a granring background
agency for funding, remember that your research protocol research objecrives
is one of many rhat reviewers will have ro read and rank. So, srudy design and rnerhod
make ir enjoyable ro read! Airn for writing that is clcar, con- statistical analyses
cise, interesting, easy ro read, and free of spel!ing and gram- sample size calculation
marica] errors. Malee sure your reasoning is logical and • ethical considerations
precise. You want ro engage the reviewers with your ideas, ., rhe role and experrisc of rearn members (about
nor ro distraer thern with errors and convolured thinking. 200-250 words)
Get advice Irorn your supervisor about writing sryle, and • srudy tirneline (about 50-100 words)
ask hirn or her fOl"exarnples of well-wrirtcn prorocols that • strengths and lirnitarions (about 250-300 words)
have been successful at granr comperirions. These should • anticipated resulrs and irnplications (about 300 words)
give you an idea of effecrive srructure, wriring sryle and • references
Aow, and the leve! of language ro use. Also consulr guide-
books and resources on granr and prorocol writing (see Each of these components is discussed in rhe following sec-
Additional Resources) for rhe principies of good expository rions.
writing, such as srarring each paragraph wirh a strong topic
senrence and using, where possible, the active rather than Project title
passive voice (see also ch. 29). The tirle should be concisc and cleady convey the central
research objective, including rhe popularíon ro be srudied.
Know the requirements and instructions of the granting In rhe case of a randomized clinical rrial, the inrervemion
agency. Early in the wriring process, become familiar with ro be srudied or tested should abo be described. Avoid
the granting agency's requiremenrs and instructions for sub- using acronyms and orher abbreviarions in your tirle. For
mitting an application, which can rake Iar more time ro our case example, the projecr title mighr be:
complete rhan you mighr expecr. Moreover, non-adherence
lO [he rules can lead ro your application being disqualified. Comparison ofambulatory blood pressure monitoring us con-
There are usually instructions pertaining LO lerigrh, formar- uentional clinic-based manual sphygmomanomerry [or atta in-
rin!:;and rhe components ro include (e.g., invesrigaror CVs, ment oftarget blood pressure in patients u/ith netoly diagnosed
an abstraer, a budger) as well as other requircrnents (e.g., the hypertensio n.
applicarion deadline, required signatures, number of cop-
ies). These insrrucrions vary according ro the granring agen-
cy and are available for reference on their respective websires.
132 © 2011 The Royal College oí Physicians and Surcso 'C¿re::¿

77 Writing a research protoco/
Project summary Research propasal
As rhe íirst section (after the rirle) thar the reviewers wil! Background. The background section extends the sta te-
read, the project summary occupies a key position in your menr of che problem wirh an in-depth review of the current
prorocol. In your sumrnary, you should airn to (1) arouse stare of knowledge on the topic-induding your pilor
interest in your projecr; (2) convince the reviewers of its work and preliminary resulrs=-and the rarionale for your
importance; and (3) provide a briefbut concise overview of srudy. For example, you should indicare why your research
your research plan, induding its objective(s) and proposed quesrion is compel!ing, why your approach ro it is ideal,
study design and methods, A well-written summary can go and why you and your tearn are well positioned ro do rhis
".a long way roward establishing your project as credible and research. There are three important parts ro rhe background
worrhy of funding. If you have completed pilor research in secuon:
the area of your proposed srudy, indude a staternent surn-
marizing this work, the progress achieved, and how ir natu- 1. A restatement of the main study question and/or
rally leads ro or supporrs your currenr proposal (see ch. 21). hypothesis and how it relates ro healrh priorities 10-
cally and/or universally (e.g., attaining rarget blood
Statement of the problem pressure can have an irnpact on the prevalence of car-
The purpose of this section is ro provide, in a succinct form, diovascular diseases, such as stroke and heart disease,
a justihcation for your proposed research and explain why which impose a significant burden on sociery). In rhis
ir is important and rnerirs funding. Typically, ir begins by example, one might propose the following:
describing the current situation and rhen outlines gap s in,
or the incondusiveness of, existing evidence. Ir may also In patients with hypertension who areflllowed in an out-
question existing knowledge in lighr of recent evidence patient clinic, does routine monitoring with sphygmoma-
from other studies-perhaps even from your own prelirni- nomen) as compared with scheduled 24-hour ambulatory
nary work. monitoring result in a higher proportion ofpatients achieu-

To use our example, your staternent of the problem mighr ingpredetermined blood pressure targets?
explain that preliminary data Irorn your dinic show that only
60% of patients have achieved their rarget blood pressure by 6 2. An extensive Iiterarure review that describes che cur-
months atter rhe initiarion of antihypertensive drugrherapy, rent evidence and/or knowledge of the research prob-
and that this success rate is consistent wirh rates reported in lem or question, identifies gaps in that knowledge,
published studies. You ernphasize the irnportance of attaining and states whar questions or controversies remain un-
target blood pressure with regard ro primary and secondary resolved. Ir is important ro enlist a health librarian ro
prevention of cardiovascular endpoints such as stroke and help wirh your literature search (see ch. 7): a sysrem-
heart disease. Typically, when patienrs are íirst started on atic and thorough search at the starr of the research
anrihypertensive therapy, rheir blood pressure is checked every process is invaluable and can save time latcr if you
6-8 weeks during rhe hrst 6 months. At these íollow-up visits, want ro refine yOL\[ research quesrion or objecrives
patients have their blood pressure measured with a manual and thus need ro re-consult the lirerature.
sphygmomanometer (3 measurernents 5 minutes apart).
Alrernarively, rhey can have an ambularory monitor placed for 3. A convincing argllment to justiíy your study and
24 hours; this is returned by rhe parient the next day so that rhus persuade the reviewers that it should be funded.
[he rneasurernenrs can be rerrieved, Your exhausrive literature You rnust be able ro outline clearly why your study
search shows rhat 24-hour arnbulatory blood pressure should be done; what knowledge will be obtained and
monitoring is as reliable as monitoring with a manual how ir wil] coritribute ro the existing evidence ro ad-
sphygmomanometer. You rhen hypothesize thar, in parienrs dress knowledge gaps and/or conrroversies; the poten-
wirh newly diagnosed hypertension, management by 24-hour tial benehts of your study ro the scientific and health
ambularory blood pressure moniroring will lead ro higher services cornmuniry and to parients; and how feasible
rates of target blood pressure attainment at 6 months ir is for your study, given its proposed merhodology
compared with managemem by manual sphygmomanomerer and design, ro answer the research quesrion and/or
monitoring. hypothesis.

Research objectives. Your research objectives arise from • the study design
your study quesrion(s) and/or hyporhesis. These objecrives • the srudy population
should be srared clearly and concisely, specifying what is ro • rhe merhod of recruitment
be described, measured, deterrnined, identified and (in rhe • operacional definitions of variables, including
case of a study hyporhesis) confirrned or disproven. Borh ourcomes ro be studied
general and specific objectives should be identified. Beware • the proposed inrervention (if applicable)
of formularing roo many objecrives, or objectives that can- • data collection methods and management
not feasibly be achieved within rhe scope of your project. • sample-size calculations
Research grants are ofren ranked poorly if the research • proposed statisrical analyses
objectives are nor attainable on the basis of the study plan.
They should be SMART: Specific, Measurable, Achievable, Sorne of these items are described further below.
Relevant and Timely. I
Study designoThis subsection should identify the srudy design

General objective. Your staternent of the study's general chosen (e.g., case-control, experimenral/inrerventional study).
objecrive should encapsulare whar will be srudied and the lf a less robust design has been chosen (e.g., case-control as
knowledge you expect ro gain. For example: opposed ro a prospective cohort study), rhen an explanation
as ro why rhis design was chosen in preference ro other pos-
Our goaL ú to determine whether 24-hour ambulatory blood sibilities should be provided (e.g., because of resource limi-
pressure rnonitoring results in better blood pressure control tations or ethical considerations).
than blood pressure monitoring by clinic-based manual sphyg-
momanometer in patients toith neuily diagnosed hypertension. We will carry out a randomized clinical trial whereby consecu-
~ tive patients referred to an Internal Medicine clinic far hyper-
o
e,
Specific objectiues. Your description of specific objectives tension management will be randomly assigned to periodic
should make the general objective more precise wirh monitoring 01 blood pressure with a 24-hour ambulatory
o
~
respccr ro measurable endpoints and should preview ele- blood pressure monitor vs manual sphygmomanometer during
o, ments of rhe study designo For example: a 6-month treatment periodo
l. lO determine and compare the proportion 01patients with Study population. Be as clear as possible in your deseription
newly diagnosed hypertension who attained target blood of rhe population in which your study will be carried out.
pressure at 6 months with 24-hour ambulatory blood pres- Also describe rhe source popularion from which the study
sure monitoring vs office-based manual sphygnzomanometry. popularion is derived, as well as the sampling merhcds used
2. To determine the percentage change in mean arterial pres- ro obrain rhe study population (e.g., a random sample from
sure at 6 rnonths after initiation 01therapy compared with a larger population, or a series of consecutive patients
at tbe start 01 therapy in patients whose hypertension was attending a clinic). Provide details on the following, as
managed with 24-hour ambulatory blood pressure moni- appropriare for your srudy design:
toring vs patients whose hypertension was managed with
office-based manual sphygmomanometry. • rhe study group definirions, e.g., cases vs controls in a
case-control study), unexposed vs exposed individuals
Study design and methods. The merhods section describes (in a cohort study); conrrol vs experimental group (in
in a concrete and objecrive rnanner the procedures that will an interventional study);
be used ro achieve your study objectives. Ir is the manual of • inclusion and exclusion criteria, recruitment and
operarions and henee a very irnportant pan of rhe prorocol. enrolrnenr procedures and, if applieable, marching
Ir should contain sufficienr detail and instruction so rhat if facrors for cases and conrrols;
another invesrigaror were to repear your study he or she • rnerhods of randomization and allocation.
would obrain comparable resulrs. Ir is universally accepted
by rhe scienriíic comrnuniry that rhe rnerhods secrion Reviewers will always pay close attention ro the study
should include a detailed description of the following: population to assess rhe potential generalizability and rele-
vanee of results and ro detect selection or ascertainment

77 Writing a research protoco/
bias, which occurs when sorne members of rhe so urce pop- dures and instrurnenrs have previously been tested by you or
ularion are less likely ro be induded in the study rhan orhers your research ream, include derails on your findings wirh
and, as a result, rhe srudy fails ro represent equa!ly all groups regard ro me accuracy and reproducibiliry of their rneasure-
ofindividuals from the source popularion. menes or results.
You should a!so describe in the data collecrion secrion
Operational definitions 01 al! variables. Ir is importanr ro rhe rnethcds you propose ro use ro safeguard your data
provide operarional definitions of a!1 variables ro be stud- against biases that may rhreaten rhe validiry of your research
ied, especially outcornes. Give a dear descriprion of what and render your resulrs inaccurare, Common sources of
rype of variables will be srudied, whar is undersrood by each bias to consider are described in Texrbox 17.1.
variable, and how data on variables will be collecred,
recorded and analyzed. You should also explain rhe validiry Data management. 1his secrion should describe your proce-
and reliabiliry of rhe proposed definirions and rneasure- dures for data emry (e.g., who will enrer rhe dara, whether
rnents. data emry will be blinded, and what software will be used)
In our example, blood pressure-relared measuremenrs along with rhe measures you will take ro ensure rhe corn-
are the primary outcome variables, but you would also plereness and accuracy of rhe information being eI~rered
need ro specifically state wherher rhe mean, systolic or día- (e.g., duplicare enrry of data, cross-validation)." You should
srolic pressure will be measured. Moreover, you would need also describe where rhe data will be stored and whar securi-
to provide a clear descriprion of rhe procedure and process ry measures will be underraken to protecr patienr confiden-
for recording blood pressure by borh manual sphygmoma- tialiry, such as storing information securely in a locked
norneter and 24-hour arnbularory blood pressure monitor- filing cabiner in a locked office or using password prorec-
ing (e.g., Who will record rhe blood pressure by manual tion on a secure computer nerwork.
sphygmomanomerer? Will ir be mcasured in borh trial
arrns? How frequentlywill blood pressure measurernents be Statistical analyses. This section, which describes your pro-
done by eirher rnethod? \'(1hich brand model will be used posed data analyses, is besr written in conjunction with a
for 24-hour ambularory blood pressure measurernents? Has statistician, It should include derails on rhe variables to be
rhis model been validared? Is ir reliable across diflerent used to compare the groups, a description of your summary
study populations/) Deíinirions that are standardized and/ measures (e.g., odds ratio, risk ratio, hazard ratio), your
or well described in the Íirerarure should be described brief- methods of analyses (e.g., t test, logisric regression analysis,
ly, wirh supporting references. survival analysis), a justincarion of adjustmenrs for pre-
Protocols whose operational definirions are imprecise- defined confounders, and a descriprion of how you \ViII
e.g., "Demographic variables will be considered" or "Blood deal wirh missing dara. This section should also specify the
pressure will be measured according ro standard dinic pro- alpha leve! ro be used (rhe cur-off probabiliry value for sta-
cedures"-do not receive favourable rankings from review- tistical significance) and wherher rwo-railed statisrical tests
ers, since rhe relevance of rhe variables ro rhe srudy will be used. The srarisrical sofrware that you inrend ro use
objecrives cannor be assessed. Moreover, vague or incorn- should also be referenced.
plere descriprions of variables do not allow for the unirorrn
execution of a study prorocol across different sites. Sample size calculation. Early in rhe planning srages of your
srudy and before you begin ro write your research prorocol,
Data co!!ection. Ir is essenria! ro describe in detail how data enlisr the help of a statistician to assisr wirh issues such as
will be collecred and recorded. Describe your dara collecrion derermining an appropriare sample size ro ensure rhar your
rnerhods (e.g., patient inrerview, chart review, self-adrninis-
rered quesrionnaire), who will be collecring the dara (e.g.,
research nurse, each parricipanr), and the data collection a "Duplicate elata entry" refers to the process whereby elata Irom case
report forms obtained in a study are independently ente red by two
instruments you will use (e.g., quesrionnaires, inrerview
individuals and compared to ensure the accuracy 01 entry. "e ross-
guide, medical record exrraction form). If rhese procedures validation" elescribes a process where either the source material for data
have been srandardized and described in rhe lirerature, pro- entry into spreadsheets or the content 01 the spreadsheets themselves
vide and describe the perrinent references. If rhese proce- are reviewed by study personnel (usually an investigator or coordinator)
to ensure accuracy.
© 2011 The koyal College 01 Physicíansand Surgeons 01 Canada 135

rhe Research Guide: A primer for residents, other health care trainees, and practitioners
TEXTBOX 17.1: SOURCES OF BIAS2
SELECTION BIAS
This bias results from a systematic error in the procedures or factors used in the selection of subjects, such that the study sarnp]
is unrepresentative of the population of interest. This means that the observed association between, for example, an exposun
and disease, is different for those who actually participated in the study and those who should theoretically have been eliqibh
for inclusion but did not participate. Referral bias, which occurs because people who are referred to a study are often differen
from non-referred individuals, and non-respondent bias, which occurs because subjects who choose to respond to a call te
participate in research studies are generally different frorn those who do not respond, are among the various types of selectior
bias.
MEASUREMENT (OR INFORMATION) BIAS

This bias results from a systematic error in the collection of information or data. Its consequences will vary, depending on how
the inaccurate data relate to the study exposure and/or outcome(s) of interest. Examples of measurement bias are recall bias
(e.g., cases-individuals identified as having the disease under study-tend to recall past exposures more completely and accu-
rately than controls) and instrument bias (which occurs when a study instrument such as a faulty sphygmomanometer leads
to inaccurate measurements being recorded; because it affects all study groups equally, the resulting measurement error will
tend to bias the study result toward the null, underestimating any real difference).
INTERVENTION BIAS
This type of bias arises from a systematic error in how an intervention (as in a randomized controlled trial) is carried out, or in the
manner in which the study groups were exposed to the intervention. Examples of intervention bias include contaminstion
bias, which occurs when members in the control group inadvertently (or not so inadvertently) receive the intervention or
treatment. thus minimizing the difference in outcome between the intervention and control groups, co-intervention bias,
which OCCUI"S when interventions other than the study treatments are applied differently to the study and control groups (this is a
serious problem when double-blinding is absent), and withdrawal bias, which occurs when outcomes for subjects who leave
the study (drop-outs) are substantially different from the outcomes of those who remain in the study.
TEXTBOX 17.2: PITFALLS IN PROTOCOL WRITING'
• Starting too late.

• Failing to provide a dear research question.
• Setting too many objectives, or objectives that are beyond the scope of the project.
• Failing to provide the context or "big picture" of your research.
• Failing to present a clear and precise rationale for your approach.
• Failing to cite key studies or present a thorough review of the knowledge in yaur tapic area.
• Giving too much detail on minor issues and not enough on major ones.
• Writing in a rambling style.
• Making mistakes in grammar and spelling .
•• Failing to pravide a sample size calculation and power calculation.
*See also the "Ten cornmon research pitfalls" listed at the end of chapter 1.
136 © 2011 The Royal Coliege of Physicians and Su ~eor.se: (¿e-e::¿ I

77 Writing a research protocol
study will have sufficienc scacisrical power. 1he srarisrician Co-investigaror, Co-invesrigarors work closely with che
can help wirh rhe wriring of chis and the other analyrical principal investigaror and make a significanr conrriburion
pares of rhe merhods, which should describe your sample ro che inrellecrual direction and conduce of rhe research.
size calcularion and inelude details on how you arrived ar Some co-investigarors may bring very speciíic experrise ro
rhe estimares of difference ro be detected (e.g., che mini- rhe tearn (e.g., mechodological or sraristical).
mum difference in percenrage change in mean arterial
blood pressure expecred berween rhe rwo srudy groups) Collaborator. The main role of collaborarors in a research
and che statisrical assumprions made regarding rhe distri- projecc is ro provide a specific service (e.g., access ro equip-
burion of variables and your chosen levels of significance, as ment, provision of speciíic reagenrs, rraining in a special-
well as references for your rnerhods of calcularion. ized rechnique, statistical analysis, or access ro a patierit
popularion) .
Ethical considerations
Erhical considerations apply ro all rypes of healrh research, Study timeline
ranging from experimenral intervention trials ro dacabase Ir is irnportanr tO specify che esrimared rime, usually in
studies. As a resulr, you should specify wherher approval has weeks ro months, required for cornplerion of the various
been received, is pending, or will be soughr from an appro- srages of your srudy (e.g., time for patient recruitrnent,
priate ethics review comrnittee. If such approval is not neces- time for dara colleccion and enrry, rime for analysis, etc.).
sary (as in a meta-analysis of published daca), then you rnust This will help che reviewers ro gauge che feasibiliry of your
state thar chis is che case and explain why. Moreover, you study and assess the duracion of che funding suppon thar
should also describe any erhical issues relared ro your study: will be needed. For example:
rhese mighc pertain ro recruirrnent srraregies, inelusion and
exclusion criteria (e.g., che participation of vulnerable sub- This study will require 3 years (36 months) jor completion: 2
jecrs such as children, or exclusions based on race or sex), months jor hiring and training ~fstudy nurses and the d~vel-
potential risks and benefirs co study participanrs (especially opment of a computerized data entry jorm; 24 months jor
in inrervention studies), che de-linking of idenrifying infor- recruitrnent ofpatients and jollow-up, 4 months jo;- data entry
marion when working with adrninistrative dacabases, and and cleaning, and 6 months jor completion ofdata analysis
any issues concerning che righcs of subjecrs. You should also and manuscript preparation.
indude, in an appendix, che consenr forms you inrend ro use
and any approval lerters from erhics comrnirtees, if these are Strengths and limitations
available ar che rime of your grant submission. Ir is good praccice ro highlight rhe strengrhs of your pro-
posal and ro indicare its limitations and how these will
Team members affecc your scudy. As rhe reviewers read your protocol, rhey
In chis section, you should identify rhe members of the will identify lirnirations and will inevirably have someching
research tearn and describe in specitic terrns their role on ro say (since ir is much easier ro criticize someone else's pro-
rhe tearn and rhe expertise rhey bring ro che project, You rocol rhan ro write one). Ir is rherefore ro your advanrage ro
want ro be able ro convince che reviewers thar your tearn is anticipare and pre-empt criticism by identifying rhose li mi-
well positioned and equipped ro see rhe srudy ro a success- rations yourself, assessing rheir potencial irnpacr, and sug-
fui and timely complecion. Three main roles in a rypical gesting alternative strategies along wirh their advanrages
research ceam are as follows: and disadvanrages. The reviewers will appreciare rhar yOL!
are being proactive and can anticipare and address pirfalls
Principal investigator. The principal invesrigaror is primarily and obscacles.
responsible for rhe inrellecrual direcrion of che research projecr
and oversighc of che execurion of che protocol. The principal Anticipated results and implications
invesrigaror also assumes adrninistrarive responsibiliry (e.g., This paragraph is the last section of che prorocol and is
fund allocarion, hiring of srudy personnel, research ethics sub- meant ro surnrnarize and ernphasize rhe significance of
mission) for che projecr. Some studies may have CViO or more your research in terrns of its anticipated results and implica-
"ce-principal invesrigarors." tions. When writing chis section, rry ro answer rhe follow--
© 2011 The Royal College 01 Physicians and Surgeons 01 Canat.ia 137

ing question: "How will results of your study be usefu! to package such as Endnore or Reference Manager right fron
clinicians and patients, ro policy-rnakers, rhe medica! corn- rhe starr of the wriring process: this will save you time in rh:
rnuniry at !arge, and ro future research efforts?" Sorne long run. Most universiry or hospiral libraries offer work.
researchers consider the questions, "So what?" and "Who shops on using referencing or citation sofrware.
cares?" as they write this section, Although this is a small
paragraph, it is the last thing the reviewers read, and you
want ro leave thcrn with the impression rhat your area of Conclusion
research is irnportant and should be funded! For example: Cleady, wriring a good research prorocol requires starring
eady, being organized, focusing on timelines, reading and
Hypertension is a major public health problem afficting 1in 3 thinking crirically, recruiting help from orhers (e.g., librari-
adult Canadians. The results 01 this study ioill improue the ans for lirerarure searches, statisricians for analysis, rncrhod-
management oj'blood pressure in outpatient clinics by ... ologists for study design and conducr), seeking guidance
from supervisors, mentors and collaborators-and, most
References importanrly, committing time and attention ro the writing
Refer ro and follow the granring agency's specifications and process. 1hese elernenrs will help you avoid the pitfalls list-
instrucrions wirh regard ro rhe formatting of references. ed in Textbox 17.2. Often the rnost diflicult pan is simply
Enrer your references into a citation managemenr sofrware ro ger started! •
REFERENCES
--
1. Singh S, Suganthi P,Ahmed J, Chadha VK. Formulation of health research protocol-a step by step description. NTI Bulletin
2005;41 (1-2):5-1 O. Available from: http://medind.nic.in/nadt05/il/nact05il p5.pdf
2. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Oeslgning clinical research: an epidemiologic approach. 3rd ed.
Philadelphia: Lippincott Williams & Wilkins; 2007.
ADDITIONAl RESOURCES
Fathalla MF, Fathalla MMF. A practical guide for health researchers. Cairo: World Health Organization Regional Office for the
Mediterranean; 2004. Chapter 5: Writing the research protocol; p. 65-71. Available from: wvwv.emro.who.intldsaf/dsa237.pcJf
• An extensive and practical guide on the entire research process, including how to plan the research, write the protocol, apply
for funding, implement the study, analyze and interpret the data, report the results, and write and publish a scientific paper.
Mclnnes R, Andrews B, Rachubinski. Guidebook for new principal investigators: advice on app/ying for a grant, writing papers, setting
up a research team and managing your time. Ottawa: Canadian Institutes of Health Research. Institute of Genetics. Available from:
wvwv.cihr-irsc.gc.ca/e/27491.html
• An excellent overview of how to write research protocols for grant applications.
Pan American Health Organization. Guide for writing a research protocol. Washington (DC): The Organization; n.d. Available from:
www.paho.org/English/HDP/JDRlRPG/Research-Protocol-Guides.htm
• A very good overview of the main steps in developing a research protocol.
University College London Hospitals: NHS Foundation Trust. Guidelines for comp/eting a research pro toco/ for observationa/ dies.
London: Medical Statistics Unit.University College London Hospitals R&D Directorate; 2006. Available from: wvwv.sld.cu/galeriaslooi
sitioslrevsa Iud/g uidelines_for _observationa I_studies.pdf
• An overview of writing research protocols for observational (non-intervention) studies (e.g., case-control or cohort) and "he
biases that should be addressed.
138 © 2011 The Royal College 01 Physicians and Surgeons o Cc~a-c¿

77 Writing a research protocal
SUMMARY CHECKLlST
o Gather all the information you will need to write your pro ocol. in 'uding any specific formats for target audiences.
O Give your project a clear, explicit title.
O List the members of your research team and their role (inves igator, co-investigator, collaborator)
O Write a brief project summary.
O Write a succinct problem statement
O Write up your background statement, using the results of your literature review to justify your study.
O Clearly articulate SMART research goals.
O Describe your study design, including appropriate details about your population, settings, relevant variables, recruit-
ment, any interventions, etc.
O Describe how you will collect and manage your data.
O Describe your data analysis methods and any relevant calculations.
O Describe how ethical issues will be managed.
O Construct a realistic study timeline.
O Describe anticipated results and their significance.
O Describe the strengths and weakness of your approach.
O If applicable, create a budget for your project.
O List your references in an appropriate formal.
O Show your protocol to your team and mentors. Revise.
-.
© 2011 The Royal Colleqe 01 Physicians and Surqeons 01 Canada 139

140 © 1011 The Royal College of Physicians and Surgeons 'Cc-~

18
Getting to "approved": Concepts, guidelines and
processes in ethics applications
Julia Spence, MD, MSc, FRCPC
ILLUSTRATIVE CASE
A third-year surgical resident is asked by his supervisor, who has expertise in medical education, to work on a research project
with her. The proposed project will randomly assign junior surgical residents to one of two educational programs to teach
chest tube placement in the trauma care setting. One group will be trained using a high-fidelity trauma simulator (educational
intervention group); the other will be trained using a simple model of a chest wall (control group). AII junior surgical trainees
at two hospitals will be asked to participate. After the training phase of the study, participants will perform a simulated
trauma resuscitation in which they will be required to place a chest tube. A trained observer masked to the group assignment
will assess performance using a validated checklist Residents wil/ be given feedback on their performance. Funding for the
project is available, and so no formal grant application is needed. However, the supervisor asks the resident to start work on a
submission for Research Ethics Board review and institutional approval. The resident is not sure what this involves.
• Although applying for ethics

approval for )'our research proposal can be a daunting pros-
whether they are patients or healrhy volunreers. The erhical
conduct of chis research dernands rhat both researchers and
pect, ir affords an opportuniry for you ro consider your those who oversee research hnd a balance berween, on the
project from an indispensable perspective: rhat of the par- one hand, prorecting research parricipants to the point
ticipant. Clinical science canriot progress wirhout the par- where research becomes impossible and, on the orher hand,
ricipation of willing patienrs and healthy volunteers; exposing participanrs ro needless or unacceprable risks. le is
however, because this participation alrnosr invariably also importanr ro understand chat research does not
involves some degree of risk, the erhical oversight of health
sciences research is crucial. This chapter will explain why
CHAPTER OBJECTIVES
erhical review is needed, outline narional and internacional
standards that guide erhics review, discuss the basic erhical After reading this chapter, you should be able to:
principIes of research involving human parricipants and • discuss the research ethics application and review process
describe rhe process of obtaining Research Ethics Board • list important components of ethics review
and institurional approval for a research project. • describe issues that should be discussed with research
participants during the consent process
• list applicable national and international guidelines for
Why is ethics review required?
the conduct of research
The developrnent of generalizable knowledge ll1 clinical • describe the role of research administration in the ethics
healrh science requires research involving real people, approval process
,_._---~-----
KEY TERMS
Assent Conflict of interest Privacy Tri-Council Policy Statement
Autonomy Confidentiality Research administration Vulnerable populations
Benefit Informed consent Research Ethics Board (REB)
Beneficence Justice Respect for persons
Concern for welfare Minimal risk Risk
© 2011 The Royal College of Physicíans and Surgeons of Canada 141

:' •..@l\:~:':~"'1""""'>n'.,~ '" .,
---------_--- - -
necessarily benetit rhe individual participant: rather, it aims As a researcher, you are responsible for undersranding 4
to benefit society as a iohole. Research participarion is, and applying nor only the ethics guidelines and regulations ~
rherefore, fundamemally altruistic. 1his lS In for the jurisdicrion in which you are carrying Out your proj-
contradistinction ro the eommonly held belief that ecr, but also those associated with rhe conditions of your
individuals will direcdy benefir from participarion in funding. 1his applies ro granrs accepred from government ~
research (i.e, "the therapeutic misconceprion."l) sources frorn orher countries. For exarnple, research funded ~
In addirion ro having no guarantee ofbenefit, [hose who by the US government, or conducted at insriturions receiv- 4
participare in research are at risk ofbeing exploited. Formal ing US government funding, must comply wirh applicable ~
erhics review seeks ro minimize chis possibility by ensuring US federal regularions in addition ro rhe TCPS. 4
the independent assessrnent of all proposed studies. 1he
review process aims ro ensure thar [he potential risks of The Tri-Council Policy Statement •
research are acceptable in ligh( of [he potential benefirs and In 1998, rhe Canadian Insritutes ofHealrh Research (for- ~
that rhose risks and benefits are disrributed fairly among all merly known as the Medical Research Council), rhe Natu- •
eligible groups. Ir is irnperative thar porentially vulnerable ral Sciences and Engineering Research Council of Canada, 4
populations not be unjustly burdened by participation in and the Social Sciences and Humaniries Researeh Council
research that mighr nor be ofbenefir ro rhern. of Canada adopted the TCPS; amendments fol!owed in 4
A key funerion of ethics review is ro ensure that rhere is 2000, 2002 and 2005,10 and a second edirion was released t
an adequare process ro obtain informed
pants must be given sufiicient inforrnation
consent. Partici-
regarding the
in 2010.11 The TCPS sets out a srandard for the conduce of
research involving human subjects in Canada. As a condi- •4
purpose, merhods, risks and benefirs of the research projecr, tion offunding from any of the three councils, ir is required
1
and about rheir rights and obligarions, so that rhey can that, at a minimum, researchers and their institutions apply
make an informed
The principIes
decision about wherher ro participare.
of informed consent wil! be discussed in
the ethical principles and the anides
research conducted under its auspices.
of rhis policy for all •
~
more detail later in the chapter. 1he TCPS supports and encourages
rhe need for generalizable knowledge musr be balanced by
research; however,
•4
respecr for human dignity. The second edition describes •
Standards for the ethic:al conduct of
three principies: respect for persons, concern for welfare,
health sciences research t
and justice (internacional guidelines and earlier versions of
Many regularory frameworks for research oversighr arose in rhe TCPS have also framed the firsr twO of rhese conceprs in
response ro infamous exarnples of abuse of vulnerable the language of autonomy and beneficence)."?
research participants.l " Foundational documents indude
the Decfaration 0lHelsinki,5 the Belmont Report." those por- • In rhe context of research, rhe principie of respect
rions of the Unired States Code 01 Federal Regulations con- for persons incorporares rhe obligarion ro respecr
cerning the "prorecrion of human subjects.Y" and rhe rhe auronomy of inclividuals in order ro ensure rhar
Council for Biomedieal Research Involving Human Sub- they are able ro make clecisions wirhour interference
jecrs (CIOMS) lnternational Ethical Guidelines [or Biomed- or undue iníluence. 1his principle recognizes rhar
ical Researcb lnvolving Human Subjects? parricipation in research can be direcr, or rhrough
In Canada, the field of research policy and research eth- rhe use of dara or biologic marerials, indirect.
ies review is relatively new; most guidelines have been in Furrhermore, researchers are obligared ro prorecr rhose
place only since the late 1990s. Central among rhese is the who, by virtue of impaired decision-making capacirv,
Tri-Council Policy Statement: Ethical Conduct [or Researcb have diminished auronomy.
lnvolving Humans (TCpS).IU.II Canadian researchers are • The principie of concern for welfare implies that
obligated ro cornply with applicable regularions (relared ro individuals and groups who participare in, or
both federal and provinciallrerriroriallegislarion) as well as are affecred by, researeh musr be proeecred mm
ro rhe TC'PS. Beeause regulations and guidelines are updar- exploirarion and unacceprable risk. 1he bese possible
ed regularly, ir is irnportanr ro be aware of rhe faer rhar eth- balance berween risks ro participants and porential
ies policies can evolve, even over the course of a project. socieral beneíits of a proposecl srudy musr be soughr.
I
142 © 2011 The Royal College 01 Physicians and Surgeons 01 Ce"13;;:¿¡ •
78 Cooceots. gu "defines and processes in ethics app/ications
Guidelines for clinical trials

• The principie of justice relates ro the obligarion ro
crear poremial parricipanrs faidy and equirably and Clinical rrials are research studies rhar assign human partic-
ro ensure rhar groups are neirher unduly burdened ipanrs or groups ro healrh-related inrervenrions. Because of
wirh rhe risks of research nor denied rhe benehts and rhe rime and resources necessary w complete clinical trials,
knowledge gained by research. rhey are not usually carried our by trainees. In sorne cases,
rrainees mighr be involved wirh clinical rrials being carried
1he applicarion of these principies maxirnizes the benefirs OL![ ro seek regulawry markering approval for investiga-
and minimizes the harms of research; this in mm helps ro rional drugs or devices, or for novel, unapproved uses of
engender public trust in the scientiíic process, which in drugs and devices; such srudies must be conducred in corn-
irself is beneficial to sociery. pliance wirh Canadian federal regulations." If applicable,
The TCPS provides guidance for researchers and REBs researchers should also be knowledgeable about rhe Inter-
in the following areas: narional Conference on Harmonisarion _ Good Clinical
Pracrice (ICH-GCP)' guidelines, which have been adopr-
• research erhics framework and review ed by Healrh Canada.'" Good Clinical Pracrice (GCP) is an
• informed consem and departures from general internarional erhical and scienrific qualiry standard for
principies of informed consent designing, conducring, recording and reporting trials rhat
• faimess and equiry in research parriciparion involve rhe participarion of human subjects. Canadian
• privacy and conhdenrialiry researchers should also be complianr wirh rhe Dec!aration 01
• conflicrs of interest Helsinlei 5 and the CIOjl¡JS lnternational Ethical Guidelines
• multi-jurisdicrional research jor Biomedical ResearchInvolving Human Subjects?
• research involving Aboriginal peoples
• qualirarive research
Tip
• use of human biological marerials
• human generic research Many REBs and institutions require certification of study
• clinical rrials investigators and staff in Good Clinical Practice (GCP)
guidelines before they undertake a clinical trial. Many one-
day accredited courses are available
Tip
Many REBs require that an ethics tutorial, such as the one

offered by the Tri-Council or the US Nationallnstitutes of
Research Ethics Board Review
Health, be completed befo re a project can be approved and What is a Research Ethics Board?
funding released. Research erhics review is essentially a peer review process.
The members of a Research Ethics Board (REB) rneer reg-
A summary and tutorial for the TCPS document can be ulady and discuss proposed research projecrs from the
found at www.pre.ethics.gc.ca/eng/education/tutorial- viewpoint of the poremial participants. Instirurions
didacticiel/ appoinr the REB to review rhe ethical acceprabiliry, scien-
tific validiry and feasibiliry of all research involving humans
A similar course is available from the NIH at http://phrp. conducted within their jurisdiction or under rheir auspic-
n ihtra ining. com/users/log in. php es." REBs are independem and accoumable for their deci-
sions. Alrhough the number of members and rheir arcas of
The Royal College of Physicians and Surgeons of Canada experrise will depend on rhe range and volume of research
also provides an excellent resource al http://rcpsc.medical. wirhin rheir insritution, in Cariada rhe minirnal require-
org/bioethics/extended-primers/research-ethics-full_e.php
a The lull na me 01 the ICH is the International Conlerence on

Harmonisation 01 Technical Requirements lor Registration 01
Pharmaceuticals lar Human Use. See www.ich.org/home.html
© 2011 The Royal College of Physicians and Surgeons 01 Canada 143

The Research Guide: A primer for residents. other hea!th care trainees, and practitioners
mene for an REB is five mernbers, who must include (1) ar specific issues related ro each prorocol. The review process
least rwo members with research expertise, (2) onc who is allows REBs and invesrigarors ro identity issues that may
knowledgeable in erhics, (3) one who is knowledgeable in affect parricipants, ro balance differing perspectives, and ro
the law, and (4) one who has no afliliation with the insritu- consider potencial solutions. Many REBs are moving
tion (cornrnuniry mernber). Ad hoc advisors may be roward a model of greater collaborarion with borh research-
appointed ro provide scientific expertise reguired Ior spe- ers and potenrial research participanrs, rhrough borh for-
cific proposals. mal and informal mechanisms.
Independenr review allows rhe project to be examined
from rhe participant's vicwpoinr. Regardless of how wel!
Tip
inrentioned they are, researchers have multiple, legitimate
Many REBswill allow researchers to observe a meeting; interesrs (induding expeditious study cornpletion, fund- I
many request or require principal investigators to present ing, career interests) that may be perceived ro be in conílict
and discuss their protocols, especially in the case of ethically with rhe inreresrs of participants, Independent review helps
cha!lenging research. However, researchers are not allowed ro ensure that rhe best interests of parricipants are prorect-
to observe deliberations or voting directly re!ated to the ed. [t seeks to maximize rhe socieral value of research while
I
review of a project with which they are associated. minimizing risks and ensuring thar those who agree ro par-
ticipate will be treated with respect. In so doing, indepen- J
dent review also ensures a greater level of public «

What i:ype~ of studies must be reviewed accounrabiliry. At a minimum, the issues oudined in the t
by an F:'EB? following sections will be considered by an REB in rhe
l\.ccording to rhe :rcps (Anide 2.1), all research rhat course of a review.
involves human subjects reguires review and approval by
an REB befo re it can be ini tiared. Research involving
Tip
,•
human rernains, cadavers, tissues, biological fluids, ernbry-
os or tetuses must also be reviewed by an REB. However, Seek quidance from your institutional REBas you develop •
research about living individuals that is based on legally your protocol. A non-scientific co!league may also provide •
accessible and puhlicly accessible inforrnation (e.g., a sys- valuable content and language guidance when developing •
t
ternatic review, documents, records, etc.) does not always a proposed consent formo
rcquire ethics review (Anide 2.2).11 Performance reviews
or testing wirhin normal educational requirernents should
not be subject ro REB review. Qualiry assurance/improve- 1he research question. Rescarch in rhe health sciences is
ment (QA/QI) projects might not require review, and of value to sociery when it leads ro porenrial improvemenrs
many institurions provide specitic guidance ro determine in health or health careo Without rhis potential for socieral
wherher a project is considered QNQI or research. value, research panicipams may be assurning risk ro no
purpose. The importance of rhe quesrion addressed by a
research project is rhus an imporrant erhieal considerarion.
Tip
If, given the nature and scope of your proposed project, you
Tip
are in any doubt about the requirement for ethics review,
seek guidance from your local REB. Many REBsrequest a lay sumrnary of the protocol that
clearly defines the research question. Ensure that your
summary is written in language suitable for a non-scienti: e
What do REBsconsider when they conduct audience. This ensures that all members of the REB,bo h
. ?
a revrew scientific and non-scientific, will understand the significance
TI1e rhree principies of respect jor persons, concern jór we!fore of your project.
and justice provide rhe foundarion for al! research erhics t
review. On rhe basis of rhese principies, REBs consider rhe
í44 © 2011 The Royal College 01 I'hysicians and Surgeons oí Canaca

78 Canee rs, 9 ¡defines and proeesses in ethies app/icatians
Scientific validity. Alrhough sorne invesrigarors mighr fee! directlv benehr rhose in similar circurnstances. In short, vul-
that ir is not rhe business of an REB ro evaluare rhe scien- nerabiliry is minimized and study integriry is maximized by
rific rnerits of a study, scienrific validiry is fundamental ro ensuring that che cargec group for participacion is appropri-
rhe ethical conduct of research. REBs should ensure that are for the srudy quesrion, design, risks and benefics.
members wirh expertise in research and merhodology are
presem during che review process ro ensure rhar scientihc Parrnerships. Researchers rnusr consider, and work cor-
validiry is cornperently considered. Unless a research proj- roborarive with, rhe communiries chey serve. 1his may
ecr has a dear potenrial ro generare reliable and valid results enrail involving parients or cornmuniry members in che
that are generalizable ro che greacer population, ir does nor planning and conduct of a srudy. Ac the conclusion of che
warrant che exposure of parricipanrs ro any pocencial risk. research, the cornrnuniry is also enrirled ro fair and, in some
cases, tangible benefirs of che research. Expecrarions in chis
Risks and benefits. le is dear thar participation in virrually regard should be discussed befo re che projecc is begun.11
any research projecr encails risk, however minimal thar risk
may be. In Canada, rninimal risk research is defined as Respect for participants. Ir is important thar iridividuals
research in which che probabiliry o[ possible harms implied are rreated with respecc chroughouc all the phases of a scudy,
by participation is no greacer chan that encountered by che including during recruitment (regardless of wherher par-
parricipant in rhose aspecrs of his or her everyday life thar ticipation is agreed ro or dedined), scudy participacion and
relate ro the research. II follow-up, and afcer che formal commitrnenr has ended.
An integral component of the REB review process is che Researchers must not lirnit their responsibiliry ro che moni-
discussion of porential risks and benefirs as well as proposed roring of individuals during che trial (e.g., for adverse reac-
mechanisms ro minimize potential risk and maximize tions, adverse evencs, change in status), but also rnust
poremial beneíit. 1he porential risks and benefirs of consider halring che rrial if aggregace outcornes from the
research should be discussed in lay terrns during the currenc study or data obtained from other sources raise
informed consent process. Risks are not limired ro physical safery concerns or make the study unnecessary. Research-
risks: they may indude psychological, social, economic and ers must also respecr the choice of any participant who
privacy risks. Where available, risk estirnation should be decides, at any rime, ro withdraw from the scudy.
based on current daca. Similarly, the potential benefirs of Ac che end of che study, parricipanrs and concerned corn-
research should be described ro che parricipanrs. Common- munities should be informed abour the ourcome of the
Iy provided secondary benehts, such as rernunerarion, the research. Having assumed che risks associared with research
provision of specialized care or increased surveillance, parricipation, they have a righr ro undersrand the results and
should not be considered pan of che risk-benefic evalua- their implicarions for health care and healrh care policies.
tion, since they do nor justify risky research. 1he overall Considerarion should also be given ro the ongoing care
benefics ro sociery are assumed and are gene rally not indud- of research parricipanrs ar the rime of tri al complerion. 1his
ed in che REB discussion or consem process. may require referrals ro primary care providers and, in more
complex situations, ongoing arrangemencs for access ro
Participant selection and recruitment (see ch. 20). Partici- medicarion, interventions or treatrncnts thar should have
panc seleccion and recruitrnent must ensure the fair selec- shown ro benefic the participants or research population.c'?
rion of participanrs and che equirable disrribution of Having respecr for parricipants also means following a
research risks and benefits. Study participants should be fair and chorough process for obcaining inforrned consent,
drawn from rhe least vulnerable popularion rhat will enable respecríng privacy, and keeping personal informarion con-
valid condusions ro be reached. Some potencial participants fidencial. These imporrant echical requirernents are dis-
may have vulnerabiliries related ro cognicive abiliry, age, cussed in the following secrions.
clinical status, social marginalizacion, economic disadvan-
cage or incarcerarion. 1hese participants should not be Informed consent. 1he informed consenr process is fun-
recruired specificaLIy because of rhese condicions; however, damental ro the erhical conducr of research. le recognizes
potentially vulnerable parricipanrs may be recruited if rhe rhe auronomy ofindividuals with decision-making capaci-
knowledge gained from their participacion is likely to ry. For the researcher and rhe research participanr, the dis-

cussion that occurs during rhe recruitrnent and consent research ro be conducted without free and informed con-
process provides an opportuniry ro discuss rhe srudy design senr or wirh delayed consent. Consent for ongoing parrici-
as well as the righrs and obligarions of rhe participant. pation must be obtained ar the earliest opporruniry after
The imbalance of power berween researcher and parrici- enrolment. Guidance should be obrained from the TCPSll
pam, and the porenrial for parricipants ro be unduly influ- and local applicable regulations.
enced by thar imbalance, should always be considered,
especially when a trearing physician is conducring a study
Tip
that involves his or her own parient popularion. An estab-
lished patienr-physician relarionship can make research Most REBswill provide sample consent forms. In general,
participants particularly vulnerable during the informed consent forms should be written at a reading level
consent process. appropriate for the proposed study population. Ensure that
Consent documenrs must disclose informarion relevant they are proofread carefully befo re submission.
ro rhe research study in a complete and accurate fashion
while srriking a balance berween exhausrive disclosure and Local REBswill provide guidance regarding waiver of
readabiliry. Standard componenrs of a consent form are consent, substitute decision-makers, and consent in
provided in Table 18.1. Mosr REBs are able ro provide sarn- emergency health care situations. Be sure to obtain the
pie consenr forms rhat demonstrate suggesred wording and guidance you need early in the planning stages of your
formars. study.
For the informed consent process ro be valid, the poren-
rial parricipant must have the capacity ro undersrand and
.- appreciate the informarion presenred. He or she must vol- Privacy and confidentlality, Issues of privacy and confi-
untarily agree ro participare wirhout being unduly inílu- dentiality arise with respecr ro research participanrs ano
enced or coerced. In sorne contexrs the approval of leaders the sponsors of research.
(e.g., households, school principals, elders, erc.) mighr need First, respecr for the privacy of research parricipants and
ro be obtained befare individuals are contacted: however, rhe confidentialiry of their personal data is riot onlv cthi-
each individual must provide independent consent for par- cally imperarive but, at many sites, is subjecr ro federal,
ticiparion. For persons from whom consent cannot be and/or provincial or rerriroriallegislarion. Researchers are
obrained, whether for reasons of age, mental capaciry or responsible for the securiry of any personal inforrnarion,
severe loss of funcrion, straregies rnust be developed ro including healrh inforrnation, that they collecr rhroughour
obrain informed consent from legally authorized represen- the life of the study and somerimes for many years beyond:
tatives and ro give opportunities for potencial participanrs that is, during the recruitrnenr, enrolmenr, collection, anal-
ro refuse parricipation or ro assent ro participation. Thar is, ysis, srorage and destruction of data." This obligarion
some individuals who do nor have the capaciry for gen u- applies ro identiíying or identiíiable inforrnarion, such as
inely informed consent mighr have suflicienr capaciry ro names, address, full posral codes, telephone and fax num-
determine that they do not want ro participare, or, wirh the bers, email and IP addresses, URLs, healrh card numbers,
guidance of a surrogare decision-rnaker, may be able ro images, social insurance numbers, medical record nurnbers,
assent ro participare. Provision for assent is commonly dates of birrh, and any health-relared informarion. Investi-
included in ethics applications for studies that recruit garors must be in compliance wirh the duties ser OL!( in
minor and/or adolescent parricipants, in conjunction with locally applicable federal, provincial and rerriroriallegisla-
rhe informed consent of the legally aurhorized representation, rhe TCPS and local applicable policies. An excellenr
tives of porenrial parricipants. In certain circumsrances, resource for privacy-related Bes!
issues is the documenr
waiver of consenr may be perrnirted (e.g., for charr reviews Practices flr Protecting Privacy in Health Research! as well as
in which privacy prorection has been maximized). In spe- the TCP5.11
ciíic situarions-such as rhose involving emergency care, Second, researchers rnust also consider me impacr o any
when rhe prospecrive participanr cannot provide consent, contractual obligarions wirh sponsors with respecr o th
imrnediate intervention is required, and rhe research may confidentialiry of data and proprierary information. ¡~.:-
provide a real possibiliry of beneíir, REBs may perrnit Such agreemenrs can ser lirnits on rhe righr ofinitial analvsi
146 © 2011 The Royal College 01 Physicians and Surgeons o; Cc~

78 Concepts, ~J;delines and processes in ethics app/ications
• Table 18.1: Elements commonly included in letters of information and/or research participant consent forms
Item Explanation
Study title Full title and working ti-le
PI-incipal investigator Name, credentials and contad information
______
Co-investigators
· .
Names, credentials and contact information
_
Study coordinator Names, credentials and contact information
--------------------------------
24-hour contact Name of someone whom the participant can contact with urgent questions or concerns
about the study at any time
Study sponsor In general, as defined in ICH-GCP

.._--- ------
Conflict of interest Declaration as consistent with institutional policy
Study information
Introduction General information about the specific research as proposed
.,
Background and purpose Information about why the study is being conducted, and about the standard (non-
experimental) treatment
Description of the methods

What will happen during the study, how may people will take part. the responsibilities
of participants, and details about study visits and procedures
"t
-e
Potential harms The potential harms of participation, both significant and common. Literature should be
used, when possible, to provide estimated rates of occurrences.
eti
This section may include reproductive risks, or these may be listed separately -
Potential benefits The potential benefits of individual participation
Alternatives to participation Other choices available to the potential participant
Privacy and confidentiality The impact of the study on the privacy and confidentiality rights of the participant
Communication with primary care or How the results and therapy with be communicated with the participant's doctor, if this
treating doctor is agreed to by the participant
Cos s of participation/reimbursement Any costs, reimbursement and remuneration associated with participation
In case of injury, illness or harm Recourse available in the case of harm
Participation and withdrawal An explanation that participation is voluntary, and that the participant has the right to
withdraw without penalty
New findings 01' information What the participant may expect with regard to being informed of new findings or
information that has a bearing on the study
Study results How (or whether) study results will be communicated to the participant
---_._-------- -------------- --------- -_._._------- -------
REBcontact information Information about the ResearchEthics Board review and contact information in case the
participant has any questions regarding his or her rights
Statement of consent and signatures Frequently contains a summary of the rights and obligations of the participant.
The signature of the participant and of a witness to the consent procedure or to the
signature is frequently obtained. This section may also require that the person who
obtained consent sign and dates the consent formo
--------------------- --- - ------

and interprerarion of rhe clinical trial dataset. As pan of research and rheir personal, institucional or orher interesr
ensuring rhe inregriry of rheir research, ir is imporranr for Conflicts ofinterest-a!so ca!!ed "competing interests'í-s-do m
researchers ro seek assurances rhar rheir contractual righrs necessarily imply wrongdoing, and the recognition, disclosui
and obligarions are consisrenr with local srandards, policies and management ofpotentia! conflicts ofinterest mitigate tI,
and rhe TCP5. As will be discussed larer in rhe chaprer, risk of scientijic misconduct. Contiicting roles, duries arn
review by your insrirurion's research administration office responsibilities musr be assessed ro ensure that the interest
will cnsure rhar insrirutional policies are mer regardíng of poremial parricipants and the inregriry of research ar.
confidentialiry agreemenrs and privacy-related issues preserved. Ir is rhe responsibiliry of researchers, REB mem
arising from rransfers of personal informarion and bers and insritutions ro idenrify real, potential or perceivec
Giological material, and from the publicarion of nndings. coníiicts of interest associared with rheir roles and ro estab-
lish mechanisms ro declare and mallage these issues as they
What should I do if I need to change my protocol arise.
or study procedures?
REBs make decisions regarding rhe acceptabiliry of sug-
Tip
gesred deparrures from originally approved research.!' In
general, there are rhree caregories ro consider: (1) unantici- It is your responsibility to be aware of your institution s
pared or unexpected evenrs; (2) changes from rhe approved guidelines and requirements for the declaration and
prorocol arising from ongoing feedback; and (3) unavoid- management of conflicts of interest.
able, single-incident deviations from rhe approved proto-
•. _ col. AlI prorocol

approved by rhe approving
addrcssed immediately
amendments must be reviewed
REB. Issues ofsafety should be
by the research team and reported
and
ctinical trial registratíon
Clinical trial regisrries permir web-based access ro informa-

promptly lo the REB. rion regarding ongoing clinical trials so rhar informarion is
available abour trials and rheir results.? Trial registration is
rhe accepted inrernarional srandard. According ro rhe
Tip
TCPS, ''Al I clinical rrials shall be registered befo re recruit-
Participant safety is paramount. Immediate changes to a ment of the first trial participanr in a recognized and easily
protocol that are necessary to ensure patient safety should web-accessible public regisrry" (Article 11.3).!! Trial regís-
be implernented, and REBsshould be notified as soon as tration helps ensure that all srakeholders-from patients,
possible. ro c1inicians, systernatic reviewers, developers of clinical
pracrice guidelines, and policy-makers-can make thern-
selves aware borh of ongoing rrials and oE the exisrence of
What is involved in continuing ethics review? trials whose resulrs are never published. This awareness
The REB is responsible for making a final dererminarion as serves a number of purposes. Publication bias and selecrive
ro the nature and frequency of continuing erhics review. Ar reporting-rhar is, failure ro publish negarive srudies or
a mínimum, an annual status or end-of-study repon is studies wirh less promising resulrs-disrorr rhe body of
rcquired. Annual renewal may require, bur is nor limited available evidence.P:" Non-disclosure agreemems wirh tri-
ro, documentarion regardíng the conduct of rhe study, al sponsors can conrribure ro rhe suppression of negarive or
including the number of participants recruired, timelines, inconclusive findings; trial regisrrarion, in randem wirh rhe
adverse evenrs, unanricipared hndings and an update development of editorial policies rhar insisr on registration
regarding rhe currenr literature. and fuI! access to dara as a requiremenr for publicarion,
seeks ro combar rhis problem. As wel! as improving aware-
ness of trial resulrs, regisrrarion of ongoing rrials helps
Conflict of interest
researchers avoid unnecessary duplicarion and provid
A conflict of interest can arise when activi ties or situations opporruniries for collaborative work and rhe identifica ion
place persons or insriturions in a real, potenrial or perceived
confiicr berween rheir duties or responsibiliries relared ro b Exarnples include www.clinicaltrials.gov and www.contro ed r 2 5C::-
148 © 2011 The Royal College of Physicians a d Surqeons o: (¿-¿::a

78 Conceo I guidelines and processes in ethics applications
of potential research gaps. Ir is hoped that registration wil! grants or agree ro conduce research on behalf of a sponsor
also lead tO improved qualiry of trials by idemifying rneth- have appropriare access ro resources and parricipanrs as
odologic problems or bringing ro light prorocol changes wel! as me skills and knowledge necessary ro conduct the
made during the COLtrSeof a tria!. Finally, tri al registrarion research in an efficiem and safe manner. In general, institu-
al!ows potential panicipants w idemify trials that may be tions receive and adrninisrer funds on behalf of research
applicable to thern. rearns. Ir is essemial, rherefore, rhar instirurional and ethics
In compliance with rhe requiremems of the WHO and review be conducted.
the International Cornrnirtee of Medical Journal Edirors, To be eligible ro receive and adrninister funds from rhe
researchers should register their trial befo re they start ro federally adrninisrered agencies (CIHR, NSERC and
recruit parricipants, and provide rheir REB with rhe regis- SSHRC), insriturions must agree ro comply with a nurn-
tration nurnber.!" According to rhe WHO, "a clinical trial is ber of agency policies set out as schedules ro a Memoran-
any research study that prospecrively assigns human parric- durn of Understanding berween the agencies and
ipants or groups of humans ro one or more health-related instirurions." In a similar fashion, instirutions thar receive
interventioris to evaluare the effects on healrh outcornes." funding from US governmem sources must provide assur-
One example of a clinical trial registration sire is Clini- ances that the use of rhe funding wil! be in compliance
ca!Trials.gov. Open ro over 170 countries, this site provides wi th the specified regulations."
repon summaries for registered clinical trials and observa- Clinical trial applications to Health Canada are required
tional studies. The site includes information on panicipam for studies in tended for use in regularory marketing approv-
.,
flow, baseline characterisrics,
mation.'" Alrhough information

outcorne measures, staristics
and adverse events as well as pertinenr adrninistrarive
trial, according ro the US Food and Drug Administration

may be subrnirted for any
infor-
al in Cariada. This includes Phase 1, II and III clinical tri-
als," including trials thar test the safery and effectiveness of
marketed
indications.
drugs ourside the parameters of rhe approved
--c
e
Amendmems Act of2007d the responsible parry for a study
Contracts and agreements
eti
--
must register and repon results for trials, including, but not
.....
limited to, trials of drugs and biologics of controlled clinical Your institution's research adrninistration staff provide a
investigations (excluding Phase 1) or of a product subject ro variery of services on behalf of the institution and its
FDA regulations. In general, the most responsible parry is researchers-including yOLl. They have experrise in rhe
required ro subrnit "basic resulrs not later rhan one year preparation and review of agreemems, including:
afrer rhe 'primary cornpletion date' or rhe date that the final
subject was examined or received intervention." • clinica! study agreements
• confidemialiry/ non-disclosure agreemems
• material transfer agreemems
The role of institutional research
• service provider/independent contractor's agreements
administration
• peer-reviewed funding agreemems
Institutions are responsible for all research conducted with- • basic science research agreemems
in rheir jurisdiction or under their auspices.P:" That is, • inter-institurional agreemems
they are responsible for the conduct of research conducted • intellecrual properry agreemems (e.g., licence
by rheir faculry, staff or srudenrs, regardless of where rhe agreements)
research is conducted. Institutions are rcsponsible for • privacy/data sharing agreements
ensuring thar research is conducted in compliance with
applicable policies, guidance docurnents and regulations. Review by institucional representatives ensures cornpliance
This includes oversight relared ro funding, good clinical with applicable policies, including those related ro transfers
practice and privacy legislation. Insritutions are also of personal information and biological material, liabiliry
responsible for ensuring that research teams who apply for and publication.
Researchers must consider the impact of contractual
e WHO International Clinical Trials Registry Platform. www.who intlictrp/en/
obligations with sponsors, who may seek ro lirnir the right
d See http://clinicaltrials.gov and http://prsinfo.clinicaltrials.gov/fdaaa.html of inirial analysis and inrerpreration of rhe clinical trial

The Research Guide A primer for residents, other health care trainees, and practitianers
Conclusion
dataset, Ir is imporrant rhat researchers ensure the integrity
of their research. Assurances should be sought that contrac- Preparing a research ethics board and instirutional applic
tual righrs and obligations are consistcnt with local stan- tion are essential sreps in the conducr of research. Althou¡
dards, policies and the TCPS. Review by instirutional ir may be perceived as burdensome, it al!ows researchers
represenratives ensures that insritutional policies are rnet view a srudy protocol from the perspective of the partic
regarding agreements and privacy-related issues arising pant, ro minimize risks, and ro maximize safety and tl
frorn transfers of personal information and biological mate- porential benefirs of the research. Ethical principles shoul
rial, and with regard to publication. be considered when the protocol is being designed; ethk
should not be an afterthoughr. The Tri-Council Policy Stat,
Insurance ment: Ethical Conduct for Research lnvolving Humans" prc
Researchers who are contracted by third-party sponsors vides important information for Canadian researcher:
should ensure that appropriate insurance is in place that Those planning ro conduct clinical trials should also seel
will cover any potential claims resulting from the project, specific rraining in the International Conference on Har
Of note, in the case of peer-reviewed funding, the research- monizarion-Good Clinical Practice (ICH-GCP)13 guide
er's home institution is considered the sponsor and is there- lines to ensure rhat regulatory requirernenrs are met
fore responsible for providing insurance. Finally, researchers should be aware of both the implica.
Researchers should ensure rhat they have insurance cov- tions and rhe obligarions associated wirh funding sources.
erage. They may be covered under institucional insurance, As applicable, al! necessary contracts should be reviewed by
as employees. However, in the case of physicians, insurance your institution ro ensure that any required agreements are
may not be available in this fashion. Ir is your responsibility in place. •
ro contact your insurer, including the Canadian Medical
Protective Association (CMPA), to inquire as ro the type
and nature of insurance coverage you have with respect to
rcsearch, since nor all trpes of projecrs may be covcred.
© 2011 The Royal College 01 Physicians and Surgeons 01 Cana da

150
78 Co I ~'-' -aelines and processes in ethics app/ications
CASE POSTSCRIPT
Although the potential risks arising frorn participation in this research project appear to be minimal, various issues should
be reviewed before the protocol is submitted to the REB.Although the staff supervisor may perceive the project as a quality
improvement effort that does not require REBreview, if the student intends to present or publish the results, and thus
disseminate the information to a broader audience, it willlikely be deemed research. If the trial is to be conducted at more
than one site, investigators should make themselves aware of the protocol applications and policies for individual and multi-
site reviews. (Of note, variability in REBreview and requirements among sites may add to the logistical challenges, including
the time necessary to obtain all required approvals.)
From a scientific perspective, the question appears to be important and scientifically valid.
How will residents be recruited for the study? The REBmight consider residents to be a potentially vulnerable population
insofar as they are in the process of completing mandatory training requirements. Might they feel pressured to participate
in a study conducted by a colleague or staff supervisor? Would using a neutral third party (a research coordinator) to obtain
consent help to minimize such influence 7 15 there a strong expectation on the part of the researcher that 100% recruitment
will be achieved, or do the residents perceive such an expectation to exist?
The residents willlikely have questions about the potential consequences of participating or not participating. They-like the
REB-will want to know the risks and benefits. Will participation have an impact on their training? Will it affect their current
or future evaluations? If it will (or is perceived to), how can this be managed? Will participants be permitted to withdraw from
the study if they choose? If so, can they withdravv their data?
The protocol states that residents will be provided with feedback. Who will provide this feedback, and how? For example,
is it appropriate for a resident colleague to have access to the performance ratings of his or her peers? Will the assessments
become part of their permanent training record? 15 this reasonable 7 What are alternative strategies?
What other privacy and confidentiality concerns need to be addressed? The researchers must give consideration to the
number of residents available for the study, and to whether potentially identifying demographic data are being collected,
grouped and reported in a manner that will maintain privacy. (In general, the minimum size for grouped data should be five,
i.e., n = 5.)
How will consent be sought? Informed consent will be required if this is a research protocol. Consent guidelines for the
respective REBsshould be followed The consent form should be checked for spelling, grammar and readability. In this
case, the general reading skills of the participants will be felt to be high, and most REBswill allow for a more sophisticated
description of the study, as well as of associated risks and benefits, privacy issues and the rights and obligations of the
research participant.
Finally, in assessing the potential risks to participants and the ethical issues arising from the protocol, consideration should be
given to holding a group discussion to enable adequate input from all parties before REBapproval is sought.
© 2011 The Royal College 01 Pl1ysicians and Surgeons 01 Canada 151

The Research Guide: A primer for residents, other health care .rainees, and practitioners
REFERENCES
1. Emanuel EJ,Wendler D, Grady C. An ethical framework for biomedical research. In: Emanuel EJ,editor. The Oxford textbook of
clinical research ethics. 1st ed. New York: Oxford; 2008. p. 123-35.
2. Rothman DJ. Research ethics at Tuskegee and Willowbrook. Am J Med. 1984;77(6):A49.
3. Curtis H. Getting ethics into practice Tuskegee was bad enough. BMI 2004;28;329(7464) 513
4. Goldby S. Experiments at the Willowbrook State School. Lancet. 1971; 1(7702):749.
5. World Medical Association. WMA Declaration of Helsinki-ethical principies of medical research involving human subjects.
October 2008. Available from: www.wma.netlen/30publicationsll0policieslb3/index.html
6. (US) National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report:
ethical principies and guidelines for the protection of human subjects of research. Washington: The Commission; 1979.
7. (US) Department of Health and Human Services. Code of Federal Regulations. 21 CFR 50, 1996 Oct 2. Report No. 61(192).
8. Department of Health and Human Services. Code of Federal Regulations. 45 CFR Part 46, Protection of Human Subjects. 15-1-
2009.20-1-2010.
9. Council for International Organizations of Medical Sciences (CIOMS). International ethical guidelines for biomedical research
involving human subjects. Geneva: CIOMS; 2002. Available from: www.cioms.ch/publications/layout_guide2002.pdf
10. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, Social Sciences and
Humanities Research Council of Canada, Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS),
1998 (with 2000, 2002, 2005 amendments). Ottawa: Interagency Secretariat on Research Ethics; 2005. Available from: wvvw.
pre.eth ics.gc.cale ng/archives/tcps-eptc/Def au!tI
11. Canadian Institutes of Health Research, Natural Sciences and Engineering Research Council of Canada, and Social Sciences and
Humanities Research Council of Canada, Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, 2nd ed.
Ottawa: Panel 011 Research Ethics; December 2010. Available from: www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcp52-
eptc2/Defaultl
12. Canadian Institutes of Health Research. ClHR Best Practices ior Protecting Privacy in Health Research. Ottawa (ON): The Institutes;
2005. Available from: www.cihr-irsc.gc.ca/e/documents/et_pbp_nov05.:._sept2005_e.pdf
13. International Conference on Harmonization of Technical Requirements for the Registation of Pharmaceuticals for Human Use.
Guidance for industry Good clinical practice: consolidated guideline. ICH topic E6. Ottawa: Health Canada; 1997. Available
from: www.hc-sc.gc.ca/dhp-mps/alt_formats/hpfb-dgpsa/pdf/prodpharma/e6-eng.pdf
14. For case law examples and the Nancy Olivieri case see http://canadianhealthcarelaw.com/index.php?option=com_content&view=
category&layout=blog&id=99&ltemid=83&lang=en
15. Naylor D. Early Toronto experience with new standards for industry-sponsored clinical research: a progress report. CMAJ.
2002; 166(4):453-56.
·16. DeAngelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International
Committee of Medical Journal Editors. CMAJ. 2004; 171(6):606-7
17. Foote M. Clinical trial registries and clinical trial result posting: new paradigm for medical writers. Biotechnol Annu Rev.
2006;12379-86.
18. Steinbrook R. Public registration of clinical trials. N Engl J Med. 2004;351 (4):315-7.
19. ClinicaITrials.gov. About the ClinicalTrials.gov results database. Available from: www.clinicaltrials.govlct2/info/results
20. Memorandum of Understanding on the Roles and Responsibilities in the Management of Federal Grants and Awards. 2010.
21. Canada. Food and Drug Regulations (C.R.e., c. 870) Part C, Division 5: Drugs for clinical trials involving human subjects.
Available from: http://laws-lois.Justice.gc.ca/eng/regulations/C.R.C.,_c._870/page-397.html#h-251

The following websites and articles provide more information and case studies regarding research ethics.
Evolution of protections for research participants

University of Waterloo. Office of Research Ethics. Evo/ution of protection of human pertiopents in research [updated 2009 March 26]
Available from: http//iris.uwaterloo.ca/ethics/human/resources/index.htm
The Jesse Gelsinger case (gene therapy/conflict of interest)

New York Times. See articles compiled at: http//topics.nytimes com/topics/reference/timestopics/people/g/jesse_gelsinger/index html
Johns Hopkins University lead paint study

Alliance for Human Research Protection. Johns Hopkins University experiment on /ead paint puts chi/dren in danger. See articles
compiled at: www.ahrp.org/children/HopkinsEndangersphp
The need for contract review and publication agreements

Naylor D. Early Toranto experience with new standards for industry-sponsored clinical research: a pragress report. CMAI
2002; 166(4) 453-56
Ownership of tissue, consent and contracts

Grady D. The lasting gift to medicine that wasn't really a gift. New York Times. 2010; Feb 1. Available frorn: www.nytimes.
com/20 10/02/02/health/02seco.html
• The case of Henrietta tacks unknowing donatian to science of the HeLa cell.
Skloot R. Taking the least of you. New York Times. 2006; Apr 16. Available from: www.nytimes.com/2006/04/16/
magazine/16tissue. htmlrpaqewantede 1
• For an important case concerning ownership of prastate tissue samples, see "The case that could change everything," p. 7.
Privacy
Canadian Institutes for Health Research. ClHR best practices in protecting privacy in hea/th research. Ottawa The Institutes; 2005.
Available from: www.cihr-irsc.gc.ca/e/documents/et_pbp_nov05_sept2005_e.pdf
• An excellent resource for the management of privacy issues in the context of research.
The Tuskegee syphilis study

Tus egee University. Research Ethics: The Tuskegeesyphi/is study. Available from www.tuskegee.edu/global/story.asp7s=1207598
White RM. Unraveling the Tuskegee study of untreated syphilis. Arch Intern Med. 2000; 160(5):585-98

SUMMARY CHECKUST
o Review the relevant institutional ethics statements for your jurisdiction (eg. TriCouncil, NIH). Complete any online tuto
rials that are available.
O Review your research protocol, and identify any ethical issues that it raises.
O Revise your protor.ol to incorporate the management of ethical issues, such as informed consent.
O Apply to all of the relevant REBsfor your study, using the application forms they require.
O Note the number and details of your ethics certificates.
O Register your study, if applicable, in relevant databases.
O If you make any changes to your protocol, compose an amendment to inform your REB.
O Complete your regular review or end-of-study declaration for the REBs.
.-
154 © 2011 The Royal College 01 Physicians and Sur<;=-o-s:::= 2.-..=::2

19
Research funding and other resources
G. Mark Brown, MSc
Eddy S. Lang, MDCM, CCFP(EM), cSPQ
ILLUSTRATIVE CASE
A senior Pediatrics fellow is interested in investigating the prevalence of occult hypoglycemia among infants who present
to the emergency department with vomiting and probable gastroenteritis. She soon realizes that gathering and analysing
the data, and ultimately disseminating the results, will require significant resources. First, in order to achieve the requisite
sample size. a method to screen and recruit a sufficient number of eligibie families for inclusion in the study is needed.
Second, some aspects of data entry and analysis will need to be contracted to someone with specialized expertise in this
area. Third, if the project were to be accepted for presentation at the annual scientific meeting of the Canadian Paediatric
Society, the costs of registration, travel and accommodation will need to be considered. Although some of the expenses of
the project might be covered at no cost to the resident through
source to cover the rest.
her hospital or university, she will need to find a funding
.,
•••••
e
11 Research requires fundinq; Even The costs of research -c
simple projecrs need sorne degree of financial suppon along As the research road map presented in chapter 1 of rhis ev
"Where do 1 ...,
-
with human resources. If you are wondering guide shows, questions of resources and funding presem
start?" or "\'V'ho can help me?" you are on che right rrack. eady and often in the COLmeof research projecr. As a junior
•••••
However, rhe firsr step is ro draw up a budget for your proj- researcher, one of your firsr steps on rhe road is to identity
ecr. Indeed, this is a requiremenr of many Research Erhics and approach a potencial supervisor-rhar is, a íaculry
Board applications. This chapter will provide a framework
for planning resources, reducing coses and securing the
CHAPTER OBJECTIVES
necessary operaring funds and human resources. Novice
researchers often underestimare rhe resources needed ro After reading this chapter, you should be able to:
complete a study, Good will and rhe contriburions of vol- • describe the wide range of formal small-grant funding
untcer collaborarors are helpful, bur rhey are not sustain- opportunities that are potentially available to suppon: the
able. Nor do rhey cover material resources. Any experienced research projects of health care trainees and residents
c1inical researcher will confirm that, as with a home renova- o discuss creative methods for enlisting human resources
tion, the successful cornpletion of a project rypically cosrs or accessing necessary infrastructure to advance a
more rhan anrlcipated. Even the srnallesr of projecrs require research project
some labour, by experts and orhers, along with access ro • list other creative and potentially unconventional
technology, space and time. Although funding can be dif- strategies for injecting funds or other financial and
hcult ro obrain, ir is nor impossible; in fact, in sorne cases "in-kind" support into the conduct or dissemination of
novice researchers find it easier ro access rhe resources rhey research
need than rheir senior colleagues do.
--------- -----------------------------,
KEY TERMS
Budget Human resources
Collaboration Sources of funding
Financial requirements Volunteers
Grants
© 2011 The Royal College of Physicians and Su.qeons of Canada '155

mernber who will guide and supporr you in the conduce of Human Resourccs deparrmenr, ro avoid any unpleasant
your research projcct (see ch. 3). Alrhough nnding an advi- surprises,
sor wirh a srrong publishing record is imporranr, so too is When rhe research is over and rhe resulrs are analyzed,
evaluaring his or her funding record. Does he or she have you aren't quire Iinished. Disserninating your srudy resulrs
funding, or work wirh a research group or laboratory rhar will require rime and energy, and can involve furrher
rourine!y secures grants? If so, are rhese grams a porenrial expense. For example, is rhis work ro be published? Where
source of funding for your projecr? For example, exisring will ir be presenred, and in which formars? Is travel required?
operaring grams mighr provide funds for your proposed Once you know what your cosrs are, you have done half
research in whole or in parro Ar a minimum, your supervi- the work of planning your projccr funding. Now yOl! need
sor should be able ro provide suirable guidance for obrain- ro be aware of how you can cut cosrs and whar funding is
ing your own funding and help you ro rroubleshoor available. These issues are the focus of rhe rernainder of rhis
problems rhar arise. In rhe end, remember rhar funding is chaprer.
nor rhe responsibiliry of rhe supervisor, alrhough securing
funding wirhour rhe he!p of a supervisor is highly unusual.
TEXTBOX 19.1: POTENTIAL HUMAN AND FINANClAL
Nexr, consider how your research question will be oper-
RESOURCE REQUIREMENTS OF RESEARCH
arionalized. As you assess your srudy design, as derailed in
chaprer 9, rhe key is ro determine whether ir is feasible. Ar Human resources
rhe rnost pracricallevel, rhis means having adequare money Chart abstractors/reviewers
and personnel ro do che job properly. Key ro this is input Data collection
from frcnrline research stafl. Even if rhey are not directly Data entry
involvcd in your project, experienced research staff can Enrolment and screening
V"; advise you abour potential barriers ro executing your study, tibrarian: literature review
o and abolir possible soluticns. If your study design is roo Statistician study design and analysis
o.. arnbiticus, ir is also likcly ro cosr roo much and ro require Studyadministl"ator
o
~
roo many people ro be achievable.
Srep 5 of [he research road map-c-developing an ourline Financia! requirements
tl.. for rhe projccr-is all abour planning. Texrbox 19.1 lists Administíative and office supplies
tlie human and other resources rhar mighr be needed for Consultants: database, statistics
even a modest research project. Careful thoughr ac rhis Enrolment and data collection
stage wiU not only prevenr rnistakes and spare you some Printing and publication
headaches and cosr overruns, but \ViII also make your Statistical software
research experience more enjoyable and more rewarding. 1.2 Surveyadministration
A cornerstone of planning is rhe crearion of a realistic bud- Travel
ger: evely grant applicarion you write will require one.
Your merhodology and research prorocol will dicrare
rnosr of your costs. Aside from the rype of study you
Managing resources to cut costs
choose-a randornized controlled trial will be more expen-
sive ro conducr rhan a survey, for example-quesrions of Human resources represenr a potentially sign;ncam cosr
cost arise from how [he data collection and analysis will be and an adminisrrative challenge for any research projecr.
done (and by whorn), the sample size necessary ro achieve However, rhere is a brighr side, for rhis is also an arca where,
adequare srudy power, who will handle rhe adrninistrarion with the help of your supervisor and a lirrle creativiry, you
and paperwork, and wherher outside consulting and spe- can substanrially cut cosrs.
cialized knowledge are needed. The grcatesr research costs Alrhough rhis mighr be your firsr experience managing a
are labour, labour, and labour. The arnount thar contracred research projecr, your supervisor should be familiar wirh
work is worrh is dicrarcd by professional srandards and the research process and the area of healrh your quesrion
locallabour agreemems. Again, ir', always best ro go ro peo- probes, He 01" she should know whom ro approach for help
pie with experrise in this area, such as your insrirurion's and what resources are available ro facilitare your work.
156 © 2011 Ihe Royal College 01 Physicians and Surqeons o' Ca aca
19 Research funding and other resources
these same fields can rap larer. Alrhough volunreers can be

Moreover, wirh a rrack record of successful funding, your
soughr from rhe generalisr training programs, a more prom-
supervisor should be able ro guide you ro rhe institutional,
ising avenue is ro seek out specíalty-specífic interese groups
governmenral and industry granrs relevanr ro your srudy.
ro focus your cal! for volunreers (e.g., an Emergency Medi-
You will also undoubredly be surrounded by orher residenrs
cine Interese Group). 111is is an attracrive oprion when
or health care rrainees wirh a wide range of backgrounds,
funding is very lirnited; an important caveat, however, is
so me of whom may have signincanr research experience.
"You ger whar you pay for."
Use rheir knowledge and experience ro your advanrage. A
If you are lucky enough ro have one in your deparrment,
senior rrainee mighr have won rhe granr you seek in a preví-
Research Associare programs4.5 are a unique resource for
ous year and be able ro provide you wirh valuable insighrs.
volunteer help.These programs are departmenral or univer-
Anorher mighr know how besr ro conducr a literature
siry-based organizarions of volunreers, typically medical or
search; anorher, wirh a masrer's degree in epidemiology,
nursing srudents, managed by experienced medical
mighr be willing ro save you the cosr of oursourcing sratisri-
researchers. Volunreers receive rraining specihc ro their par-
cal advice. Participare in journal dubs and other depare-
ricular duries and fill many screening, srudy enrolrnenr and
mental acrivities where you are likely ro develop your own
dara collecrion roles. 1hey receive rhe same benehts as rhe
expertise, And consider thar, alrhough your peers are busy
srudenr volunreers alluded ro previously, but management
people, they are also a potencial so urce of volunreer labour.
by rhe program coordinaror removes much of rhe adminis-
Resource-sharing is rhe basis of collaboration and is
trative stress of coordinaring what could be several srudents
ofren a means of reducing cosrs. Col!aboration can bring
needed experrise ro a study, obviaring rhe expense of out- for your one srudy.
sourcing. Ir can also make use of exisring capital and

research infrasrrucrure that mighr be impracrical or impos- The budget
sible ro duplicare on your budger. 1his is rhe same principal
If you have been rhinking abour cosrs rhrough rhe planning
that al!ows many small, university-builr sarellires ro ger into
and developmenr of your srudy design, rnost elemenrs of
space: they piggyback on the efforrs of larger commercial
rhe budget should already be derermined. Now ir is rime ro
outhts, who pay rhe bulk of rhe signincanr launch cosrs. In
estimare rhe specific costs associared wirh each aspecr of rhe
rhe held of healrh research, rhis could mean associaring
project. Sran wirh a lisr of everyrhing you wanr and pare ir
wirh a research nerwork doing similar work (see ch. 4) and
down ro whar is absolurely necessary. Include a conringen-
possibly aligning your research quesrion wirh rheir inter-
c)' budger for unforeseen expenses; wirh the proper plan-
esrs. Of prime considerarion in such relarionships are ques-
ning and input from experienced personnel in chis area,
rions of ownership of research, recognirion and authorship
these should be minimal. Your budger should speciíy a
(see ch. 3), expecrarions for which need ro be discussed
minimum and maximum: your actual cosrs should fall
before the research begins.
somewhere in berween. Discuss rhe budger wirh your
Even more valuable than collaborarors are volunteers,
supervisor and gechis or her opinion on whether rhe roral
the majoriry of whom are interested students who can be
arnount is feasible. Table 19.1 provides examples ofbudger
invaluable ro you: rhey generally have faidy flexible sched-
iterns that mighr be involved in a residenr research projecr,
ules and may be able ro give you their rime for a low. rare-
wirh cost esrimares in 2010 Canadian dollars.
or even for free. Alrhough they may have limired healrh care
knowledge and ofren limired research acumen, srudenrs are
always looking ro gain relevant experience, particulady if Formal funding mechanisms
they wanr ro ger into professional healrh care programs
.Why seek formal funding? 1he simple answer is rhat a lor
and/or graduare school." Similarly, undergraduare medica!
oE rhis rype of money is available. Anorher reassuring aspecr
srudenrs hoping ro eriter a particular specialry wanr ro meet
is that rhe exercise of applying for formal funding forces
residenrs and sraff ro enhance rheir chances ofbeing accepr-
you ro go rhrough a clearly laid-our process rhat, regardless
ed. However, this is a rwo-way street. Healrh researchers
of the outcorne, wíll likely make an)' subsequenr funding
have a vesred interest in providing such research experience
requests, formal or orherwise, 'easier ro prepare. However,
because ir atrracts srudenrs ro rheir profession and held of
this rype of financial support has orher benetirs as wel!. 111e
srudy. Ir also develops a pool of research experience thar
157
© 2011 The Royal College 01 Physicians and Surgeons oí Canada
• Table 19.1: Sample budget items, with associated cost estimates
Commercially available online survey program $20-$50/month
Medical record search (first 100 free) $3/chart

-----
Use of prepared and validated data collection or survey tool $250-$500
Survey methodologist
$50-$80/hr
A quantitative survey specialist will provide advice regarding survey methodology
to ensure survey validity and maximize response rates.
Statistical analyst
$50-$100/hr
To address study design and quantitative analysis; responsible for statistical
programming and data analysis.
Reference database
$150-$350
To generate a database of studies retrieved through searches, select studies for
inclusion in the study, and provide citations for a manuscript.
Duplication costs at $0.20/page.
Distribution of information, assessment and data collection tools to study $50-$500+

participants and reviewers. Distribution of studies to blinded investigators
assessing items for inclusion.
Administrative costs
Acquiring contact information and correspondence (e.g., survey implementation); $25-$40/hr

blinding investigators responsible for assessing studies for inclusion; organizing
volunteer resources.
Office costs
$100
Including occasional photocopies, printing, faxes, telephone contacts and
teleconferencing.
Database support
$30-$40/hr
An information technology specialist will generate and manage an electronic
database of data retrieved from studies selected for inclusion.
Travel and media preparation
$400-$1000
For presentation of the study findings at your specialty association's annual
conference
Poster for results dissemination $250-$500
applicarion process involves, peer review, giving you a valu- Formal applicarions also involve srepwise planning and
able opportuniry ro obtain an expert opinión on your pro- deadlines rhat will help ro ger you on track, on rime and off
posal. \'{1eak proposals are ofren rightly rurned back at this ro a good starr with your research. Finally, winning a grant
point. However disappoiming this may be, ir is better ro ofren brings presrige and credibiliry. When seeking grams,
discover (he shoncomings of your project now, rarher than Iook for resources designa red for junior invesrigarors, new
after rhe money is spem with a poor resulr ro show for ir. researchers or residenrs and orher healrh trainees.
158 © 2011 The Royal College oí Physicians and Surgeons of Canada

Many privare foundations exist ro promote researeh and

Formal funding can have disadvanrages, though.
health care investment; these are usually affiliated with hos-
Application timelines are sometimes restricrive, forcing
pitals or disease-specific soeieties. Examples include rhe
you ro work according ro someone else's schedule. AIso, if
Canadian Caneer Soeiery, rhe Heart and Srroke Founda-
you don'r find out abour the granr deadline with enough
rion, rhe Kidney Foundation of Canada and the Alzheirn-
lead-tirne you might find yourself scrambling to pull
er's Sociery of Canada. In Omario, the Physicians' Services
everything rogether at the last minure-or being lefr out
Incorporared Foundation is a unique physieian-eenrred
enrirely. (Large porrions of your project proposal can
resouree for grants. Certain specialties also have related
ofren be used ro draft a grant application, saving valuable
foundations rhat offer researeh funding, such as rhe Cana-
time when deadlines approach.) Also, no matter how
dian Foundation for Women's Healrh (rhe foundation of
wonhy your proposal , you willlikely be competing with
rhe Soeiery of Obstetrieians and Gynaecologists of Cana-
numerous other applicants. Fewer rhan 50% of grant
da). However, specialties ofren have no direct relationshi p
applications are successful. However, dcn't think of your
with related organizations, and an initial funding seareh
work on an unsuccessful proposal as wasted; rhe docu-
might be broadened ro find these funding opportunities
menrs you prepare for one application can ofren be tai-
(e.g., the Canadian Paediatrie Sociery and the Sick Kids
lored for another, and the time you invest in honing your
Foundation borh provide awards and grants for Pediatrics
ideas should enhance your next projecr's chances of
research). Unless you have a specihc organization in mind,
success.
use searehable databases of granting facilities-for there are
Examples of rhe many sources of funding are listed in
literally thousands of foundations. Excellent resourees
rhe Additional Resources secrion at the end of this chapter.
include the Communiry of Science, Imagine Canada and
Important first-line sources are federal and provincial
GrantsNet. Moreover, many universities have a person des-
agencies that fund science, social science and medical
ignated as the grant advisor whose role ir is ro idemify
research, often through an institute or foundation. These
appropriate potential sources of funding for a given project.
include the Social Sciences and Humanities Research
Council (SSHRC), rhe Natural Sciences and Research
Council of Canada (NSERC) and the Canadian Institutes Informal funding mechanisms
of Health Research (CIHR). At the provincial level are
Not all sources of funding have ro be organizations with
organizations such as Alberta Innovares Health Solutions
formalized grants and grant application proeesses. Billing
(a division of the Alberta Heritage Foundation for Medical
groups and "practice plans" are clinical practiees in whieh
Research), Ontario's Minisuy ofResearch and Innovation,
physicians develop a pool of funds for a common use. Fre-
and Quebec's Fonds de la recherche en san té.
quently this is ro promote advaneemem of their field with-
National societies and associations offer grants for
out the eonsrraints of ourside agencies. Consequenrly,
research in specific fields. The Royal College of Physicians
research eondueted rhrough rhis mechanism must have rel-
and Surgeons of Canada has a grant that rargets resident
evanee ro clinicians. An exarnple might be research rhat can
research specifically, as do specialry societies such as the
help improve clinical effieiency, for exarnple rhrough novel
College of Family Physicians of Canada (CFPC) and rhe
triaging, the use of electronie darabases or the developmem
Canadian Association of Emergency Physicians (CAEP).
Consider related specialties when your research project of clinical practice guidelines.
Indusrry is also a source of funding-one with deep
spans more rhan one field, as in our case scenario (the Pedí-
pockets but a profit-making agenda. At issue with this
atrics fellow could seek funding in both Pediatrics and
mechanism are significant coneerns related ro ownership of
Emergency Medicine). Check out rhe website of your
the study resulrs and porential contiicts of interest (see ch.
nacional association for more information. In facr, spend
18). If you pursue indusrry sponsorship, you must ensure
some time 011 the website of your universiry, medical
that your funding or gram is unrestricted. Local pharrna-
school or depanment. Universities usually have a research
ceutical companies and other biomedical manufaeturers
office dedicated ro facilirating grant and funding effons
will often provide small « $5000), unrestricted researeh
rhar can provide, as well as information on speeifie funding
grants ro junior researehers who apply. An exarnple might
sources, grant mentoring programs and experrise ro point
be a grant from an ultrasound manuíacturer ro assess how
you in the right direction.
9
Conclusion
frequendy ultrasound is used at the bedside in emergency
departments. Review your university's requirements regard- Funding research takes time, effort and-most irnportant-
ing the kind ofIegal agreements that musr be in place before ly-planning. However, rhe cost of research, whether
indusrry or other privare funding may be accepred. financial or human, should not be seen as a major impedí-
Finally, consider the graduare-srudent approach of beg- ment. There are many sources of grants and a deep pool of
ging and borrowing. The contacrs you make in orher labs, cxpertise and labour ro ger the job done correcrly, on time
with other students and with other researchers open doors. and on budget. •
Equipmenr is ofren not in use, space is sometimes unoccu-
pied and people are frequenrly willing ro do an hour or two
of work simply for the experience.
CASE POSTSCRIPT
Proper study design and the development of a solid research protocol would have averted most of the problems the
Pediatrics fellow is now facing. However, if enough volunteers can be found to identify and recruit eligible participants for
entry into the study, the project might still be feasible. Determining whether either the pediatric or emergency departments
have Research Associate programs could solve the human resources and organizational challenges this represents. Data
entry can be tackled in a similar manner, although analysis often requires specialized expertise. A collaborator may be able to
provide this without additional cost, o/ the resident could spend some time learning the material herself. Finally, funding can
be sought from appropriate organizations or the hospitals research foundation to cover the costs of study execution and the
dissemination of results.
REFERENCES
1. Hebert RS,Levine RB,Smith CG, Wright SM. A systematic review of resident researchcurricula. Acad Med. 2003;78(1 ):61-8
2. Barletta JF.Conducting a successful residency research project. Am J Pharm Educ. 2008;72(4):92.
3. Sparano DM, Shofer FS,Hollander JE.Participation in the academic associate program: effect on medical school admission rateo
Acad Emerg Med. 2004; 11(6):695-8
4. Hollander JE,Valentine SM, Brogan GX Jr.Academic associate program: integrating clinical emergency medicine researchwith
undergraduate education. Acad Emerg Med. 1997;4(3):225-30
5. Hollander JE,Singer AJ. An innovative strategy for conducting clinical research: the academic associate programo Acad Emerg
Med. 2002;9(2): 134-7.
National Organizations
Canadian Coalition for Global Health Research

www.ccghr.ca/def auIt.cfm ?control=fu nding&la ng=e&su bnav-roadma p
Canadian Health ServicesResearchFoundation

W\NW.chsrf.ca/funding_opportun ities/index_e.php
Canadian Institutes of Health Research(CIHR)

W\NWcihr-irsc.gc.ca
• CIHR has a master list of funding opportunities that is updated on a regular basis.
160 © 2011 The Royal College 01 Physicians and SLfgeo - o' ¿-ss ~
~
Knowledge Translation (KT) Canada

http://ktclearing h o use. ca/k tca nada/ed ucati o n/fu n d ing
Natural Sciences and Research Council of Canada

www.nserc-crsng.gc.ca
• NSERC provides a list of new funding opportunities and general application information.

http://rcpsc.medical.org/awards/wightman_e.php
• K.J.R. Wightman Award for Scholarship in Ethics
Social Sciences-and Humanities Research Council (SHHRC)

www.sshrc.ca/site/apply-demande/program_index-index_programmes-eng.aspx
• SSHRC provides a general page on programs available for researchers to apply to.
Provincial
Alberta Innovates Health Solutions
www.ahfmr.ab.ca/grants.php
Nova Scotia Health Research Foundation

www.nshrf.ca
Ontario Ministry of Research and Innovation

www.mri.gov.on.ca/eng lish/prog ra msldefa u It.asp
Physicians' Services Incorporated Foundation

www.psifou ndation .org/
Quebec Fonds de la Recherche en Santé

www.frsq.gouv.qc.ca/en/i ndex.shtml
Medical schools and universities

The Research Institute of the McGi11 University Health Centre
htt p :l/m u h C. ca/resea rch/d ash boa rd
University of Alberta Research Services Office

www.rso.ualberta.ca
University of British Columbia: Office of Research Services

www.ors.ubc.ca/index.htm
University of Toronto: Research and Innovation Office

www.research.utoronto.ca/for-researchers-administrators/funding-sources
Disease and specialty-specific funding

Alzheimer Society of Canada
wvvw. a Izhei me r.ca/en 9 Iish/resea rch/resp rog -aw a rds09 .htm

-~------- ~-~
Canadian Association of Emergency Physicians

www.caep.ca/template.asp?id=16A0191 B258246CDB2FF9FD2FD4FEBCC
Canadian Heart and Stroke Foundation

www.heartandstroke.com/site/c.ikIQLcMWJtE/b.3479073/k.99CAlFunded_Research.htm
College of Family Physicians of Canada

www.cfpc.ca/Family_Medicine_Resident_Awards/
Kidney Foundation of Canada

www.kidney.ca/Page.aspx?pid=362
Obstetrics and Gynaecology-Canadian Foundation for vvornen's Health

http://cfwh .org/i ndex. php? paqeeresearch- fund ing&h I=en_ CA
Pediatrics-Canadian Pediatric Society

WWw.cps.ca/english/Residents/Grants.htm
Sick Kids Foundation

www.sickkidsfoundation.com/gra nts/
Foundations
Chronicle of Philanthropy
http://philanthropy. co m/ 9 rants
Cornmunity of Science
http://fundingopps.cos.com
• The Community of Science site provides a wealth of resources for the world's researchers including links to funding
opportunities.
Council on Foundations
www.cof.org
Foundation Center
http://fou ndationcenter. o rg
GrantsNet
http://sciencecareers.sciencemag.org/funding
• This site allows you to search through thousands of funding opportunities in nearly all scientific and medical fields.
Imagine Canada
www.imaginecanada.ca
• A searchable database of thousands of Canadian foundations and grants; requires a subscription to access, which many
universities hold.
SUMMARY CHECKUST
o List all of the resources needed for your research project. Review them with your supervisor and team and revise.
O (reate a budget that describes the amounts and the flow of resources for the project.
O List all of the relevant sources of funding for the project. Prioritize them.
O Apply for funding.
O Ensure all funds are managed professionally and accounted foro
O (reate a report summarizing all your resources used for the project.
O Return unused funds.

20
Sampling, recruiting and retaining
study participants
I<aberi Dasgupta, MD, MSc, FRCPC
ILLUSTRATIVE CASE
A second-year resident in Internal Medicine is interested in learning more about factors that influence adherence to
antibiotic therapy and their impact on the likelihood of recurrent infection and antibiotic resistance. He has framed a
research question as follows: "Among patients from the inpatient clinical teaching unit discharged home on oral antibiotic
therapy, does a follow-up visit with the treating resident increase the likelihood of adherence to antibiotic therapy?" He
expects that the results of his study might alter the discharge planning approach at his institution and beyond.
• A clinical study that relies on

primary data collection always presents a challenge. There
tients who are treated with intravenous
of a variery of infecrions (e.g., pneumonia,
antibiorics for any
urosepsis) and
are many threats to both the inrernal and externa] validiry of then discharged home wirh a prescription for oral anribiot-
a srudy rhat can limit rhe scientiíic validiry of the íindings ics. The source population is the popularion from which
and thus the conclusions rhat may be drawn from rhern. rhe researcher acrually recruits rhe srudy participanrs: in
1he process of clinical research is rime-consuming and can the case example, inpatients at rhe resident's hospital who
be frustrating. Nonerheless, a well-designed primary study are treared for che infecrions of interese.
involving the collection of clinical data can provide valuable Next, the researcher needs to esrablish inclusion criteria
information, as well as rnuch satisfacrion ro rhe researcher. ro more sharply define rhe study sample, keeping borh sci-
Moreover, clinical research in its many forms-including
clinical rrials, prospective cohort studies, case-control stud-
CHAPTER OBJECTIVES
ies and cross-secrional srudies-are rhe only means of
addressing so me kinds of research quesrions. After reading this chapter, you should be able to:
• explain the difference between a target population and
a study sample and the relevance of this distinction o
Defining the study population
answering a research question
Clinical research requires rhe participation of a sample of • identify potential sources of selection bias
real people. Consrrucring that sample involves, in rhe first • describe the importance of an adequate sample and the
place, defining rhe target population ro whom the factors that enter into a sample size calculation
researcher wishes ro generalize his or her results. In our • apply strategies to enhance recruitment and limit losses
illusrrative case, that popularion mighr be hospital inpa- to follow-up
KEY TERMS
Exclusion criteria
External validity
Internal validity
Losses to follow-up
Study sample
Target population
'-,
Generalizability Selection bias
Inclusion criteria Source population
© 2011 The Royal College of Physicians ancl Surgeons of Canada 163

entific and logis tic considerations in mind. Hu11ey and col- Screening for study participants
Ieagues' describe rhese as rhe speciíic demographic, clinical, Reliable and comprehensive screening procedures for sal
geographic and temporal characteristics that are borh rele- ple selecrion-i.e., mechanisms for identifYing eligi~
vanr ro the research question and logistically feasible ro parients-are crirically irnporrant ro any study thar seeks
include. In rhe case exarnple, the resident would need ro recruit parricipants for the purpose of data collection.
specify the period during which rhe participanrs will be screening procedures are inconsistent or prone to error, ti
recruited, such as a single rnonrh or season. He will need ro conduce and cornpletion of the study can be seriously con
decide whether he is interested in patients wirh any condi- promised, if not doomed ro failure. lf patients wirh a speci
tion requiring antibioric treatrnenr, or only in those with ic characterisric are preferentially included or excluded, tl
one or more speciíied conditions. He will need criteria for resultant selection bias can jeopardize the study's extern:
establishing rhe presence of infection (e.g., culture vs clini- validiry and hence irs general izabili ty. Ideal approaches ger
cal diagnosis). Orher patient characteristics will also need erally involve the elimination ofhuman facrors, such as rel)
to be specified. For example, will a11age groups be eligible? ing on members of the healrh care ream ro alert rhe researc
What abour patients who are riot capable of giving group ro potentially e1igible patients who mighr be recruit
informed consent beca use of derncntia or mental i11ness? ed for rhe study. Ideally, procedures that incorporate sorn.
In contrasr, exclusion criteria are intended ro rule out kind of electronic health record are best suired ro rhe sys
individuals who, although rhey may fulfi11 inclusion crite- tematic idenrificarion of porenrially e1igible srudy subjects.
ria, possess characteristics thar are unsui table on derno- In our case example, a daily review of al! admissions anc
graphic, clinical or geographic grounds. Thus, parients who diagnoses would help rhe resident identify patients whe
are likely ro be lost ro fo11ow-up (see later discussion), or mighr be approached for recruirment as they enrer the pre-
cannot provicle good-quality informarion, or are at risk for discharge phase of rheir hospital stay. In settings that lack
adverse events if they follow srudy procedures, might be an e1ectronic health record, ir is worrhwhile ro design a pro-
excluded. For exarnple, the resident in our example might cess map thar describes rhe movement of relevant patienrs
opt ro cxclude institutionalized patients from the study on rhrough the healrh care systern, wirh an eye ro identifYing
rhe grounds that, alrhough they may have an infection opportune mornents ro initiate recruitment, alongwith key
requiring inpaticnt treatrnent (e.g. aspiration pneumonia), personnel ro do so. Appropriare promprs can take the form
rhey may for logistical reasons be unlikely ro adhere ro of strategically placed reminder posters or colourful mate-
u_
4-'
U
study procedures and to attend fo11ow-up appointments.
Narrowly defining a study population improves the
rial appended ro parient charrs ro encourage healrh care
providers ro conract the research team, alerting rhern to the
"'C
::J interna1 va1idity of a research project: that is, the less potentially eligible patients and al!owing thern to explore
diverse rhe sarnple, the more readily we can atrribute an inclusion and exclusion crireria wirh the relevant parties.
c: observcd effect ro the variables and population under study.
o For example, the resident in our example could choose ro
u focus on women over the age of 65 years with urosepsis.
There is a good chance, if the study is free from other sourc-
Identifying the participants
Even with well-thought-our study population definirions

es of bias and demonsrrares a strong effect of resident fol- and inclusion/exclusion criteria, recruitrnent can still be
low-up on adherence, rhat this effect will hold true for prone ro selection bias. If this occurs, your srudy sample
orher women wirh these infections in that particular ser- mighr not truly represent your rarget popularion. Suppose,
ting. However, such a narrow definition wil! reduce the for exarnple, that you decide ro recruit patients from the
externa1 validity, or genera1izability, of the findings ro clinical reaching unir in your hospital, which serves French-
other groups (e.g., men, children, younger women) or to and English-speaking populations. lf you are nor fully Hu-
parienrs wirh orher iníecrions (e.g., pneumonia). ent in French, you may have difficulry recruiting
The care taken in defining the research question, rarget French-speaking patients and, as a resulr, will end up wirh a
popularion, source popularion and eligibiliry criteria wil! preponderance of English-speaking patients in your srudy
help ro optimize rhe generalizability of a study's findings. sample. This is of concern only if rhere is a possibility that
The goal is ro ensure rhat rhe sample of parricipanrs that is the impact of the intervention (i.e., fol!ow-up with a rrear-
ultimarely selected reflects rhe popularion of interest, to ing team member) will differ berween French- and Eng-
whom the study's results are intended ro be applied. lish-spealcing patients. If this is rhe case, you should involve

20 52""';1 '~g 'éCrui ing and retaining study participants
a colleague with strong French-language skills ro assist wirh inform by a member of the health care ream that he or
recruitment. she will be approached to parricipate in a research srudy and
In a similar vein, one su'ategy thar can help reduce selec- rhat a verbal approval be obrained before they are
tion bias is ro recruit patienrs consecutively in order to approa hedo
mimimize facrors rhat might inBuence rhe rnake-up of rhe The difhculry inherent in recruiting participanrs through
selected group, For example, patients discharged on eve- physicians and clinic s[aff is highlighted by Butt and col-
nings and weekends may differ from other patients in that leagues;' who demonstrate how, in a comrnuniry-based rrial
rhey require a family mernber ro escorr rhern home outside targeting posrrnenopausal wornen, fewer than 4% of par-
of regular work hours, rhus reBecting a more fragile health ricipanrs were recruired through physician recruiters and
status, In our case example, the procedural and logis tic rhe rnost successful method of recruitment was newspaper
implications of consecutive recruirrnenr would include vis- advertisernent. However, this approach would not be
iting the hospital ward daily ro determine who is being dis- appropriate for all research scenarios, including our illustra-
charged on antibiotics and then approaching rhese patienrs tive case. Thus, in a srudy thar requires recruitrnent from a
ro invite them ro participare in the study parient care serring, strong working relationships and sim-
You will need ro keep a log of acceprances and rejections ple reminders are crirical.
so that you can ultimately determine wherher rhe patients Other strategies that can enhance recruirrnent and dis-
who were enrolled in your srudy differ subsrantively from courage loss ro follow-up include payments ro parricipants
those who were not. Significant differences wil! rhreaten the ro cover cosrs such as rransportation and parking, and lor-
external validiry or generalizabiliry of your results. General- teries with small prizes (e.g., movie rickets, gifr certifica tes)
ly speaking, people who agree to participare in scientific for staff who facilitate recruitrnenr, provided rhat these
studies (i.e., volunreers) differ in so me respects from the strategies are perrnitted by rhe local Rescarch Ethics Board.
general population. 1he goal is to lirnir porential differences In our exarnple, however, givcn that rhe residenr will be
as much as possible and to be aware of thern when they do assessing whether follow-up with a trearing tearn mernber
exist so that your findings can be inrerpreted appropriately. enhances adherence ro antibiotic rherapy, ir may be prob-
lemaric ro cover transporration costs, given that the visir is
part of the intervention: reimbursernenr for parking do es
Recruitment not reflecr real-world practice and could chus introduce
To increase rhe likelihood thar potencial candidates wil! bias.
agree ro participare, make your srudy procedures as simple
and consistent as possible. For example, parients might be Lossesto follow-up: another potential
more likely to agree ro a relephone follow-up interview rhan source of selection bias
ro an in-person inrerview, Egually important, however, are
your inrerpersonal skills. A kind, warm and courteous rnan- TI1e shorrer the period berween recruitrncnr and follow-up,
ner can go a long way toward encouraging interest in your the easier ir wil! be ro avoid losses ro íollow-up. \V'hen we
srudy and a wil!ingness ro participare. Alrhough you might "lose" patients, we cannot be confident abour [he associa-
not want ro explicitly state your underlying hypothesis, tions that we identify among [he remaining parienrs, since
parienrs may be more receptive to a study designed ro those lose ro follow-up migh[ have sysremarically difíerenr
improve patienr care in general, although you must take care ourcornes than parricipants who remain in [he study until
ro indicare on [he consent form rhar participaring in rhe the cnd. For cxarnple, in a placebo-controlled tri al asscssing
study will have no impact on the qualiry of care received. the impact of a drug ro treat heart failure, medication side-
As imporrant as it is ro rnainrain cordial relationships effects migh[ lead ro more drop-ours from rhe trearrnerit
wirh potential participants, your inreractions with rhe arm than rhe placebo armo 1his can Jead to rhe faulry con-
health care ream-including nurses, clerical sraff, patient clusion that [he study drug was as well tolerated as placebo,
assisrants, residenrs, medical srudents and attending physi- when in facr the differen[ialloss ro follow-up introduces a
cians-are also crucial. Good relarionships can determine bias rhar makes rhe experimental drug's profile appear more
whether rhese individuals become key allies in study recruit- favourable than ir really is,
rnent or impede [he process. Moreover, many Research In our case example, the resident could plan ro assess
Erhics Boards reguire rhar a porenrial srudy parricipant be antibioric adherence through a telephone-based interview,
© 2011 The Royal College of Physicians and Surgeons oí Canada

165
The Research Guide: A primer ter residents, other health care trainees, and practitioners
Participanrs contacred wirhin a few days of the estirnared and srudy parricipanrs will be dictared not only by consid
date of antibiotic rherapy cornpletion will be more likely ro erations of what the ideal study design would be, but als.
remember che resident, to answer his questions, and ro pro- by what kind of study is feasible to conduct while stil
vide accurate information than panicipants contacted, say, answering your question-or, if necessar)', an alternativ.
three monrhs later. The resident will therefore have ro rrack question that )'ou also consider ro be irnportanr. Specifical
participanrs and rheir timelines rneticulously, to ensure thar ly, logistical or feasibiliry issues such as your operating bud
informarion is obtained in a timely fashion. Spreadsheers get, the number of eligible participams available and the
and built-in time alerts may be useful in this regard. In a time frame are critically imporrant. Given thar rhe timé
rigorous study, any loss to follow-up should be the result of frame for trainee projects is generally short and rhe budger
a genuine inabiliry on the pan of the patient to comply limited (or non-existenr), pracricaliry is the order of the da)'.
with or tolerare the intervention (although this is probably Let us suppose that your discussions with a statistician
abona tide outcorne rather than simply loss to follow-up) suggest that you will require 40 patienrs in each group to
as opposed to poorly planned or execured srudy procedures. answer )'our research question, or 80 parienrs in total. Your
In studies with longer fol!ow-up periods, it may be wise institution has sorne funds for residenr research projecrs,
to maintain contact with participanrs regulady by phone or and so )'ou have approxirnately $2000 (O use. You assess rhe
rhrough newslerters. As discussed wírh reference to recruit- situatiori and realize that a clinical trial will be possible only
ment, sorne insrirutions perrnit researchers to encourage if you are able to dedicate at least rwo years to rhe project,
ongoing participarion through incentives such as lorreries taking into account the time required for erhics approval,
for small prizes (e.g., gift certificares for books or movies) recruitment and data collection, data entry, analysis, and
fol' those who complete follow-up procedures. Another manuscript writing. A more like!y scenario is that you will
useíul strategy can be to obtain contact inforrnation nor have been able ro piece togerher a three-rnonth period dur-
only for the participant but also for two or rhree friends or ing which )'ou will collecr rhe data.
Iamily mernbers, to increase the likelihood of being able to In this situarion, a cohort srudy may be possible, albeit
Iocate participants who mcve during the study so thar rhe challenging. You estímate thar approximately 20 parienrs
applicable study outcornes might be determined. are discharged on amibiotic therapy on rhe medical teach-
ing ward at your base hospital each rnonth. If they are all
How many subjects will you need? accepted to be in your srudy, you would still be shorr at least
20 patients. Your options are to indude another hospital,
Alter rhe careful work of idemifying the correct source pop- which will involve rravel or finding a collaboraror at rhar
ulation for your study, the ncxt step is to calculare what rhe hospital, or to indude anorher type ofhospital ward. In an)'
ideal sample size would be (see ch. 25). At this point, con- case, )'ou will need to balance logistical and methodological
sultation wirh a sratistician will be useful. You will need to factors as you come up wirh a plan rhat is both practical and
be well prepared for this meeting, as rhe statistician will has the capaciry to answer your research question.
require sorne vely specific inforrnation to be of help. In our
case exarnple, he or she would need to know, based on rhe
Conclusion
resident's lirerature review, an estimare of the minimally
irnporranr difference in antibiotic compliance rates artribut- To sorne degree, in conducting a research project as a train-
able to thc inrervention, rhe acceptable error around rhe ee, you are fulfilling rwo roles: thar of a researcher, and that
estimare, and rhe number of addirional variables the resi- of a research coordinator. Researchers idemify rhe relevant
dent would need to examine as well as the desired study research questions, design the studv, obtain rhe necessary
power thar would establish the probabiliry of a Type JI error funding, analyze the results, write che manuscripts and ,
(failing ro dcrnonstrate a difference when one exisrs) as well present the findings. Research coordinarors actually exe-
as rhe alpha level, which is an estimare of the probabiliry of a cute the study (albeit under the researcher's supervision)
Type J error (dernonstrating a diífercnce by chance when wirh respect to recruitrnent and study prornorion, data col-
rhere is no significant diíference). lection and data emr)'. The opportuniry to fulfill borh roles
The resulrs of your sample size calculation miglu send provides key insights that can rnake )'ou a berter researcher
you back ro thc drawing board. Your actual srudy design in rhe long rerrn and will certainly increase your awareness
166 © 2011 The Royal Col!ege of Physicians and Surgeons oí Canaoa

20 Sa'T:J - ~, ecruiting and retaining study participants
of rhe cornplexities of study execution, particularly with insiahts when, as a health care professional, you evaluare
respecr ro rhe identification, recruitrnent and rerention of me eviden e base relevant ro rhe care of patients. •
participants. 111is experience will provide you with valuable
CASE POSTSCRIPT
The resident combines forces with some of his colleagues and, with guidance from a willing and capable supervisor, embarks
on a randomized controlled trial to address his research question. The inclusion criteria are admission for community-acquired
pneurnonia for individuals living independently who are 18 years of age or older. Potential participants who are dependent on
alcohol or illicit drugs are excluded. Outcomes of follow-up at the teaching clinic within 2 weeks after hospital discharge are
compared with those of "usual care" and follow-up with the patient's own physician. The research team assesses adherence
through telephone interviews and by means of pharmacy dispensing records (the participants having consented to the
research team contacting their pharmacist). The research team discovers that roughly 75% of patients report completing their
antibiotic regimen as prescribed and that there are no important differences between groups.
REFERENCES
1. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Oesigning c/inica/ research. 3rd ed. Philadelphia: Wolters Kluwer
Health/Lippincott Williams & Wilkins; 2007.
2. Butt DA, Lock M, Harvey BJ. Effective and cost-effective clinical trial recruitment strategies for postmenopausal women in a
cornmunity-based, primary care setting. Contemp Clin Tria/s. 2010;31 (5):447-56.
Hulley SB. Oesigning c1inica/ research. 3rd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wi:kins; 2007. Chapter 3,
Choosing the study subjects: specification, sampling, and recruitment; p. 27-36.
Kleinbaum DG, Sullivan KM, Barker ND. A pocket guide to epidemi%gy.

8, Were subjects chosen badly? Selection bias. p. 127-38.
New York: Springer Science-Business Media; 2007. Chapter _.
• These two references provide helpful guidance on the selection of study samples.
SUMMARY CHECKLlST
o Determine your study's target population.

O Determine the source population.
O Establish the inclusion and exclusion criteria for your study sarnple.
O Draw up a budget for the study.
O Establish your recruitrnent targets.
O Determine your time frame for recruitment.
O Frame strategies to use to maximize recruitment.
O Frarne strategies to use to minimize loss to follow-up.

.--
+J
U
""C
:l
e
O
U
168 © 2011 The Royal College of Physicians and Surgeons 01 Canada

21
Pilot studies
Bart J. Harvey, MO, PhD, f\I1Ed, FACPM, FRCPC
ILLUSTRATIVE CASE
While preparing the protocol for a full-scale randomized clinical trial to assess the effectiveness of gabapentin for the
treatment of menopausal hot flashes, a Family Medicine physician-researcher wonders what efforts would be required
to recruit the necessary number of consenting participants When she raises this question at a departmental research
meeting, one of her colleagues suggests she consider conducting a pilot study so that she might gain some experience and
insight regarding this and other aspects of the proposed study. She thinks this is a great idea and asks her colleague if he
would assist her.
• Pilot studies are "pre-studies" rarive research because rhe rheorerical scheme of a qualita-
designed and carried our in prepararion for a more expen- rive srudy cominues ro be developed rhroughour irs conducr
sive full srudy; rhey are a separare and exrernal' "dress and analysis (see also ch. 16). Furrher, Warson and col-
rehearsal,"? Pilor srudies are imended ro provide researchers leagues" poim out rhar a qualirarive srudy conducred ro
wirh an opporruniry ro test (tri al run, or try out) one or inform the developmem of a quesrionnaire or any orher
more aspecrs of the amicipared full study's proposed proro- aspecr of a furure quanrirarive srudy should nor be described
col. 1hey allow invesrigarors ro more economically idenrify as a pilor study, because ir "clearly generare[s] findings thar
any potenrial barriers and any areas requiring revision in have value in rheir own righr." With these considerations in
rhe proposed prorocol thar, if lefr uncorrecred, could mind, rhe remainder of this chaprer addresses the use of
impede rhe successful cornpletion of the anricipared full pilor studies in quanrirarive research.
srudy. In fact, findings from a pilor srudy somerimes dern-
onstrate rhar rhe planned full srudy is, as proposed, is not
CHAPTER OBJECTIVES
feasible (as a resulr, pilor srudies are somerimes referred ro
as feasibility
Simon'
studies).
highlighrs rhat pilor srudies are parricularly use-
• discuss the range of purposes for conducting pilot
_.
fui ro assess rhose aspecrs of a proposed full srudy that are studies and identify when a pilot study would (and
"novel, untested, cornplex, or innovarive." In this lighr, would not) be indicated;
Hinds and Catcuso" poim out that pilor srudies are not • discuss how pilot studies are designed and conducted;
always necessary, especially when rhe researcher already and
possesses sufficienr experience using rhe proposed merhods • discuss issues related to the planning and conduct of
in a comparable popularion and an analogous serring. pilot studies, such as ethics review, obtaining funding
Morse-" argues rhat pilor studies are nor applicable in quali- and reporting pilot study findings.
,-------- ----------- _.- .. __ --_._-------_._-_

.
KEY TERMS
Acceptability Pre-studies Safety
Barriers Preliminaryestimate Scientific design
Feasibility studies Processes Suitability
Operation Resources Trial run
© 2011 The Royal College of Physicians ancl Surgeons 01 Canada 169

------------- -
A pilor srudy is nor carried out ro answer rhe research

The purposes of pilot studies
question morivaring rhe ulrimare full srudy but, insread, is Pilor srudies serve a wide range of purposes, which can be
conducted to address scientific or operational issues and ro categorized into at least six main areas:
idenrify the need for any revisions in rhe study procedures
and prorocol. Accordingly, a pilor srudy is disrincrly differ- l. To test and assess one or more of rhe processes rhat
ent from, and this term should not be used ro describe, a will be required ro successfully complete the proposed
small srudy that simply has:1[1 inadequate sample size.2,B A main study.
pilor study yields insighrs into rhe scientific design and 2. To evaluate rhe safety, acceptability and suitability
operation of rhe anticipated full study, but nor into out- of any of the aspects of the proposed main study,
come measures relevant to the research question of rhe 3. To test and assess the proposed and potencial out-
anricipated full study, because its findings are almost invari- come measures.
ably garhered from an underpowered number of pilor par- 4. To estimare the resources (e.g., fUnding, stafi, time)
ricipants.l=? needed ro successfUlly complete the proposed main study.
Various authors":'? have described a wide range of pur- 5. To provide evidence of the feasibility and worth of
poses thar pilot studies can serve. However, before we dis- the proposed ful! study.
cuss these, ir is important to note several study designs rhat 6. To provide data (e.g., a preliminary estimate the out-
can be confused wirh pilor srudies. These are "exploratory" come rneasure) ro assist in planning of the proposed
srudies,3.11,20"proof of coricept" srudies21,22 "adaptive tri- main study (e.g., ro calculate the needed sample size).
als"23,24"interna] pilots"25,2Gand "process evaluarions."27-29
Unlike pilor srudies, these rypes of studies are intended to Each of these purposes will be addressed in turn in the rol-
identiíy or measure an effect, and not ro trial a proposed lowing sections.
protocol for an anricipated full study. Explorarory studies
are typically used ro generare hypotheses that may poten- Testing and assessing the proposed study
tially be pursued in fucure research.3.11,20Proof-of-concept processes and procedures
studies are conducted ro provide a preliminary assessment Perhaps the rnost common purpose of a pilot study is ro ,
of a proposed incervencion to estimare the magnitude of its provide an opportuniry ro test and assess oue or more of the
.-
+J
potencial effect and determine whether further, more rigor-
ous, srudy is warranted.":" Adaptive and internal pilot
processes
such as:
being considered for rhe anticipated full srudy,
U studies are planned assessmenrs made in the midst of a ful!

~ srudy ro inform possible mid-srudy adjustrnents, usually of • identirying and recruiting a sufficient number and
"'C rhe sample size needed ro achieve a sought-after sratisrical diversiry of eligible study participants (as well as
c: power.23-2GProcess evaluations, which often employ rneth- applying exclusion criteria);
o ods similar ro those used in pilot srudies, are carried our in • if applicable, acquiring, preparing and administering
u conjunction
supplemencary
with, and not prior ro, a full study ro provide
information and insights ro inform rhe
the drug or other intervention
placebo or other cornpararor)
(and, if applicable, the
ro be studied;
interpretarion of rhe srudy's results, parricularly those • assessing the degree ro which participanrs receive (and
porentially influenced by the delivery and receipt of the accept) any applicable rreatrnent or other incervencion
intervention under srudy.27-29Of course, as Power and col- (i.e., adherence):
leagues29 point out, even full studies employing a process • assessing the degree ro which parricipancs receive any
evaluation can benefit from a planning pilot/feasibiliry competing treatrnent or orher intervention (i.e., co-
study. It should also be nored that a pilor study can be nest- intervention and contamination);
ea witliin a full srudy ro provide, for exarnple, insights • enabling scudy personnel, including co-invesrigators,
regarding other potencial fucure full studies. (An example is ro gain familiariry in working wirh one another and
the work by Klymko and colleagues.") applying the procedures ourlined in rhe proposed
study protocol:
• if applicable, identifying, recruiting and moniroring
additional scudy sites;
170 © 2011 The Royal College 01 Physicians and Surgeons oí Ca ada

27 Pifot studies
• obraining, using, maimaining and swring any These rhree elernenrs are discussed in rhe subsecrions below.
instruments and orher equipment needed, particularly
rhose necessary ro rnake srudy measurernents; Identifying, recruiting, ga.lJ1.1J1gconsent f1'0111, and
• thoosing between rwo or more possible data collecrion reta.ining srudy p ar ti cip ants. As Ross and colleagues"
merhods (e.g., interview vs self-adminisrered survey); highlighr, recruiring a sufhcient number of srudy parrici-
• assessing rhe Ieasibiliry, accuracy and cornplereness of panrs presems many chaI!enges, and recruirmenr problerns
.data collection and emry (arid, if applicable, che abiliry are a Trequem reason for underpowered and failed srudies,
[Q march and combine data Erom differenr sources); and The identificarion and recruitrnent of participanrs into a
• analyzing rhe data, including rhe developrnenr and srudy should rherefore always be carefully planned and
assessmenr of the pilor srudy results rabies. pilored. Even rhough the final sarnple size required for rhe
anticipated EuI!study mighr nor be known during rhe pilor
These assessrnenrs should be conducred explicidy and study, ir is importanr ro determine wherher a suirably
sysrernarically, such rhat each question ro be addressed by diverse group of eligible participams can be iden rified and
the pilor study is clearly identified and arriculated in the recruited rhrough available venues. A pilor srudy also pro-
pilor study plan. l11aba;le and colleagues" recommend rhat vides ah opportuniry ro test the procedures ro be used ro
researchers speciíy "success criteria" before launching a pilor approach porenrial particípanrs and obrain rheir consent ro
srudy. However, even wi rh a priori identification oE success enrer the srudy. Experience gained rhrough a pilor study
criteria, pilot studies should be considered ro be primarily should better enable researchers ro estimare rhe arnount of
descriptive; accordingly, invesrigawrs should be alerr for rime needed and rhe abiliry [Q recruir an adequare number
and take every opporruniry ro observe, docurnent and of eligible parricipants, The pilor srudy also provides an
assess, in as rnuch detall as possible, each srep taken and opportuniry for invesrigarors ro assess and receive feedback
process used during the pilor study, about rhe proposed consent form and any informarional
Alrhough many oE rhe pilor study assessrnenrs will resulr marerials given ro porenrial participanrs abour rhe srudy.
in quantitarive measures-such as recruirrnent rates, per- Pilor studies can also be used ro estimare rhe proportion of
cemage of ineligible parricipants and average nurnber of recruired parricipants who will fail comply with any of
n
[Q
questionnaire irerns not cornpleted by respondents-e-inves- the study requiremenrs, including rhe proportion of rhose
rigarors should also consider garhering q ualirative feedback, recruired who will wirhdraw befo re rhe end of rhe srudy as O
:l
insighrs and advice Erom study participanrs,
invesrigarors ro gain as complete an undersranding
sraff and co-
as pos-
well as rheir reasons for doing so.
However, because pilot srudies provide only estimares, e,
sible oE rhe proposed srudy's operation and funcrioning. For recruitrnent rates, Iike all orher aspecrs of a study, should be e
n
example, if asked, people who decline ro participare in rhe
srudy mighr explain that the excessively long enrolment
regulady rnonirored rhrough the course of rhe fuI! srudy
(see also ch. 23). The imporrance of ongoing srudy moni-
_.
.r+
quesrionnaire or rhe frequency of follow-up visits dererred roring is illustrared by the experience of Burr and col- ~
rhern Er0111 paniciparing. Similarly, srafF mighr suggesr thar leagues," who pilored a proposed recruirmenr srraregy and lO
preparing Iollow-up visir rnarerials in advance would found that rhe required number of parricipants could be
improve efficiency and reduce rhe rime demands on each idemified and recruired wirhin the rime frame of rhe pro-
partlc¡ pan t. posed fuI! study. However, eady in che course of rheir full
As Lancasrer and colleagues' suggesr, some srudy pro- srudy ir became clear that the expected recruirrnenr rate was
cesses are of sufficienr importance ro warrant particular not being achieved. As a resulr, supplernenrary recruitrnent
arrention in a pilor srudy. These include: rnerhods, which required the acquisirion of addirional
funding, had ro be idenrified and implemented, which ulti-
• idenrif)ling, recruiring, obraining consem frorn, and marely enabled rhe srudy ro be successfully complered with
reraining srudy parricipants; rhe required number oE parricipams.
• the use of data collection insrrurnenrs, including Specific cypes of studies, such as surveys, medical record
quesrionnaires; and reviews and adrninistrative data srudies, pose unique
• if applicable, the randornization and allocarion of "recruirrnent" challenges that should be considered and
participanrs inro srudy groups. piloted before rhe proposed main study is implemented.
© 2011 The Royal College 01 Physicians and Surgeons oí Canada 171

Although these srudy designs are addressed in chapters 11, unamicipared answers. However, Lancaster and colleagues'
12 and 13, respectively, several aspects are highlighted here. stress thar pilot studies, because of their relatively small
Surveys, particularly rhose administered "rernotely" .sample 'sizes, are generally not an appropriate means ro
rhrough rhe postal sysrern or the Internet, require rhe iden- assess the validiry and reliabiliry of study instruments (see
tification of an appropriare lisr of eligible recipients (i.e., a also ch. 10).
"sampling frame"). This may require researchers ro gain
permission from one or more organizations that possess Testing randomization, blinding and concealed alloca-
such a list (e.g., membership or enrolment lists). In addi- . tion procedures for a study. As discussed in chapter 9,
tion, a suitably inviting introducrory communication and two critical components of randomized trials are rhe appro-
survey insrrumenr is usually necessary so rhat an acceptable priare randomization and concealed allocation of partici-
response rate can be achieved. A pilot study provides an panrs into study groups. Much of the srrength of
imporranr opporruniry ro test these and orher components randomized trials is derived from rhe cornparabiliry of rhe
of rhe proposed ful! survey. In addition, pilot testing pro- groups being srudied, but this is highly dependenr on an
-' vides an initial estimare of rhe fuI! survey's amicipated appropriare randomization process being carried out to
response rate, assign participants ro study groupS.34 However, the uriliry
Like surveys, medical record reviews and adrninisrrative of randomization is highly dependenr on allocation con-
data srudies often require researchers ro obtain permission cealrnenr procedures rhat keep investigarors, healrh care
ro gain access ro the necessary data records. In this regard, providers and participanrs unaware of upcoming assign-
pilot studies provide an imporranr opportuniry ro explore ments so that rhe group that any parricipant is assigned to
importanr aspects of the proposed srudy, such as the pro- carinot be manipulared.v Pilot studies provide an opportu-
cesses thar will need ro be foUowed, rhe time required ro niry for investigarors ro test the proposed procedures for
obrain the needed medical records, the cosr of retrieving randomization and allocation concealment. Investigarors
rhese records, and rhe feasibiliry of locating aU requesred can also pilot and assess any blinding of participanrs, health
rnedical records. Payment may also be required ro gain care providers or assessors that is being considered for the
access to administrative data and, in many cases, to gain rhe anticipared full study. Blinding, like randomization and
assistance of specialized staff at the agency holding rhe allocation concealment, strengthens the methodological
needed data. Because rhese specialized staff are frequendy
....,
lI_
u
in shorr supply and very busy, a requesr for their involve-
rigour of a randomized
rhe opportunity
trial,36 and so ir is irnportanr
ro pre-rest, during a pilot study, rhe ade-
ro take
rnent is ofien required weeks or even monrhs in advance. quacy of these processes, including the effectiveness of rhe
~ placebo and its similariry ro the interven tion.
'"'C Testing data collection instruments (including ques-
e tionnaires). Another irnporranr aspect of any srudy is the Assessing the safety, acceptability and suitability
o manner in which the required data are collected from and of the study and its components
u ,abolit each study participanr. Pilor studies provide an Pilor studies also enable an assessmenr of rhe acceprabiliry ~
. irnportanr opporruniry ro test rhese processes, especial!y of the proposed fu!! study and irs various components ro ~
when different staff wi!! be collecting the data or when the relevant stakeholder grou_ps, such as porential participanrs, •
participanrs thernselves wiU provide the data on self- parienrs, parents, research collaborarors, healrh care pro-
adminisrered
i
surveys or other forms. As Lancaster and col- viders, institutional managers and communiry leaders. •
-:leagues outline, piloting provides an irnportanr This assessment is parricularly irnporranr for any rrear- •
:opportuniry ro ensure rhar data collection instrurnents are ment or orher interventions being studied.! i For example, •
cornprehensible and appropriate, and rhat data iterns are invesrigarors may wish ro assess whar proporrion of eligi- _
well delined, undersrandable and presenred in a consisrenr ble parricipanrs consenr ro receive the proposed interven-
manner. Furrher, Simon3,33 suggests that pilor studies rion and, in rhe case of randomized trials, the comparison
.enable the researcher ro explore practical matters, such as inrervention (ofien a placebo). Pilot studies also allow
wherher rhere is enough room on the data collection form investigators rhe opportuniry ro seek feedback from eligi-
for al! rhe data received and wherher any questions tend to ble individuals who decline to participare so thar modifi-
elicit no answer, multiple answers, qualified answers, or cations ro improve srudy recruitrnent might be idenrified.

27 Pilot studies
A pilor study also provides an opportuniry ro gather infor- Identífyíng and estimating the required resources
mation abour rhe safery of the proposed intervenrion, For Pilor srudies also provide an opporruniry ro idenrify and
example, a pilot srudy conducted by Bressler and col- estimare rhe resources necessary ro successfully carry out
leagues37.:lS led ro the cancellation of the corresponding full rhe proposed full study-e-borh rhose that are anticipated,
srudy because the pilor resulrs suggested rhat the rreatrnent such as srudy sraff and needed supplies, and rhose not fore-
under srudy would not be beneficial and could possibly be seen by rhe investigarors. For exarnple, Beebe's pilot study''"
harmful. identified rhe need ro provide suitable footwear for study
Pilot srudies also enable investigarors ro determine rhe parricipanrs. 1he resource estimares that resulr from pilor
acceprabiliry of other aspects of the full study's proposed studies can inform the budget requests rhat accompany
prorocol, such as rhe kind and arnount of data collected research funding applications.
from and about participanrs, rhe frequency and lengrh of
study visits, and the intrusiveness of ourcorne and orher Provídíng evidence of the feasíbility and
srudy measures. If any of these are found ro effect parrici- worthíness of the proposed fuI! study
pam drop-out rates, investigators will usually idenrify and Several authors4.".u.'5.'6.39 have highlighted the fact rhar
implement srudy rnodifications and further pilot test these pilot study resulrs can provide valuable evidence of the fea-
before initiating rhe revised full study. sibiliry and worthiness of a proposed srudy. Being able ro
Gardner and colleagues,'? Hinds and Gattuso" and Jai- describe findings from a pilot study can be particularly use-
rath and collaborators" also discuss and highlight rhe value fui, and is ofren required, in funding applications. More-
of pilot srudies in assessing the adrninistrarive environment over, rhe cornpletion of a pilot study provides evidence that
as well as the staff orientation and educarion required ro the study tearn possesses the necessary skills and expertise
successfully conduct the proposed full study, Difíiculties in ro successfully carry Out the proposed study, Some
any of these areas can adversely affect any aspect of a study, authors'v'? have suggested rhar this benefit is enhanced
including access ro organizational resources and rhe identi- when rhe resulrs of rhe pilot srudy are published in the
fication and recruitment of a sufficient number of study peer-reviewed literature. (See the larer discussion of publi-
participants. cation issues.) 1he successful cornpletion of a pilot srudy
and the experience it provides can also be instrumental in
Identifyíng, testíng and assessíng proposed and convincing others, such as institucional adrninisrrators and
potentíal outcome measures [hose who might porenrially recruit porenrial parricipants,
Several authors,·,o.4o have discussed the role that pilot stud- of rhe study's potencial and feasibiliry.'7
ies can play in identifying, testing and assessing proposed
and potenrial outcome measures. Although this can be Provídíng data estímates to ínform the design of
designed as a specific pilot study objective, it may arise by the maín study
serendipity. For example, Beebe'" describes how, while her One of the commonly described reasons for conducting a
pilot srudy accomplished rradirional objectives regarding pilor study is ro provide estimares ro inform planning of me
the feasibility of a proposed full srudy of a walking pro- proposed fuI! study, 1hese otten include measures of central
granú effect on the well-being of individuals with schizo- tendency (e.g., mean or percemage) and variabiliry (e.g.
phrenia, alternative outcome measures, such as increased standard deviation) of the outcorne measure(s) ro assist in
Hexibiliry, increased energy and reduced psychiatric symp- rhe esrirnarion of rhe sample size needed for the anticipared
rorns were also idenrified by pilot study parricipanrs during full study. However, beca use rhe number of individuals
exit interviews. included in a pilot srudy is usually (and appropriately) small,
In conrrasr, Arnold and colleagues'? describe several the resulting estimares are generally quite imprecise. As a
piior srudies thar spccifically included objecrives ro assess resulr, several authorsI0,41-44 have stressed that estimares aris-
proposed study outcome measures and, in some cases, rhe ing from pilor srudies should be treated very conservatively.
mechanisms believed ro be responsible. Feeley and col- Because of the resulting imprecision of rhe estimates, several
leagues" and Lancasrer and colleagues' also discuss rhe aurhorsi'r" have recommended thar best- and worsr-case
porential irnportance of pilor srudies in assessing rhe suir- scenarios should be considered when using pilor study esti-
ability of proposed outcome measures. mates ro calculare rhe sample size needed for the corre-

sponding fuI! study. For example, Browne" points out rhat Determining the sample size of a pilot study
the smal! size of pilot studies ofren resulrs in underestimates Alchough rhe sample size necessary for a full study is
of the variance ro be used to inform the calculation of the usually calculaced ro achieve a prespecified sraristical
fuI! study's sample size, which would lead ro an underpow- srudy power, determining the sarnple size needed for a
ered sample size for the fuI! study, To guard against this . pilor srudy is less srraighrforward. However, rhe deter-
porential underestimation, Browne'" has recommended minarion of sample size for both types of srudies
rhat the upper limit of the 80% confidence inrerval of rhe should be guided by the same general rarionale: ro
pilot srudy variance should be used ro err on the side of enable th e srudy ro achieve its objectives."? In this
overesrimating the underlying variabiliry of rhe dara. regard, full studies are designed ro enable a sufficienrly I
Leon and colleagues? and Kraemer and coauthors" precise measure of the outcome of interesr. As dis-
describe the use of pilot srudies ro provide an estimare of cussed earlier, the objecrive of pilor srudies is usually ro
rhe anticipated size of effecr/difference. However, rhey provide experience and insight concerning scienrific
warn rhar, given the relatively small size (and resulting and operarional aspecrs of rhe anricipated full srudy, As •
imprecision) of pilot srudy esrimares, these can also be very a result, pilot studies require a sample size rhat will «
misleading, capable of borh under- and overestimating rhe enable their objecrives ro be achieved. In this regard, 4
actual effecr size. SimilarJy, they suggest that 95% conri- several authors3,9,l2,39,43 have provided advice ro consid-
dence intervals be calculated so that the fuI! range of values er when selecting the sarnple size for a pilot study.
4
consisrent with the pilot study's result can be ful!y and Leon and colleagues? ernphasize that "the pragmarics of 4
explicitly appreciated. Further, Leon and colleagues? stress recruitment and the necessiries for examining feasibiliry" ,
that an agreed-upon dinically meaningful difference-and should guide the sample size of a pilot study. ]airath and e
not the resulr determined imprecisely from the pilot colleagues" also point out that the sample size may be t
srudy-should be used to calcula te rhe sample size needed guided by a "predeíined time frame for data collection or
for the full study. Lentli" also provides valuable advice an arbirrarily selected proportion of the parent study sarn- •
about how ro reasonably estimare rhe sample sizes needed pie size." Hertzog'" suggests that rhe width of resulting •
for fuI! studies. confidence intervals should be considered when deciding
Recognizing the imprecision of estimares resulring from on the sample size of a pilor study so rhat suirable estimares
t
..-u
pilor srudies, orher aurhors stress rhe need ro be circum- regarding the anticipa red full srudy can be deterrnined.
·- specr; as Lancasrer and col!eagues note, "rhe analysis of a Although Feeleyand colleagues" rernark rhat usual sample
pilor srudy should be mainly descriptive or should focus on sizes for pilot studies are berween 10 and 40, they also •
:::l confidence interval estimarion" and "resulrs from hypothe- stress that additional factors such as human and financial ••
-c sis resting should be treated as preliminary and interpreted resources, timelines and the proposed fuI! study objecrives
e wirh caution." (p3ll) should also be raken inro accounr. Further, Sirnon" sug-
o gesrs rhar researchers consider nor only rhe number of pilor
u Practical considerations in the conduct
srudy parricipants
sonably represent
but also rheir diversiry, so that rhey rea-
rhose expected ro be recruired inro the
of a pilot study
anticipated full srudy.
As is the case wirh any orher srudy design, pilot studies pose Alrhough any estimare of sarnple size for a pilor srudy is
practica! challenges. These include determining rhe sarnple just rhar-an estimate-investigators should endeavour
size, obtaining approval from a Research Ethics Board, and ro recruit rhe number of parricipants they \ViII need ro
obraining funding ro suppon rhe work. Specific issues con- meer rheir pilot study's objecrives. For efficiency's sake,
cerning publication should be raken into account by however, the ultimare number of pilor participants should
researchers conremplaring a pilor srudy. Finally-and be only a fraction of rhe number anricipared ro be required
imporrandy-one should be aware of the issues arising for the full study. In addition, if researchers are able (Q
from rhe use of data from pilor srudies in the full proposed conduct a large pilor srudy wirhour exrernal funding,
srudy and rhe inclusión of pilot-study parricipants in rhe potencial funders approached for rhe full srudy ma)' gues-
full proposed srudy. Each of rhese pracrical considerarions tion wherher rheir suppon is really required.
will be considered briefl.y in rhe fol!owing secrions.
174 © 2011 The Royal College of Physicians and Surgeons oí Canace

27 Pi/ot studies
Ethics approval for pilot studies new a- is also substantive, correct, and expected ro be of
Like all srudies involving human parricipanrs, pilot srudies interese ro orhers. Further, several authors="!' have pro-
require Research Erhics Board (REB) approval. The proposed specific guidelines for rhe publicarion of pilot studies.
posal submitted for REB review should include a clear out- For exarnple, \'(farson and colleagues' suggesr rhat pilot
line of rhe objecrives of rhe pilor study and the rype of studies ma)' not be suitable for publicarion beca use rhey are
informarion ir is intended ro provide, along with a descrip- not primarily inrended ro generare generalizable results.
tion of and rarionale for the anticipated full srudy' Tha- However, rhey and others-!':" poinr out rhat ir may be
bane and colleagues" poinr out thar pilot studies are not appropriare ro publish pilor studies for certain purposes,
specifically addressed in any of the commonly cited such as ro inform the research communiry of rhe kinds of
research ethics guidelines. In addirion, rhey highlighr the research quesrions being explored and ro bring operacional
imporrance of disclosing ro potencial participants rhar the issues ami barriers ro the atrention of interested investiga-
study is a pilor and not the full "substantive" study. To tors who mighr benefir from rhis informarion.II-13,1)'4l
address rhis, they propose a sample wording for a pilor In rhis contexr, the primary focus of such a publicarion
srudy consent forrn." A furrher considerarion is rhat, should be rhe objectives that rnotivated the pilor srudy, and
alrhough ir is not clear whether pilor studies conducred in not rhose of the anticipa red full study.2,7,15Alrhough rhe
advance of an anricipated clinical trial require registrarion, results arising from rhe pilor srudy that are relevanr ro the
researchers should seek relevanr advice before iniriaring a full study's objecrives (i.e., applicable hyporhesis resring)
pilor study ro ensure that their full study will remain eligi- could be presented in a pilot srudy publication, these should
ble for publicarion in a peer-reviewed journal. nor be rhe main focus of such a publicarion but should be
cornplernenrary ro the pilor study findings that are perti-
The availability of funding for pilot studies nent ro the feasibiliry, acceptabiliry and other operacional
Alrhough guidance abour seeking and obraining funding objectives addressed by the pilor srudy.1.2,15Ir should be
for research studies is addressed in detail in chapter 19, ir noted that many pilor study publications include a review
should be ernphasized thar funding ro supporr the conducr of che relevant literature; this fearure mighr make these sub-
of pilot srudies is ofren available from research funding missions more attracrive ro journal editors.
agencies and other organizarions, such as the Canadian Watson and colleagues" and Morse," however, highlight
Insri tutes for Health Research, the Albena Herirage Foun- the lirnired amount of space available in peer-reviewed pub-
dation for Medical Research and Physician Services Incor- lications and stress thar any pilor studies that are published
porated. Invesrigarors are encouraged ro contact their local should add value ro the exisring lirerarure. Further, ir is
research adrninistrarion office and ro consulr relevant fund- important rhat pilor studies be clearly identified as such
ing agencies ro determine whar funding might be available when they are published, especially ro avoid porential con-
ro support rhe conduce of rheir pilor srudy. fusion if a publicarion describing rhe anticipated full study
does not follow." Sorne aurhors/:'? have also warned rhar
Reporting and publishing pilot studies the publication of a pilor study may affecr the chances of
\Vherher rhe resulrs of pilot studies rnerit publication in rhe the anticipa red full srudy being published, as rhe larrer may
peer-reviewed Iirerature continues ro be a controversial topic be deemed a duplicare or redundant publicarion. Lancast-
debated by a variery of aurhors.2,S-R,II-15,40
Some2,7,14have higher' and Becker" also warn of the potential for published
lighred the imporrance of drawing a clear disrincrion between pilor study results ro be inappropriarely included in sysrem-
small, underpowered, hypothesis-testing studies incorrectly aric reviews and rnera-analyses.
terrned "pilor studies," and genuine pilot srudies that have as Thabane and colleagues" have adapted the CONSORT
their primary objective rhe resring and assessmenr of various guidelines ro develop a checklisr of irerns for authors ro con-
scienrific and operational aspecrs of an anricipared full srudy sider when reporring thc resulrs of a pilor srudy.
The general consensus concerning the publishing of
pilor studies is that there are cerrain circumsrances in which Can pilot study participants and their data be
rheir publicarion is nor only appropriare but should be included in the full study?
encouraged. Morse asserts that journal articles of any kind Several aurhorsl.9.11,16.31have discussed rhe issue of wherher
should be published only if rhey communicare somerhing the data from parricipants in a pilot srudy should be includ-
© 2011 The Royal Coliege 01 Pl1ysicians and Surgeons oi Canada 175

~ ~- __ -
- --
ed in the subsequenr full study. Some9,11 point out rhat ir

Conclusión
would be inappropriare ro indude the pilar study partici- Pilor studies provide an irnporrant opporruniry for
pants and their data if rhe methods of rhe full study are researchers ro test out various scientific and operacional
modiíied in lighr of the pilot study: because these twO aspecrs of a proposed full srudy, The informarion and expe-
groups would have participated in studies wirh differing rience gained rhrough pilor studies can not only enhance
protocols, the resulring data would not be comparable. rhe likelihood of the anticipa red study's success, but can
Further, Lancaster and co-authors' argue that, even if the also provide valuable evidence ro funders and orher deci-
rnerhods are largely lefr unchanged, beca use the decision ro sion-makers of rhe feasibility and soundness of the pro-
proceed with the full study is ofren dependent on the pilot posed srudy. •
srudy results, the pilar srudy participanrs should not be
included because of the risk of selection bias and the
increased likelihood of Type 1 error (i.e., a false-posirive
full study result).
CASE POSTSCRfPT
The Family Medicine physician-researcher works with her colleague to design and carry out a pilot study to estimate how
long it would take to recruit the number of consenting participants necessary for the proposed full-scale randomized
clinical tria!. Although she and her colleague are able to successfully recruit the required four study participants per week
in the full study, the other practitioners who agree to recruit participants are much less successful, each recruiting an
average of less than one participant per month. As a result, she and her colleague revise the study's recruitment strategies
to include running advertisements in local newspapers, thus enabling the clinical trial to be completed successfully.
REFERENCES
.- 1. Lancaster GA, Dodd S, Williamson PRoDesign and analysis of pilot studies: recommendations for good practice. J Eva/ Clin Pract
2004; 10(2):307-12
2. Brooks D, Stratford P.Pilot studies and their suitability for publication in physiotherapy Canada. Physiother Can. 2009;61 (2):66-7
3. Simon S. Pilot study. Updated 2008 July 14. Available from: www.childrensmercy.org/stats/plan/pilotasp (accessed 2011 Aug 4).
4. Hinds PS,Gattuso JS. From pilot work to a majar study in cancer nursing research. Cancer Nurs. 1991; 14(3): 132-5.
5. Morse JM. Appropriate use of pilot studies. J Nurs Scho/arsh. 2001 ;33(4):307.
6. Morse JM. The pertinence of pilot studies. Qua/ Hea/th Res. 1997;7(3):323-4.
7. Watson R, Atkinson 1,Rose K. Pilot studies: to publish or not? J Clin Nurs. 2007; 16(4):619-20.
8. Perry SE.Appropriate use of pilot studies. J Nurs Scho/arsh. 2001 ;33(2): 107.
9. Lean AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in c1inicalresearch. J Psychiatr Res. 2011 ;45(5):626--9.
10. Arnold DM, Burns KE, Adhikari NK, Kho ME, Meade MO, Cook D, et al. The design and interpretation of pilot trials in c1inical
research in critical ~are Crit Care Med 2009;37(1 Suppl):S69-S74.
11. Connelly LM. Pilot studies Medsurg Nuts. 2008; 17(6):411-2.
12. Feeley N, Cassette S, Coté J, Héon M, Stremler R, Martorella G, et al. The importance of piloting an RCT intervention. Can J 'urs
Res. 2009;41 (2):85-99.
13. Gardner G, Gardner A, MacLellan L, Osborne S. Reconceptualising the objectives of a pilot study for clinical research. Int J Nurs
Stud. 2003;40(7):719-24
176 © 2011 The Royal College of Physicians and Surgeons of Cenaca

27 Pilot studies
14. Thabane L, Ma J, Chu R, Cheng J, Isrnaila A, Rios Lp, et al. A tutorial o p.iot 51 ies: t' e what, why and how. BMC Med Res
Methodol. 2010;101.
15. Becker- PI Publishing pilot intervention studies. Res Nurs Health. 2008;31 (1): 1-3.
16. van Teijlingen E, Hundley V The importance of pilot studies. Nurs Stand. 2002; 16(40) 33-6
17. van Teijlingen ER, Rennie AM, Hundley V, Graharn W The irnportance oí conducting and reporting pilot studies: the example of
the Scottish Births Survey J Adv Nurs. 2001 ;34(3) 289-95
18. Prescott PA, Soeken KL The potential uses of pilot work. Nurs Res. 1989;38(1 ):60-2
19. Grady D, Hul/ey SB. Implementing the study and quality control. In: Hulley SB, Curnrnings SR, Browner WS, Grady DG, Newrnan
TB. Oes/gning clinical research. 3rd ed. Philadelphia (PA) Lippincott Williams & Wilkins; 2007. p 271-89.
20. Stewart PW Srnal/ and/or pilot studies: GCRC protocols which propose "pilot studies." 2004. Available from: www.
cincinnatichildrens.org/research/cores/gcrdprotocols/Small_or _Pilot_Studies. htrn (accessed 2011 Aug 4).
21. Lawrence Gould A. Tirning of futility analyses for "proof of concept" trials. Stat Med. 2005;24(12) 1815-35.
22. Srnith MK, Jones 1, Morris MF, Grieve Ap, Tan K. Irnplernentation of a Bayesian adaptive design in a proof of concept study. Pharm
Stat. 2006;5(1) 39-50
23. Coffey CS, Kairalla JA. Adaptive clinical trials: progress and challenges. Orugs R o. 2008;9(4) 229-42
24. Wang M, Wu YC, Tsai GF. A regulatory view of adaptive trial designo J Formos Med Assoc. 2008; 107(12 Suppl) 3-8.
25. Friede T, Kieser M. Sample size recalculation in internal pilot study designs: a review. Biom I 2006;48(4):537-55.
26. Lachenbruch PA, Wittes J. Sorne aspects of the application of internal pilot studies. Biom I 2006;48(4):556-7.
27. Oakley A, Strange V, Bonel/ C, Al/en E, Stephenson J; RIPPLE Study Team. Process evaluation in randornised controlled trials of
cornplex interventions. BMI 2006;332(7538)413-16.
28. Campbel/ NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to
improve health careo BMI 2007;334(7591 ):455-9
29. Power R, Langhaug LF, Nyarnurera T, Wilson D, Bassett MT, Cowan FM. Developing cornplex interventions for rigorous
evaluation-a case study frorn rural Zirnbabwe. Health Educ Res. 2004; 19(5):570-5.
30. Klymko KW, Artinian NT, Washington OG, Lichtenberg PA, Vander Wal JS. Effect of impaired cognition on hypertension
outcomes in older urban African Arnericans. Medsurg Nurs. 2008; 17(6):405-10.
31. Ross S, Grant A, Counsell C, Gillespie W, Russelll, Prescott R. Barriers to participation in randornised controlled trials: a systematic
review. J C1in Epidemial. 1999;52(12):1143-56.
32. Butt DA, Lock M, Harvey BJ. Effective and cost-effective clinical trial recruitrnent strategies for postmenopausal women in a
comrnunity-based, prirnary care setting. Contemp C1in Trials. 2010;31 (5)447-56
33. Sirnon S Design and analysis of pilot studies. Revised 2009 July 8. Available frorn: www.childrensrnercy.org/stats/weblog2004/
PilotStudy.asp
34. Schulz KF, Grirnes DA. Generation of allocation sequences in randornised trials: chance, not choice. Lancet.
2002;359(9305)515-9.
35. Schulz KF, Grimes DA. Allocation concealrnent in randornised trials: defending against deciphering. Lancet.
2002;359(9306) 614-8.
36. Schulz KF, Grirnes DA. Blinding in randomised trials hiding who got what Lancet. 2002;359(9307):696-700
37. Bressler NM, Maguire MG, Murphy PL, Alexander J, Margherio R, Schachat Ap, et al. Macular scatter ("grid") laser treatment of
poorly dernarcated subfoveal choroidal neovascularization in age-related rnacular degeneration. Results of a randornized pilot
trial. Arch Ophtha/mal. 1996; 114(12) 1456-64
38. Ferris FL 3rd, Murphy RP.The peril of the pilot study. Arch Ophthalmol. 1996; 114(12) 1506-7.
39. Jairath N, Hogerney M, Parsons e Tlle role of the pilot study a case illustration frorn cardiac nursing research. Appl Nurs Res.
2000; 13(2):92-6
40. Beebe LH. What can we learn frorn pilot studies? Perspect Psychiatr Careo 2007;43(4):213-8.
© 2011 The Royal College oí Physicians and Surgeons of Car.ada 177

The Research Guide: A primer for residents, other health care trainees. end practitioners
41. Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA Cau ion regarding the use of pilot studies to guide power calculations
for study proposals. Arch Gen Psychiatry. 2006;63(5) 484-9.
42. Browne RH. On the use of a pilot sample for sample size determina ion -,Stat Med. 1995; 14(17): 1933-40.
43. Hertzog MA Considerations in determining sample size for pilot studies. Res Nurs Health. 2008;31 (2): 180-91.
44. Vickers AJ. Underpowering in randomized trials reporting a sample size calculation. J Clin Epidemial. 2003;56(8):717-20.
45. Lenth RV Some practical guidelines for effective sample size determination. Am Stat. 2001 ;55(3): 187-93.
Feeley N, Cossette S, Coté J, Héon M, Stremler R, Martorella G, et al. The importance of piloting an RCT intervention. Can J Nurs Res.
2009;41 (2):85-99
Gardner G, Gardner A, MacLellan L, Osborne S. Reconceptualising the objectives of a pilot study for dinical research. Int J Nurs Stud.
2003;40(7):719-24.
Hinds PS,Gattuso JS. From pilot work to a major study in cancer nursing research. Cancer Nurs. 1991; 14(3): 132-5
8 These two artides provide informative examples of actual pilot studies.
Lancaster GA, Dodd S, Williamson PRoDesign and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract.
2004; 10(2):307-12.
Prescott PA, Soeken KL. The potential uses of pilot work. Nurs Res. 1989;38(1 ):60-2.
• These four articles provide particularly detailed and informative discussions about pilot studies.
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP.et al. A tutorial on pilot studies: the what, why and how. BMC Med Res
Methodol. 2010;10) 1
+J
U
:J
""C
c.: SUMMARY CHECKUST
O O Thinking about your proposed full study, list the possible benefits of conducting a pilot study.
U O If you did a pilot study, list the elements of your protocol that you wouid assess and how you would do so.
Be sure to consider, where applicable, the following:
- participant recruitment
- testing data collection and instruments
- randomization, blinding, and concealed allocation
- safety
- performance of outcome measures
- sample size calculation
- resource requirements
- overall feasibility of a larger study
- overall worthiness of a larger study
O List any decisions to be made as the result of a pilot.
178 © 2011 The Royal College oi Physicians and Surgeons o' C~"B=C
22
Data collection and data management
A. Curtís Lee, PhD
ILLUSTRATIVE CASE
A resident in Otolaryngology has designed a needs assessment survey to evaluate curricular changes in her training
programo After taking the steps to refine her study goals and the questionnaire design, she implements her survey but
finds that the response rate is very low. To make matters worse, many of the survey responses are incomplete and she is
having difficulty interpreting her data. She's not sure what to do next. and wonders what she should do differently when
she attempts her next research project.
11 Data collection is a fundamental design will foster effective data collection and prepare
component of the research process. 1his chapter offers information for subsequent analysis.
guidance on optimizing data collection so rhat you can get 1he purpose of a data collecrion rool is ro obtain a sarn-
rhe most out of the information you gather. Data collection ple of data rhat truly reflecrs what is being measured; there-
forms thar are well planned, structured and organized can fore, its design must be valid, reliable and practica!. In the
make rhe process easier, enhance rhe qualiry of the data you context of data collection, validity refers ro the assurance
collect, and make subsequent steps such as the analysis and rhat the data collected will comprehensively and accurately
inrerprerarion of resulrs more reliable, more valid and more represent what is being measured. Reliability ensures that
readily achieved. A1though many issues concerning data the data are consistent and the resulting conclusions repro-
collection will have been addressed inyour study protocol ducible. Practicality is also essential ro success: ease and
and ethics applicarion, when you get down to rhe derails of efficiency of use and the abiliry to analyze rhe resulting data
designing a data collection rool ir is helpful to bear certain rnust be factored into the designo
practica] considerations in mind.
CHAPTER OBJECTIVES
Data collection forms
Data collecrion forms can be as diverse as the research pur-

• describe key steps in developing data collection tools tha
will enable you to collect reliable and valid information
--
poses rhey serve. For exarnple, a form can be designed ro that can be thoroughly and appropriately analyzed (see
guide observations of parients, or the assessment of trainees also ch. 10);
vis-a-vis recommended practice. Specific eypes of informa- • discuss the importance of well-organized data collection
tion from patient records and case files can be organized for the successful completion of a research project;
through data collecrion forms. Online and paper-based • apply practical, common-sense tips in creating effective
surveys are orher examples of data collection tools. What- data collection tools; and
ever rhe eype of rool rhat is used, carelul organization and • describe the basics of preparing data for analysis.
!~E'I;TERr\~s
---l
Data cleaning Framing Relevance
Data element Likert scales Reliability
Double entry Practicality Validity
Face validity Questionnaires
--- ------ --- ------------- -_j
© 20 11 The Royal ColJege of Physicians and Surqeons of Canada 179

General considerations
ple, if a srudy is comparing residents in their firsr year '
Contextualizing your data collection
residency wirh rhose in their fifth year, ir would be essenti
Your data collecrion form must be developed and/or adapt- ro collect year-of-rraining data from each respondent.
ed ro fir your research question and methodology. Consider When you collecr demographic data, ask only for info
the type of informarion rhat you need and irs source. For mation rhat is perrinent ro your research quesrion and th.
example, if your study involves a survey rhar is to be disrrib- will acrually be used in the analysis. Creare atable ourlinin
ured ro a variery of parienrs, the form will need to be feasi- rhe data rhat you hope to collect, and reassess whether a
ble for the average person to use. Conversely, if your data data to be collected are necessary. Eliminare rhose quesrion
collecrion tool is inrended for use only wirhin a research that do not contribute to the purpose of the srudy, Includ
ream (such as form to extract data from medical records), ing irrelevanr quesrions of a privare or sensirive nature in ,
yOLlmay wanr to focus your efforts on oprimizing func- survey is not only unethical but is likely to increase rhe ran
tionaliry wirh respecr ro data extraction rather rhan on sirn- of non-responses ro particular items or of refusals ro com
plifying the wording and polishing the visual presentarion. plere or submir the formo A furrher reason to be judicious ir
To rake another example, if your study requires the collec- rhe range of quesrions posed is expediency: balance the
rion of exrensive qualirarive data, such as those collecred in tempration to ask quesrions that "rnight" be used agains:
a strucrured inrerview, you will need a form that is quite the practicaliry of simplifying and reducing the effon
different from one inrended for purely numerical dara required for effecrive data collecrion.
entry, If you are using an esrablished tool, consider the spe- Consider privacy legislarion and ensure rhat your rnerh-
cific contexr of your study and adapr the form as necessary. ods of data collecrion and storage adhere ro rhe confidenti-
Ir is rare for an exisring tool ro be suitable for a research aliry standards ser out in your prorocol and approved by
projecr wirhout adaptation and further assessment of its your Research Erhics Board.
reliabiliry and validiry in the proposed study environment.
If a predesigned form is chosen to be used, seek copy-
Designing a data collection form
righr clearance and ensure rhat proper attribution is made
ro the crearor or distributor of rhe formo Alrhough many 1he design of your data torrn or survey insrrumenr-irs
data collecrion instrurnenrs can be found on the Incernet, strucrure, organizarion, formar and visual presemarion-
,,_ many are proprierary and may have licensing cosrs associ- can assist data collecrion by making ir easy ro follow and
ared wirh their use. enhancing rhe clariry of individual iterns. A well-planned
design can also Iacilirare the inrerpretation of the results by
Identification of respondents
making rhe data easy ro classify and retrieve. For exarnple, a
Each cornplered data form or record should be assigned a well-designed interview form is a communicarion tool that
unique idenrihcarion code that allows the researcher, if nec- lends clariry and structure to the inreracrion, helping the
essary, ro trace each response to its source (an individual or researcher to pose questions that are direct, well focused
group). Identiíicarion numbers can also be creared to clas- and unambiguous. Similarly, an effecrive self-adminisrered
sify parricipants (e.g., control group or trearrnenr group), questionnaire should be clear and ro rhe point: rhe appear-
identify rhe session or rime of assessrnent, and so on. By ance of rhe instrurnent should not be distracting or its
helping to preserve rhe anonymity of the respondents, the organizarion confusing. 1he organizarion of the insrru-
use of idenrification numbers not only prorects rhe confi- rnent should be logical, helping ro focus the participanr's
deririaiiry of responses bur also reduces the potencial for attention and to ensure rhat the type ofinformarion that is
bias in rhe analyses. being asked for is clear. Finally, the form should enable
answers to be recorded in a manner that facilirares the data
Collecting demographic information exrraction and analysis.
Demographic informarion is perrinent ro rhe generalizabil- To ensure that your data collecrion form will result in
iry of findings and to subgroup analysis. Consider whar data that are clear, useful and require minimal correcrion or
in formarion is necessary ro collecr and, as appropriate, elirnination (data cleaning), keep the following factors in
make responses ro rhese quesrions rnandarory For exarn mind:

2 Data collection and data management
1. Purpose ornprehensive list. This said, however, rhe analysis

• Relevance. Each data element requesred on adara would be more accurare wirh a eomprehensive lisr of
collecrion form should have a legirimare bearing "forced choices" as long as rhe list matched expected
on rhe research quesrion. 1hose who fill out the choices.
form should understand why they are being asked
the question and how ir contribures ro rhe research 3. Organization
projecr's goals. • Instructions. \'(Ihen designing a form, ensure that
• Framing. Iterns included on rhe data collecrion form brief, adequate and clear instrucrions are provided and
should be framed in such a manner that ir is clear ro can be easily aecessed.
respondents whar rhey are being asked ro do, or whar • Grouping of questions. Use headings ro group irerns
kind of informarion is being soughr. logieally and ro facilirare smooth rransitions from
seerion ro seerion. Use "skip" questions ro enable
2. Clarity respondenrs ro avoid sections rhar are not applicable
• Simplicity. Each itern should have a single foeus: ro thern.
ask one quesrion at a rime. Complex quesrions ofren • Order of questions. When creating a quesrionnaire,
lead ro ambiguous or cornplex responses that are ask rhe imeresting and crucial questions firsr ro
difiiculr ro inrerprer or abstraer, For exarnple, "double- capture the interest of the respondenr. Some
barrelled" quesrions (i.e., those that ask rwo quesrions designers prefer ro colIect simple information such
at once) should be avoided. See chapter 10 for furrher as demographics first while others prefer ro ask the
diseussion of itern designo demographics questions last in order ro gradually close
• Plain language. Use complete senrences, simple the survey. However, if crirical inforrnation is included
senrenee structures, and familiar words. in the demographies, ir is besr ro ask rhese questions
• Freedom from ambiguity. Phrase questions in a firsr. In addition, sensitive questions are usually
manner thar enables only one, easily undersrood, lefr until the latrer portioris of a questionnaire ro
interpreration of what is being asked. minimize rhe chance that respondents will give up on
• Precision. Response fields should be designed ro the quesrionnaire when lírtle of ir has been cornplered.
enable rhose who complete rhe form ro record
responses that accurately refiecr their imended 4. Format
meaning. For example, respondems who have no To rnake your data eollecrion form look professional and
opinion on a particular issue should be able ro record optimize the design of rhe form ro record dara, consider
"no opinión" rather than being foreed into selecting the following recommendarions:
an answer that implies agreemem or disagreemenr. o Use a legible rypefaee in a size rhat will be easy for al!
• Comprehensiveness. Questions should aceoum users to read.
for aII possible choices. For exarnple, if you are o Use spaee ro define and separare rhe quesrions.
asking trainees ro ehoose rheir area of specializarion o If the data form is presented direcdy ro rhe srudy
from a list, all specialties and subspeeialries must be participanrs, avoid splitting quesrions over a page
accounted for on rhe list. break.
o 1he oprions in a list of ehoiees rnust be murually o Use page numbers and/or provide a guide to indicare
exclusive and logieally organized. the percentage of the survey 01' questionnaire rhat is
o A eommon rnerhod of ensuring comprehensiveness lefr ro complete (or, in a prinr documem, rhe eurrent
when asking questions is the use of the option page and total number of pages). This is espeeially
"Orher." Although rhis allows for grear Hexibiliry, it irnporrant if rhe srudy parricipants are filling out [he
also presems challenges for analysis. For exarnple, if data colIecrion form thernselves,
you ask participants ro indicare their favourire food o Use bold and italic judieiously ro provide clariry and
and the majoriry of responses are included in the emphasize imporrant derails.
"other" eategory, the analysis will be mueh more o Use bold and italie judieiously ro provide clariry and
complex than the analysis of ehoiees from a simple, ernphasize imporranr derails.
© 2011 The Royal College 01 Physicians and Surgeons 01 Cana da 181

• Use tables sparingly but effective!y to guide responses.

will he!p ro enable this direcr comparison. For example, iJ
For exarnple, when asking a series of relared Likerr-
you adminisrered a survey rwo years ago and wamed ro
rype questions, group thern in a table formar. Do not
track changes from year ro year, you may wanr ro ask rhe
use too many tables, however, as they will crowd rhe
same quesrion, worded in rhe same way, and do a direcr
data collection too!.
comparision. If wording changes are made ro the question,
• Consider using colour-coded paper to he!p organize
on rhe orher hand, you will be unable to derermine wherher
your data collection, For example, highlight
any changes in the responses reflect the participanrs' inter-
irnportanr or mandatory questions with a different
pretarion of rhe revised phrasing of the quesrion, or acrual
colour than the less importanr questions.
changes in behaviour over time. Many useful quesrion
scales have been used in past research and are available in
Question formats
the literature, rhese should be idemified and assessed before
Whenever possible, avoid open-ended quesrions on your you decide ro creare questions de novo.
data collection form (see also ch. 10). Questions with
selecred responses (such as multiple-choice quesrions, Finalizing the data collection form
drop-down se!ection boxes or Likert scales) rarher rhan
Once a drafr of the dara collecrion form is created, it should
open-ended quesrions are easier to answer, and the respons-
be reviewed rhoroughly from rhe following perspecrives:
es are easier to categorize, code and analyze. Analyzing fre-
quency counrs of a limited number of specific selecrions is
• Face validity. The data form, survey or quesrionnaire
simple and unambiguous.
should look professional and conrain no spelling,
Likerr scales provide respondems wirh a limited number grammarical or formatting errors. Ir should seem
of ordered responses to indicate their answer ro a posed
evidenr to anyone filling it out rhat it is indeed
question, such as their currenr state of hea!th (e.g., exce!-
appropriare for the stated purpose of rhe research.
lent, very good, good, fair, poor) or their currenr leve! of
• Relevance. Each question in rhe form should relate
pain (e.g., none, mild, moderate, severe). They might also
to rhe research goals and contribute to answering (he
be used to enable respondems ro indica te their leve! of
research question(s) posed in rhe project proposal.
._ agreemem with a posed staternent (e.g., strongly agree,

agree, disagree, strongly disagree). Other common
used wirh Likert scales perta in to rhe respondem's
anchors
percep-
• Clazity, Ask a knowledgeable colleague who is nor
direcrly associated with rhe projecr ro review the
form ro test whether his or her interpretarions of
tions of importance or frequency in relation ro a sraremem
rhe questions are in line with your intenrion, An
or series of staternenrs. For example, "Rare our leve! of saris-
approach that can be he!pful in identif)ring unclear
facrion wírh the following staternenr. I like buffalo wings:
or poremially minsterprered quesrions is to have rhe
srrongly agree, agree, disagree, srrongly disagree." When
reviewer ralk our lo ud as he or she works through the
using Likerr scales, researchers often need ro determine
draft survey instrument. After examining rhe form,
wherher ir is berrer to offer rhe oprion of a mid-range or
your reviewer should also undersrand rhe purpose of
neutral answer or to force the respondent to choose berween
rhe study and the process required of rhe panicipams,
a posirive and negarive response. For example, a rypical Lik-
and should have a clear undersranding of all irems and ~
en itern would like!y have a respondenr "srrongly disagree," rheír contributions to rhe research project.
"disagree," "neirher agree nor disagree," "agree" or "srrongly
After you have received feedback from one or more
agree" wirh a statemenr. If rhe researcher would like rhe
knowledgeable colleagues, your draft instrurnenr could t
respondenr ro have only four options, "neirher agree nor
then be reviewed by being adminisrered in one or more ~
disagree" would nor be included. To reduce the variabiliry pilot projects in which sample data are collecred from sui _
of inrerpretarion, scales should have no more rhan five able respondents (see ch. 21). 4
oprioris.'
If your research projecr builds on currem research, or
you amicipare rhar you will want ro compare your resulrs
if
From data collection to analysis •
wirh those of previous research, using the same wording of
As you creare your data collecrion form, anticipare how che ,
quesrions and response oprions as in rhe previous research
dara will need to be prepared for analysis. Quantirarive dala
182
© 2011 The Royal College 01 Physicians and Surgeons oí Ca-e::"
Lata co/lection and data management
are analyzed using programs such as Microsofr Excel, SPSS Once the data are enrered, they should be reviewed and
and orher statisrical analysis packages. Microsoft Excel dara cleaning should be done. In addition ro reviewing each
rends ro be rhe simplesr tool for data entry and is sufhcienc quesrionnaire as soon as possible afrer ir is received, simple
for most basic research projecrs. Several user manuals are descriptive sraririsrics should be cornplered on rhe enrire
available, and Excel's online learning resources are very data ser so thar inconsisrenr or unreasonable values mighr
helpful." Because rhe data requirernents for a particular be idenrified. Data cleaning is rhe derailed process of
projecr can be quite specialized, only a few general tips on reviewing collecred data ro idenrily any incorrecr and/or
using Excel will be described here. inconsisrent enrries in prepararion for the analysis; if obvi-
To illusrrate che use ofExcel for data collecrion and man- ously invalid responses are found, rhey need ro be eirher
agemem, consider rhe residenr in our case example, who is correcred 01' eliminared prior ro analysis. For exarnple, if a
evaluaring curricular changes. She creared and disrribured a group of medical residents are asked for rheir ages, and a
quesrionnaire and used ir ro collect informarion penaining value of "8 years old" appears in the resulring data ser, rhis
ro assessrnents within the curriculum. The spreadsheet in invalid value should, if possible, be corrected or removed
Table 22.1 shows some of her dara. Nore rhar in spread- prior ro analysis. Incorrect 01' inconsistent data items can
sheers of rhis kind ir is rypically besr ro organize the data porentially be corrected by rechecking the original data
using one row per respondent and ro assign each individual enuy form (ro correct adara entry error) or even by return-
itern ro be analyzed ro its own column. In the sample case, ing ro the respondenr ro clarify, correct or confirm che
che researcher has 10 respondents, allocared ro rows 3 ro 12. response.
Each quesrion is given a differenr column. A sample offour Data enrry errors can be assessed by having rhe data (or
questions is shown in the figure. at leasr a portien of it) independently ente red by rwo differ-
Each column should conrain consisremly labelled infor- ent people (double entry) and the rwo sets of enrries corn-
marion, and a glossary 01' standard lisr of rerrns rhat will be pared. Ir is also recommended ro verify a sample of data by
accepted on rhe input form should be used ro ensure consis- randomly selecring as many data as possible within the
tent data input. For exarnple, if you are collecring informa- Excel file and comparing thern wirh the original data source
tion about Canadian universities, all Canadian universiries (e.g., rhe original, complered questiorinaires).
should be lisred as possible oprioris in a consistenr rnariner, A copy of the raw data should be kept and a back-up
such that responses rhar are nor included on rhar lisr would copy should be rnade before the process of data cleaning
be rejecred by the input formo In Excel, using drop-down begins.
lists for selections of rhis kind is recommended. In our case
exarnple, Excel is able ro count che number of "Yes" and
"No" responses because rhey have been entered consisrent-
ly. NorerharrheformulaincellF22 (=countif(E3:E12,"Yes")
Conclusion
In rnosr healrh researcb projecrs a considerable arnounr of
_.
counrs the nurnber of "Yes" responses in column E, rows 3 time and effon is directed toward developing rhe research
ro 12, ro rhe question "Do you prefer the new currículum?" quesrion and ensuring that the research proposal receives
lf a "Y" were eme red rather rhan "Yes" ir would not be erhical approval. A reasonable arnounr of efforr will also be
couneed: hence the need for consisrency. required on rhe design and organizarion of data collecrion
If dates are used, use a consistenr, unambiguous formar tools ro ensure that these initial efforrs result in data rhat
such as 05-]ul-2010 or YYYYMMDD. If you are analyzing can lead ro successful research outcomes, and ro ensure rhat
frequencies, ensure that all subrnitred responses fit into fixed an instrument acceptable to the ethics review comrnirtee
caregories. Numeric data should be entered as numbers, not will be submitted with your application for ethics approval.
with words, ro enable addirion and other cornputation with- Carefully designed data forms can enhance the qualiry of
in the spreadsheet. Sorne common descriprive srarisrics are your findings: by srandardizing data collection, they can
included in the spreadsheer exarnple in Table 22.1. make rhe analysis and interpretation of resulrs more reli-
able, more valid, and more readily achieved. 11
aSee http://office.microsoft.com/en-us/excel-help/
© 2011 The Royal College 01 Physicians and Surgeons oí Canacla 183

The Research Guide: A primer for residents, other health care rai ees, and practitioners
• Table 22.1: Example of a spreadsheet used for data management
i"
¡.--... ! A ,B: e
---i--------------.L-----' ---.--------
: D J
, E I i II
' -. __ ...
F .__
J
! 1 ! Evaluation of the curriculum changes (Survey questions 5 and 6) i
r--------r------------'-----' --------. ---¡-·-----------r, ---·---------·---·---1
:,'. i, I! 1 On average, how ': I1 ', i,
¡ ¡ ¡ ¡ manyassessments how many ¡ On average, ¡ i When d.id you 1
2 i l! were yo~ able to ¡ assessments we~e you I I start usmg the !

1 ¡ I I make usmg the OLD ¡ able to make usmg the I Do you prefer the ¡ new I
: ¡ 1, training program? 'NEW training program? I new curriculum? i curriculum? I
i-···-··-·----+-----·--·---·---·----!-----------f-------- ..----------.--.-- ..--~-.- ...----.------ ...-..-----.-·-¡----------·-------·---j-·--·-----------·-1
: 3 ; Respondent 1!
['·--·------·-I·--------·--·-----··-----,------··-··
i 5 ! 8 ¡ Yes i 15-Sept-10 ¡
..---¡-----.-.--.-------------¡----.-.-------------------.,---------·------·---1------·-------·----1
: 4 i Respondent 2, 1 4 I
¡-·-·-------·-¡-·-·---·---------·-- ..I------··-··-r--····--.-.---------- ..-----.---.---¡-.---.--
7
..-.--.---- .... ------··------[·---------··--·--·----·------1-·-
i Yes ¡ 16-Aug-10
..----------- ..- '--"'1
¡
: 5 r
[----··--_·_--1----_····-----------------1------
Respondent 3! ! 3
---f--------------------------------r----------------------------------------¡------------------------
¡ 8 j Yes i 17-Sep-1 O
.. --:--------------------------.j
¡
¡
:..- -_._--
6
", ._---
_._.
-- -_.---_._
.._.,
! Respondent 4
.._------j_ .._------------------_.
__
[ ¡ 4 :
._.+---------_.-----_._-_. ------------,-_.
8 ! -----_._--
Yes r 09-Jul-10
_._-----_.¡---_._---_._-------_ .. _---.. _--!:
i 7 ! Respondent 5 ¡ ! 10 1 9 i No : 19-Jul-l O i
¡--·---·-·-¡----·-------··-·-·~---·--·----·i·---------·-------.-.-----¡..--- ..-.---------.---------------.¡- ..---·--··----·-----·-·-+···--·-·----·-·-·--------1
:
¡----- 8
---------- i Respondent
r- -- .-_.--- --------
6!
----- -lo. ----------1---------
! - ..-9-- -------------1-----
[
-
8 !
-- ----_- --..- ._o
No
..~-----.~-. ! 01-Sep-l0
-------. -----T·--·--··--· --..------.-.·------·-1
I
. 9 • Respondent 7 ! 6
..._. __ ._-_----- ... _-- -
8 Yes 23-Jul-l0 ¡
10 Respondent 8 6 No 1O-Oct-l O
11 . Respondent 9 5 8 Yes 23-Aug-l0

_.1
!
12 6 7 Yes 1O-Oct-l O
13
14 : Average 5.3 7.7
6
•,
.----.---- - .----.- •• o .__,_.-.-- •• - •• --.-.-. ----.- •• --- -----.- -'-T"-- ..--- -------.-
---..
..·_·_·1--·--·_·· .- - .- .... --.--.-------

10
.. --.
I
¡---.----.- .. --- .. -
9 1<- =MAX(D3:D12)
----.----.-·oO-. ---- .. - ..-,--.----.-.----oO.---.---.----. "1 .. - .. -- ... -- ... --.-.. . -- -!
M_ ,
17
18
••••••
¡
: Std Dev
, --_.-
Median
.-- - I
!
.- •• -- ••• -;--_.--
¡
.. _- ---_ ••
2.67!
---.-
5
__ o •• --.-- .---:--.
i
- ••• _-. ---- ••••
0.82
-.--.----.-.--
8
•• ----- •• --.--:.-
i <- =STDEV(D3:D12)
••
k =MEDIAN(D3:D12)
---
¡
•• -.-----.------.------
LI ---------- -- •• - •••. - •• -.-. -
4
:--------------;-.~.-------.-.----- ... ----~.-.-.----.---~-------_.--------------- .• --~--- · · · · J ---·---·---f--· -------------
, 19 : Skew ! ! 0.43 ¡ -0.81 ,<- =SKEW(D3:D 12) ¡ .
.._----_.--.. --~-~----------.----..-- ----------+----------.---- ..---------.------__¡-----.---------·~·-----------------_·-----~--i----------------·----·--·---·-:··---·----------·------·l
1
•
.:
20 !
r------·--------;-----------------r-------·
: 21 1
Kurtosis
,t
'
i 1
¡
0.18
I
,
¡
·-·1------·--------------·---------1'-------------·-----------------¡---·------·-----------·--·~---r---·----------------.--¡
:
! 1,24 i<-=KURT(D3:D12)!
:
¡
I
•
¡ ~----._--------~-------l-----_------ -------
--~---- --------- --- -------~----- J-------------------!
, i . !! 1 <- =CORREL ¡ _ ! <- =COUNTIF :
: 22 ¡ Correlation ¡ ! .70 ¡ (C3:C 12,D3:D12) : # Yes - 7 ; (E3 E12, "Yes ") ,
i.. I
.:•. ._._ .. l'
"'!. . ,
.....J_-.----------------------------------!----------------------.--.---------.1: L---------.-------.---.,-------------.------.--;
I :
23 I t test 0.01! <- =TIEST , # No - 3 : <- =COUNTIF

, " : (C3C12,D3D12,2,2) : - : (E3E12,"No")
------_!_------ ---- : ~ ~ ~ . ___________j L.- __ ~ ~
••
41
•
,
.ti
CI
,
••
184 © 2011 The Royal College of Physicians and Surgeons o' Cc~a3
22 Data co/lection and data management
REFERENCE
1. Wasek PA. Practicalconsiderations in designing data collection forrns. Inf 0, 701 Hosp Epidemiol. 1990; 11(7):384-9.
Dillman DA, Smyth JO, Melani Christian L. Internet, mail and mixed-mode surveys. the tailored design method. 3rd ed. Hoboken (NJ):
John Wiley; 2009.
• The best resource for designing surveysand creating forms for research.
Jamieson S. Likert scales:how to (ab)use them. Med Educ. 2004;38(12): 1217-18.

• Providesexamples of scalesand practical methods of improving the use of Likert scales.
Wasek PA. Practicalconsiderations in designing data collection torrns Inf Control Hosp Epidemiol. 1990; 11(7):384-9.
• A good guide to designing forms.
http://people.usd.edu/-bwjames/tuVexcel/
• An online tutorial helpful for beginners in Excel
www.baycongroup.com/eIO.htm
• An online resource for Excelusers.
rv
O
:::s
EXERCISE: ASSESSING YOUR DATA COLLECTION FORM a.
Consider the following questions: e
n
,....
1. If all the questions on your data collection form are answered, will you have the information you need to answer all of your
_.
research questions? :J
~
2. Would someone not involved in your study interpret your questions in the way you intend?
3 Will your data collection form be clear to all users (e.q., interviewers, data abstractors, and study participants)7
4. Can your data collection form be practically irnplernented? How long does it take to fill out? Are participants likely to
complete the entire form? How much work will it take to extract data from the form 7 Have you tested all of these aspects
in one or more pilot administrations, using a group of suitable respondents?

The Research Guide: A primer for residents, other health care trai ees. and practitioners
SUMMARY CHECKLlST
o Consider your research protocol, identify all ot the data that you will need to colleet, and determine how you will
colleet it.
O Identify how many data eolleetion instruments you will need.
O Seleet, adapt or design eaeh instrument, keeping in mind best praetiees and the elements needed to create an
effeetive tool.
O Revisit the context and protoeols of your study. Decide how the instruments will be administered, by whom, and
when. Revise as needed!
O Pilot your instruments and protoeol. Revise again as needed.
O Finalize your forms prior to ethies review.
O Manage your data in a relevant spreadsheet suitable for analysis.
O Clean your data.
O Ensure your data are always kept safe and seeure .
.• -
186 © 2011 The Royal eollege 01 Physicians and Surgeons o; e¿-¿=c

23
Managing and monitoring a study
Stacy Ackroyd-Stolarz, MSc, PhD
ILLUSTRATIVE CASE
During an intense, month-Iong research rotation, a resident develops a study protocol and prepares a submission to his
institution's Research Ethics Board. Developing the protocol takes far more work than he had anticipated, but after some
minor clarifications it receives full approval. Now he's ready to do the study, but his rotations and call schedule for the next
four months allow little or no time for it. He is being encouraged to present his findings at the departmental Research Day,
but he doesn't know how he will manage to finish the study by then. He begins to wonder why physicians bother doing
research.
11 Earlier chapters in this guide • Oversight of:

described the essential first steps in planning a research - study team
study, including finding a research supervisor (ch. 3,) for- - srudy conduct
mularing an "operational" research question (ch. 6), deter- • Monitoring of:
mining which methodological approach ro take (ch. 9) and patient enrolment, allocation and follow-up
developing a study protocol (ch. 17). However, the day-to- financial expenditures
day conduce of a study also needs careful planning. The data qualiry and securiry
. practica] aspecrs of how a srudy is carried out can have a adherence to study prorocol and to timeline
significant impact on its overall qualiry and scienrific
CHAPTER OBJECTIVES
rigour. This chaprer provides an overview of study manage-
rnent and of the various responsibilities that ir enrails. After reading this chapter, you should be able to
• describe the responsibilities of a principal investigator;
Responsibilities of study management

• discuss the distinction between designing and
conducting a research study;
_,
When you assume the role of principal investigator, the • describe key considerations in managing resources,
overall conduct of the srudy becomes your responsibiliry, including time, for research;
frorn assembling rhe srudy ream ro fulfilling post-study • discuss the importance of, and processes for, monitoring
obligarions such as ensuring the secure storage of data and the progress of a research study;
proper srudy closure wirh the Research Ethics Board. • summarize issues related to data management and
Essenrially, these duries involve the key functions of over- security; and
sigh[, moniwring and communication, as follows:1.2 • identify online resources for new investigators
KEY TERMS
Authorship Monitoring Recruitment Study identifiers
Communication Oversight Research Ethics Board Study team
Data security Personnel Research coordinator Threats to internal validity
¡
Funding Personal identifiers Research diary Time management
Information technology Principal investigator Statistician
_. - _o. - ._o _. ._. . _j

I

• Cornrnunication with: if you have a positive experience with research during your
study team (including preceptor) rraining, you will probably be much more interested in
study participants (where applicable) incorporaring research into your professional career (see ch.
Research Erhics Board 32)-and an important component of rhat posirive experi-
program director ence is tackJing a project thar can be realized within a rea-
relevant institutional departrnents sonable rime.
These responsibilities can be challenging during a busy Expecting the unexpected

training program, bur you can take practical steps to make Although it may seem obvious that a study prorocol must
them more manageable. The following sections describe be followed as written, the realiry is that unexpected situa-
I
various aspects of research management that are important tions almost invariably arise in the conduct of a study that -,
to consider as you gear up for your project. The sample make it diíiicult, if not impossible, to adhere ro rhe prorocol
project checklist provided at rhe end of the chapter lisrs the in every detall. In sorne cases, an amendment to the proto- I
main tasks and functions for which you will be responsible, col will be necessary. Consider the following examples:
either direcdy or through the supervision of others on rhe
team. Also provided are a templare for keeping track of • You have designed a study that involves a series of
expenditures and a sample project rirnetable. monthly follow-up interviews with patients rhat
take an average of 30 minutes to complete. Aírer
Keeping it manageable collecting data for a couple of months, you realize rhat
One of the mosr irnporrant resources that you will need to a majoriry of study participants are withdrawing from
manage during rhe COLme of your research project is the study after rhe Iirst follow-up interview.
time-that of others, and your own. Developing a project • You want ro conduct a survey of Iourth-year residents
thar is feasible to complete during the allotted time during in Internal Medicine across Canada. The Research
your rraining may mean tackJing something smaller in Erhics Board has approved rhe study, but in view of
scope than you (or your preceptor) had initially imagined. the personal nature of the questions has requested
However, rhe Jessons learned in a small study are just as thar you seek wrirten informed consent from each
valuable as those gained from a large one, since rhe process participant.
·- and tasks are similar. Whether you undertake a health • In a study involving review of electronic healrh
record review, a survey, or a randomized controlled trial, records, the rime it takes ro extract the data is 45
you will have ro develop a srudy protocol and will benefit minutes longer per record than originally assurned.
from the experience of preparing a submission ro a With 500 records ro review, you're going ro run out of
Research Ethics Board. (Increasingly, institutions require funding to pay rhe reviewers.
that even small srudies involving healrh record reviews
undergo erhics review.) Regardless of the scale of your Each of these examples has potential solurions. Alrhough
study, you will have ro ger ir up and running, monitor ir on so me problerns should be prevenred during the protocol
a regular basis, and ensure rhat íollow-up (if required) is development phase (e.g., the situation in the hrst example
carried out and rhar the analyses are complete. could have been avoided by pilor resting rhe follow-up
~
Your h.rst foray into health research has a betrer chance questions and interview schedule), there must be a deíensi-
4Jl
of being a posirive experience if you can complete your ble scientific and erhical rationale for any protocol arnend-
project during training. Many a rrainee has had every inten- rnents or changes in study rnerhods, and these rnust be ~
rion of h.nishing a srudy once he 01' she starts practice, only documented. Moreover, any changes must be approved by
[Q find rhar work dcmands make ir impossible ro give the your Research Ethics Board before rhey are implemenred.
project rhe artention ir deserves. This means thar either the
precepror has [Q finish rhe project, or rhat ir is never corn- Working effectively with your study team
plered-a resulr that is not only frustrating for everyone A study tearn rypically includes investigarors and srudy
involved but that also means thar parients enrolled in the staff (e.g., a research coordinator). Orher personnel, such
srudy may have been needlessly exposed ro risk. Moreover, as human resources managers, database analysrs and tech-
188 © 2011 The Royal College of Physicians and Surgeons Di Canada

23 Managing and monitoring a study
nicians, rnight not be part of rhe sciemific tearn, but are rnost eITlcient use of your time (and that of the sratistician),
irnportanr to rhe overall operation of rhe study. In all train- bur wil] also ensure rhar the data are in a usable formar and
ing programs rhere are people who can help you with your of sufficiem qualiry ro be analyzed.
research project. Alrhough you must retain oversight and The key ro a successful research project involving a tearn
primary responsibiliry for your study, ir is borh reasonable of people is routine and effective cnrnmurrication. Even if
and wise to seek assistance from others. Help is available you are doing most of rhe work on your project, ir is still
from many sources, including senior trainees, research imporrant [Q keep in rouch with orhers to ler thern know
coordinarors, statisticians, informatíon technology (IT) how things are progressing. A brief email on a weekly 01'
personnel, health records technicians, and your preceptor, monthly basis is ofren all rhar is needed. For studies involv-
Research Director and Program Director. Most people ing larger rearns, regular face-ro-face meerings are an excel-
involved in research are more than willing to rake rhe time lent strategy for building a more effective tearn. Using a
to answer quesrions or direct you ro another resource. Find- consisrent format and routine timing of any srudy-related
ing out about rhe experrise available in your depanmem, updates or meetings will often save you time by avoiding
faculty and institution will save you time. potential miscommunication. You will also be responsible
Ir is important to consider the nature of your relarion- for ensuring timely correspondence wirh cenain deparr-
ship with each person who is assisting you. For example, rhe ments in your instirution, such as Research Finance for
team might include a medical studenr working as a vol un- funded studies or IT for studies involving institutional
teer to help with data collecrion in order to get some data. In addition, you will need to ensure that tearn mern-
research experience, or perhaps you have sufficienr funding bers are able to contact you at any time during rhe srudy if
to hire a research assistant. In eirher situation, ir is irnpor- rhey have questions or no longer wish to participare.
tanr to: (1) define responsibilities, (2) agree on remunera- Anorher irnportant srudy-relared task is negotiating
rion, if any; (3) moniror rhe quality and quantiry of the authorship of any published results. Ir is always best ro
person's work on a regular basis; (4) provide constructive have rhis discussion early so that expectations are dear and
feedback; and (5) be available ro answer quesrions related ro consistent. There are excellenr referel1ces describing guide-
lines and other considerations for aurhorship.Y' 1he Inrer-
your study
Srarisricians are an invaluable resource during rhe devel- national Committee of Medical Journal Edirors
opment of your protocol for such tasks as calculating an recommends that authorship be "based 011 1) subsrantial
adequate sarnple size. 1hey can advise you on appropriate contributions ro conception, design, acquisition of data, or
statisrical analyses and on the particulars of data colleccion analysis and interpretarion of data; 2) drafting the anide or
to ensure that you are collecting the right information to revising it critically for imporranc inrellectual contenr; and
enable a meaningful analysis. In some situations, statisti- 3) final approval of the version ro be published."3 Decisions
cians are involved as consulrants who provide advice at the abo lit rhe order of aurhorship on a final paper may change
beginning of the srudy and conducr analyses at the end. over time ro reflect rhe relarive contriburion of each invesri-
Usually this is done on a contractual basis, with professional gator during the conduct of rhe srudy and preparation of
fees ranging from $50 to $150 per hour. Your departrnent the manuscript.
or preceptor may have some discretionary funds to cover
these coses, In other situations, a sratistician becomes
Keeping data secure
involved as a member of the research tearn and contributes Chapter 22 of rhis guide provides excellenr inforrnation on
his or her services as pan of the role of co-investigator, Stat- data management frorn the perspective of facilitaring high-
isticians sometimes have a graduate srudenr conduct analy- quality and reliable analyses. In your role as PI you wil! also
ses under supervision. You can also direcdy contacr graduate be responsible for patienr confidentialiry and data securiry,
programs in statisrics, public healrh, psychology 01 orher and will need to ensure rhar everyone associated wirh rhe
disciplines aryour universiry to find graduate srudents who srudy adheres to rhe requiremenrs of your Research Ethics
might be interested in assisting you with the study analyses. Board with respect to data acquisition and storage.
In any case, ir is imporrant ro involve a statisrician early in Stringent requirements for ensuring data securiry have
the development of the srudy rather than waiting unril you evolved in recenr years in randern wirh an increasing
have collected all of rhe data. This will not only make rhe sensiriviry ro issues surrounding privacy and patient

189
conhdentialiry? When personal information rhat can Ir is very common to run into unanticipated delays in
identify individual parienrs (e.g., a healrh insurance research projects. Sorne of these-e-such as a prolonged
number) is contained in an insritutional database, ir is review time by your Research Erhics Board-will be out of
common pracrice to remove these personal idenrifiers and your control. Others will be entirely within your control,
creare a unique study identificarion number or code for each bur not necessarily any easier ro manage. A1ways factor in
patient record extracted from the darabase for the purpose of rime for delays, even if you can't anticipare exactly what
research. The code for marching the study identiíier with those delays mighr be. One strategy to prevent elevenrh-
rhe personal identifier rnust be srored in a separare, secure hour panic is ro creare a timetable for your projecr early on.
place. Ir is generally prohibired to remove data collecrion This will help you ro break rasks inro manageable parrs and
Iorrns or electronic data rhat contain participant informarion to plan around rhe c1inical and educational demands of
from an institurion. You are responsible for ensuring that all your rraining programo You may be able ro use rhe tirnera-
srudy-relared correspondence and data are archived ble ro negoriare protecred time for research al' critical poinrs
according to these requirements. during the projecr (see Table 23.2).
Managing the study finances Monitoring protocol compliance and study

Many excellent resident research projects are completed progress
using exisring deparrmental resources and do not require Once your srudy is up and running, ir is imporrant ro moni-
addirional fuuding. Sorne deparrments or faculries protor its progress on a regular basis ro identiíy problems,
vide seed funding for trainee research projects. Check with ensure rhat the protocol is being implemented correctly,
your Program Director or Research Director and precep- and ensure that the study can be completed in the rime
tor to see if funding mighr be available for your project. allotted and with the resources available. Moniroring can be ,
There are also a number of institucional, provincial 01' sirnplitied by esrablishing a routine that dcesn't necessarily
nacional funding opporrunities for which you may be eli- require a lor of your time. The frequency and intensiry of e
gible (ch. 19). The Research Services office at your univer- monitoring willlikely need to be higher al' the beginning of
sity and/or hospital will likely have information on these rhe srudy so rhat unanticipated problems are spotted eady. •
funding agencies. Your monitoring wilI cover quantirarive aspects such as the
,,_ Awards for research projecrs generally do not go directly number of enrolmenrs, surveys cornpleted or healrh records
~ ro an investigator, bur are managed rhrough an institution reviewed, but will also need ro include an assessmem of rhe '
u such as a hospital or universiry, Ir will be irnportanr for you qualiry of the data collected (e.g., laboratory resulrs, corn- ,
"'C
:s to familiarize yourself wirh rhe specihc requirements of plereness of surveys , etc.). For example, in studies involving •
each funding agency you plan to approach. For exarnple, participam recruitment, you will need to monitor rhe •
e sorne funding agencies will allow an organization ro keep number of patients enrolled per week 01' rnonrh to ensure
o any unspenr monies. Orhers require unspent monies to be rhat rhe rate is keeping pace wirh your target. If you need ro
U returned, If you are successful al' securing funding for your recruit 200 participanrs in one year bur by the end of the •
research project, it is importanr that yOL!check with your Iirst month have only enrolIed 5, rhen you will need to re-
departrnent abour the processes for establishing and rnoni- evaluare your recruitrnent srraregies, the srudy period and/ ••
toring a research account. Often a departrnenral adminis- or sample size considerarions. Similarly, you will need ro I
rrator will be able ro assist you wirh setting up an account monitor the dropout or losr ro follow-up rate for enrolled ••
and monitoring morithly staterncnts, but you must retain patients. In studies involving surveys, the response rate
oversight (see Table 23.1). needs ongoing monitoring, as does the response to individ- ••
ual questions (ro detect problems of non-response).
Managing time Every study faces porenrial threats to internal validiry, ••
Training is a demanding phase of your career, not least wirh such as selection bias, maturation (normal changes over
respecr ro rime management. Even if your longer-term time), repeared testing, regression roward me mean, and
career aspirarions do not include research, the experience in changes to the rneasurement instrumento Mosr often, mese
time management rhar you will gain during your first potenrial threars are dealt with in me design of a srudy or are
research project will serve you well in the future. controlled for during rhe analyses. A1rhough a separare chap-
190 © 2011 The Royal College 01 Physicians and Surgeons of Cenada

• Table 23.1: Sample line items for study budget
L. _ COST
PROJECTED ACTUAL VARIANCE

BUDGET ITEM
Wages (and benefits)
Administration
Inlormation Technology
Research assistantlcoordinator
Other types of compensation
Consulting lees
Research participant honoraria
Statistical consultation
-¡ I
!
Transcription services ~.__ . ~... __J_.__... ._~_~_.__
~L. ~ .1._.__ ~_ .__ ._. ~_ ._.
Supplies and general expenses
--- ---- --- --,'_.
¡;
Stationery
··---··M~~;ii-~~~--·-·-·--·-·---~~~--~--~--- -...~--.---.-.--~---.-- ------+-.- ..- ··-·-----··----·---·-1--.- ...--.--....---.. -... -----¡--- ..---.---.------.-
o.
....-...-.-.- -..-..-..- ..---.- -.---.-.----.-.-...----.-.-

..-.-..-.--.-.. . -...-i·-- --.- ...--...- ..------.--.-.---.--. ··--1'·- -- ---- -.---.-.- .- - - .. .o.
Long distance and lax costs lor data collection ,

Photocopying -.- --: - -- ---.o --- ,.-- ----.-.--.-.- ¡._. ~ -_~~_~~:~~~-~ _
..•~~~::,"idg;:, .._ - =
-_.._. ... +.__._.
_1 ~~~~ --~~l-=_~=.•.~..j_..•..•...
.. .~.
~ ::--~
.__. -_._.__[1._. .._. __ . __.__
_ ... _~.. __._ .1.._._._. ... _ _ _._ ..... ! .. . .__. .,_ .. -- - - ..---. -- --
Equipment
--- ---_----- --¡-- ---------_._-_ .. --
Telephone ___
L .__
!_.
Computer
1
I -- __ J. _i
.. ¡-.
1
Printer ! ---- -------------- .__ .. _._.~l.__ ... ... _.
---------------
___
. ._. _..__.._L ...
i i
Dissemination expenses
Travel
Mileage
Per diem lor meals
Flight
Accommodation
Other ___ • .l _
Poster production ... I- -_ ..

Manuscript preparation I i
----------- ------_--_.- .. _----------_ -
.. .-- .- ---- -- ---------- --------_. - -__ .- :_--------- ---------_._----------~ ..---------- _--
.~-_. o_o •• _. _
Review by prolessional copyeditor I

-- -.--_. --_. -_. ---- ---
.1
¡ ,
--------------
Other costs
Development 01 website lor ongoing dissemination
____
Costs lor lab tests,
o - - .0._
diagnostic imaging etc.
._.
Total expenses
-- . -- . i
ln-kind or other contributions
---, . --
TOTAL COSTS
© 2011 The Royal College of Physicians ancl Surgeons 01 Canada '191

Conductinq
._-_ ... ------- -_ "------_--- -----------. ----_--- -----------_
---l
YEAR ONE July Aug Sept Oct zr
Jan Feb Mar Apr May ro
Pre-study
---;-- -- -------------í ---- ---------;----------------1--
: Prepare REB ! Submit to ¡ Revisions
! subrnission : REB
! ¡
! as per
¡ REB
:
'R
B -¡M~~i~¡thstudy
: investigators to
----------_- -,--- --------------- ---
June
-- --------------------------,
i
•¡¡¡t
:;:o
ro
V1
ro
OJ
I
E : establish roles and n
zr
A C"
Consult with statistician i ; responsibilities
applicable) K Estabhsh routine study-
ro C\
I
N e
Identify potential , related communication w D._
funding sources : (formatltimíng) i N
ro
Develop study i VI
CJ
»
timetable : ¡ ""O
-_.···----T---------------,- _ i :
--:---·----··--~------------l---------------
3 """'
¡ ¡ ----------1------------------ -------------------------- _1 "C :3
: ' HICe and , Start data collection ¡ Data collection , ro ro
"""'
!:!'. -+.
, train Develop and initiate monitoring ¡ Monitoring: ! 3 O
study staff ' regimen • recruitment (includes response rate for surveys) """'
ro
...• """'
ro
, Set up • adherence to protocol CJ V1
: research • data qua lit y D._
5!: ro
, account • consistency of clinical and lab procedures and/ ro ::l
or assessments by multiple assessors
..•.. .--+
_V1
O
, • confidentiality ""<
O
I • CJ .--+
study budget
...• ::::r
ro
¡ Routine contact
¡ • study team
with: sO """'
::::r
ro
¡ i • preceptor
• REB (as needed)
-<ro OJ
::::r
-------L - ------J------ ..l L _ • participants (as needed)
CJ
...•
VI "
OJ
~.
. _._- -------- ------,-------------------------------------------
: YEAR TWO July Aug: Sept Oct Nov Dec Jan e """'
ro
Feb Mar Apr May June a.
--
B
- _._.----.-- --- . ----¡-------------,----------.-----_._-------
------1 '< CiJ

Monitoring: , R ::l
ro
,. recruitment (includes response rate for surveys) i E ¡ ro
e _V1
N :. adherence to protocol i A
o OJ
• data quality , K ::l
D._
1, • consistency of clinical and lab procedures and/
""O
, or assessments by multiple assessors
CiJ
1· confidentiality
:. study budget "
.--+
Cj-:
¡ Routine contact with: o
::l
i. study team
~
i. preceptor Vl
[. REB (as needed)

!. participants (as needed)
!
----------
Submit request for annual approval to REB
----1------- -----¡--------- -------1- --,--------------,
¡ Data : Prepare abstract lor presentation in -- Present study --- -----------

I
--¡-~b;;;¡¡-~i~d~--¿I~~-r~t;;-REB-~~d-a_;:Zhi~~---------¡
I analysis
!
: January Familiarize preceptor with study ! documents(or make arrangements to have it !
I I
: Synthesize results and review with 'documentation I done) !

preceptor \l\Iork with study team to ¡ ¡
.ant ma
Sta - t
.anu sc np prepare documents for "
_ archiving
Complete follow-up for parncípants Revise rnanuscript and prepare
- . --- .i '-- 1.. _ J . : .:_ íor submission
ter could be devored ro rhese conceprs, rhey are raised in rhis psychornotor performance of rhe procedure and a
secrion ro highlighr how rhe day-ro-day decisions or evenrs rnultiple-choice questionnaire designed to examine rhe
that occur during rhe irnplemenration of a study can adverse- cognitive aspecrs of rhe procedure. 111is is done on a
ly affecr its interna] validiry. The following exarnples describe monrhly basis for one yeaL
diflerenr situatioris rhat could pose a porenrial threat.
Ir is irnportanr ro documenr any decisions or events thar
• Selection. Alrhough you designed your srudy wirh mighr pose a rhreat ro rhe interna] validiry of your srudy
rigorous inclusion and exclusion criteria ro minimize and ro actively consider rheir porential irnpacr as you inter-
selecrion bias, you discover on follow-up rhat several prer your resulrs. One valuable rool to help wirh rhis is a
enrolled patienrs failed ro mention pre-exisring research diary. Virrually every invesrigaror runs inro
medical condirions rhar should have excluded rhern unforeseen circumsrances that may necessirate a change-
from paniciparing. some minor, orhers more significam. For instance, a change
• Instrumentation. You have enrolled 60 patients rnighr be made to adara collecrion form or the riming of a
wirh asrhma from a family pracrice ro participare in follow-up interview mighr have ro be altered to increase rhe
a home-based inrervenrion for self-managemenr thar yield of complered inrerviews. In the weeks and monrhs
will take place over 12 monrhs. The prorocol requires rhar follow, ir is ofren difficulr ro recall the rationale or exact
that patients monitor rheir respirarory funcrion on a circumsrances for a given change. By documenting all
daily basis using standardized, calibrated spirornerers, changes, along with the merhodological insighrs thar you
As rhe study proceeds, 6 of rhe study patients (10%) gain as rhe projecr moves along, you will have a record of
have problems with rheir spirornerers, which rnust the process and a rarionale for each specific change. When
be replaced. Unfortunately, similar spiromerers are the rime comes ro wrire a manuscripr or respond ro derailed
no longer available because the hospital has swirched questions from an editor, the diary you have kepr will be
suppliers; rhey must be replaced with models from a surprisingly helpfu!.
differenr company.
• History. You are conducting a survey of residenrs'
perceptions of the irnpact of working hours on parienr Conclusion
safery in rhe hospital, using a sarnple of residenrs from As a healrh care professional, you are poised ro ask saliem
all residency programs from 2 consecurive years. Near questions that have the poteritial ro lead ro improvemems
the end of the nrsr year, a patient dies in hospiral in patienr careo A well-plannecl and well-execured research
because of a medication error rhar is attributed to an study can help you ro answer those quesrions. Hands-on
orcler given by a residenr post-call. The family involves involvement in your own study will provicle not onlv
rhe media and rhe incidenr attracts considerable public insight into the challenges of conclucring research, bur also
arrenuon. the rhrill of crearing new knowledge or undersranding old
• Repeated measures, You are involvecl in a medical problems in new ways .•
cducarion research study examining retention of
procedural skills. The prorocol involves re-tesring
CASE POSTSCRIPT
When the resident creates a timetable, he is struck by how much needs to be done to complete the study in the time he has
available. He also realizes that the project seems less daunting and easier to schedule if he breaks it down into a series of
srnaller tasks He speaks to his senior resident, who tells him about the excellent support she has receivecl from the Research
Coordinator in the department, especially with regard to documentation for the Research Ethics Board and the archiving of
study records. His preceptor gives him the name of a statistician, who subsequently helps him with data analyses. The resident
is able to present the final study results at his departmental Research Day. He is even more surprised to realize that his project
has given him a new appreciation of the way in which research influences his thinking about clinical practice.
© 2011 The Royal College of Physicians alld Surgeons of Cenada 193

-----~ -- - ---
The Research Guide: A primer for residents, other health care uainees, and practitioners
REFERENCES
1. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Designing clinical research: an epidemiologic approach. 3rd ed.
Philadelphia (PA): Lippincott Williams & Wilkins; 2007.
2. Wolf OC, Katz S, Safdi MA, Sandler RS, Lewis JO. Site organization and management. Inflamm Bowel Dis. 2005;11 Suppl
1 S29-33.
3. International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical joutnels:
ethical considerations in the conduct and reporting of research. authorship and contribution. Philadelphia: The Committee; 2009.
Available from: www.icmje.org/ethical_lauthor.html
4. Tsao CI, Roberts LW. Authorship in scholarly manuscripts: practical considerations for resident and early career physicians. Acad
Psychiatry. 2009;33(1 ):76-9.
5. Canadian Institutes of Health Research. C1HRBest Practices for Protecting Privacy in Health Research. Ottawa (ON): The Institutes;
2005. Available from: www.cihr-irsc.gc.ca/e/documents/et_pbp_nov05_sept2005_e.pdf
Canadian Institutes of Health Research. CIHR knowledge translation publications. Available from: www.cihr.ca/e/29484.html
• This link provides a list of resources on knowledge translation activities relevant to CIHR. This has become an increasingly
important consideration for funding agencies.
Canadian Institutes of Health Research. Research in ethics: web-based tutorials [Internet] Available from: wwwcihr.ca/e/30489.html
• This link provides a list of web-based resources on research ethics, including a link to an online tutorial for the Tri-Council Policy
5tatement. Ethical Conduct for Research Involving Humans (TCPS) This tutorial has seven sections that can be completed at
your own pace. You can receive a certificate of completion once you have finished the tutorial. It takes approximately two hours
to complete.
"'
+al
.... Mclnnes R, Andrews B, Rachubinski. Guidebook for new principal investigators: advice on applying for a grant, writing papers, setting
U up a research team and managing your time. Ottawa: Canadian Institutes of Health Research. Institute of Genetics. Available from:
:s www.cihr-irsc.gc.ca/e/27491.html
'"'C • This guidebook is designed for health researchers, including basic and clinical scientists. It provides guidance on grant writing,
e managing a research team or laboratory, writing papers and time management. This is a useful and easy-to-read resource,
o particularly for those interested in pursuing further research.
EXERClSE: QUESTIONS TO CONSIDER
1. Develop a timetable for your research study. Build in additional time (at least 25%) for inevitable delays.
2. Talk to trainees nearing the end of their training to get their input on dealing with unexpected challenges,
suggestions for useful research resources at your institution, and time-management strategies.
3. Talk to your Program Director and Research Director and to research personnel in your departmen to learn ore
about the resources available to help you with your research project.
194 © 2011 The Royal College 01 Physioans and Surgec"So; ¿-~

SUMMARY CHECKLlST
Pre-study
O Develop protocol
O Consult with statistician (if applicable)
O Develop study procedures (e.g., data collection form, mechanisms for tracking progress, etc)
O Identify potential sources of funds
O Develop study timetable (plan fOI' delays)
O Ethics submission and approval
O Approval date _
O Determine roles and respnsibilities of study team
O Determine method(s) and timing of routine study-related communications (e.q., bi-weekly updates)
Start-up
O Hire and train study staff (if applicable)
O Establish research account (if applicable)
O Account number _
O Develop and initiate monitoring
Ongoing
Routinely monitor:
O Recruitment of study participants/response rate for surveys
O Adherence to protocol
O Data quality
O Consistency of clinical and laboratory procedures and/or assessments by multiple assessors
O Confidentiality
O Study budget
O Other
Maintain relevant corresponden ce with Research Ethics Board regarding.'
O Request for annual approval
O Amendments to protocol and/or consent forms
O Reports of serious adverse avents
O Study closure
Schedule routine meetings and/or contact with preceptor and study team
Post-study
O Complete follow-up for participants (e.q., communicate study results)
O Perform data analysis (with statistician if applicable)
O Review study documentation with preceptor
O Archive all study documents as per institutional requirements
© 2011 The Royal College of Pllysicians and Surgenns oí Canada 195

196 © 2011 The Royal College 01 Physicians and Surgeons 01 Cariada

24
Data analysis: Descriptive statistics
ILLUSTRATlVE CASE
A General Internal Medicine resident, completing a rotation at a community health centre, wants to determine what
proportion of people served by each of the centre's two practice groups have received the recommended preventive
services during the previous two years. After speaking to his field supervisor and deciding to carry out a review using the
centre's electronic health record database, he wonders what procedures and techniques will be needed to summarize the
data once they are collected.
• Quantitative health studies address

CHAPTER OBJECTIVES
research questions abour people and conditions by collect-
ing and analyzing data obtained by counting, observing After reading this chapter, you should be able to:
and measuring. In chapter 8 we examined the types of data • list and describe key descriptive statistics (e.g., measures
collected in quantirarive research. The present chapter is of central tendency and variability) used in health-related
the Iirsr of rwo that introduce and discuss rhe statistical quantitative research studies;
rools and techniques used ro analyze rhese data. The ana- • describe the situations and circumstances in which each
lytic techniques used in qualitative health studies are pre- of these descriptive statistics should and should not be
sented and discussed in chaprer 16. used;
The srarisrical techniques used ro analyze quantirarive • describe the characteristics of normal and skewed
data generally fall into rwo main caregories: descriptive distributions;
statistics, which, as rhe name suggesrs, enable collecred • describe the difference and the relationship between a
data ro be summarized and described; and hypothesis- sample and a population;
testing or inferential statistics, which comprise a diverse • explain how each individual sample provides a single
ser of tests and rechniques rhat enable comparisons and
other analyses ro be complered. In addirion, and relared ro
estímate of the quantity being estimated and how (and
why) the estimates arising from multiple samples are
_.
borh of rhese uses, statistics also enable a dererrnination of distributed around the "true" (population) value;
rhe precision wirh which any measure has been made. In • state what is measured by the standard error (SE)and
general, rhe rnost important role of srarisrical testing is ro which factors affect its size: and
assess whar role chance may have played in rhe collected • define the confidence interval (el) and describe what
.... dara and rhe degree ro which ir mighr be a plausible expla-

narion for rhe observed study results.
,-----------------------------------------------
KEY TERMS
Biological variation Graphical displays
information it conveys and how to use it when reporting
study results .
Outlier Skewed (non-normal)

Census Hypothesis-testing (inferential) Parametric statistical tests distribution
Central tendency statistics Percentile Standard deviation (SD)
eonfidence interval (el) Interquartile range (IQR) Population Standard error (SE)
Descriptive statistics Mean Precision Variability
Dispersion Median Random variation Variance
Frequency Mode Range
Frequency distribution Non-parametric statistical tests Sample
Gaussian distribution Normal distribution Sample size

----- .---
This chapter and the next have been writren from the What do you make of rhese results? Probably not much:
perspective of a health researcher, rather than thar of a star- it's hard ro make sense of a list of numbers, even when rhere
istician, From a researcher's perspective, statistics should be are only 40 of rhern. Almost all of us need some sort of surn-
a means ro an end, and not an end in itself This chapter mary before we can discern patterns within a set of data.
strives ro introduce statistics as a "ro 01 box" from which The subsequent sections of this chapter will, using mainly
tests and techniques may be selected and used ro complete this sample data ser, describe several graphical and numeri-
rhe needed data analyses. In working wirh any set of tools, cal statistical techniques that can be used ro describe and
one requires an understanding not only of what the avail- summarize data.
able tools are and how they should be used, but also of
which tools should be used for which tasks.
Data distributions
This chaprer discusses descriptive statistics, while the
next discusses the staristical rnethods used for hypothesis Graphical displays are frequently used to summarize and
testing, as well as rhe concepts of precision and starisrical presem data. An example is rhe frequency distribution
power, and rhe determination of a study's required sample shown in Figure 24.1, which depicts the 40 heart rates
size. However, before beginning, it is important ro reiterare obrained ar the beginning of the cardiac rehabiliration ses-
a therne that recurs throughout this guide: researchers sion.
should ensure rhat they consult with someone knowledge-
able about statistical merhods during rhe planning and Figure 24_1
A frequency distribution of participants' heart rates (n = 40)
clesign stages of rheir research projecrs! Only then can they
be confident that rhey will collect the necessary data, using
6
rhe right techniques, and then be able to analyze their find-
5
ings appropriately ro obtain a valid answer ro their research
4 ------ -- - - - e-.-e--
quesrion. G
fi¡ 3 --- - ------ ------.-----.e-.e-e-----
:J
ir
~ 2 --e- - - .- -- ••• e e-e e- - e $- - e

Descriptive statistics 1 e------- e-e---e.-e e-e-oe-e-e-e-. e-e 0-- -- e
O+-------.-------.-------~------_r---
.- Making
from multiple
sense of group data (e.g., measurements
persons, or rnulriple measurements
taken
raken
65 70 75 80 85 90
Heart rate (beats per minute)
from the same person) usually requires the use of descrip-
tive statistical techniques that summarize values into a This graphical plor depicts several characteristics of rhe
more readily understandable formo Descriprive statistics collected data. For one thing, we can see the frequency of
provide the most basic form of data sumrnarization and individual data values, or how often each value occurs in
should be rhe starting point uf any data analysis. Descrip- rhe data set (e.g., 5 participants had a heart rate of 81 bprn,
tive staristics summarize data and allow thern ro be 4 had a heart rate of 80 bpm, and 4 had a heart rate of 82
expressed in a few, easily communicated numbers. bpm).
As an example, let's consider a cardiac rehabilitarion spe- We can also see that most of the data poinrs fall between
cialist who wishes ro assess various characteristics of program 79 and 82 bpm. In statistics, rhis clustering is called the
participants, One of the healrh variables she wishes to exam- central tendency of rhe data, which means rhat the data
ine is baseline (pre-exercise) heart rate (in bears per minute, tend ro fall near the central value. We can also see the degree
bprn). In a group session atrended by 40 program parrici- ro which the data poinrs are spread. The values range from
pants, she obrains rhe following heart rate values: 65 to 90 bprn, and most cluster berween 76 and 87 bpm.
1his spread of rhe data points is called the variability or
76,81,65,86,70,79,83,87,71,80,90,77,71,84,78,77, dispersion of the data.
73,85,80,81,82,74,81,90,79,82,82,74,86,80,83,77, As illustrated in the next rwo figures (which are based on
81,81,78,83,79,85,82,80 differem sers of hypothetical data rhan our main exarnple)
we can depict differences in dispersion by graphing rhe fre-

24 Data analysis: Descriptive statistics
quency disrriburion of rhe data. In Figure 24.2, values are (i.e., coun ) o values. In our hean rate example, the mean
righdy clustered around 80 bprn, with few values lower hearr rate is 79. bpm: the rotal of rhe individual values
rhan 75 bprn or higher rhan 85 bprn. In conrrast, Figure (3193) divided by the number of values (40). Depending
24.3 depicts a wider range of values; the graph includes on rhe conrexr, we may round up such a result and repon ir
hearr rates lower rhan 60 bprn and higher rhan 100 bpm. more sirnply as 80 bpm. 1his is oíten referred to as rhe arith-
rnetic mean because rhere are other means (e.g., geomerric
Figure 24.2
mean, harmonic mean) thar are used in special situations;
A "narrow" frequency distribution of participants' heart rates
rhese are beyond the scope of chis chaprer.
The median (abbreviated Mdn) is the middle valué of a
sequential (i.e., ordered) ser of data, or its 50rh percen-
tile-rhe value rhar divides the distriburion into an upper
and lower half (i.e., rhe value at which half of the data poi n ts
are grearer rhan rhe median val ue and half are less). When
we have an odd nurnber of sequential data points, rhe mid-
die value is easy ro find; however, when we luye an even
SS 60 65 70 75 80 85 90 95 100 105 number of data points, as in our example of 40 heart rates,
there is no single middle value. In this case, the median is
Figure 24.3 the average of rhe rwo values closesr ro rhe middle of the
A "wide" frequency distribution of participants' heart rates
sequencial set of data: these rwo values would be rhe 20rh
(80 bpm) and Zl st (81 bprn). By averaging rhese [WO val-
ues, we obtain the median value of our data ser: 80.5 bprn.
ee 657071 71 7374747677 77 77 78 78 79 79 79 80 80 80 80
(j)
::J
rr
81818181 ...
1:
u,
The rnode, which doesnt have an abbreviation, is the

rnosr frequendy occurring value in a ser of data. In rhis
SS 60 65 70 75 80 85 90 95 100 105
study ofhearr rates, the mode is 81 bprn, a value reponed
by 5 participants. 1he mode is rarely used ro describe bio-
medical data, alrhough it is useful for reponing disrribu-
rions that have rwo or more peaks (e.g., a bimodal
When describing data samples, researchers (and aurhors) distribution). each ofwhich is indicared by its mode. Figure
should report borh of rhe fol!owing: 24.4 depicts a bimodal disrriburion of hearr rares, with
modes at 60 bpm and 80 bprn.
• a measure of central tendency
• a measure of variability (also called dispersion or spread) Figure 24.4
A bimodal distribution of participants' heart rates
14
Measures of central tendency
12
The centre of a ser of data is usually reponed as one or more 10
of rhree descriprive sratistics: mean, median, and mode. C
e 8
T·~··········
(j)
The besr way of describing the centre of a particular set of g. 6

data depends on how rhe dara points are disrribured, as dis-
cussed later in this chaprer.
The (arithrnetic) mean (ofren abbreviated as x or as M)
,t 4
O
SO
•...)
55
...
60
"'.~
65 70 75 80 85 90 95
is the average value and is calculated by adding al! of rhe
individual values and dividing this toral by rhe number

· .,
------------- --------------
Measures of variability (dispersion) thar me 75th percentile (berween the 29th and 30th value
1he variabílity of the values wirhin a set of data is usually is 83 bpm. As described earlier, we also find rhat rhe 50r
indicated by one or more of rhree descriptive sraristics: percenrile (the median, berween the 20th and 21st dar
range, interquartile range, and standard deviation. poinrs) is 80.5 bpm.
1he range is rhe difference between the lowest and high-
Figure 24.5
est values in adara set. However, because the range is
Quartiles of a distribution (n = 40; 1 quartile = 10 data points)
defined by the rwo most extreme values, ir can sornetimes
be misleading. For example, if a few individuals in our 6 -----.------.---¡-------, -----.---
exarnple had extrernely low (and/or high) hearr rares, the 5 · ~?~t!:J~·~~~~ e---:-~~Js-t~~~
percentile : : percentile
range would be large, even though only one or rwo values
c-, 4 -----------f-----e-e-.+----------
were responsible. In sciemific publications, the range is usu- u "
~ 3
:J -·-----·---------.--e-e-e- •• ---------
ally written as an "inrerval" in which the minimum value is ir
1':' 2
u.
---·-----e---e----e e-e-e e-e e--..e e------e
"
followed by the maximum value, such that the scale of the
1 -----ee-e-e-e-. e-.-e-e-.' .-e-e e -- e
data is obvious. For example, data values from 10 to 15 have
a range of 5, but so do values from 95 to 100. Instead of 65 70 75 80 85 90
reporring the range as 5, we provide the minimum and Heart rate (beats per minute)
maximum values so that the reader will know where those 5

values are locared on the rneasurernent scale. 1he standard deviation (abbreviated as either SD or s:
The Interquartlle range (IQR) is the range between rhe describes the average "distance" between each data poirn
datas first quartile (25th percentile) and third quartile (75th and rhe mean value of the distribution. We'll illusrrare what
percentile)--the middle 50% of data values. As the name -rhis means in a momento 111eSD (and the mean) should be
implies, a data set can be divided into four quartilcs, each of used and reported only when rhe data are normally distrib-
which contains one-íourth of the values in a sequentiallist ured (or nearly so). 1his will be discussed with an example
of data points. 1he IQR is composed of the rwo middle shorrly,
quartiles of a given data ser. 1he interquartile range is ofren The process used to calculate the standard deviarion for a
reponed wirh rhe median. As is rhe case wirh the range, we data set such as our sample heart rates can be illusrrated
.- usually repon rhe inrerquartile inrerval rather than simply

the difference berween the 25th and 75th perceneiles. For
with five steps. (Calculating rhe SD is fairly simple when
the number of data values is small, as in our example. How-
example: ever, because health srudies can involve hundreds or thou-
sands of parricipants, ir is necessary to use cornputerized
Length of hospital stay (days): median, 6; interquartile range, staristical programs (e.g., Srara, SAS, SPSS) or web-based
1 to 22. calculators that quickly and accurately process the applica-
ble formulas and calculations.)
1his example indicates several characterisrics of rhis dis-
tribution, as follows: (1) for at least 25% of rhe parients the l. Calculare the mean by adding rhe 40 heart rate values
hospital stay was 1 day or less; (2) for at least 25% it was 22 and dividing rhe total by 40 (i.e., 3193 -i- 40 = 79.825
days or longer; (3) for another 25% ir was berween 1 day bpm).
and 6 days; and (4) for the remaining 25% it was berween 6 2. Determine the disrance of eaeh data value from the
and 22 days. mean by subtracting the mean frorn ea eh of the 40
1he IQR is preferred to rhe range because it is less affect- heart rates (e.g., 90 bprn minus 79.825 = 10.175; 70
ed by extreme values. bprn minus 79.825 = -9.825).
111e frequency distriburion shown in Figure 24.5 illus- 3. Square each of these 40 calculared clifferences (dis-
trates another way of describing the data ser of 40 heart canees) from rhe mean, which converrs the negative
rates in our earlier example. 1he 40 sequential values can be values to positive values (e.g., 10.1752 = 103.53;
divided into quartiles conraining 10 values each. By count- -9.8252 = 96.53). (Note: If we don'r perform rhis
ing rhe data values from rhe lefr, we find thar rhe 25th per- step, then when we add rhese 40 differences, as we
cenrile (berween rhe l Orh and 11rh value) is 77 bprn, and will do in the next step, we will obtain a total of zero

24 Data analysis. Descriptive statistics
beca use all of rhe posirive differences will be balanced Gaussian distribution in recognirion of Johann Karl
exacdy by the negarive differences.) Friedrich Gauss, che German marhernarician who rediscov-
4. Add rhese 40 squared differences (this yields a nurn- ered ir in 1820 while graphing the variabiliry in planerary
ber called rhe sum of square differences) and divide orbirs. 1he bell-shaped curve has several characreristic
rhe rotal by rhe number of values minus 1 (11 - 1, propertles:
or 39 in rhis exarnple) ro obtain rhe average squared
disrance from the mean: 28.1. This number is known 1. 1he mean, median, and mode are equal in value.
as rhe variance, The variance is rarely reponed as a 2. 1he frequency distribution is symmetrical about irs
numerical value; we use the terrn ro describe rhe dis- centre (rhe mean).
persion of data po ints, as in the sentence, "The rwo 3. As depicred by Figure 24.6 below, the area under rhe
disrriburions exhibired differenr variances." (The sum curve can be precisely defined by rhe mean and stan-
of rhe squares is divided by n - 1 rarher than by n dard deviaricn, as follows:
because rhis slighr increase in rhe average "distance" • abour 68% of rhe data points fall within plus or
provides a more accurate estimare of rhe "underlying" minus 1 SO of the mean
dara variability; in this case, of all cardiac rehabilita- • abour 95% of the dara poinrs fal! wirhin plus or
rion parricipants-nor jusr of rhe 40 srudied here.) minus 2 SO of rhe mean
5. Finally, n.nd rhe square root of rhe variance (the aver- • abour 99.7% of the dara points fall within plus or
age squared difference) ro derermine rhe standard de- minus 3 SO of the mean
viarion: 5.3. Calcularing rhe square root compensares
for our squaring the differences in srep 3 and provides Some biological data fall into normal distriburions. For
us wirh a measure of data variabiliry rhat is in the example, blood pressure, hearr rate, body remperature, and red
same units as the actual data: in rhis example, hearr or white blood cel! counrs all rend ro be normally disrribured.
rate in bpm.
As discussed aboye, the standard deviarion indicares the Figure 24.6

arnount of variabiliry wirhin a ser of dara. In our example, Areas under the curve in a normal distribution
the SO of 5.3 bprn suggesrs rhat the data poinrs are, on

average, abour 5 bpm away from the mean value. The SO
can also be used ro compare disrriburions. For exarnple, a
distribution of hearr rares wirh an SO of 2.0 would be less
dispersed (i.e., narrower) rhan our distriburion, which has
~68% ol-+!
an SO of 5.3 (see, for example, Figs 24.2 and 24.3). As we : data points :
will discuss in rhe nexr secrion, perhaps rhe greatest value of
-3 SO -2 SO -1 SO mean +1 SO +2 SO +3 SO
the SO is the facr rhat ir enables us ro determine what pro-
+- 95%01data points ----+
portien of values are included wirhin certain porrions of a +-----99.7% of data points ----"+
normal disrriburion.
Skewed distributions
In contrast ro normal disrributions, skewed distributions
Types of distributions
are asymmerrical. This asymmcrry enrails importanr con-
Alrhough many rypes of dara distributions exisr, the fol- siderarions in communicaring rhese distributions and in
lowing sections wil! discuss in detail only rwo basic ones: selecring and performing rhe various descriprive statisrical
normal and skewed disrriburions. analyses. To illusrrate, ler's rerurn ro our hearr rate exarnple.
Imagine rhar, at the next rehab session, 8 parricipanrs who
Normal distributions had been absent from rhe previous session in order ro corn-
In view of irs unique properries and widespread use, ir is pere in a local road race ask ro be induded in rhe rneasure-
imporrant ro undersrand rhe normal (or "bell-shaped") dis- menr of hean rates. The hearr rates of rhese adclirional
rriburion. The normal distribution is ofren called rhe parricipanrs were 48,55,60,63,64,65,66, and 67 bpm.
© 2011 The Royal College 01Physicians and Surgeons 01Canacla 201

Figure 24.7 shows the data set for the 48 parricipants. Figure 24.7
A skewed distribution of participants' heart rates (n = 48;
The heart rates of most of rhe additional 8 parricipants, the 8 additional heart rates are included within the bracket
who are very fit runners, are lower than the heart rares of on the left side 01 the distribution)
the original 40 participants. Because each of these 8 addi-

rional hearr rates is lower than 70 bpm, the lower portien of
the original distriburion has been further extended (i.e., the », 4 -.--.-----------.-----.--.---------._.---------
v
e
distribution is now skewed ro the left, in what is called a ~ 3 -------------------------.-.- ..--- -.- •• __ .-------
.._----.---.-_.- .........._..--.
ir
left-skewed or negatively skewed disrriburion). Data values l': 2 ------------.---.--.--
u, ••• _••••• ----.• -----
so extreme that they don'r appear to be pan of the rest of the _ .. - - - ----
distribution are called outlying values or outliers. The

45 50 55 60 65 70 75 80 85 90 9
skewness of the disrribution exerrs marked effects on the
measures of central tendency and dispersion.
What happens ro rhe mean value when rhese S addition- Figure 24.8
al heart rates are added? Will ir increase, decrease or remain A relatively normal distribution of participants' heart rates
(n = 40)
the same?
If we perform rhe appropriate
the mean has decreased
calculations, we find that
from 79.8 bpm ro 76.7 bprn.
6 - . __~_ea~rl:~dian __ =~-~~
>- 4 . . .. ._.__. .__ _._ •..• .__ .. _.
Examining the rwo frequency distributions (Figs 24.7 and
~ 3 -. 8 ••••
24.8) shows us that rhe mean of 79.8 bpm reasonably :J
cr
reflecrs the centre of [he original distribucion of 40 heart
~ 2 --e-- -. -. --e-e-.-e- •• e----.-e-- -
-- -- -- •
1 e- --.----.. e e ... e •..•• -•• e •• e-e e-o•• -$ - -.
rares; however, because this mean has shifted ro the ourer
edge of rhe main cluster of hearr rates, the revised mean of
o+----.----.----,------,---~
65 70 75 80 85 90
76.7 bprn poorly reílects the centre of the extended group,
which I!OW inclucles the toad racers.
1nis shiEr has occurred because rhe mean is srrongly age distance of all data points from the mean (and thus
iníiucnced by extreme values, namely, rhe 8 additional increase the value of the SD).
.- heart rare values from rhe more physically fir road racers.
Also, what happcns ro rhe median when rhese 8 addi-
In contrast, as Figure 24.9 illustrates, the interquarrile
range (IQR), which is based on rhe rank-ordering of rhe
rional hearr rates are included? Will ir increase, decrease or data and not on the acrual values, continues to accurarely
remain rhe same? reflecr rhe actual variability of the data. Notice also rhar the
The median is always less sensirive ro extreme values standard deviation doesn't indicare on which side of rhe dis-
than rhe mean. As a resulr, adding the 8 addirionallower rribution the outliers are found. In contrasr, rhe interquar-
heart rates shifts rhe value of rhe median only slighdy, Erom tile range shows rhar rhe addition of the 8 mararhoners'
80.5 bpm to 79.5 bpm. Unlike the corresponding mean of hearr rates has skewed rhe distriburion ro the lefr, and rhar
76.7 bprn, rhe new median value of79.5 continues ro rea- rhe 25rh percenrile has shifred from 77 bpm (3.5 bprn below •
sonably reHecr rhe centre of rhe data, remaining within the rhe median) ro 72 bprn (7.5 bpm below rhe median). How-
main cluster ofhean rate values (see Fig. 24.7). ever, the distance of the 75th percentile from the median is
As nored earlier, a skewed distriburion wil! also affect rhe unchanged: in borh distributions, rhe 75rh pcrcenrile is 2.5
measures of dispersion. For example, although the standard bpm higher rhan the median. We would expect that this dis- I
dcviation of the original 40 heart rares is 5.3 bpm, ir tance berween the median and rhe 75rh percenrile would be •
increases subsranrially ro 8.9 bpm when the 8 addirional the same in borh sers of data, given rhat all of rhe additional
lower heart ratcs are included. This in crease occurs in pan 8 heart rares were lower values and no new values were add- •
because rhe higher dara values are now farther from the ed ro rhe upper portion of (he distriburion. ~
mean, which has decreased by more rhan 3 bprn. The Many types ofbiomedical data fall inro skewed di rrib -
increase in the SD also occurs in pan because many of rhe tions. For exarnple, laborarory values for which me clini- i
addirional S values have Eunher extended rhe lower portion cally normal (i.e., expected) range lies near zero (e.g., serum
of rhe disrribution; as a resulr, rhese values are also farrher bilirubin or urinary prorein concentrarions) generally pro-
Erom rhe mean. These increased distances in crease rhe aver- duce skewed distributions because abnormal values can be
202 © 2011 The Royal College 01 Physicians and Surgeons o Cera::;;

24 Data analysis. Descriptive statistics
In skewed disrriburions, the mean and rhe median will

Figure 24.9
Quartiles of a skewcd distribution (11 = 48;
differ from one anorher, beca use the mean is drawn in the
1 quartile = 12 data points) direcrion o (he skew. Unlortunately, even when distribu-
tions are obviously skewed, many research publications do
6 72 i5 the 25th: : 82 is the 75th
.: percentile
not appropriately provide median values and are even less
5 percentile .:
••• likely ro indude che inrerquarrile range. In such circum-
.
¡;-4
••••••
~ 3
:J
f2 .:. . stances, rhe distriburion

ed mean and· standard
of the values implied by the repon-
deviation can srill be quite
• • • _ ••• :••••••••
,
~.
'
o inforrnarive, especially when certain values are implausible
O
95
(or even irnpossible).
45 50 55 60 65 70' 75 80' 85 90
For example, ler's imagine that a repon indica tes rhat the
mean size of a particular rype of turnour (n = 20) ar the ti me
quite high on one side of the distribution but can never be of diagnosis is 40.1 mm, with a standard deviation of 42.0
less rhan zero on the other side. Other examples of health mm. Are these descriprive sratistics consistent with a nor-
care data that might be expected ro produce skewed distri- mal, or a skewed, distribution of values?
butions indude birth weighr and number of days in a hos- The distribution rnust be skewed. If ir were a normal
distribution, we would expect that 68% of the data points
pital stay.
The mean and the standard deviation should be used would be within plus or minus 1 standard deviation of rhe
mean (i.e., 34% of the data points would fall within an
only ro describe data rhat are normally distributed (or nearly
area 1 SO aboye the mean, and 34% would fall wirhin an
so). As illustrated by the heart rate example, rhe more skewed
area 1 SO below the mean). In this example, however,
rhe distribution, the more misleading the mean and SO will
be in describing rhe central tendency and the variabiliry of a subtracting the SO (42.0 mm) from the mean (40.1 mm)
set of data. In fact, the use of these (\NO statistics, by definí- yields an impossible turnour diarneter of-1.9 mm! There-
tion, implies rhat the distribution is generally bell-shaped fore, this distribución rnust be positively skewed, with
and is centred at the mean, 68% of the data values being some large turnours pulling rhe mean ro rhe right. In such
wirhin plus or minus 1 SO from the mean, 95% within plus a case, we would expect that rhe median tumour size
or minus 2 SOs from the mean, and more than 99% wirhin would be less than the reponed mean tumour size (40.1
mm). Looking below at the acrual list of rumour sizes (in
plus or minus 3 SOs from the mean. A review of rhe fre-
quency distribution (Fig. 24.9) presented aboye shows rhat mm), the median tumour size can be seen ro be 17 mm
rhe disuibution of 48 heart rates, induding rhose of the road (rhe average of 16 mm and 18 mm, the l Oth and 11 th
racers, is not normal/bell-shaped. Skewed distributions such data points):

as rhis one are best described by the median and the inter-
910 11121213 141415161821214555758095
quartile range. Because most biological data are not normal-
ly distribured, rhe median and inrerquartile range should 105 160
appear more commonly in rhe biomedical literature than
In sorne instances we can also recognize a normal or a
rhe mean and standard deviarion.
skewed disrribution sirnply by considering whar is being
measured. For example, what kind of distribution would
Recognizing skewed distributions. As stated earlier, ir is
you expect ro arise from a cornmuniry health survey ques-
imporranr ro recognize skewed (non-normal) distribu-
tion thar asks respondenrs how many cigarettes they
rions. One way of doing so is ro visually inspect a graphical
plot of the data. Unlortunarely, research publications usu- smoked during the previous day?
Because the prevalence of smoking is rnuch lower than
ally do not provide this; instead, rhey provide only mea-
50% in many populations, we should expect rhat rhe most
sures of central tendency and variability. However, by
frequendy reponed answer will be zero. If most of rhe
understanding the meaning conveyed by these mensures,
we can determine wherher a disrribution is normal or respondenrs ro our survey are non-smokers, then we should
expecr rhe distribution of the data ro be skewed ro rhe right
skewed.
(positively skewed). Most of the values will be zero, and rhe
© 2011 The Royal College of Physicians and Surgeons al Canada

203
positive values will reflect the reported smoking habirs of usually because rhey don'r understand mese concepts. Sum-
the minoriry of respondents who are smokers. marizing skewed disrriburions with the mean and the SO is a
Figure 24.10 depicts such a distribution, arising from me very common error associated with statistical reporting in the
responses of 1000 surveyed individuals. In this example, health research literature. After all, if rhe mean and standard
would you expecr che mean, or che median, ro be larger? deviation for me smoking daca depicted in Figure 24.10 (6.2
and 12.4, respectively) were reponed, this would imply that
Figure 24.10
che data follow a distribution similar ro me one shown in Fig-
A skewed frequeney distribution (n = 1000)
ure 24.11, which rhey certainly do notl 1his furrher ernpha-
800
sizes why rhe median and interquartile range (O and 0-5,
700
600
respectively) should be used insread to describe mese data.
>,
u
e 500
Q)
::l
tr 400 Figure 24.11
2
u.. 300 A normal distribution (mean = 6.2, SD = 12.4)
200
100
O
O 10 20 30 40 50 60 70 80
Smoking history (cigarettes per day)
In this case, we would expect (he mean to be larger than

the median. If we are correcr in our assumption that more
than half of rhe survey respondents are non-smokers, rhe
median will be zero. The mean will be larger than zero
beeause of the minority of resporidenrs who repon having -30 -20 -10 O 10 20 30 40
smoked some nurnber of cigarettes during the previous day.
1he distriburion of data also determines which staristical

Descriptive statistics, distributions,
tests should be used for their analysis. Parametric statisti-
reporting, and statistical testing
cal tests should be used only ro analyze cominuous data
In keeping with the goal of accurate and informative health rhat are normally distributed (or at least nearly so). For con-
research reporring, the distribution of rhe underlying data tinuous data that are not normalIy distributed, rhe corre-
should guide rhe selection of the staristics and statistical sponding non-parametric statistical tests should be used.
tests used to summarize and analyze the data. For example, These rypes of tests are discussed in more derail in chaprer
we could describe thc inicial ser uf 40 heart rares as folIows: 25.
"1ne mean (SO) heart rate was 79.8 (5.3) bpm." In con-
trasr, we would most accurarely describe rhe expanded ser
Descriptive statistics for other types
of 48 heart rates, which produced a skewed distriburion,
of data
with the folIowing senrence: "The median heart rate was
79.5 bpm (IQR, 72 ro 82 bprn)." So far, we have discussed descriptive statistical techniques
Of course, the original set of 40 heart rates could also be for continuous data (e.g., heart rates). Before we conclude
describecl as follows: "The median hearr rate was 80.5 bpm this portien of the chaprer, however, we should briefiy con-
(lQR, 77 ro 83 bprn)." However, bccause the original set of sidcr sratisrical concepts speciíic ro other rrpes of data (e.g.,
40 hearr rates produced a reasonably normal distribution, it is nominal, dichoromous, ordinal, and discrere data). These
most appropriarely described using the more frequendy used rypes of data are also discussed in chaprer 8.
mean and standard deviation. Reporting the data by using rhe
mean and me SO shoulJ imply rhar the data are normally Nominal data
disrributed (or at least nearly so). However, not all authors Nominal data fall inro caregories thar are named bur have
adhere ro rhese principIes for reporting descriptive statistics, no inherent order. These data are usually summarized as
204 © 2011 The Royal College 01 Physicians and Surgeons o' Canaca
simple counts or as proportions or percentages of rhe For exarnple, are we sure rhat rhe disrance berween 5
enrire sample. For example, a srudy of rhe freguency of (excellenr) and 4 (very good) is rhe same as the disrance
ABO blood rypes being used for rhe 415 rransfusions berween 1 (poor) and 2 (fair)? If not, ir may make more
adminisrered in a hospiral lasr year mighr repon that 190 sense ro describe the data by using the median, as follows:
(46%) of the 415 were rype O, 166 (40%) were rype A, 35 "The median response was 4 (very good) on a 5-poinr scale
(8%) were rype B and 24 (6%) were rype AB. Graphically, ranging from 1 (poor) ro 5 (excellent)." Perhaps ir would
these data can be depicted with a box, dot or pie charr thar be even berrer ro present these data as caregories: "Of the
displays the proportion represented by each nominal ca re- respondents, 30% reponed rhe placemenr site as excellenr,
gory, or wirh a bar chart that uses bar heighr ro indicare rhe 40% as very good, 15% as good, 9% as fair, and 6% as
size of each nominal category (as eirher a count or percent- peor." Graphically, ordinal data can be illusrrated with dor,
age). Figure 24.12 depicrs these data in a pie charr. pie or bar charts. Figure 24.13 depicrs our example wirh a
bar charr.
Figure 24.12
A pie chart of ABO blood types used in hospital transfusions Figure 24.13
A bar chart of trainee evaluations of their experience
at a placement site
AB
45
B
40
35
30
<J!.
o ¡;.
e
25
(l)
::J 20
cr
e
u, 15
A lO
5
O
Excellent Very good Good Fair Poor
Like nominal data, dichotornous data are usually surn- Discrete data
marized with simple counrs or percenrages of each of rhe Discrere data are nurnerical counrs (i.e., whole numbers)
rwo caregories. They are ofren graphically depicred with on a scale with known and equal disrances berween each
simple, 2-component box, dor, pie or bar chart. value. Examples of discrere data include rhe number of
Ordinal data
stairs climbed during a l2-minure exercise test, the num-
ber of members in a family, and rhe number of decayed,
_.
Ordinal data fall inro inherendy ordered categories. As missing 01" filled reerh. Because they represenr numeric
such, although ordinal data may be summarized with the counrs, discrere data are ofren summarized with measures
same approaches used for nominal data, rhey can also be of central rendency (e.g., mean, median, and mode) and
subjecred ro analyses that take into accounr the additional dispersion (range, standard deviarion, and interquarrile
information provided by the rank order of the categories. range).
As an example, ler's imagine that a placemenr site evalu- Oepending on the range of values, discrere data may be
arion form conrains rhe following question: "Overall, how depicted graphically as a freguency distriburion or as a bar,
would you rate your experience at rhis placemenr sire?" dot or pie chart. However, as is true in every circumsrance,
Trainees answer by selecring an option on a 5-poinr scale rhe choice of summary rechnique should be guided by rhe
ranging from 1 (poor) ro 5 (excellent). BecJuse rhe ordered researcher's purpose and rhe needs of rhe inrended audi-
categories are numbered, we could summarize rhe data ence. For exarnple, alrhough the lengrh of time in hospital
with rhe measures of central rendency (e.g., mean, 3.7) (in days) can be summarized by calculating a mean, a value
and dispersion (SO, 0.4) thar are used ro describe continu- such as 4.8 might be oflimited use tO rhe hospiral leader-
ous data. However, doing so carries the irnplicir assump- ship team who are trying ro undersrand the length of hos-
tion thar health has been measured on a cominuous scale pital admissions and how they mighr be shorrened. The
from 1 ro 5. 15 that a good assumprion? mean of 4.8 days cerrainly doesn't accurarely represenr any
© 2011 The Royal College 01 Physicians ancl Surgeons 01 Canacla 205

specitic hospital stay! 1hese data may be more useful for To begin ro answer rhis question, we must recognize that
health planning purposes if they are depicted with a chart any given sample is simply one of many possible samples
il!ustrating thar 40% ofhospital admissions are less rhan 2 mar could have been studied. If we had measured heart
days long, 45% are 2 ro 5 days, 10% are 5 ro 14 days, 4% rares a day earlier, or a day larer, we mighr have recruited a
are 14 ro 30 days, and 1% are longer than 30 days. slighrly different sample of cardiac rehabilitation program
parricipanrs from the same underlying population. There-
fore, ir rnakes sense ro think of each of rhese possible sarn-
Descriptive statistics summary
pIes as simply providing an estimare of the "rrue" underlying
Numerical and graphical rnethods of summarizing data are population mean. For example, measuremenrs of rhe heart
importanr because they provide effective means of cornrnu- rates of 5 other samples of cardiac rehabilitation program
nicating and idemifYing patrerns in data. In fact, because participants could have resulted in the following mean I
most of us can'r make sense of more than a few iterns of heart rates: I
data at a time, summarizing data wirh descriptive statistics 80.1 bpm

and graphical techniques enhances our abiliry ro recognize 78.9 bpm
underlying parrerns in the data. Furrhermore, we must 81.0 bpm
remember rhat the selection of descriptive statistics that are 79.5 bpm
used (O summarize data should be guided by rhe rype and 81.4 bpm
disrribution of those dara.
However, the sumrnarized data usually describe only a Why should G samples produce G different, albeit sirni- ,
sample of persons, not the entire population of interest.
Therefore, statistics obtained from samples should be con-
siderecl simply as estimares of rhe true values íound in the
lar, rneans? The answer is random or biological
We expect rhat rhe heart rare of any person will vary from
mornent to moment and thar the variation
variation.
in hcart rate
,
e
population. This concepr of estimarion will be discussed in from one person to anorher will be even larger. 'Iherefore, I
the next secrion. even under stringent study procedures, the mean heart rate •
of each sample wil! vary according ro precisely when each •
person's heart rate is measured and exacdy which persons •
Measuring and interpreting data
.-
~
variability
are selected for and participare in the study, Although
mean heart rate of each sample is a reasonable, independem
rhe ~
u In rhe previous secrion, we discussed how a set of data can estímate of the "true" underlying population mean, rhese •
::s
""C
be surnmarized with descriprive statistics such as the mean variations-borh within and berween the individuals srud- •
and rhe standard deviarion or median and interquarrile ied-cause sarnple means ro differ from one another, ti!
range. However, given rhe pracrical problems associated With this fact in rnind, which of rhese many possible
'-=
o wirh measuring large populations, ir is rarely possible for a sample means wil! be closest ro rhc "true" underlying popu-
u dara ser to indude
of interesr.
observarions from every eligible person
(If every eligible person were induded, we
lation mean? If we consider that rhe means of all of rhese
samples differ from one another only because of random •
would have a census.) More realisrically, researchers select a (chance) variation, and thar the mean of each sarnple is a •
sample of individuals who, ideally, are representative of the reasonable, independent estimare of the "rrue" underlying •
entire population of interesr, obrain data from thern, and population mean, then rhe "centre" of these many sample •
rhen "generalize" rhe resulrs from rhis representative sarn- means (i.e., the mean of the means) should be the besr esti-
ple ro the entire popularion of interesr. Consider rhat a sin- mate of rhe "rrue" underlying population value. An illustra- 111
gle raste of wine can sell rhe botrle because the taste is tive frequency distribution (Fig. 24.14), crea red by plorring •
reprcsenrative of rhe enrire bortle. mulriple independent sample mcans, is shown below, •
In our heart rate study, for example, we measured me heart This frequency distribution illusrrates several conceptS.
rates of a sample of 40 (or 48) participants in a cardiac reha- First, this group of sample means is clearly "cenrred" ar O
bilirarion program and calculated me mean. But this is only bpm. 1bis finding indicares rhat rhe means of rnosr of our
me average heart rate of this particular sample. How dose is samples are close ro 80 bpm and suggesrs that 80 bpm is rhe •
this sample mean ro me true mean heart rate of rhe emire underlying "rrue" mean heart rare of rhe popularion. e-
population of cardiac rehabilitación program parricipants? ond, rhis distribution exhibirs less variation (i.e., rhe values
206 © 2011 The Royal College of Physicians and Surgeo s 01Ca~¿:l:

24 Data analysis.·Descriptive statistics
are closer ro the centre) than does rhat of any of the indi- SE = SO of the sample
vidual samples of heart rate distributions (for an example, -J sample size
see Figure 24.15). The reason for rhis reduced variation is
that calculating a mean "averages out" individual extreme To illusrrare, ler's consider our hean rate exarnple, in
values, rhereby causing the mean of a sample ro be (by deíi- which 40 cardiac rehabiliration program participants had a
niriori) less extreme rhan the individual values within thar mean heart rare of 80 bprn with an standard deviation of
sarnple. This fact explains why the means of samples are 5.3 bprn. Whar is rhe esrimared SE for this sample? To
clusrered more closely around the population mean rhan begin, we must determine the square rcot of rhe sample
are rhe individual heart rares of any given sample around irs size. Using a calcularor wirh the square roor íunction, we
sample mean. find rhat the square root of 40 is 6.3. Dividing 6.3 inro 5.3
Figure 24.14
bprn, rhe SD of rhe sample, yields a result of 0.8 bprn, rhe
Frequency distribution of sample heart rate means esrimated SE of the population. This example furrher illus-
(the 6 sample means are indicated by arrows)
trates rhar we should expect much less variabiliry among
sample means (i.e., rhe SE) rhan among the individual data
values in any single sample (i.e., the SD).
Like rhe standard deviation, rhe standard error provides
an estimare of variabiliry-in this case, the variabiliry
expected among the means obtained from various sarnples.
Ht And, as is true for rhe SD, perhaps the greatest value of rhe
78 77 78 79 80 81 82 83 84 SE is rhe fact thar ir enables us ro determine rhe proportion
Heart rate (beats per minute) of sample mean values that wil! be expected ro fall within
certain portions of the normal distribution (e.g., 68% of all
Figure 24.15 possible sample mean values will occur within plus or
Frequency distribution of participants' heart rates minus 1 SE of the "rrue" [population] mean value),
One way of expressing the precision with which a mean
has been estimated is ro repon the mean and the corre-
sponding standard error, as follows: "The estimared mean
(SE) heart rare of the popularion is 80 (0.8) bprn." Howev-
er, in rhe health sciences, the SE is nor rhe preferred way of
reporting the precision of an estimare (alrhough ir is corn-
monly used as a descriptive statistic in basic science jour-
62 64 66 68 70 71 72 74 76 78 80 82 84 86 88 90 92 94 nals). Journals and health scientisrs prefer thar aurhors

report the mean plus or minus approximately 2 SEs; this
interval is called rhe 95% confidence interval (el).
Standard error
Confidence intervals
We can estimare the variabiliry among this hypotherical The characteristics of the normal distribution allow us ro
group of sample means just as we dererrnined the standard determine confidence intervals, Like the individual data
deviation for a sarnple of individual measures. Such an esti- poinrs in a sample, 95% of al! possible sample means wil!
mare of rhe variabiliry among sample means is called the fal! within approximately 2 SEs (or more precisely, l.%
standard error (SE) of rhe mean. Of COLtrSe,the SE would SEs) on eirher side of rhe true mean (see Figure 24.16).
nor be very useíul if we needed to conduct multiple studies Therefore, the range encompassed by l.96 SEs aboye and
ro derive an estimare ofit. In realiry, a single study provides below the mean of any individual sample has a 95% proba-
enough information ro allow us ro estimare rhe standard biliry of including rhe "true" (population) mean (i.e., the
error of che mean. To do so, the SD of che sample is divided mean of rhe means). Thar is ro say, only sarnple means that
by the square roer of the sample size: lie at rhe extreme ends of rhe hyporhetical normal distribu-
tion of mulriple sample means would be more than 1.96

The Research Guide: A primer for residents, other health care trainees. and practitioners
SEs away from the "true" mean. This interval, which cap- Ase narrow, or wide, confidence intervals preferable?
tures the true mean wirh a probabiliry of 95%, is known as Narrow confidence intervals are preferable beca use rhej
a 95% confidence interval. (Probabiliry is a conrinuous reflecr a more precise estimare, given that the widrh of rhe
numerical quantiry, ranging from zero ro 100%, indicaring el indicares the range of values consisrent wirh a given esti-
the chance rhat an event wiIl occur; it is determined by mare. The "homogeneity" of a Cl's upper and lower values/
dividing the number of times an event is observed by rhe lirnits is also important, A homogeneous el indicates a
total nurnber of times the event could have occurred. For consistent result (i.e., the upper and lower values borh lead
example, if out of 10000 children born in a province, 5100 ro the same general condusion), irrespective of rhe size of
are boys, then rhe probabiliry of a boy is 0.51.) rhe Pvalue and the widrh of the intervalo
If narrow Cls indicare a more precise estimare, how can
Figure 24.16 the widrh of the el be reduced?
Area under the curve of the distribution of sample means
Remember that a 95% confidence inrerval is calcula red
as rhe mean plus or minus 1.96 SE and thar the SE is calcu-
lated as foIlows:
SE = SD of the sample
-J sample size
The confidence interval reílects the size of the standard

-3 SD -2 SE -1 SE mean +1 SE +2 SE +3 SE
error, which in turn reílects the standard deviation and,
.0(- 95% of study means ->-
more irnporrantly, the size of the sample. Therefore, we can
The 95% Cl for our heart rate example wirh 40 partici- reduce rhe widrh of the el by increasing rhe sample size,
pants can be calculated as foIlows: because a larger sample wil! result in a smaller SE.
Finally, the upper and lower bounds of the confidence
el = mean plus or minus (2 x SE) interval can provide valuable information. For example,
= 80 plus or minus (2 x O 8) bpm let's consider a study report srating that a new type of che-
= 80 plus or minus 1.6 bpm morherapy reduced rumour size by a mean of20 mm (95%
= 78.4 to 81.6 bprn el, -10 ro 50 mm). These results indicare thar the cherno-
therapy is consistent with an average reducrion in turnour
Inrerpreted mathematical!y, a 95% confidence interval size of20 mm, a finding rhat appears quite promising (i.e.,
caprures the "true" (popularion) mean 95% of rhe time. In of c1inical interesr). However, the 95% el indicares that rhe
practical terrns, we recognize that the "true" (population) chemorherapy is also consistent with an average increase in
mean might not be exactly equal ro the individual sample turnour size of 10 mm (rhe lower bound of rhe el is -10
mean of 80 bpm, but we know there is a high probabiliry mm, a change in the opposire direction), as wel! as wirh an
(i.e., 95%) that it is no lower rhan 78.4 bpm and no higher average decrease in tumour size of 50 mm (i.e., a heteroge-
than 81.6 bpm (rhe 95% el). Pur another way, rhere is neous el).This study has obtained an imprecise estimare of
onlya 5% probabiliry (chance) that the "rrue" (population) the chemorherapy's effectiveness, most likely beca use the I
mean lies outside this interva!. sarnple was too smal!.

In general, researchers should calcula te and provide In conrrasr, anorher srudy report indicares thar a new
95% confidence intervals when reporting esrirnated treat- medicarion reduced systolic blood pressure by 2 mm Hg
ment effects. For example, imagine that a journal anide (95% el, 1 ro 3 mm Hg). Although this resulr suggests that I
includes che following staremenr: "The new drug reduced rhc medicarion wilI probably reduce blood pressure, rhis I
toral cholesrerol concenrrations by a mean of 20 mg/dL reduction is unlikeIy to be larger rhan an average of 3 mm
(95% el 12 ro 28 mgl dL)." This sentence suggesrs that the Hg, a reduction that wil! generally be perceived ro be too
srudy drug wiIl probably reduce cholesterol by an average smal! to be of c1inical importance. In this example, the
of ar leasr 20 mg/dL bur not by more rhan 28 or less than results indicare thar the srudy has very precisely measured
12 mg/dL. the effectiveness of the new medication, probably because

the sarnple was very large. Ir is also possible, in fact, that rhe precision of an estimare. Ir approximates rhe range of values
sample size for this study was roo large, given rhar it is rarely rhar are consistenr wirh an individual result (i.e., ir brackers
necessary ro make an estimare as precise as chis one (i.e., our besr guess of rhe "true" population estimare).
plus or minus 1 mm Hg). When dererrnining the sample
size for a study, researchers generally strive ro balance rhe
Conclusion
benetir of having a sufficient number of study parricipants
with the risks and COS[Sassurned by eacb srudy parricipant This chaprer provides an overview of the techniques used ro
(i.e., researchers should strive ro achieve a sarnple size rhat is summarize quanritative data and ro enable readers ro
neither roo small nor toO large, bur jusr right). become more comforrable anc! comperenr carrying out,
inrerpreting and reporting c!escriptive statisrical methods
and resulrs. In particular, atter reading this chapter you
Data variability summary
should be betrer able ro critically discern rhe "srory" behinc!
In this secrion rhree irnporranr concepts have been dis- commonly used c!escriptive staristical procec!ures and
cussed: standard deviarion, standard error of [he mean and results,
confidence inrerval, For example, afrer reac!ing rhis chaprer, you'Íl be much
Standard deviation is a measure of the variabiliry (disper- berter equippec! ro interpret results such as these:
sion) observed among individual measuremems in a sarn-
pie. Standard error of the mean is a rneasure of the Length of hospital stay (days): mean, 13; median, 6; standard
variability (dispersion) that we might expect among sample deviation, 12.
means when we estimare population values from sarnple
values. You'll have noticed the large difference berween [he val-
So, which will be larger, the SD or rhe SE? Recall bow we ues of the mean and the median and condude that this
estimare the standard error of the mean: difference has occurred beca use the distribution of data is
positively (right) skewed. Your condusion is funher con-
SE = SD of the sample firmed by the large standard deviation and the impossible
V sample size negative values rhat would occur ar a point less rhan rwo
standard deviations below rhe mean. Of course, rhis makes
Clearly, the SD rnusr be larger rhan the SE, because we perfect sense beca use you recognize that rhe variable "lengrh
calculare the SE by dividing the SD by the square roer of of hospital stay in days" can be measured only by posirive
the sample size. However, warch out for SE being used as a values, negative values being impossible.
descriptive sratistic ro irnply more precise rneasurernents. The next chapter will build on yOLlr undersranding of
The confidence interval (CI) is another measure of the descriptive statistics ro enable the use of additional srarisri-
variability (dispersion) rhar can be calculated wben popula- cal tests and techniques ro carry out hyporhesis tesring and
tion values are esrirnared from sample values. Tbe size of rhe determine the sample sizes needed ro ensure sufficiem sra-
95% CI is approxirnately twice the SE on eirher side of [he tisrical power. •
estima te. The CI is [he preferred value for indicaring [he
CASE POSTSCRIPT
After consulting his supervisor and the department's statistician, the resident learned that he would need to calculate the
95% confidence intervals for proportions to determine and illustrate sufficiently precisely the proportion of people in the
2 practices who had.received the recommended preventive services in the previous 2 years. For example, he found that the
proportion of eligible persons who had received the seasonal influenza immunization in the 2 practices was 98/211 (46.4%)
and 149/262 (56 9%), respectively. In addition, he was able to determine that the 95% confidence interval for these 2
percentages are 39.6% to 53.4% and 50.6% to 63.0%, respectively.

•
"
Acknowledgements
AII referenced web-based calculators were located at and accessed from www.statpages.org.
The development of this chapter was informed by: Harvey BJ, Ancker JS, Bairnsfather S, Bukowski JA, Hudson S, Lang TA, et al.
Statistics for medical writers and editors. Rockville (MD): American Medical Writers Association; 2009.
Glantz SA. Primer of biostatistics. 6th ed. New York: McGraw-Hill Medical Publishing Division; 2005.
Lang TA. How to display data in tables and graphs. In: Lang TA. How to write, publish, and present in the health sciences: a guide for
clinicians and laboratory researchers. Philadelphia: American College of Physicians; 2010. p. 67-100.
• This chapter provides a great overview of the principies and techniques of summarizing data using tables and graphics.
Lang TA, Secic M. How to report statistics in medicine: annotated guidelines for authors, editors, and reviewers. 2nd ed.
Philadelphia American College of Physicians; 2006.
• This book is a comprehensive reference discussing how statistics and the results of statistical tests and procedures, including
descriptive statistics, should be correctly reported.
Norman GR, Streiner DL. Biostatistics: the bare essentials. 3rd ed. Shelton (CT): PMPH USA; 2008.
• These three books are quite comprehensive, while also being readable and understandable (and include informative discussions
concerning descriptive statistics)
Norman GR, Streiner DL. POQstatistics. 3rd ed. Shelton (CT) PMPH USA; 2003.
Tufte E. The visual display of quantitative information. 2nd ed. Cheshire: Graphics Press2001.
• This is considered a classic text, written by one of the "qurus" in the field. I recommend reading it cover to cover.
Wainer H. Visual revelations. graphical tales of fate and deception from Napoleon Bonaparte to Ross Perot. New York: Springer-
Verlag, 1997.
• This is an easy-to-read, comprehensive and informative text by an author who, like Edward Tufte, is "visionary" in the art and
,,- science of describing quantitative data.
+J
V
:l
-c
e
O PRACTICE QUESTIONS
U
The answers to these questions are provided below.
Descriptive statistics
1. What measure identifies the middle value (the 50th percentile) in a set of data?
2. What measure identifies the lowest and highest values in a set of data?
3. What measure is determined by adding all of the values in a data set and dividing that total by the number oí values?
4. What measure identifies the 25th and 75th percentiles in a set of data?
5. What measure identifies the most frequent value in a set of data?
6. What measure is the average distance from the mean value in a set of data?
210 © 2011 The Royal College of Physicians and Surgeons of Cenada

24 Data ana/ysis: Descriptive statistics
7. Respondents to a community health survey were asked to repon 0.\ a y cigarettes they had smoked during the
previous day. With Figure 24.10 in mind, what measures of centra tendency and data variability best summarize
"number of cigarettes smoked" 7
8. In the Women's Health Initiatives Trial, women who were randomly assigned to receive hormone replacement therapy
(HRT) had heart disease more frequently than did women who did not receive HRT. What measures of central tendency
and data variability best summarize had heart disease?
9. The following statement appeared in a journal article: "Tumour size: mean, 46.7 mm; median, 21.3 mm; SD, 34.3
mm." What measures of central tendency and data variability best summarize this result?
10. The following statement appeared in a journal article: "Gland diameter: mean, 18 mm; median, 17 mm; SD, 2 mm."
What measures of central tendency and data variability best summarize this result?
Measurement variability
1. What measure is an estimate of the amount of variability expected between sample rneans?
2. What measure approximates the range of expected sample means that is consistent with an individual result (i.e., it
brackets our best guess of the "true" population mean)?
3. What measure is an estimate of the variability (dispersion) among a group of individual rneasurernents?
4. Researchers conduct a study to assess the association between age (in years) and the occurrence of osteoporosis
(i.e., low bone density) among a sample of 100 women. The mean bone mineral density of this group of women is
0.6 qr/cm' with a standard deviation of 0.1 qr/crn-. What is the standard error of this measurement7
5. For the study described in question 4, what is the 95% confidence interval of this measuremenP
6. For the study described in question 4, if the sample size remains unchanged at 100, but the standard deviation is
0.3 cr/crn'. what is the standard error of this measurement?
7. For the study described in question 4, if the sample size is increased to 400 and the standard deviation is 0.2 gr/
cm'. what is the 95% confidence interval of this measurement7
8. Which of the following confidence intervals is narrowesP
A. -10 to-2
B. -5 to O
e -3 to 1
D. -1 to 2
E. 2 to 4
9. Which of the following statements is false?
A. A larger sample size makes the standard error smaller
B. A smaller sample size makes the confidence interval wider.
c. A smaller standard error makes the confidence interval wider
D. A larger standard deviation makes the standard error smaller.
E. A smaller standard deviation makes the confidence interval wider.
The Royal College of Pllysicians and Surgeons of Canada 211

Answers to practice questions
Descriptive statistics
1. Median
2. Range
3. Mean
4. Interquartile range
5. Mode
6. Standard deviation (or variance)
7. Median and interquartile range (beca use the large number of expected zero values will cause the distribution to be
negatively skewed to the left)
8. Percentages (to present two dichotomous categories)
9. Median and interquartile range (because the data distribution appears to be negatively skewed to
the left)
10. Mean and standard deviation (beca use the data appear to be normally distributed)
lVIeasurement variability
1. Standard error of the mean
2. Confidence interval
3. Standard deviation
4. 0.01 gr/cm2 (the standard deviation divided by the square root of the sample size)
5. 0.58 to 0.62 (the mean plus or minus twice the standard error of the mean)
.- 6. 0.03 qr/crn' (the standard deviation divided by the square root of the sample size)
7. 0.58 to 0.62 (the mean plus or minus twice the standard error of the mean)
8. E 2 to 4, which is only two units wide
9. e A smaller SE yields a narrower (not a wider) confidence interval (recall, the CI equals the mean plus or
minus twice the SE)
SUMMARY CHECKlIST
o Review your research protocol and the types of data to be collected to identify the descriptive statistics that should be
used.
O Review your draft data analysis with a statistician, methodologist, members of your study team, and your supervisor.
O Revise as need.
212 © 2011 The Royal College 01 Physicians and Surgeons o' C¿~G;:¿
25
Data analysis: Hypothesis testing, sample size
and study power
ILLUSTRATIVE CASE
The Generallnternal Medicine resident we met in the previous chapter has determined that the proportion of eligible
people served by a community health centre's two practice groups who had received seasonal influenza vaccination was
98/211 (464%) and 149/262 (569%) Now, he'd like to determine whether the two practices actually differ from one
another in their ability to deliver this preventive service. He decides to arrange another meeting with the department's
statistical consultant to determine how to do this.
11 Quantitative health studies address addresses rhe conceprs of precision and statistical power
research questions about people and condirions by collect- and describes how ro determine the sarnple size necessary
ing and analyzing dara obrained by counting, observing ro provide a specified leve! of precision and/or srudy power.
and measuring. Building on the previous chaprer, this However, before beginning, ir is important ro reiterare a
chapter introduces and discusses the statistical rools and therne thar recurs throughour rhis guide: researchers should
rechniques used ro analyze dara arising from quantitative
healrh srudies. 1he analyric rechniques used in qualirarive
CHAPTER OBJECTIVES
healrh srudies are presented and discussed in chaprer 16.
As nored in the previous chaprer on descriptive sratistics, After reading this chapter, you should be able to:
the srarisrical techniques used ro analyze quanrirarive data • discuss probability and the process of hypothesis testing;
generally fall into rwo main caregories: descriptive statistics, • describe how a study result and the standard error are
which, as rhe name suggesrs, enable collecred data ro be used to determine the test statistic and the resulting
summarized and described; and hyporhesis-resring or iníer- probability (P) value;
ential statisrics, which cornprise a diverse ser of statisrical • define the alpha (0.) level and explain how it is used to
tests and rechniques thar enable comparisons and orher determine whether a given study result is statistically
analyses ro be complered. In addition, and relared ro borh significant;
of these uses, srarisrical merhods also enable a determina- • discuss Type I and Type 11 errors;
tion of rhe precision with which any measure has been • discuss "multiple testing" and its effect on statistical
rnade. In general, rhe most imporranr role of statisrical test- testing;
ing is ro assess whar role chance may have played in the col- • explain the difference between statistical significance
lecred dara and rhe degree ro wbich ir mighr be a plausible and clinical importance;
explanarion for the observed srudy results. • define statistical power and describe how it is
Like rhe previous chaprer, this chaprer has been written determined, along with the factors that affect a study's
from a pracrical perspecrive and strives ro introduce "sratis- power and how they relate to one another;
tics" not as an end in irself bur as a "rool box" from which • discuss the effect of sample size on a study's power, and
rhe healrh researcher may selecr tests, rools and rechniques list the factors to consider when calculating a study's
ro complete the necessary analyric tasks. Ir aims ro provide sample size and how they relate to one another; and
the reader with an overview of the starisrical rools that are • list common statistical tests, both parametric and non-
availabie for hyporhesis resring, how rhey should be used, parametric, and discuss the factors used to determine
and which rools should be used for which rasks. Ir also which test should be used under a given circumstance.
© 2011 The Royal College of Physicians and 5urgeons of Canada 213

!KEYTERMS
I Alpha (ex level) Multiple testing Sample size
Analysis of variance (ANOVA) Non-parametric s a istical tests Spearman rank correlation
Beta (B) level Null hypothesis Statistical power
Chi-squared (X 2) test One-tailed tests Statistical significance
Correlation Paired t test Test statistic
Clinical importance Parametric statistical tests Two-tailed tests
Confidence interval Pearson product moment Type I error
Hypothesis testing correlation Type 11error
I Kruskal-Wallis test P value Unpaired t test

l_ Mann-Whitney U test Regression Wilcoxon signed-rank test
consult wirh someone knowledgeable about srarisrical Given this distribution of mean heart rates, how proba-
merhods during the planning and design srages of rheir ble is ir thar a randornly selected sample of cardiac rehabili-
research projects! Only then can rhey be confidenr rhat rhey tation program participanrs would have a mean: heart rare
will collect rhe data they need using rhe righr techniques of83 beats per minute (bpm)?
and rhen be able ro analyze their findings in a manner that Recalling whar we know abour rhe normal disrribution,
results in a valid answer ro rheir research quesrion. we can assurne that a sample mcan of 83 bpm, although
Ir should also be noted that intermediare and advanced possible, would be very improbable, given thar 83 bpm is
rnerhods of data analysis used ro assess and adjusr fOI con- much higher rhan nearly al! of the mean values in the distri-
founding and effect modihcation are beyond rhe scope of burion. In tacr, a mean hearr rate of 83 bpm is 3.75 (3 -i- 4
chis chapter, good discussions of these techniques are avail- 0.8) SEs (standard errors oírhe mean) higher rhan 80 bprn, 4
able, however, in orher resources.!" the mean of rhe distriburion of means (i.e., the mean of the •
,
means). As such, the probability of observing a mean this •
"extreme" bychance is less than 1 rime in 1000 (P < .001).
Statistical hypothesis testing
.- As discussecl in the previous chapter, ir is possible ro esti-
In contrasr, how probable is ir thar a randomly
sarnple of cardiac rehabilirarion program
selecred
parricipants t
mate a distribution of sample means in which each value is would have a mean heart rate of79 bpm? 1
the mean valuc of difíerent samples from rhe same po pula- As Figure 25.1 shows, this value is locared near rhe cen- ,
tion. For example, Figure 25.1 illusrrates the distribution tre of rhe distriburion; in facr, ir lies slighdy more rhan 1 SE
of rneans that rnight arise from a series ofheart rate studies. below rhe mean of the means (i.e., it is among the possible
means closest ro the centre value as predicted by rhe proper-
Figure 25.1 ties of rhe normal distribution). Therefore, rhe probabiliry
Frequency distribution of mean heart rates from
that a random sarnple would have this mean hearr rate is
several samples
quite high (P= .21).
Hypothesis testing
As an illustration ofhow the disrribution of sample means •
can also be used ro inform srudy resulrs, ler's consider a
hyporherical study conducred ro determine whether me
heart rates of parricipants in a cardiac rehabilitarion ro-
gram increase between the beginning and end of a rehab
session. To answer this quesrion, we could rneasure e
78 77 78 79 80 81 82 83
heart rate of the program parricipanrs m-ice: once whe
rhey arrive at a rehab session, and again ar me end o. me
session. The truncated rabIe below illustrares whar me
resulting data might look like.
214 © 2011 The Royal College 01 Physicians and Surgeons o C2~,,::¿

25 Data ana¡ysis:'j¡ ~ esis testing, sample size and study power
ine rhar we calculared rhe SD and found rhat ir is 10 bprn.

Participant Beginning End Difference However, ir's more imporrant rhar we know how much rhe
- -
1 79 87 +8 mean heart rare differences mighr vary from sample ro sarn-
2 80 78 -2 ple, rarher than how much rhe individual hearr rate differ-
3 76 87 +11 ences mighr vary within any single sample.
_-_--_----- --- ---_---- --------
- - - - To determine how much rhe mean differences berween
rhe beginning and end of a rehab session heart rares from
several sarnples mighr vary, we calculare rhe corresponding
40 standard error (SE) of the mean for these differences. As you
recall, the SE can be esrimared by dividing the SD (10 bpm)
We calculare rhe mean difference or change in hearr rate by by rhe square root of the sample size ("Y40 = 6.3). This
summing the differences berween the 40 pairs of heart rates calcularion yields a standard error of 1.6 bpm for chis
(given in rhe founh column aboye) and dividing that (Oral by hyporhetical exarnple. Recalling rhe properríes of the normal
40, rhe size of rhe sarnple. In this case, imagine that rhe result- distribution, we know thar 68% of sample means will fall
ing mean difference is + 5 bpm-thar is, che heart rate of cardi- wirhin plus or rninus 1 SE of the true (population) mean-in
ac rehabilirarion progra.tn parricipanrs is, on average, 5 bpm this case, wirhin plus or minus 1.6 bpm. The hyporherical
faster ar the end of a rehab session than ir was at the beginning. distribution of the mean heart rate differences, assuming rhar
But is this 5-bpm difference in hcarr rate the result of rhe there rruly is no difference berween the values obrained at [he
rehab session (i.e., a real difference) or rhe result of chance? beginning and end of rehab sessions, is shown in Figure 25.2.
A common scienrific approach allows a question such as So, is a 5-bpm difference berween rhe hearr rares at rhe
this (O be explored by looking at the other side of the coin. beginning and end of rehab sessions the result of the rehab
Thar is, we start with the assumption thar participarion in session (i.e., a real difference) or the resulr of chance?
the rehab session has no effecr on heart rare-s-that the heart Because rhe SE is 1.6 bpm, the observed mean hearr rate
rate at rhe beginning is, on average, equal ro the hearr rate ar difference of 5 bprn for rhe sample of 40 participanrs is an
the end. This assurn ption constitutes what is called a null improbable resulr, Why? Figure 25.2 shows that a mean dif-
hypothesis. Our goal will be (Odetermine wherher the data ference of 5 bpm is slighdy more than 3 SEs Iarger (han the
allow us (Orule out or rejcct rhis null hyporhesis and thus be resulr that would be expected if rhere were truly no difler-
fairly sure thar the in crease in hearr rate is indeed artribur- ence berween rhe heart rares rneasured ar rhe beginning and
able ro parricipation in rhe rehab session and nor to chanceo end of a rehab session (i.e., the probabiliry oí ir occurring by
111is process is referred ro as hypothesis testing. chance is less than 1 time in 250).
\YJeknow that heart rates differ from time to time, even Figure 25.2
if a rehab session is nor involved. That is (O say, we expect Estimated frequency distribution of mean
differences between beginning and end
hean rates ro be differem within or berween persons simply
heart rates (SE = 1.6 bpm)
because of random and biologic variation. Ir's possible that
we mighr observe one of these chance variarions and mis-
rakenly assume that ir was caused by participation in a rehab
session. It is irnportant ro determine wherher an observed
resulr may have arisen by chance so that we can avoid draw-
ing incorrect conclusions, especially when those conclu-
sions will affecr people's health. To determine whether
chance is responsible, let's ask rhe following quesrion: If
parricipation in a rehab session truly has no effect on heart
-------~~------------------~---
rate, how large might the mean heart rafe difference be, bpm -4.8 -3.2 -1.6 O + 1.6 +3.2 +4.8
simply because of randorn variation alone? On the basis of SE -3 -2 -1 mean +1 +2 +3
[he conceprs rhat have already been discussed, we know Heart rate difference (end - beginning)
rhar we can measure the variation among these 40 differ-

ences berween the beginning and end of a rehab session by \YJe can explain rhis irnprobably large mean hearr rate
calcularing [he standard deviarion (SD). In rhis case, imag- difference in one of rwo ways:
© 2011 Ihe Royal College of Physicians and Surgeons of Canada 215

The Researeh Guide: A primer for residents, other health eare trainees, and praetitioners
l. The null hyporhesis is true (i.e., the rehab session observed. In our exarnple, this is rhe probabiliry that we
rruly has no effecr on heart rate), and this improb- would observe a mean difference berween heart rates mea-
ably high mean difference berween stressed and sured at the beginning and end of a rehab session of 5 bprn
relaxed heart rates (5 bpm), while improbable, has if, in real i ry, the rehab session has no effect on heart rateo
sirnply occurred by chanceo Because it is a probabiliry that can be inrerpreted as a per-
2. The null hyporhesis is not true. We can no longer centage, the P value will always be a fraction bérween O (no
accepr the idea that rhc heart rares measured at the chance of occurring) and 1 (certain ro occur). A large Pval-
beginning and end of a rehab session do not differ ue means that rhe results could plausibly have occurred by
from one anorher. chance (i.e., there is a reasonably high probabiliry that the
observed results arose sirnply by chance). A small P value
1'0 determine rhe probability of observing a resulr as means that the observed resulrs probably did not occur by
extreme as the one observed, we calculare the number of chance (although, no matter how small the P value, rhere
SEs berween rhe mean heart rate difference under rhe null will always be a possibility rhat the results arose by chance).
hyporhesis (O in this case) and rhe mean difference that we As a rnarter of convention, researchers commonly ser the
observed (5). In the exarnple aboye, the SE is equal ro 1.6 alpha (ex) leve! (the leve! at which results will be classified as
hprn, and so we use the following formula: sratistically signihcant) at .05. APvalue rnust be lower rhan
rhe alpha leve! ro be declared sratisrically significant. This
(5 O- 0).;- 1.6 = 3.125. means thar researchers are willing ro accept a 5% chance
that they could wrongly conclude that an observed result
The result of this calculation is called a test statistic, By was real when, in facr, it was due ro chanceo In other words,
locating chis value in a sratistical rabie (in this case, the t if there truly is no real effect, the observed result (01' one
rabie), which can be found in many sratistics books or more extreme) will occur by chance less than 1 time in 20
online," we find rhat rhe probability that such an extreme (5%). (The risk of wrongly concluding rhat an observed
value eould occur by chance is approximately 34 in 10000 result is real is increased by the number of statistical tests
(i.e., 0.0034). That is ro say, only 0.34% (about one-third thar are conducted-e-ir is 5% for the first test conducted,
of 1%) of rhe values within chis normal disrribution are plus almost an additional 5% for each subsequent test, so
more improbable than the one we observed. when rnultiple statistical tests are conducted and assessed
The terrn eommonly used ro describe rhe probabiliry special care should be taken. These inelude using an appli-
that a value could occur by chance is the P value. In the cable technique ro adjust for multiple testing, such as Bon-
contexr of our hyporhetical exarnple, the P value is the íerroni's correction or the Holm adjustment.)
probabiliry of detecring a mean difference berween stressed Ir should be stressed that rhe size of rhe J> value deter-
and relaxed heart rates of 5 bprn 01' more when there tru!y is mines on(y whether a result could have occurred by chance
no diffirence. if the null hypothesis is true. In fact, one of the rnost corn-
A result is considered ro have reached statistical signifi- mon staristical reporting errors in the literature is ro con-
canee if rhe observed result is unlike!y ro have happened by fuse sraristical significancc-a small Pvalue-with clinicaI
chance alone. In rhe example aboye, we would conclude importance. The size of rhe P value does nor indicare the
thar the difference berween the hearr rates of cardiac reha- importance or unimporrance of the results. Starisrical sig-
bilitation program participanrs measured at rhe beginning nihcance is not the same as clinical importance. One char-
and end of a rehab session are srarisrically significant. That acteristic of hyporhesis testing is rhat small and c1inically
is, iris unLike!y to be explained by chanceo trivial differences can be statistically significanr if me srudy
sample is large, and that clinically irnportant differences
The Pvalue can be missed iE(be srudy samplc is smalL In read, me lini-
The P value is the probabiliry that, if the null hypothesis is cal importance of a study resulr should be dererrnined bv
rrue (i.e., rhere rruly is no effect), chance alone could have the observed srudy "effecr" (e.g., rhe size of me difiere:
produced a result as extreme 01' more extreme than the one berween study groups). For example, a decrease o: ~
Hg in mean systolic blood pressure would zene y ;::0: be
a For example. at hrtp:l/statpages.org/pdfs.html considered c1inically irnporranr even if che P -alue associar-
216 © 2011 The Royal eollege of Physicians and S:..rg2 ==:a-a:c

25 Data analysis: 'ypomesi: testing, sample size and study power
ed with the result is sratisrically significanr-even as low as SEs) oE che null hyporhesis value of zero. As a result, rhe
.001. When writing about P values, researchers should be probabiliry of observing a mean heart rare difference at leasr
careful nor ro lead readers ro believe rhat sraristically signifi- this extreme by chance is now 9.6% (P= .096).1his Pvalue
cant results are aurornatically clinically important-or vice suggesrs that rhe mean heart rate difference of 5 bpm can
versa! So, although Pvalues should be incorporared into the now be expecred [O occur by chance alone approximare!y 1
interpretarion of study resulrs, biological plausibiliry and time in 10. Therefore, this result (because of its larger SE) is
rhe clinical imporrance of the observed result should also be no longer statisrically significam when compared wirh rhe
considered. convemional alpha (ex)leve! of .05.
Our example oE hyporhesis testing about a difference
Figure 25.3
berween hearr rates measured at the beginning and end of a Estimated frequency distribution of mean differences
rehab session is a sirnplified version of a paired t test (also between beginning and end heart rates for possible
samples (SE = 3.0 bpm)
called Srudenr's paired t test). 1his is used because rhe dara
are paired: rwo measurements are taken from the same per-
son at different times, or participanrs in rhe srudy groups
are "matched." Orher statisrical tests use similar approaches
[O compare observarions (collected data) ro rhe results rhat
would be expected under the null hyporhesis and then ro
srandardize this difference by dividing ir by the applicable
measure of variabiliry (e.g., the SE). bpm -9.0 -6.0 -3.0 O +3.0 +6.0 +9.0
Ler's consider another ser ofhypotherical heart-rare dara: SE -3 -2 -1 mean +1 +2 +3
Heart rate difference (end - beginning)
Participant Beginning End Difference

1 79 91 12
Other statistical tests
2 80 71 -9
3 76 87 11 Other statistical tests use merhods similar ro those illustrat-
- - - - ed by the exarnples presemed aboye. AH of these tests derer-
(}
- - - - mine the difference berween the observed resulrs and the O
-
results that would be expected by chance if the null hyporh- ~
-
40
-
88
-
77
-
-11 esis were true (i.e., if there truly was no effecr); rhey rhen o,
I
srandardize this difference berween observed and expected e
results by dividing ir by rhe applicable measure of variabili- n
r-?
As was true of rhe original ser of data, rhe mean difler- ty (e.g., the SE). Researchers selecr the appropriare starisri- -.
ence berween rhe hearr rates ar the beginning and the end of cal test ro use by derermining rhe srudy design (e.g., me
a rehab session for rhese 40 cardiac rehabilitation program nurnber of groups being compared) and rhe level of mea-
parricipanrs is also +5 bpm. However, imagine rhat, in this surernenr of the ourcome variables (e.g., nominal, ordinal,
sample, the variabiliry oE the differences in heart rate is discrere or coritinuous data). A guide for se!ecring rhe
higher, wirh a standard deviation (SO) of 19 bpm (insread applicable staristical resr-adapred from Glamz5-is pre-
of the SO of 10 bpm in the original data set). 1his higher sented in Table 25.1.
SO results in a correspondingly higher SE oE3 bpm (insread
of the SE of 1.6 in rhe original data ser). As Figure 25.3 Analysis of variance
depicts, rhe disrriburion of mean heart rate differences for Analysis of variance (ANOVA) is used ro compare rhe
possible samples around rhe null hyporhesis is now more means obrained Eram rwo or more groups, derermining
strerched out. whether rhe variarion among the mean s is grearer than
Because the variabíliry in the heart rare differences is mighr be expecred given rhe arnount of variation inherent
higher and the corresponding standard error (SE) of the in the underlying data. However, the result of an ANOVA
mean for these differences is larger, rhe mean hean rate dif- indicares only wherher a sratistically significanr difference
ference of 5 bpm is now wirhin 2 SEs (acrually, ir is ar 1.67 exists: ir doesn't indicare which group or groups diffcr from
© 2011 The Royal College oí Physicians and Surge'ons oí Canada 217

• Table 25.1: A guide for selecting the appropriate statistical test
Type of comparison being carried out
Three or more Two matched Multiple measures Association

Level of outcome Two independent independent (dependent) in the same .between two
measurement groups groups groups individuals variables
Normally distributed Unpaired t test Analysis of variance Paired t test Repeated-measures linear regression or
continuous data* analysis of variance Pearson product
moment correlation
Nominal Difference of propor- Chi-squared McNemar's test

tions or chi-squared contingency table
contingency table analysis
analysis
Ordinal' Mann-Whitney rank- Kruskal-Wallis test Wilcoxon signed-rank Friedman statistic Spearman rank
sum test test correlation
Survival time Log-rank test
Notes:' if the data are not normally distibuted, rank the observations and use the methods for data measured on an ordinal scale.
t Or interval data that are not normally distributed.
rhe orhers. For rhis reason, an ANOVA that has yielded a berween rhe rwo measures; -1 indicares a perfecr inverse/ e
statisricaIly significanr resulr is usually followed by furrher indirect relationship (i.e., as one measure increases, the oth- !
rests thar, ideally, account for the applicable number of er decreases); O indicates no relationship (i.e., the observed ~
pair-wise comparisons (e.g., Tukey, Newrnan-Keuls, relarionship is consistenr wirh whar would be expecred
Holrn-adjusred t tests) ro derermine
from the others beyond what would be expected by chance
which graups differ from random variarion): and intermediare
ing a parcial relarionship
values indicar-
berween rhe rwo. 1here are rwo
,
«
(i.e., their difíerences are srarisrically significanr). main rypes of correlarion: paramerric and nonpararnerric.
Pearson product mornent correlation is a paramerric •
~
Unpaired ttest m~rhod rhat assesses rhe acrual measured values for each
The unpaíred ttest is a special case of ANOVA and is used variable, while Spearman rank correlation, an example of
.....,
"'- ro determine the srarisrical significance of the difference a non-pararnetric method, assesses the rank-order of rhe •
u berween rhe means obrained frorn rwo (and only rwo) values for each variable. Pearson product momenr correla- t
~ independenr samples. In contrast, rhe hearr rate examples tion assumes that each of the rwo sets of dara being assessed e
"'C aboye involved paired t tests because rhe heart rates are are normally distributed (01' nearly so), whereas Spearman ,
e pairs of measuremenrs raken fram the same persons, not rank correlation can be used on conrinuous ordinal data
•
01'
o measurernenrs raken from 2 different graups of partici- rhat follow any distribution patrern. The correlarion coeffi-
u panrs.
•
Figure 25.4
Correlation A scatter plot of the number of graduates per year
Correlation enables the assessrnent of rhe relarionship

25 -----.---.--- ..
---.--.----
berween rwo quanritarive measures. 1he data can be graph-
+
ically depicred in a scatter plot in which one variable is ~ 20 ----. ---.-.-+-.----~--
plotted on each axis. Figure 25.4 plors rhe number of grad- ~ 15 -._!-+--+---+ + +
uares by year. lf a porentially causal relarionship were being 01 + + + +
'O 10 ---.----- ... -. __!_L .. _!__~.+_L_~_~ _
--- ..
assessed, rhe exposure/predicror (i.e., independenr) vari- Oi +
.a + +
able would be plotred on the x axis wirh the outcorne (i.e., E 5 -.-----------------.----- .. ._----+
::J
deperident) variable on the y axis. Z
O +-----,-----~----_r----_.------~----.
1he level of associarion/correlation berween rhe rwo 1980 1985 1990 1995 2000 2005 2010
variables is quanrified with a correlarion coefficienr ranging Year
from -1 ro 1: 1 indicares a perfect direcr relarionship
218 © 2011 The Royal College of Physicians and Surqeons ¡C¿;-~

25 Data analysis: ,,:oorhesis testing, sample size and study power
cient for rhe data depicted in Figure 25.4 is -0.70 (by both ferences berween rhe expecred proporrions or categorical
rnerhods), which indica tes an inverse relarionship (i.e, the counts and (he proporrions of data obtained from rwo or
number of graduares is declining over rime). more independenr samples. Ir produces a P value rhat
assesses (he staristical significance of these differences (i.e.,
Regression (he probabiliry that a difference as grear as the one observed
Regression also enables rhe relationship berween quanrira- could have arisen by chance).
tive measures (O be assessed. However, unlike correlación,
regression should be used only (O assess porenrially causal
Interpreting study results
relarionships berween one or more exposure/predictor vari-
ables and an outcorne variable (e.g., effect of drug dosage Ler's consider furrher how sraristical rnerhods are used (O
on urinary output; effect of age, height and weighr on assist in rhe interpretation of srudy results. Imagine thar a
blood pressure). In rhese circurnstances the data can be clinical rrial, conducred (O assess the efficacy of radiothera-
graphically depicred in a scarter plot with rhe exposure/pre- py (a dichorornous indicaror variable) (O reduce dearh from
dictor (i.e., independenr) variable plotred on rhe x-axis and breast cancer (a dichorornous outcorne variable), provides
ourcorne (i.e., dependent) variable on rhe y-axis (Figure rhe following resulrs:
25.4). If there is no relarionship berween rhe rwo variables/
measures, rhe resulting scarrer plor will approximate a hori- Alive Dead Total
zontal lineo In conrrast, rhe data depicted in Figure 25.4 Radiotherapy 24 16 40
suggesr rhat rhe number of graduares is declining over time No radiotherapy 16 24 40
(wirh an esrirnared slope of -0.33 graduates/year). Regres- Total 40 40 80
sion compares the observed data againsr what would be
expected if rhere were no relationship, by dererrnining the
probabiliry of the observed slope differing from zero (i.e., These results indicare that the dearh rare in the group
rhe slope of a horizontal/Har line). receiving radiorherapy was lower (40%; 16/40) rhan in the
comparison group (60%; 24/40). Although these results
Non-parametric tests appear promising, rhey (Iike all quantirarive study resulrs)
Like Pearson product mornent correlation, the t tests and can be explained in one or more of rhree "vays. TI1e hrst
ANOVA are examples of pararnetric statistical tests explanarion, of course, is that the results are real-s-rhar in
because rhey are intended for use only with norrnally dis- rhis example radiorherapy does, in facr, lead (O a reduced
tributed (or nearly so) data that can be described by param- rate of death from breast cancer. However, there is no \\·ay
eters (measurable characreristics) such as group means, SD ro directly measure or orherwise determine me degree (O
and variance. However, when study dara are not normally which this explanation accounrs for any set of observed
distributed, researchers should use alrernative non-para- quantirative srudy results. Insread, we rnust assess this
metric statistical tests. Some examples of nonparamerric explanation indirecdy by firsr considering rwo alternarive
tests are the Wilcoxon signed-rank test (the non-paramer- explanations.
ric alternarive (O the paired t test), rhe Mann-Whitney U The first of rhese is rhat the srudy was flawed, either in its
test (the non-pararnetric alternarive (O the unpaired t test), design or in irs conduct, and thar rhis resulred in a biased
rhe Kruskal-Wallis test (the non-pararnerric alternative ro result. For example, if rhe allocation process was flawed,
Al"JOVA) and the Spearrnan rank correlarion (which, as resulting in a disproporrionarc number of parients expecred
nored above, is the non-pararnerric alternative (O rhe Pear- ro die being allocated ro the comparison group, this "selec-
son producr moment correlation). tion bias" in how patients were allocated (O the rwo srudy
groups could reasonably explain rhe apparem efficacy of
Chi-squared <X2) test radiotherapy. To assess the degree (O which study errors,
Probably the most widely recognized non-parametric sta- biases and confouncling might account for rhe observed
tistical test is the X" (chi-squared) test of proporrions, resulrs, we would need (O use critical appraisal rechniques,
which is used (O assess rhe statisrical significance of resulrs such as those discussed in chapter 9 and described in rhe
obtained with categorical data. 1his test determines the dif- CONSORT (Consolidated Standards ofReporting Trials)
© 2011 The Royal College 01 Physícíans and Surgeons 01 Canada 219

Srarernent." Such derails would indude, for example, me advance. When one-tailed tests are used, however, the
so urce of random numbers, how rhe allocation schedule should be dearly idenrified as such and their use justified.?
was kepr secret from those who enrolling patients and rhose As discussed earlier, this probabiliry (i.e., P value) need
who assigning patients ro the srudy groups and, if applica- ro be sufhciendy low that chance is consiclered an improb
ble, the success ofblinding patienrs. able explanarion for rhe observed resulrs before researcher
The second alrernative explanarion is chance-rhar rhe condude that rhere is an alrernarive explanarion for rh.
observed excess of dearhs in rhe comparison group is simply observed resulrs-i.e., that the observed resulrs reflecr ;
the result of a larger proportion of srudy parienrs who were rrue effecr: in rhe case of rhis example, this alrernarive expla
ultimarely destined ro die being randomly allocared ro that narion would be that radiorherapy really do es reduce the
group. Even with thc use of scientifically sound and rigor- risk of dying of breast cancer. As nored earlier, this cut-of
ous randomization processes (i.e., all patients have rhe same value, the alpha (a) level, should always be selected by the
chance of being assigned to rhe trearment or comparison researchers before the srudy. The traditional rhreshold (the
group), rhe distribution of patients who are ultirnatcly des- a level) for indicaring srarisrical significance is .05. Howev-
tined ro die may, by chance, be irnbalanced. As described er, using rhis rhreshold does nor guaranree rhar rhese con-
aboye, rhe probabiliry rhat such an unbalanced allocarion dusions will be correcto In facr, if we reject the null
may have occurred can be dererrnined by sratisrical resting. hyporhesis every rime we find a P value of .05, we will be
More specifically, applicable statistical tests answer rhe wrong 5% of rhe time: recall thar a P value of .05 means
question, "lf there truly is no difference berween the groups rhat the null hypothesis would produce resul ts at leasr as
(i.e., rhe so-called "null hyporhesis"), what is rhe probabili- extreme 5% of the time. This rype of error-of mistakenly
Iy of observing a result at least as extreme as rhe one conduding that rhere is an effecr when in fact rhere isn't
observed, simply by chance alone?" one-is called a Type I error,
In rhe example study, if radiotherapy truly has no efrect, On the other hand, researchers can make a differenr kind
then we would expect 20 paticnts in each group to have a of misrake. They can incorrecdy condude thar there is no
recurrence, because (he 40 (i.e., 16 -;-24) patienrs with a efíect when in fact rhere really is one. This rnistake is
recurrence would be expected ro be evenly divided berween referred ro as a Type II error,
[he 2 study groups. However, the observed results in our Researchers can never know for sure wherher rhey've
.- hypothctical srudy difl-er from this "expected" amount by 4

patients each. By complcring the appropriate starisrical test
made one of these errors in their srudies. But ro avoid these
potenrial errors rhey try ro set an a level thar represents an
(in rhis case, a chi-squared test) eirher by hand or by using acceptable trade-off berween the risk of wrongly conclud-
suitable computer software," chis probabiliry is found ro be ing that rhere is a real effect when such an effecr in fact
0.117, or 11.7%. Thar is, assuming thar the null hypothesis doesn't exisr (a false-positive result) and the risk of failing ro
is true (that radiorherapy truly has no effect on the risk of detecr a real eflect when such an effect in fact do es exisr (a
dying), a difFerence as large as the one observed could have false-negative result).
occurredsimply by chance more rhan 1 rime in 10 (i.e., P> .1). Although rhe mosr frequenrly chosen a level is .05, there
Ir is imporranr ro note that P values like this one should are circumstances in which either a higher a level or a lower
almost always be based on a rwo-tailed hyporhesis test. a level may be more appropriate, depending on rhe balance
Two-tailed tests are used when differences can porentially berween rhe consequences associated with false-positive
occur in either direction: ro take our exarnple, rhe dearh results and those associated with false-negative resulrs. For
rare could be eirher higher or lower in rhe radiotherapy instance, in the presented example, the researchers might
rhan in rhe comparison group. Two-tailed tests give rhe have selected a lower alpha leve! (e.g., .01) so that rhey
probabiliry for a difference in eirher direcrion and so should could funher reduce the risk of wrongly concluding thar
be used when thc clirecrion of a diflercnce is unknown," rhe radiorherapy is effective, particularly in lighr of irs
whereas one-tailed tests should be used only in those infre- potencial side-eflecrs, cosrs and lirnired availabilirv, How-
quent insrances when rhe direcrion of rhe resulr is known in ever, even if the Pvalue is as low as .01, chance mighr still be
responsible for rhe observed resulrs (in mis case, ir is a low
probabiliry: 1 chance in 100). As such, (he probabiliry of
b Such as the web-based calculator at www.statpages.org/ctab2x2.html cbance being responsible for observed resulrs never goes ro

25 Data analysis: 'ypothesis testing, sample size and study power
zero because ir is always possible for chance ro be responsible by defini ion, significant at the .05 leve!. That is, rhe null
for the observed resulrs. However, the key quesrion is how value of zero (borh groups are the sarne), will occur by
probable is ir thar chance could accounr for the observed chance less [han 5 rimes in 100.
resulrs. Alrhough confidence intervals are related ro hypothesis
In contrast, a study examining the health benefit of eat- tesring and P values, they focus attention on the health
ing fruits and vegerables may benefir from a higher a leve! effect; on the esrirnated size of rhe effect, or rhe size of the
(e.g., .10) beca use the risks associared wirh failing to detecr difference berween the groups; and on che precision of the
the benefir of rhis dier (i.e., a false-negarive result) ourweigh estimare. Reviewing this effect size and rhe precision with
rhe risk associared with falsely concluding that such a dier is which ir is esrimared allows us to better determine rhe clini-
beneficial (j.e., a false-posirive result). Ir should be no red cal imporrance of the resulrs, As a result, researchers are
that Type I and Type II errors have a reciprocal relationship: advised (and even required by 111anypublications) to repon
when rhe probabiliry of one is decreased, rhe probabiliry of a confidence interval to indicare the precision of rhe esri-
the other is reduced-hence, there is a "trade-off" berween mated effect size. By completing the applicable staristical
rhern. procedure by hand or by using suirable compurer software,"
The relationship berween a study's result and rhe under- che 95% el in our exarnple can be dererrnined to be -0.02
Iying "truth" can be illusrrared as follows. to 0.40, indicating that rhe "true" difference in dearh rates
will be within this range with a probability of95%.
The truth
The study result There is an effect There is no effect When the P value is not statistically significant
True-positive statistical False-positive statistical In the radiotherapy example, the P value was sufficientÍy
Looks like an effect
result result (Type 1error) large (i.e., .117 or 11.7% or P > .1) rhat we would accept
False-negative statisti- True-negative statistical chance as a reasonable explanation for rhe observed resulrs.
Looks like no effect
cal result (Type 11error) result So, should we now conclude rhat radiotherapy is ineffective
at reducing the risk of dearh due to breast cancer? As a hrst
step, ler's reconsider the apparent effect of radiotherapy, As
Alrhough P values refl.ecr the probabiliry that an we nored aboye, rhe results of rhe study indicare thar receiv-
observed result may occur by chance, they do not indicare ing radiorherapy reduced the rate of dearh by 20% (40%
how precisely an observed difference or effecr has been mea- versus 60%). So, although rhe observed results were nor
sured (i.e., the range of values consisrent wirh the study's sratistically significant (i.e., rhe P value was not less rhan
result). In rhe previous chapter we discussed the use of con- the chosen alpha leve!) , they suggest a clinically imporrant
fidence intervals to identity, at a given probabiliry, the effect.
range of values within which the "true" (popularion) value Analogous to rhe discussion aboye, al1 exarnination of
will be found. In a similar manner, confidence intervals can this staristically non-significanr result requires [he consider-
be used to specify the researcher's confidence (i.e., range of ation of three possible explanations. The first explanarion,
values) in an observed difference or effect. of COLme, is rhat the observed results are real: thar radio-
As an example, recall the srudy repon from the previous rherapy is, in facr, not effecrive at reducing the rate of dearh
chapter rhat contained the following sentence: "On aver- from breasr cancer. Again, however, there is no way to
age, rhe new drug appears to lower parienrs' toral cholester- directly measure or orherwise determine whether this
01 concentration by 12 to 28 mg/dL." This staternent is far explains the sratistically non-significanr resulr, Insread, this
more inforrnative rhan a staternent such as rhe following: deterrnination is made indirectly by assessing each of rhe
"The new drug resulrs in a sraristically significant reduction rwo alrernative explanarions.
in parienrs' toral cholesrerol concentration (P= .01)." As we The hrsr of rhese alternative explanarioris is thar the
discussed in rhe previous chapter, rhe confidence interval srudy was fl.awed, either in its design or in irs conduce,
tells us whether rhere is an effecr and also tells us the plausi- resulting in a biased result and in a failure ro detect a real
ble magnitude of this effecr. In addirion, rhe el also tells us difference berween rhe groups. For example, if a larger
indirectly about srarisrical significance: rhis is because, in a
study of differences, a 95% eI rhat does not include zero is, e Such as the calculalor al http://statpages.org/ctab2x2.html

number of participants in the comparison group were lost or desirable difference if one truly exisred. Traditionally,
ro follow-up, reducing rhe number of deaths being recog- researchers consider a study power of 80% to be the mini-
nized in this group, this "measurernent bias" could poren- mum reguired, alrhough higher values are commonly used
rially account for the apparent ineffecriveness of (e.g., 90% or 95%). As such, this study's statistical power of
radiorherapy. Again, ro assess rhe degree ro which study 34.4% helps to explain why this observed 20% difference
errors, biases and confounding mighr account for rhe in dearh rares was nor statistically "detecred." Nor enough
apparent lack of effecr, we would need ro know and criti- patients were studied ro have a reasonable chance of derecr-
cally assess rhe specitic details about rhe research design and ing ir. Similarly, the wide (i.e., imprecise) confidence inter-
activities, such as those discussed in chapter 9 and described val presented earlier (i.e., - 2% ro 40%) suggests a study
in the CONSORT Sratement.f In this case, a "best-case/ rhar lacked sufficient statistical power. So, how could such a
worst-case analysis" (i.e., sensirivity analysis)" could be situation have been avoided?
done to estimare rhe potential impact the differences in the
rare of patients lost to follow-up might have had on the
Estimating the sample size for a study
observed results.
The second alternarive explanation is, once again, To avoid conducting a study only to find rhat it has insuffi-
chanceo But instead of asking, as we did aboye, what the cient power, researchers are strongly encouraged-and
probability is that the observed results arose by chance, we even expected and morally obligated, so that patients are
ask what rhe probability is that the study failed to detect a not put at risk unnecessarily-to specify the desired study 1
resulr of a given size if such an effecr rruly exisred, This power and ro calculare the sample size needed to achieve I
probability is called the beta W) leve! and is relared to sta- this power when planning the study. In our example, how
risrical power, which is the probabiliry of a study derecting a many patients would have been reguired ro provide suffi- ,
difference of a given size, if one truly exists. Study power is cient study power? To esrimate this number, the applicable
egual to the complement of the beta level, 1 - beta. statistical calculation could be completed using a suitable
statisrical procedure." In our example, ro have an 80%
chance of derecting a 20% difference in the rate of death
Statistical power, beta (~), Type 11
from breast cancer at an alpha leve! of .05, the power calcu-
error, and confidence intervals
Iarion indicares rhat each of the 2 groups should have at
,,-
To determine the statistical power of our hypothetical leasr 107 patients, As you would expect, rhis sarnple size
study (i.e., rhe probability that it would detect an effecr of a wil! change if any of the specifications are changed (Table
certain size iE it acrual!y exisrs), we must identity and speci- 25.2). For exarnple, to derect a smaller difference in death
f)r four characteristics of the srudy: (1) the minimum differ- rates (e.g., 60% versus 50%), at least 407 patients would be
ence berween the groups that would be considered needed in each study group. Similarly, the use of an alpha
clinicaHy imporrant
data variability
to detecr: (2) the esrirnated arnount of
(e.g., standard deviation), (3) the sample
leve! orO.01 would reguire each study group to have ar least
154 patienrs, Further, ro achieve a statistical power of90%,
-
size of each srudy group; and, (4) the alpha (a) leve! used to each group would reguire ar least 139 patients.
declare a statistically signiíicant result (i.e., ro condude Alrernatively, researchers may choose ro determine rhe
rhere is a difference). With rhis informarion, ir is possible to sample size needed ro ensure a sufiicienrly precise estimare
estimare a study's sraristical power." (i.e., a sufficiently narrow confidence interval). Using a
This calcularion gives a result of 34.4%, meaning that web-based calculator' indicares that the difference in dearh
rhe power of our hypothetical study to detect a 20% differ- rares could be estirnated with a precision of abour plus or •
ence in death rates using an alpha leve! of .05 and a total minus 10% if each group included 200 participanrs (as
sarnple of 80 was about 1 in 3. So, whar <loes rhe staristical compared with rhe precision of plus or minus 20% [rhe
power of a study tel! us? Firsr, ler's consider how high rhis 95% confidence interval of -2% ro 40% nored aboye) rhar
probability would need to be for us ro condude that rhe was possible with 40 indivíduals in each study group).
study had sufíicient statistical power to detect a reasonable
e Such as the web-based sample size calculatar avaíta 'e a: ••••••••••••

d A web-based calculator lar comparing two proportions is available at statpages.org/proppawr.html
vvvvvv.stat.uiowa.edu/-rlenth/Power!lndex.html Such as the one available at http://statpages.arglctab2x2.n:r
222 © 2011 The Royal College oí Physicians and Surqeons o' Ca'B~2 ~
25 Data analysis: , tbesis testinq, sample size and study power
• Table 25.2: Variables ineluded in statistieal ealculations for a eomparison of 2 pereentages using a ehi-
squared test and eaeh variable's effeet on the desired sample size for ea eh group
Variable* Group 1 (%)
I Group 2 (%)
I Alpha (a) I Power (1 - p) I Sample size (n)
0.05 0.8 107

2-tailed test 60 40
I I I
1-tailed test 0.05 0.8 86
60
-
40
I I
-l- diflerent 60 50 0.05 0.8 407
I
LAlpha 60 40 0.01 0.8 154
------
t power 60 40 0.05
. _____ L________.
0.9
I 139
._.
*Values in bold have been varied lrom the íirst line to show how changes in each variable altee! sample size as determined using wwwstatpages.org/proppowr.html.
Determining rhe sample size necessary ro ensure sufhcienr Conclusion

estimare precision can also be carried out for the planning This and rhe previous chaprer were wrirren ro provide a
of surveys. For example, the resident in our illustrarive case better understanding of sraristics-s-nor ro enable readers ro
should determine how precisely he would like ro determine become staristicians, but to become more comforrable and
what proportion of patients in each of rhe practices has comperent designing, conducting, interpreting and report-
received the applicable preventive services. For example, if ing staristical methods and results. In particular, having
he wishes ro determine how patients will need ro be assessed read this chapter you should be better able ro discern che
to estimare the percentage of patients who have received "story" behind the most commonly used sratisrical proce-
colorectal screening in the previous rwo years with a preci- dures, tests and results.
sion of plus 01' minus 5% (for the resulting 95% Cl), he For example, you should now be better able ro discern
could use a web-based calculator.? If he anticipares that the "story" behind a study result such as:
about 50% of patienrs will have received rhis preventive ser-
vice, rhe calcularor shows rhar srudying 50 parienrs pro- Mean reduction in systolic Bp, 10.2 mm Hg; 95% el, 4.1 to
vides a precision of only plus or minus 15% (and only plus 16.3 mm Hg; P = .003
or minus 10% if 100 parienrs are srudied), in facr, about
400 patienrs would be reguired ro arrain the soughr-after You're now more familiar with Pvalues and 95% confi-
precision of plus or minus 5%. dence intervals and should be able ro describe the informa-
By applying these principies and procedures, researchers tion they convey abour the study's resulrs, You'Il recognize
are able ro better ensure that their srudies have an adequate that rhe mean reducrion in sysrolic blood pressure of 10.2
number of patienrs and therefore sufficient staristical power mm Hg is clinically irnporranr. However, rhis conclusion is
(and measurement precision) ro detect clinically important sornewhat ternpered by the fact thar the lower bound of rhe
differences if rhey rruly exist. In facr, if a proper sarnple size 95% CI suggesrs rhar rhe mean reducrion in sysrolic blood
calcularion is completed during rhe planning of a srudy and pressure may be as small as 4.1 mm Hg. On rhe orher hand,
if the required nurnber of subjecrs is recruited into the you undersrand rhat the mean reduction could be as large
srudy, a calculation of power after rhe cornpletion of rhe as 16.3 mm Hg. Finally, you recognize rhat rhe reponed P
study should never be needed. As such, posr hoc calcula- value indicates thar the observed results are very unlikely ro
rions of study power should, ideally, never be necessary. have occurred by chance (only 3 times in 1000).
Alrhough the example radiotherapy srudy compares the Although chis and the previous chapter are intended to
differences berween rwo percentages, the same principies expand your sratisrical knowledge and skills, you are encour-
and analogous procedures are available for other study situ- aged ro continue to enhance your statistical capabilities regu-
arions, such as correlation coefficienrs, regression slopes lady, borh to solidiíy rhern and to further expand rhern, After
and resring the differences in means and rates. reading these rwo chaprers, readers should be berrer at under-
standing, designing, conducring, interprering, reporting and
cririquing analytic merhods and resulrs and will have gained
more confidence in acrively inreracting wirh and, dare I say,
guesrioning others about selecring, interprering and reponing
9 Such as the one at http://statpages.org/confint.html statistical procedures, tests and resulrs. B

CASE POSTSCRIPT
After consulting his supervisor and the department's statistician, the resident learned that he would need to use the chi-
squared test to determine whether the observed proportion of preventive services received differed significantly between
the health centre's two practices. After doing so, he found that the proportion of eligible persons who had received the
seasonal influenza immunization differed by 10.4% between the two practices: 98/211 (46.4%) as compared with 149/262
(56 8%)-a difference that would have been expected to arise by chance with a probability of .026. He also calculated the
95% confidence interval for the observed difference between these two proportions to be 0.9% to 19.7%.
Acknowledgements
AII referenced web-based calculators were located at and accessed from: www.statpages.org
The development of this chapter was informed by: Harvey BJ, Ancker JS, Bairnsfather S, Bukowski JA, Hudson S, Lang TA, et al.
Statistics for medica! writers and editors. Rockville (MD): American Medical Writers Association; 2009.
as well as:
Harvey BJ, Lang TA. Hypothesis testing, study power and sample size. Chest. 2010;138(3):734-7.
REFERENCES
1. Glantz SA, Slinker BK. Primer of applied regression & analysis of vetience. 2nd ed. New York (NY): McGraw-Hilllnc.; 2001.
2. 1(leinbaum DG, kupper LL, Nizam A Muller KE.Applied regression analysis and other multivariable methods. 4th ed. Belmont
(CA): Duxbury Press;2008.
3. Vittinghoff E, Glidden DV, Shiboski Se, McCulloch CE. Regression methods in biostatistics: linear, logis tic, survival, and repeated
measures mode!s. New York (NY): Springer Scienc:e+BusinessMedia; 2005.
tri
e
--
4. Norman GR, Streiner DL. Biostetistics: the bare essentials. 3rd ed. Shelton (CT): PMPH USA; 2008.
s. Glantz SA. Primer of biostatistics. 6th ed. New York: McGraw-Hill Medical Publishing Division; 2005. p. 446.
..¡..J
U 6. Altman DG, Schulz KF,Moher O, Egger M, Davidoff F,Elbourne D, et al. The revised CONSORT statement for reporting
::s randomized trials: explanation and elaboration. Ann Intern Med. 2001; 134(8):663-94.
-c 7. Lang T Documenting research in scientific articles: Guidelines for authors. 2. Reporting hypothesis tests. Chest. 2007; 131(1):317-9.
e 8. Fletcher RH, Fletcher Sw. Clinical epidemiology the essentia!s. 4th ed. Philadelphia (PA) Lippincott, Williams & Wilkins; 2005. p.
o 118, 121-2.
U
Glantz SA. Primer of biostatistics. 6th ed. New York: McGraw-Hill Medical Publishing Division; 2005.
Lang TA Secic M. How to report statistics in medicine: annotated guide!ines for authors, editors, and reviewers. 2nd ed. Philadelphia:
American College of Physicians; 2006.
This book is a comprehensive reference discussing how statistics and the results of statistical tests and procedures, including
descriptive statistics, should be correctly reported. Norman GR, Streiner DL. Biostatistics. the bare essentials. 3rd ed. Shelton (CT):
PMPH USA; 2008.
• These three books are quite comprehensive, while also being readable and understandable.
Norman GR, Streiner DL. PDQ statistics. 3rd ed. Shelton (CT): PMPH USA; 2003.
224 © 2011 The Royal College 01 Physicians ano Su'ge'""~5 o: ~a-¿c¿

25 Data analysis: L;~'po-nesistestinq, sample size and study power
PRACTICE QUESTIONS
The answers to these questions are provided below.
1. When reading a journal article, you encounter the following sentence "Influenza occurred less frequently in the
vaccinated group (2%) than in the unvaccinated group (30%; P = 44) " What would you conclude about
the clinical importance and statistical significance of this result?
2. You have high blood pressure, and you read the following sentence in a journal article "Our findings (P < 001)
show that the new drug lowers blood pressure by an average of 1 mm Hg." What would you conclude
about the clinical importance and statistical significance of this result?
3. As a journal editor, you receive a manuscript that contains the following statement: "The treatment regimen
prolonged the time to relapse by an average of 65 months (95% el, 45 to 85 months)." What would you
conclude about the clinical importance and statistical significance of this result?
4. The following statement appeared in a journal article "The new drug lowered the serum cholesterol concen
tration by an average of 1 mg/dL (95% el, -14 to 16 rnq/dl.)." What would you conclude about the
clinical importance and statistical significance of this result?
5. A researcher who conducted a double-blinded, randomized trial of a new intervention for reducing dizzy spells
reports that the treatment group experienced an average of 3.4 fewer dizzy spells per week than the
comparison group (standard error of the mean, 0.8). What would you conclude about the clinical importance
and statistical significance of this result?
6. How is the ability to reject the null hypothesis affected by a reduction in the a level?
7. How is the chance of a Type 1 error affected by a reduction in the a level?
8. How is the chance of a Type 11 error affected by a reduction in the a level?
Answers to practice questions
1. Clinically important: there is a 15-fold difference in the rate of influenza.

Not statistically significant (P> 05)
2. Not clinically important: the decrease in blood pressure is only 1 mm Hg.
Statistically significant (P < 05)
3 Clinically important: there is a 65-month difference in the time to relapse
Statistically significant: because the 95% el does not include the null hypothesis value of zero, this result would
be considered statisticaily significant (i e., P < .05).
4. Not clinically important: the difference in the cholesterol level is only 1 mg/dL).
Not statistically significant: because the 95% confidence interval includes the null hypothesis value of zero, this
result would be considered not statistically significant (i .e., P> .05).
5. Clinically important: an average reduction in dizzy spells of 3.2 per week was achieved.
Statistically significant: the decrease of 3.2 dizzy spells/week is 4 SEslarger than no effect.
6. Make it more difficult: a lower a level would require a lower P value.
7. Reduced chance of a Type 1 error a lower a level would require a lower P value.
8. Increased chance of a Type 11 error a lower a level would require a lower P valué.
© 2011 The Royal College 01 Physicians and Surgeons oí Canada 225

SUMMARY CHECKLlST
o Thinking about your study, write down the hypothesis or hypotheses related to your research question(s).
O For each hypothesis in your study, select an appropriate statistical test to be included in your analysis. Bé sure to justify
you choice of tests.
O Determine what you will do to avoid Type I and Type 11errors in your design and analysis.
O Review your proposed statistical analysis with your supervisor and a statistician. Revise.
O Estimate the sample size for your study (if applicable).
C')
t: •
-
....pi
U
,
::J •
'-c
c: ••
O ••
U 4
•
ti
226 © 2011 The Royal College 01 Physicians and Surgeons oí Ca-,,:::¿

26
Interpreting your research findings
Stephen Choi, MD, FRCPC
ILLUSTRATIVE CASE
A resident in Neurology has just performed a medical record review of all patients with severe head injury treated within
the past 3 years at the hospital where she is training. She has formulated the hypothesis that patients with certain clinical
variables have different outcomes than patients without those variables. She has also collected data on numerous other
patient characteristics. Now, with the help of a biostatistician, she is ready to analyze and interpret her findings and to
assess her hypothesis against the results.
Are my results significant? P values

• The collection of data, their logical
versus confidence intervals
analysis, and the interpreration of rhar analysis form the
essence of research. In quantitative research, data means The Pvallle is a measure of probabiliry rhar is used time
numbers, and analysis means statisrical calcularion. There and again in medical research, Alrhough rhe reponing of P
is no question that interprering your results dernands famil- values is ubiquitous, rhe concept of rhe Pvalue is somewhat
iariry with rhe statistical tests that were used tO generate circuitous and can take a moment [O get your head around
rhern. However, this chaprer is not meanr [O provide an at hrsr. Ir is tied [O rhe idea of rhe null hypothesis, which
overview of the mulritude of statistical tests used in rnedical for the pllrposes of a rypical clinical trial mighr be defined
research (see chs 24 & 25), but rarher ro provide you wirh
some guidance on approaching the inrerprerarion of your
CHAPTER OBJECTIVES
data. 111isprocess is the most inrellecrually engaging pan of
doing research. After all the grum work of collecting your After reading this chapter, you should be able to
data, you can pause to reílecr: "What is the discernible • determine whether your results are statistically significa
poinr of my results? Whar is irs irnporrance?" This chaprer and contrast the information conveyed by P values ersus
contrasts statistical significance wirh "real-life" or clinical confidence intervals
importance, in pan by explaining rhe difference berween • distinguish between statistically significant and clinical/y
hypothesis testing and rhe calculation of confidence inter- important results and describe the importance of
vals. Special consideration is given ro observational studies, "negative" studies
the category of research rnosr commonly performed by • list special consiclerations to take into account in
health care trainees, Finally, a guide [O preseming an ínter- interpreting data from observational studies
preration of your study [O orhers is provided, • describe how to prepare the discussion section of your
manuscript or conference presentation.
_ .. _------ ------ ---- '---1

KEY TERMS
Associations "Negative" studies Publication bias
Causal relationships Null hypothesis Statistical significance
Clinical importance Observational stuclies Subgroup analysis
Confidence intervals Odds ratio
Hypothesis testing P value

as the hypothesis "thar there is no difference berwecn out- rhar would result if one were able, hypothetically, ro test t]
comes as a result of the treatrnents being compared."1 Ir is entire population.
important ro grasp that the calcularion ofthe Pvaluc is pre- Given rhat the confidence interval do es not provide
mised on rhe assumption that rhe null hypothesis is true. In clear-cur ourcorne, its interpretation requires sorne nuanc:
other words, if we assume rhe null hypothesis ro be rrue, Here is an example used by Guyan and colleagues ro illus
then what is the probabiliry thar rhe observcd finding trate this." Let's say that you find an absolute risk reducrioi
occurred simply by chance? The lower rhe P val ue, rhe less for patients that favours rreatrnenr A over trearrnent B wid
likely ir is that the finding is attributable to chance, and the a 95% confidence interval of-1.2 ro 12. Because rhe confi
more likely thar the null hypothesis is false and should be dence interval spans O (and thus includes the null hypothe-
rejecred. This process of reasoning and calcularíon is sis), rhis is nor a sratisrically significant finding (i.e., P>.05).
referred ro as hypothesis testing. I The confidence inrerval indicares that trearrnent B might
The P value convemionally used in medical research is sligbdy increase risk by l.2%, bur it is just as likely rhat
.05; that is, a probabiliry value ofless rhan 5% (P < .05) is treatrnent A might result in a 12% absolute risk reduction .
rhe rhreshhold below which any resulr is judged unlikely ro However, rhe finding here should not be dismissed as "neg-
have occurred by chance and is rhus deemed statisrically arive" because it is more likely rhar rhe real value rhar lies
significanr. This cut-off: however, is arbirrary. The meaning within the confidence interval is probably somewhere in
of a cut-off of .05 can be put in rhese tcrrns: assuming the rhe middle, i.e., around 5%. The conclusion, therefore,
null hypothesis ro be true, rhere is a 1/20 probability that rnust be rhar a larger srudy-which would likely produce a
rhe finding observed was due to chanceo Ir you take a narrower confidence interval-is needed ro investigare the
moment ro reíiect on the definition of rhe P value, you question further. Here, the adjective "signiíicanr" certainly
should come ro rhe conclusion that the absolure numher of applies to the results, although by rhe terms of conventional
the P value for any given result does matter and rhat ir is statistics they are not.
actually rather crude ro dichotomize a value as eirher great- And so, wherever possible, repon your results with confi-
er or less than .05. Thercíore, you should always include rhe den ce inrervals, Alrhough yOl! may be familiar wirh seeing
anual P values when reporting your data, and not simply confidence intervals around estima tes such as risks and odds,
whether a finding was "statisrically signiíicant." rhey can also be calculated for resulrs such as likelihood
This dehnition of the P value and its applicability ro ratios, sensitivities and speciticities, ro name a few. When
hypothesis resring should also make you question rhe whole you are interpreting statisrical data, it is absolutely crucial
norion ofhypothesis tesring, since ir demands a binary out- that you enlisr the help of a statistician. lf you take just one
come: either accepting or rejecting a null hypothesis. Con- message from rhis chaprer, this should probably be ir.
fidence intervals-conventionally calculated as 95%
confidence imervals-provide an alrernarive to this binary
What to do if you do not obtain
analysis by presenting a mnge of plausible values for the
a statistically significant result
parameter being described.' This range is construcred ro
include within it the real value ofthe parameter. Thus, for a Ir's importanr ro realize that, alter al! your hard work, you
given 95% contidence interval rhere is a 95% chance that might not end up with a staristically significant resulr.
rhe actual value of rhe parameter being measured falls with- Although rhis can be discouraging ro the budding research-
in thar interval, The width of the confidence interval pro- er, it is nonetbeless a realiry of doing research: the null
vides a wealth ofinformation as ro how small, and how big, hyporhesis might win the day. However, the facr thar you
rhe actual value of the parameter that is bcing estirnated did not find a difference is still an importa m finding rhat
Illight be. In general, a narrow confidence inrerval reassures the world can benefit from. Ir is inadvisable ro re-run ta is-
us of che prccision of rhe estimare, whercas a wide confi- rical tests unril you hnd a way ro achieve a sratisricallv sig-
dence inrerval invites doubr as ro thar precision. The sample nificanr result, In her enlightening work, Hoto UJ read a
size used in your srudy will likely be the mosc importam papa, Trisha Greenhalgh" lisrs 10 ways of" heari ~- . e
deterrninanr of the width of rhe confidence interval, since a wriring-up of statistical resulrs (Textbox 26.1).
larger sample size usually means more outcomes of inrerest Ir is worth mentioning here the issue of suhgronp anal-
and rherefore would more closely approximate the range ysis as it relates ro points 1 and 9 in me rextbox, This rerers
228 © 2011 The Royal College 01 Physicíans ano s_ -;e:::r:s:::= :~a:;:

26 Interpreting your research findings
ro the S(l'aregy of subdividing your popularion into smaller resulr rhar, although srarisrically significant, is acrually a
groups, rypically ro determine wherher a given subgroup is false posirive finding. Do not be rernpred ro creare sub-
ar panicular risk for an ourcorne. Ofren, subgroup analyses groups that are not clinically meaningful, with rhe airn of
rnake good sense (e.g., parienrs with diabetes in a study of rrolling for P values < .05. 1his dubious practice is quite
cardiovascular disease), and when this is the case ir is opti- likely ro mislead you and your audience inro unfounded
mal ro plan for rhese subgroups apriori-rhar is, at rhe OLl(- conclusions.
ser of your srudy. Subgroup analyses can become Moreover, ir seems rhar medical journal edirors are gain-
problematic, however, when they are conducted post hoc ing a betrer appreciation of rhe irnportance of publishing
(i.e., afrer all the data have been collecred). The reason is whar are terrned "negative" studies-rhar is, studies that
rhar when you creare many srnaller groups wirhin your pop- do not dernonstrare a statistically significane difference
ulation, the likelihood of finding a statistically significane berween rwo paramerers or rhar, ro use rhe language of clin-
result becomes much higher in rhose smaller groups because ical epidemiology, cannor disprove the null hyporhesis.
rhe number of outcornes of interese is usually lower, As There is no quesrion thar publishing studies that dernon-
such, rhere is a very good chance rhar you will obtain a strare a new finding are inherendy "sexier": rhey generare
more interest in rhe medical cornrnuniry and more buzz in
the media aírer they are published. The result of this selec-
TEXTBOX 26.1: TEN WAYS TO CHEAT ON
tive publicarion of "posirive" results is ro lirnit rhe range of
STATISTICAL TEXTS WHEN WRITING UP RESULTS.
valid dara thar are available: this is referred ro as publica-
ADAPTED FROM GREENHALGH4
tion bias. Bur "negative" srudies are crucial ro our under-
standing of a topic, and are especially relevant ro systernatic
1. Abandon your a priori plans for data analysis, look reviews and mera-analyses of the lirerarure, whose results
for any relationship that gives you a P value < .05 will be distorted if negarive studies are systernarically
andchoosetofocusyourpaperonthat. excluded. Your negarive study, if ir is of high quality and
2. If baseline differences between the groups favour includes a sufficiendy large sarnple, might in fact trump
the intervention group, remember not to adjust for another study showing a posirive result when rhe lirerarure
them. is reviewed and surnmarized. By the same token, there is a
3. Do not test your data to see if they are normally sound argumene ro be rnade rhat ir is unerhical not ro pub-
distributed. If you do, you might get stuck with lish research findings-wherher rhose nndings are staristi-
non-parametric tests. cally significane or nor-since a null finding nonerheless
4. Ignore all withdrawals and non-responders from the contributes to our undersranding of the hyporhesis being
analysis. posed in the study The bortorn line is ro rake hearr if rou do
5 Plot data against each other that don't make not obtain a sraristically significanr result: be honesr and
intuitive sense in the search for a statistically don't try ro fudge the analysis ro obrain one, as the null find-
significant correlation coefficient. ing is srill irnportant and deserves ro be published if your
6. Disregard outliers when doing so favours your srudy has been well designed and properly conducred.
hypothesis.
7. Omit non-significant results.
Observational studies
8. Perform an interim analysis and alter the length
of the planned study time if that results in a Ir is highly unlikely rhar you will have rhe rime, resources or
statistically significant result. expertise ro conduct a mulri-centre randomized interven-
9. Perform subgroup analyses to search for a tional trial during rhe course of your training programo
relationship where P < .05 even though the Rarher, you willlikely focus on a srudy design rhar inrro-
subgroup is not of interest. duces you ro the whole notion of asking a research quesrion
10. If none of your results are statistically significant, try and undercaking a practica] process for testing an hypothe-
different statistical analyses in the hope of obtaining siso Because observational studies are generally less cornpli-
one that is. cared than inrervenrional trials, they tend ro be more feasible
ro manage alongside rhe many other demands of training.

Specifically, most health care trainees undenake case-control

Preparing the Interpretation section
studies and cohort studies (especially retrospective versions of Wherher you are wriring a manuscript for publicarion or
these) to fulfil me research requirement of their programo preparing a talk for a conference, symhesizing your data
Having a salid grasp of rhe fundamentals of these study inro a meaningful inrerpretation (or discussion) is funda-
designs befare you begin will save you grieflater. In panicu- men tal ta the success of either, If your research paper were a
lar, undersranding borh the advantages and Iirnitations of legal argllment, your study protocol and analysis would
observarional studies? as well as the imporranr sources of represem irs premises, and the interpreration would pres-
porential bias (i.e., selection, misclassification, confound- em your closing arguments. Here is a simple, fail-proof for-
ers)" is crucial ro interpreting your study results. Ir is impor- mula for organizing your interpretation section.
tant to keep in mind that observational studies provide us
wirh associations rarher rhan causal relationships, even • Summarize rhe mosr important results of your study,
though the strength of those associarions might suppon rhe These may be statistically significam or not,
case for causation. Because the odds ratio is one of rhe • Pur your resulrs into the context of whar is already
most common statistics used ro measure rhe association known in your topic area. This should be rhe bulk of
berween an exposure and an ourcorne," it is importanr ro the interpretation. Do your data conrradict existing
have a good grasp of what ir means. Basically, it is rhe ratio knowledge, or are rhey consisterit wirh ir? How did
of the odds of an event occurring in one group (e.g., the your study prorocol differ from that of previous
experimental group) ro rhe odds of the event occurring in invesrigarions, and how should those differences
another group (e.g., the control group). In clinical studies, iníiuence rhe interpretation of your results in light of
ir is common to see odds ratios as rhe fina! result of a logis tic rhe existing literature? Carefully situaring your studv
regression analysis that aims ro find variables that seem ro wirhin the conrext of existing lirerature should help
be independendy associated with the outcorne. Finally, you to determine what yom study adds to the field.
alrhough logistic regression and correlation analyses are • Whar are the lirnitations of your srudy? This is a
related rnerhods of finding associations berween variables, crucial section ro reflecr on and prepare carefully.
),0'_1 need ro understand the difference berween rhem.R,9 Every study, including yours, has limitations.
Simply put, logis tic regression analyses aim ro determine Imerprering your data in a cogent way dernands
rhc relation berween an exposure ami an event and, in so that you consider rhese limirations. Oversrepping
doing, suppon the case for a causal relation (e.g., smoking whar rhe data can actually say is a common pirfall
and lung cancer); correlarion, on the orher hand, aims ro for many junior scientists (e.g., making a claim of
determine rhe closeness of rwo variables without making a causality when an association berween a risk variable
case for causality (e.g., urnbrella use on a rainy day, or evalu- and outcorne is found). It is crucial ro consider the
ation scores on professionalism during residency uaining effect of missing data and not to shy away from any
and frequency of patient complaints posr-training). limitations in your study's conception and designo
w_ Perhaps the greatesr rnistake ro make in interpreting an All of this is ro say rhat it is irnporranr ro be realistic
association is ro infer causality when none exists. Andrew about your study and its contriburion ro scienri hC
Wakefield's repon of an associatiori berween the MMR vac- knowledge. You will not find rhe cure for cancer in
cine and autism, based on the accounts of rhe parents of 12 your observational srudy, but if you have carried out
children wirh autism in 1998, is an example of the potential the work carefully, you will surely contribure one
abuse ofinferring causality from a weak association berween small piece of a large puzzle that, just maybe, you
two variables. Besides this transgression, it appears rhat will choose ro spend the rest of your career helping
\'V'akefield may-far worse-have acrually fabricated his ro salve .
dara;'? this aside, rhe mere infercncc of causality in this • Finally, concludc wirh whar you fec! is rhc most
instance was, uníortunately, enough ro entice one of the irnportant finding of your srudy and poinr me way ro
world's most presrigious medical journals to publish Wake- furrher research that is needed in your tapic area,
fdd's paper. The result has been a profound and long-last-
ing effect on rhe public's trust in vaccines and on public
healrh policies surrounding vaccination. Wakefield's paper
has since been retracred by 7he Lancet. 11
230 © 2011 The Royal College oí Physicians and Surgeons ,,-2::2

26 Interpreting your research findings
Conclusion the tests used ro determine associations berween variables.

Ir's irnporrant ro diflerentiate a finding thar is statisrically As a rule, observarional srudies do nor prove causal rela-
significant from one rhar is clinically imporranr. Alrhough tionships, and so ir is irnporrant ro understand rhe lirnirs of
hyporhesis tesring is one wa)' of derermining statistical sig- when one can make a case for inferring causaliry versus
nificance, also consider wherever possible rhe use of confi- associarion. Finally, ir's importanr ro convey )'our resulrs
dence inrervals as a way of reporring and imerprering your and interpreration by wa)' of peer-reviewed publicarion.
data. As many of you will perform observarional studies as The sreps aboye will provide you wirh a useful road map ro
your firsr research endeavour, ir is importanr ro understand preseming your argumenr. •
CASE POSTSCRIPT
After performing initial statistical testing to look for associations between factors in the clinical presentation and the
outcomes of interest, the resident finds no statistically significant associations. However, she had intended a priori to examine
a large subgroup with a pre-existing comorbidity and found that there was, in fact, an association with respect to this factor
in this group of patients. Overall, the null hypothesis was true-but, for an important patient group, the association found
should prompt, the resident hopes, future clinical trials specifically for this subgroup of patients.
REFERENCES
1. Guyatt G, Jaeschke R, Heddle N, Cook D, Shannon H, Walter S. Basic statistics for clinicians: 1. Hypothesis testing. CMAJ.
1995; 152(1) 27-32
2. Gardner MJ, Altman DG. Confidence intervals rather than Pvalues: estimation rather than hypothesis testing. BMJ.
1986;292(6522) 746-50.
3. Guyatt G, Jaeschke R, Heddle N, Cook D, Shannon H, Walter S. Basic statistics for clinicians: 2. Interpreting study results:
confidence intervals. CMAJ. 1995;152(2) 169-73.
4. Greenhalgh T How to read a paper: the basics of evidence-based medicine. 3rd ed. Oxford: Wiley-Blackwell; 2006.
5. Kestenbaum B. Epidemiology and biostatistics. an introduction to clinical research. New York: Springer; 2009.
6. Rothman KJ. Epidemiology: an introduction. New York: Oxford University Press;2002.
7. Jaeschke R, Guyatt G, Shannon H, Walter S, Cook D, Heddle N. Basic statistics for clinicians: 3. Assessing the effects of
treatment: measures of association. CMAJ. 1995; 152(3):351-57.
8. Guyatt G, Haynes B, Sackett D. Analyzing data. In: Haynes RB,Sackett DL, Guyatt GH, Tugwell P,editors. Clinical epidemiology:
how to do clinical practice research. 3rd ed. Philadelphia: Lippincott Williams & Wilkins; 2006. p. 446-60.
9. Guyatt G, Walter S, Shannon H, Cook D, Jaeschke R, Heddle N. Basic statistics for clinicians: 4. Correlation and regression.
CMAJ. 1995; 152(4) 497-504
10. Deer B. How the case against the MMR vaccine was fixed. BMJ. 2011 ;342:c5347.
11. [Editors of The Lancet.] Retraction - Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental
disorder in children. Lancet 2010;375(9713) 445
ADDITIONAL RESOURCE
Haynes RB, Sackett DL, Guyatt GH, Tugwell P,editors. Clinical epidemiology: how to do clinical practice research. 3rd ed. Philadelphia:
Lippincott Williams & Wilkins; 2006.
• This seminal work in clinical epidemiology and research should be required reading for anyone who strives to understand how
research is done and the fundamental principies of evidence-based medicine.

SUMMARY CHECKLlST
o Review your results. Do they surprise you? Do any stand out? Do the data answer your questions as planned?
D Identify the results that were "significant" and those that were not.
D Write your draft interpretation of the results, summarizing them in the context of the existing related literature and
hypotheses.
O Describe the limitations of your data.
D Write a conclusion that summarizes the relevance and clinical importance of your findings .
.-
232 © 2011 The Royal College 01 Physicians and Surgeons oí Canaoa

27
Writing effective abstracts
Carolyn Brown, ELS
ILLUSTRATIVE CASE
A resident is working on a research project comparing cytologic findings after surgery for various conditions, and she
tabulates exciting interim results. She asks to discuss the results with her supervisor, who comments that they would make
an excellent poster or oral presentation at the upcoming meeting of the Society of Obstetricians and Gynecologists of
Canada (SOGC). The deadline for abstract submissions is in a weekl The supervisor directs the resident to the website for
the conference, which indicates that the conference accepts abstracts for research in progress. The website also provides
a word count and some general guidelines for abstracts for the meeting. The countdown starts for the resident to provide
a clear abstract that conveys the importance of her work and that (she hopes!) will be accepted for presentation at the
meeting.
• Abstracts serve several important journal publicarion, which is ro help rhe reader decide
funcrions rhat vary according (O conrext. In a posrer or oral whether ro read the full paper. A reader looking for an
presentarion at a meeting, rhe abstraer cornmunicares the answer ro a clinical quesrion or for current research in a par-
nature and significan ce of a research projecr in a compact ticular area needs ro glean sufficient inforrnation from the
fashion thar is quick for other researchers to read and grasp.
Through rhe presenrarions and the publication of the
CHAPTER OBJECTIVES
abstracrs in a program or proceedings, researchers learn
about current areas of investigation. Since some meerings After reading this ehapter, you should be able to:
accept abstracts of research in progress, meering abstracts • describe the requirements for abstracts for posters or
can also serve as preliminary repons. However, several stud- presentations at meetings, and for abstraets for research
ies have shown rhar only a portion of meeting absrracts- papers and other types of publications (sueh as reviews);
ranging from 34% ro 52%-are followed by rhe publicarion • describe and apply an approach to writing abstracts for
of final results in the peer-reviewed lirerature."? meetings and for publieation;
\'(1hen an abstraer accompanies a paper subrnirred ro a • use ehecklists for what to inelude in an abstract;
journal, irs funcrion is somewhat different. Most definí- • diseuss styles of abstraets;
tions state that the abstraer summarizes and condenses the • make abstraets coneise; and
research paper.6-9 While technically corren, rhis narrow • ensure abstraets can be readily found in electronie
definirion misses one of esseritial roles of rhe abstraer in a searehes.
KEY TERMS
Abstract guidelines Electronic searches MeSH
r Abstract headings
Background
IMRAD structure
Informative abstracts
Methods
Objectives
Conclusion Interim results Results
Conference abstracts Introduction Struetured abstraets
CONSORT checklist Journal abstraets Unstructured abstracts
Descriptive abstraets Keywords
-_.- ------_. __ ._ - ---- -- ---------__j
© 2011 The Royal eollege of Physicians and 5urgeons of Canada 233

abstract to decide wherher it is worth rhe time in a bus)' Most guidelines srate rhar the abstraer should summa
schedule to find and read the paper.? 1he abstraer should rize che research study following rhe IMRAD structure of
therefore be able to stand alone," as the first-and, uníorru- research rnanuscripr: Introducrion, Methods, Resulrs, anc
narely, often the only-pan of the paper most readers read, Discussion.F''!':" 1his can apply even when the acrua
apan from the title and author list. 1here are even docu- headings are nor required.
mented cases in which readers have made clinical decisions 1he introduction supplies the research question 01
from abstracts (although they shouldn't have) because the study objective as well as background explaining why the
fuI! text was unavailable.'? quesrion was srudied, (In some sryles, background and
In rhe electronic information age, the abstraer is also a objectives are broken out separately.") No more than rwo
means for porential readers to find published articles. Elec- or three sentences should be used ro cover rhis section.
tronic searches of abstracts contained in literature darabases 1he methods should cleady indica te rhe study design,
such as PubMed (MEDLINE), Web of Knowledge, and instrurnent (e.g., survey), laborarory rechnique, assignment
Scopus (to name a few) lead readers to anides that they or allocation of human or animal subjects, inrervenrion,
might orherwise be unaware of. An abstraer must therefore and outcornes measured.
be wrirten in a way rhat ensures that the paper is not missed The results should provide data for the outcornes that
in searches where it would be relevant. address the research question. All results should be suPPOrt-
ed with appropriate descriptive statistics (e.g., mean differ-
ence berween groups; odds ratio), a measure of precision of
From research to abstract
the estimare (e.g., 95% confidence interval), and a clear
The best abstracrs describe a well-designed and carefully indication of whether the result is statistically signi ficant. 11
conducted srudy that addresses an impcrtant and interest- However, it may not be possible to provide such statistics
ing problem. A good abstraer captures rhe soundness and for interim results.
significance of a good study; it cannot salvage a poor one. If the meeting guidelines allow results to be presented
In preparing a conference abstraer for the SOGC meet- in atable or graph, this is often an eflicient way ro sum-
ing, our gynecology resident goes back ro her original marize findings. Again, tables 01' graphs should indude
research question. What does the study set out ro show? results for rhe relevant outcornes only; unless the study
How does ir do rhar? Which interirn data are relevant to her had few subjects, atable or graph should not indude all
hyporhesis"!' Finally, why is the research importanr? Whar data.
has she learned from ir? AlI orher data or outcomes of the 1he conclusion should answer the research question
research, no matter how inreresring, should be set aside. and, as appropriare, explain rhe importance of the findings
011ce she has collated from her notes the material that in rhe larger research contexto Caution should be exercised
answers these questions, the resident checks che guidelines for in reaching condusions; rhese should not go beyond rhe
abstraer submissions on the conference websire.P Such guide- original research question.
.- lines may indude a word lirnir, abstraer hcadings,

acceptable acronyms and orher abbreviations, information on
a list of 1he relevance of the research should be clearly conveyed
in the abstraer, as rhis is an important opporruniry for rhe
whether tables and figures can be induded with me abstraer researcher ro communicate his 01' her firidings ro rhe con-
submission.v? and so on. Not all conferences accept interim ference cornmittee. 1he conference cornmittee judges rhe
results: if you are planning to present a preliminary report, be absrracts subrnitred to determine which will be accepted for
sure ro check the guidelines before submitting. the meeting. Reviewers on the cornrnirree usually assess
Many beginning researchers do not realize that abstract abstracts on several factors, including rhe purpose of [he
guidelines are strict and should be followed to the lerrer. research, its methodological rigour, the inclusion of all ele-
Exceedi ng the word cou nr by even one word, tor example, mcnrs, and the rclevance of rhe srudy ro rhc discipline i
can result in rejection or truncation of rhe abstraer.'? Use general and to the conference specifically. Clariry of presen-
rhe word count function in word processing sofrware ro tation may also be scored. Reviewers may recommend
determine rhe length of the abstract.Tr is also a good idea ro whether abstracrs should be accepred for a pos ter presenta-
read absrracrs from prcvious meetings" ro get a sense of che tion or an oral presenrarion (the larter is usually more pres-
required conrent and sryle. tigious), or rejected.
234 © 2011 The Royal Coilege 01 Physicians and Surgeons 01 e ar2::C

27 Writing effective abstracts
To ensure rhar your abstraer is elear and conveys irs mes- she meets vi h her supervisor, who encourages her to
sage, apply al! of rhe usual rules of good wriring (see ch. 29): complete her s udy and write a paper as soon as possible.
That means a ehallenging month ahead, as she tabulates
• Prefer the active voice." final data and writes the papero lt's almost ready to submit
• Use srrong, simple verbs, such as "ro determine," to the Journal of Obstetrics and Gynaecology Canada,
"studied," "found," "rreared." when she remembers she needs to add the abstraet. She
• Use linking words such as "however," "also," "yet," opens the abstraet she submitted to the eonferenee, seleets
"furrhermore." it, and hits "copy." Fortunately, just as she is about to
• Avoid needless phrases such as "We coneluded rhat switch files and hit "paste," she takes a moment to read
"6
it overo She realizes her abstraet doesn't follow the same
• Use terrns consisrendy. strueture as the other abstraets she has read in JOGC And
• Wrire in rhe pasr tense" (excepr for background abour the information is now out of date. In faet, in her paper she
the research quesrion, which may be in the presenr has ehanged the eonelusion.
tense),
• Avoid acronyms and orher abbreviations, as rhey creare
a diflicult-ro-read "alphaber soup."6,12,1}
From paper to abstract
• If you do use acronyms and other abbreviations, use
only well-established ones if possible (do not coin new Despite their obvious imporrance, journal abstracts are far
ones) or those specified in the conference submission too ofren unelear and incomplere.2,6,14-1(,Somerimes this is
guidelines. because the main effon is focused on rhe paper, and rhe
• Define acronyms and orher abbreviarions the firsr rime abstraer is wrirten hurriedly, as an afrenhoughr. Somerimes
rhey are used (some conference submission guidelines ir is because a previous meering abstraer has nor been updat-
allow exceptions such as HIV and DNA).6 ed. Somerimes the problem is a lack of understanding of rhe
abstracr's funcrion, or a lack of objective srandards for pub-
You may find thar, in an efforr ro include all important lished absrracrs-rwo issues rhar many journal edirors and
points, you have exceeded the word lirnit in the Iirsr drafr of research organizarions have rried to address.IO,17,18
your abstraer. The next srep is ro trirn ir down. Sran by The best approach is ro write the abstraer lasr, working
llnding ways ro say the same rhing more succincdy. Exam- from rhe complered paper. Any previous version should be
ine phrases ro see if rhe meaning would be conveyed jusr as eirher ser aside or substanrially revised. 1his means rhar
well if the phrase were dropped. Then, if need be, fOCLlSthe rime and effort musr be devoted ro the absrracr: Ir should
abstraer. Is the research question elearly stared, and are all not be forgonen until just before rhe deadline!"
subsequenr irerns ried back ro rhat question? Is there any Mosr journals inelude instrucrions for rhe abstraer-e-
exrraneous informarion thar does not add ro rhe conelu- word lirnir, structure, headings, and so on-in rheir
sion? insrrucrions for aurhors. Be sure ro consulr rhese befo re
srarting ro write or revise. For sryles of abstracts rhar may be
speciíied in instrucrions for authors, see rhe nexr secrion.
Beginning researchers ofren make rhe mistake of rrying
THE CASE REVISITED
ro boil down their enrire manuscripr into the abstraer. In
The resident writes an abstraet, following the submission the case of our gynecology resident, she may have presenred
guidelines earefully, and being sure to include all data frorn differenr caregories of patienrs, from different
information relevant to her researeh question in the IMRAD rissues, and so on. In experiments involving humans, the
strueture. She submits the abstraet by the deadline, and dara can be complicared, relaring ro severa! arms of rhe
hears from the seleetion eommittee a month later. She has srudy, side effecrs, and patierir drop-outs. An abstraer can-
been seleeted to make an oral presentation I A few months not present every detail of a complex manuscripr. You
later, the big day arrives. She triple-eheeks her PowerPoint should also be aware thar, alrhough tables and charrs may
file and boards a bus to the eonferenee. The conferenee be accepred in conference abstracts, they do not appear in
presentation goes very well. Baek at her university, abstracts for published papers.
© 2011 The Royal College 01 Physicians and Surgeons 01 eanada 235

The Research Guide: A primer for residents, other heal h care rainees, and practitioners
Whieh derails should be induded? Again, rerurn ro your headings and order of elernents. However, ir is a useful i
primary researeh quesrion and foeus on the data that answer adjuner ro rhe journal insrructions ro ensure the informa- I
rhe question. tion is induded, even if it is under a different heading or in I
Several journals and researeh groups have made specific a diflerent order.
recornrnendarions ro ensure that relevant data are nor Table 27.2 provides the abstraer headings required by 1
missed or ornitted from published absrracts. Table 27.1 rhe Annals 01 Internal Medicine for original research;: cost- •
shows a checklist from the CONSORT grouplO for the data effeeriveness studies, and systernatic reviews (induding
ro indude in an abstraer of a randomized controlled trial. mera-analyses). These headings were firsr recommended in ~
The aim of this ehecklisr is ro provide speeific instrucrions 1987-1990 by proponents of evidenee-based medicine ro
about key elernents of a trial that should be induded in the facilitare peer review, help clinical readers find anides rhat ~
abstraer. The authors argue rhar, without a minimum of are seientifically sound and applieable, and allow more pre- •
key informarion,
applicabiliry of a tri al.
ir is difficult ro assess rhe validiry and cise elecrronic literarure searches." For manuscripts
rined for journals that do not use rhese headings, the list •
des-
"
This ehecklisr mal' nor Iir the abstraer style specified by simply provides anorher useful checklist for informarion ro
your rarger journal in its instructions ro authors, in rerrns of indude in abstraets of the cypes of manuscripts eovered.
I
••
• Table 27.1: CONSORT checidist for abstracts for randornized controlled trials
Title'" Identification of the study as randomized
17
• e
Authors Contact details for the corresponding author e
Triai desiqn Description of the trial design (e.q. parallel, cluster, non-inferiority) e
Methods
Participants Eligibility criteria for participants and the settings where the data were collected
•
Interventions ; Interventions intended for each group
'•"
Objectives
Outcome
Randomization
¡ Specific objective or hypothesis
Clearly defined primary outcome íor this report
How participants were allocated to interventions
•
Blinding (masking) Whether or not participants, care givers, and those assessing the outcomes were blinded to
group assignment
Results
Numbers randomized ! Number of participants randomizecJ to each group
Recruitment ¡ Trial status
Numbers analyzed ¡ Number of participants analysed in each group
.- Outcomes
I
¡ For the primary outcome, a result for each group and the
¡ estimated effect size and its precision
Harms : Important adverse events or side effects
·----------------------r---·--------------------------------------------------------------------------.-.-------
Conclusions ! General interpretation of the results
-----------------------------1----------------------------------------------------------------------------------.-
Trial registrationt ! Registration number and name of trial register
------------------~ ........•. _------_._-----_.----_.------------------------------------------------------------
Fundingt
~_ _-------------_.~
..
i' _._----~--_._-~------------------------------------------
Source of funding
* Although they appear in the CONSORT eheeklist, the title and authors are not usually included in abstraets lar eonlerenees or journals.
t While COCJSORTrecommends that these items be included in the abstraer, many journals publish them with the author afliliations or
aeknowledgements. Check the style 01 the journal.
._---------------------------------_._--------------------------
236 © 2011 The Royal College oí Physicians and Surqeons o' c~.a::.a
" Iosr abstraer srrucrures are based on IMRAD, wirh

• Table 27.2: Annals of Internal Medicine so me variarions. 1he heading scheme for abstracts in Annals
headings for abstracts for three rnanuscript 01 Internal Medicine (Table 27.2) has the most headings of
types"
any system currendy in use." Many journals require struc-
Original research
tured absrracrs for cerrain rypes of manuscriprs (research
Background
Objective papers, reviews), and unstructured abstracts for orher rypes
Design (edirorials, brief reporrs). When writing a strucrured
Setting abstraer with many headings, ir is aceeptable ro write in
Patients
point Form (excepr thar the Resulrs and Discussion often
Intervention (il any)
Measurements require full senrences), whereas unsrructured absrracrs
Results should be wrirren in full sentences.
Limitations
Even if an abstraer is unstrucrured, the rext should
Conclusions
include all rhe elemenrs oF the IMRAD structure, in the
II the study is a randomized controlled trial, list
where the trial is registered and the trial's
appropriare order. For research paper abstracts, an appro-
unique registration number at the end 01 the priare checklist such as CONSORT is a valuable rool.
abstract.
Cost-effectiveness studies Informative vs descriptive abstracts. Abstraer sryles ean

Background
also be categorized as inforrnative or descriptive (somerirnes
Objective
Design
called indicatiuei.' Mosr abstracrs, especially for research
Data sources papers, are informative, providing the information on the
Target population srudy design, rnethods, results, and eonclusion in the
Time horizon
IMRAD structure, or following one of rhe alrernative head-
Perspective
Interventions ing structures, However, there are certain con texts in which
Outcome measures it makes more sense ro indicare rhe arricle's scope, rhe prin-
Results 01 base-case analysis
cipal subjecrs discussed, and how the topics will be
Results 01 sensitivity analysis
Lirnitations
addressed in the anide, providing an overview of the
Conc!usions paper.i'Ihis rype of abstraer is called "descriprive." Descrip-
5ystematic reviews, including meta-analyses tive absrracrs are used mainly For tradicional review arricles
Background that prcsenr a lirerarure seareh wirh clinical conclusions.
Purpose See examples in Appendix 27.1.
Data sources
Study selection
Data extraction Abstracts for systematic reviews and case reports. Many
Data synthesis recent review papers are sysrernatic reviews, in which data
Limitations
from sysrernatically identihed and selected research papers
Conclusions
____ o _
are extracted and may even be pooled and analyzed rogerh-
er in a rnera-analysis. (See ch. 15.) In rhis case, an informa-
Types of abstracts
tive abstraer provides the mcthodology and the analysis
resulrs. See the heading scheme for systernatic reviews in
_R
Before wriring the abstraer for a paper, ascertain the cype of Annals 01 Interna! Medicine (Tablc 27.2) for a good check-
abstraer and rhe formar from rhe journal's instructions for Iist ro follow in preparing abstracts for such papers. See
aurhors. 1he following oudines sorne of rhe considerarions examples in Appendix 27.1.
in preparing absrracrs of specific rypes. Anorher rype of researeh paper ofren prepared by rrain-
ees is the case repon or case series. Again, rhe abstraer con-
Structured and unstructured abstracts. Absrracts for renr and sryle diflers. 1he abstraer may be shorter than fOI
research papers may be unstructured (consisting of one original research, and some journals do not includc abstracts
conrinuous paragraph) or structured (using bolded or itali- with such papers at al!. If required, the abstraer should
cized headings that break rhe text inro secrions)." See exarn- in dude rhe following:!'
pies in Appendix 27.1.

The Research Guide: A primer for residents, other heaith care trainses, and practitioners
• objective Keywords
e case summary or case presenrarion (include patient
Sorne journals ask researchers ro subrnir a few keywords
derails)
wirh rheir abstraer. 1he usefulness of rhese keywords is
• discussion (ornirred by sorne journals)
quesrionable, as abstraer darabases such as MEDLINE
• conc1usions
(PubMed) are professionally indexed and do not use
aurhor-supplied keywords at alL However, if researchers are •
For case series, criteria for the selecrion of cases, and rhe
asked ro supply keywords, they should selecr rhree ro five ~
number of cases rhus seJected, should also be indicated. See
rerrns that capture rhe broad subject areas of rhe research,
exarnples in Appendix 27.1. ~
eirher from a list of keywords supplied by the journal, or
from MeSH if the journal does not have a lisr.
Considerations in writing or revising 11
abstracts Short versions of abstracts
Electronic retrieval Sorne journals request, in addition, an even shorter I ••

"abstraer" of rwo or rhree sentences ro appear in the rabie of
To fine! research papers, researchers seareh abstracts in elec-
contents. Read sorne of the journal's tables of contents ro •
tronie darabases sueh as PubMed, Web ofKnowledge, and
Seopus. Alrhough the best-known darabases have profes-
sional indexers who apply indexing terms to eaeh abstraer,
ger afee! for sryle and elernents covered.
Remember to revise
,
ti
rhere is a deJay in applying terms, and not al1 databases and 4!

]ournal editors ofren eomplain that abstracrs do not refiect
seareh engincs (e.g., Google) are indexed. Henee, sorne
the paper: sometimes information in the abstraer differs
e
publishers now recommend "optimizing" abstraets so thar
eiecrronic sea,..ches will find them easily." Ir is worthwhi]e
from rhe manuscript or do es not appear in rhe manuscript e
at al], Studies have found high rates of ineonsistent data "
ro consider which terrns a researcher would use te; search
and conclusions berween published papers and rheir
for anides in the pertinenr research area, and ensure that
absrracts, even in major medical journalsY-23 This is rnain- "
those terrns appear in thc abstracto For exarnple, our resi-
ly because researchers simply forget ro revise the abstraer ••
denr's abstraer should conrain terrns about cyrologic find- along wirh the manuseripr.
ings ("cytology") in rissue samples from uterine hyperplasia
While the best practice is ro write the abstraer afrer hnal- •
("uterus," "hyperplasia"), inc1uding malignant ce1ls ("urer-
izing the paper, rherc may be unanticipared changes ro a "
ine cáncer" or "urerine neoplasrns").
manuseripr before submission bur atter rhe abstraer is wrir- ••
1n medical li rerarure, ensure as well that the relevanr
ten, If so, [he abstraer should be verified and revised ro
medical subject headings (MeSH) appear in rhe abstraer
ensure it is consistenr with rhe paper, as the focus may have
(see ch. 7). 1his can be awkward if the researcher has used a
changed during revision. Afrer peer review, rhe paper may
non-MeSH rerm throughour rhe research: [or example, if be revised, and, again, the abstraer should be verified and
the paper refers ro Srein-Leventhal syndrome, bur rhe
revised as well. 1he final abstraer should match the papc[ ••
MeSH rerrn for rhe sarne entity is polycystic ovary syn-
not only in terms of data accuracy, but also in rerrns of irs
drorne. In this case, rhe best eourse would be ro use the
conclusions and tone. For example, if [he paper reaches a _
MeSH terrn in rhe paper; if rhis is nor possible, however, ir
tentative conclusion in a narrow area, the abstraer should
is good practice ro indude rhe MeSH term in brackets aíter
not state the condusion firmly or extend the conclusion
rhe first mention in the abstraer, i.e., "parienrs with Stein-
beyond whar was conc1uded in rhe papero
Levenrhal syndrome (polycystic ovary syndrome)." 1his
ensures rhat savvy searchers, who use MeSH terrns, will find
rhe paper."
a For more about MeSH, see www.nlrn.nih.gov/rnesh
238
© 2011 The Royal College 01 Physicians and Surgeo~s : Ca-¿:¿
CASE POSTSCRIPT
The resident writes a 300-word abstract for her manuscript, a case-control study showing that a significantly higher
percentage of surgical samples from a certain benign condition (uterine hyperplasia) contain malignant cells than
samples from other benign conditions. She receives constructive peer reviews and revises the manuscript and the abstract
accordingly. The paper is accepted and appears a few months later in JOGC
Through this process, the resident has demonstrated she can:

• Read and follow the guidelines on the conference website and the instructions for authors on the journal website for
word limit, style of abstract (structured versus unstructured, informative versus indicative), and other requirements.
• Focus on the research question, and data that answer it, in preparing a meeting abstract, the paper, and the abstract for
publication.
• Use a checklist as a guide to ensure she has included all the needed elements for the abstraer.
• Substantially revise the meeting abstract to accompany the subsequent manuscript, ensuring the abstract is succinct,
focused, and consistent with the paper.
• Ensure the abstract contains words that will be searched by researchers retrieving papers in the area.
REFERENCES
1. Scherer RW, Dickersin K, langenberg P.Full publication of results initially presented in abstracts. A meta-analysis. JAMA
1994;272(2): 158-62.
2. Bhandari M, Devereaux PJ,Guyatt GH, Cook DJ, Swiontkowski MF, Sprague S, et al. An observational study of orthopaedic
abstracts and subsequent full-text pubiications. J Bone Joint Surg Am. 2002;84-A(4):615-21.
3. De Bellefeuille C, Morrison CA, Tannock IF.The fate of abstracts submitted to a cancer.meeting: factors which influence
presentation and subsequent publication. Ann Onco!. 1992;3(3): 187-91.
4. Castillo J, Garcia-Guasch R, Cifuentes 1. Fate of abstracts from the Paris 1995 European Society of Anaesthesiologists meeting.
Eur J Anaesthesiol. 2002; 19(12) 888-93
5. Marx WF, Cloft HJ, Do HM, Kallmes DF.The fate of neuroradiologic abstracts presented at national meetings in 1993: rate of
subsequent publication in peer-reviewed, indexed journals. AJNR Am J Neuroradiol. 1999;20(6): 1173-7.
6. Cornett Pl. Writing abstracts. In: Witte FM, Dew Taylor N, editors. Essays for biomedica! communicators. Volume 1 oí sele -ed
AMWA workshops. Bethesda (MD): American Medical Writers Association; 2001. p. 92-100.
7. Eastman JO, Klein ER.Writing abstracts. In: Minick P,editor. Biomedica! communication: se!ected AMWA workshops. Bethesda
(MD): American Medical Writers Association; 1994. p. 143-6.
8. Day RA, Gastel B. How to write and publish a scientific paper. 6th ed. Westport (CT) and london: Greenwood Press;2006. p.
52-5.
9. Pierson DJ. How to write an abstract that will be accepted for presentation at a national meeting. Respir Care.
2004;49(10) 1206-12.
10. Hopewell S, Clarke M, Moher D, Wager E, Middleton P,Altman DG, et al. CONSORTfor reporting randomised trials in journal
and conference abstracts. Lancet. 2008;371 (9609):281-3.
11. Alexandrov AV, Hennerici MG. Writing good abstracts. Cerebrovasc Dis. 2007;23(4):256-9
12. Boullata JI, Mancuso CE. A "how-to" guide in preparing abstracts and poster presentations. Nutr Clin Pract. 2007;22(6) 641-6
13. Foote M. Some concrete ideas about manuscript abstracts. Chest. 2006; 129(5): 1375-7.
14. Hill Cl, Buchbinder R, Osborne R. Quality of reporting of randomized clinical trials in abstracts of the 2005 annual meeting of
the American College of Rheumatology. J Rheumatol. 2007;34(12):2476-80

The Researeh Guide: A primer for residents, other health eare trainees, and praetitioners
15. Krzyzanowska MK, Pintilie M, Brezden-Masley C. Dent R, Tannock IF.Quality of abstracts describing randomized trials
in the proceedings of American Soeiety of Clinical Oncology mee ings: guidelines for improved reporting. J Clin Onca!.
2004;22(10) 1993-9.
16. Ubriani R, Smith N, Katz KA. Reporting of study design in titles and abstracts of articles published in clinically oriented
dermatology journals. Br J Dermato!. 2007; 156(3):557-9.
17. Hopewell S, Clarke M, Moher D, Wager E, Middleton p.Altman DG, et al. CONSORTfor reporting randomized controlled trials
in journal and conference abstraets: explanation and elaboration. PLoSMed. 2008;5(1 ):e20.
18. Haynes RB, Mulrow CD, Huth EJ,Altman DG, Gardner MJ. More informative abstraets revisited. Ann Intern Med.
1990;,113(1):69-76
19. Information for Authors. Annals of Internal Medicine. Available from: www.annals.org/site/miscJifora.xhtml.
20. Wiley-Blackwell. Author services: optimizing your article for seareh engines. Available from: http://authorservices.wiley.cam/
bauthor/seo.asp
21. Pitkin RM, Branagan MA, Burmeister LF.Aeeuraey of data in abstraets of published researeh artides. JAMA. 1999;281 (12): 1110-1.
22. Pitkin RM, Branagan MA. Can the aceuraey of abstraets be improved by providing specific instructions? A randomized eontrolled
trial. JAMA. 1998;280(3) 267-9
23. Ward LG, Kendraeh MG, Priee SO. Aceuraey of abstraets for original researeh artieles in pharmaey journals. Ann Pharmacother.
2004;38(7-8) 1173-7.
EXERCISES
1. Choose abstracts from published randomized controlled trials (see examples in Appendix 27.1 or search for reeent artides
in your area in PubMed). Analyze these aeeording to the CONSORT extension for abstraets. Are any elements rnissínq?
How did the researeher orqanize 'lhe elernents? 15 eaeh abstraet elear, readable, and complete?
2. Have a friend ehoose a good researeh paper from your field and remove the abstraet. Read the paper and write the
abstraet. Compare your abstraet with the author's. Does your version have any omissions or oversights? Does the author's /
O")
t: SUMMARY CHECKLlST
.- o Consider where you will be submitting an abstraet of your work. Assemble any information you will need about the
required strueture, word count, etc. Review abstraets that have previously been aeeepted to guide your eomposition.
O Consider the target audienee of your abstraet. What will they know about this type of study? What will they eare
about? What results should you emphasize?
O Construet your abstraet earefully using best praetiees.
O Send your abstraet to your team, supervisors and eolleagues for review. Revise and repeat.
O Seleet key words.
O Submit.
240 © 2011 The Royal eollege oí Physicians and S n;;2C~s : C¿<Ó~2.

APPENDIX 27.1: ABSTRAeT EXAMPLES STRUCTURED INFORMATIVE ABSTRACT
Long-term effects of dihydrotestosterone treat-

ment on prostate growth in healthy, middle-aged
men without prostate disease: a randomized,
placebo-controlled tria lb
UNSTRUCTURED INFORMATIVE ABSTRACT kien A, Griffiths KA, Harwood T,Seibel Ml, Turner L,
Conway Al, et al.
Soy and the exercise-induced inflarnmatory

Background: Benign prostatic hypertrophy increases with
response in postmenopausal wornen-
age and can result in substantially decreased quality of life
Beavers KM, Serra Me, Beavers Op, Cooke MB, for older meno Surgery is often required to control symp-
Wi/loughby OS toms. It has been hypothesized that long-term administra-
tion of a nonamplifiable pure androgen might decrease
Aging is associated with increasing inflammation and oxi- prostate growth, thereby decreasing or delaying the need
dative stress in the body, both of which can have negative for surgical intervention.
health effects. Successful attenuation of such processes with
dietary countermeasures has major public health implica- Objective: To test the hypothesis that dihydrotestosterone
tions. Soy foods, as a source of high-quality protein and iso- (DHT), a nonamplifiable and nonaromatizable pure andro-
flavones, may improve such indices, although the effects in gen, reduces late-life prostate growth in middle-aged meno
healthy postmenopausal women are not well delineated. A
single-blind, randomized controlled trial was conducted in Design: Randomized, placebo-controlled, parallel-group
31 postmenopausal women who were assigned to consume trial. (Australian New Zealand Clinical Trials Registry num-
3 servings of soy (n = 16) or dairy (n = 15) milk per day ber: ACTRN12605000358640)
for 4 weeks. Parameters of systemic inflammation (tumor
necrosis factor-a (TNF-a ), interleukin-l ~ (IL-l ~ ), and inter- Setting: Ambulatory care research center.
leukin-6 (IL-6)) and the oxidative defense system (superoxide
dismutase (SOD), glutathione peroxidase, cyclooxygenase-2) Participants: Healthy men (n = 114) older than 50 years
were measured post supplementation, before and after an without known prostate disease.
eccentric exercise bout performed to elicit an inflammato-
ry response. A significant group-by-time effect for plasma Intervention: Transdermal DHT (70 mg) or placebo gel
TNF-a was observed (p = 002), with values in the dairy daily for 2 years.
group increased post supplementation and then decreasing
into the postexercise periodo Additionally, significant time ef- Measurements: Prostate volume was measured by ultraso-
fects were observed for plasma SOD (p < 0.0001) and IL-6 nography; bone mineral density (BMD) and body composi-
(p < O 0001) in the postexercise periodo Overall results from tion were measured by dual-energy x-ray absorptiometry;
our study do not support the notion that 4 weeks of daily soy and blood samples and questionnaires were collected every
milk ingestion can attenuate systemic elevations in markers 6 months, with data analyzed by mixed-model analysis íor
of inflammation or oxidative defense. However, data do sug- repeated measures.
gest that the downhill-running protocol utilized in this study
can be effective in altering systemic markers of inflammation Results: Over 24 months, there was an increase in total
and oxidative defense enzyme activity, and that the ingestion (29% [95% CI, 23% to 34%]) and central (75% [CI, 64% to
of soy may help prevent fluctuations in plasma TNF-a. 86%]; P < 0.01) prostate volume and serum prostate-specific
antigen level (15% [CI, 6% to 24%]) with time on study,
Appl Physiol Nutr Metab. 2010;35(3):261-9 but DHT had no effect (P > 02) Dihydrotestosterone treat-
ment decreased spinal BMD (1.4% [CI, 0.6% to 23%]; P
< 0.001) at 24 months but not hip BMD (P> 02) and in-
creased serum aminoterminal propeptide of type I procolla-
gen in the second year of the study compared with placebo.
Dihydmtestosterone increased serum DHT levels and its me-
tabolites (So-androstane-Jp. 17~-diol and 50.-androstane-
3~, 17~-diol) and suppressed serum testosterone, estradiol,
luteinizing hormone, and follicle-stimulating hormone levels.
Dihydrotestosterone increased hemoglobin levels (7% [CI,
a Copyright 2010 Canadian Science Publishing or its licensors. Reproduced

with kind permission 01 the publisher. b ©American College 01 Physicians. Reproduced vvith permission.
© 2011 The Royal College 01 Physiciansand Surgeons 01 Canada 241

5% to 9%]), serurn creatinine levels (9% [CI, 5% to 11 %]), Origami in outer membrane mimetics: correlating
and lean mass (2.4% [CI, 1.6% to 3.1 %) but decreased fa' the first detailed images of refolded VDAC with
mass (5.2% [CI, 2.6% to 7 7%]) (P <0.001 for all). Protocol- over 20 years of biochemical datad
specific discontinuations due to DHT were asymptomatic in-
Summers WAT, Court DA
creased hematocrit (n := 8), which resolved after stopping
treatment, and increased prostate-specific antigen levels (n =
Mitochondrial porin forms an aqueous pore in the outer
3; none with prostate cancer) in the DHT group. No serious
membrane, through which selective passage of small metab-
adverse effects due to DHT occurred.
olites and ions occurs, thereby regulating both mitochondrial
function and cellular respiration. Investigations of the struc-
Limitation: Negative findings on prostate growth cannot
ture and function of porin have been performed with whole
exclude adverse effects on the natural history of prostate
mitochondria, membrane vesicles, artificial membranes, and
cancer.
in detergent solutions, resulting in numerous models of porin
structure. The mechanisms by which this protein functions
Conclusion: Dihydrotestosterone treatment for 24 months
are undoubtedly linked to its structure, which remained elu-
has no beneficial or adverse effect on prostate growth but
sive until 2008, with reports of 3 high-resolution structures of
causes a clecrease in spinal but not hip BMD. These findings
this voltage-dependent, anion-selective channel (VDAC). The
have important implications for the wider use of nonsteroi-
barrel structure is relatively simple yet unique: it is arranged
dal pure androgens in older meno
as 19 anti-parallel13-strands, with 13-strands 1 and 19 aligned I
parallel to each other to close the barre/. The N-terminal heli-
Primary funding source: BHR Pharma.
cal component is located within the lumen of the channel, I
although its precise structure and location in the lumen var-
Ann Intern Med. 2010;153(10):621-32
ies. With the basic barrel structure in hand, the data obtained
in attempts to model the structure and understand porin
over the past 20 years can be re-evaluatecl. Herein, using the
mammalian VDAC structures as templares. the amassed elec-
DESCRIPTIVE ABSTRACTS trophysioloqical and biochemical information has been reas- ~
sessed with respect to the functional mechanisms of VDAC •
Meningococcal seroqroup e
conjugate vaccinatlon activi ty, with a focus on voltaqe-dependent gating.
in Cenada: How far have we progressed? How fa!'
do we have to go?'
White Cp,Scott J
Biochem Cell Biol. 2010;88(3):425-38
•
•
Since routine meningococcal C conjugate vaccination was
introduced into Canada in 2002, there have been a large SYSTEMATIC REVIEW •
regional variation in the routine programs, changes to the
tirninq of the infant series in some provinces, and wide dif-
ferences in catch-up programs. As immunization is viewed
Dietary interventions for fecal occult blood test
"
I
screening: systematic review of the literature"
as a provincial responsibility, less attention has been paid
Konrad G
to determining national coverage rates and the direct and
indirect effects of the widely varying provincial/territorial
Objective: To determine whether dieta/y restrictions en-
vaccination programs on the nation as a whole. Canada's
hance the specificity of guaiac-based fecal occult blood tests ~
disjointed regional immunization campaigns leave the popu-
(FOBTs) when screening for colorectal cancer.
lation at risk of disease for an extended length of time. The
United Kingdom has proven that with a pro-active approach
Data sources: PubMecl-MEDLlNE, the Cumulative Index to ~
to planning, coordination, and implementation of a national
Nursing and Allied Health Literature, and Cochrane databas- t
immunization program, excellent long-term control of inva-
es were searched.
sive meningococcal disease in a large population could be
achieved in as little as one year. A summation of the current
Study selection, English-Ianguage case series, cohort stud-
meningococcal immunization strategies used in Canada and
ies, randomized controlled trials (RCTs), and meta-analyses
an estimate of overall vaccine coverage of children and youth
were selected. Studies that did not include dietary marupi.-
is provided.
lation or the use of guaiac-based FOBTs available in / --:1-
America were excluded.
Can J Public Health. 2010; 101 (1): 12-14
d Copyright 2010 Canadian Science Publishing or its hcsnsors. ~'" - '?:;
--_----------- with kind perrnission 01 the publisher.

c Reprinted with kind perrnission 01 the Canadian Public Health e Reproduced with kind perrnission 01 the College oí Fa, J :'S.:::¿'"$ ¡;"
Association. Canada.
242 © 2011 The Royal College oí Physicians ano S· 'g_ es e' ¿-.a=¿
Synthesis: Ten case series, S cohort studies. 4 RCTs, and Conclusion: e recommend that in the case of an
1 meta-analysis were critically appraised. AII studies used abnormal adnexal mass, particularly in women who want
Hemoccult, Hemoccult 11, or Hemoccult SENSA tests. Data to preserve their fertility, frozen section histology be
from case series involving challenge diets showed no in- performed laparoscopically. A frozen section diagnostic
crease in positive FOBT results from high-peroxidase veg- procedure, instead of a regular biopsy, seems to be a use-
etables, but results varied with red-meat challenges de- fui tool during an elective diagnostic laparoscopic proce-
pending on the amount of meat consumed and the test dure in order to prevent potential morbidity as a result of
used. Case series, cohort studies. and RCTs comparing possible future laparoscopy or even laparotomy. Previous
FOBT results during restricted versus unrestricted diets con- laparoscopic procedures can cause massive adhesions that
sistently showed no differences in positive FOBT results. could impede a subsequent laparoscopic approach.
Conclusion: Most of the evidence evaluating the effect of Arch Gynecol Obstet. 2010;283(6): 1369-71
dietary restrictions on FOBT results is dated and of subopti-
mal quality. However, 4 RCTsand a meta-analysis of these
data do not support dietary restrictions when screening
for colorectal cancer. Because patient adherence can be
an issue with FOBTs, and dietary restrictions can affect ad-
herence in some populations, it is reasonable to abandon
CASE SERIES
these recommendations without fear of substantially af-
fecting specificity. Urological complications of laparoscopic ingUinal
hernia repair: a case seriesv
Can Fam Physician. 2010;56(3):229-38 Kocot A, Gerharz E\IV, Riedmiller H
Objectives: To illustrate urological complications of lapa-

roscopic inguinal hernia repair and discuss their manage-
ment.
CASE REPORT
Patients: Between April 2002 and February 2004, four
men (aged 38-63 years) were treated for serious compli-
The diagnostic and therapeutic approach of a cations 2 days to 11 years after unilateral (1 patient) or
primary bilateral leiomyoma of the ovaries: a case bilateral (3 patients) laparoscopic inguinal hernioplasty.
report and a literature review'
van Esch EM, van Wljngaarden SE, Schaafsma HE, Results: In al! cases (extra and intraperitoneal bladder
Smeets MJ, Rhemrev JP injury, purulent urocystitis due to mesh-erosion of the
bladder, secondary retroperitoneal fibrosis) open revision
Introduction: A primary fibroid (Ieiomyoma) arising from with complete drainage of the urinary tract was chosen
both ovaries is rare and can be difficult to diagnose as a as an efficacious therapeutic strategy.
result of the low incidence and its indistinctive presenta-
tion. A literature review on the diagnostic and therapeutic Conclusions: Awareness of rare complications of laparo-
approach of this rare benign tumour is presented. We scopic inguinal hernia repair may lead to early diagnosis
describe a case of bilateral primary ovarian fibroid with an and appropriate management.
unusual presentation to illustrate our recommendations
for treatment. Hernia. 2010 Jul 4. [Epub ahead of print]
Case presentation: A 37-year-old woman was admitted

with symptoms of acute severe abdominal pain. She had
a history of faint abdominal discomfort. Due to the acute
deterioration of the abdominal pain a diagnostic lapa ros-
copy was performed. A tumour arising from both ovaries
was seen and a biopsy was taken in order to decide on
íurther therapy. Histology showed a fibroid for which ex-
cision by a second laparoscopic intervention was planned.
Due to excessive adhesions conversion to laparotomy was
necessary.
f Reprinted with kind permission 01 Springer Science + Business Media. 9 Reprinted with kind permission of Springer Science + Business Media.

CONSORT checklist analysis of this sample

abstract
APPENDIX 27.2: APPLYING CHECKLlSTSTO
ABSTRACTS
• Trial designo Can be inferred; qualitative study with
questionnaire befo re and after intervention. From
results, it is unclear whether all survey respondents
EXAMPLE 1 participated in the intervention.
• Participants. Thirty-three residents.
Health promotion program: a resident well-being • Interventions. Workplace health promotion pro-
gram; little information on what this involved.
study. lowa Orthop J 2009;29:83-7.h
• Objectives. Hypothesis clear: improved workplace
Watson DT, Long WJ, Yen O, Pichora DR
hygiene could improve presenteeism (which is de-
fined).
Background: Surgical training places unique stresses on
• Outcome. Primary outcome seems to be change in
residents that can lead to decreased levels of presenteersrn.
score on SPS-6, but this is not stated explicitly.
We hypothesized that presenteeism levels could be positively
• Randomization. Not applicable.
influenced by improving workplace hygiene.
• Blinding. Not applicable.
• Numbers randomized. Not applicable.
Methods: A cohort of surgical residents was asked to com-
• Recruitment. Not applicable.
plete the Stanford Presenteeisrn Scale: Health Status and
• Numbers analyzed. Number that responded to
Employee Productivity (SPS-6) questionnaire before, and one
survey qrven.
year after the implementation of a workplace health promo-
• Outcornes, (1) Initial SPS-6 score was significantly
tion programo
lower than normal; (2) the intervention resulted in
limited, non-significant improvement in SPS-6 scores;
Results: Twenty-six of thirty-three residents responded to
(3) a subgroup change in score showed slightly better
the initial survey and reported a mean SPS-6 score of 17.3
but still non-significant improvement; (4) significant
± 4.5, well below population normative value of 24 ± 3 (p
difference between junior and senior residents. For
< 0.000'1). At one-year post intervention 25 of 32 residents
eaeh result, the effect size. precision (±) and statistical
responded, reporting a mean SPS-6 seore of 18.3 ± 4.6. The
significance are given.
mean SPS-6 score improved by 1.2 ± 3.8 (p = 0.35). Sub-
• Problems with outcomes. (1) A subgroup analy-
group analysis showed a trend toward improved SPS-6 in
sis seems to suggest not all participants underwent
those who participated in the health prornotron program (p
the intervention, and this does not make sense as
= O 15) and a significant difference when junior residents
the methods indicate testing irnprovernent after
were compared to seniors (p = 0.034). Overall, results were
intervention, presumably in all participants. (2) "A
limited by our small sample size.
trend toward improved SPS-6" implies an indica-
tion of improvement, but the result is not statisticaily
Conclusions: Presenteeism seores for surgical residents at
significant and should not be represented as mean-
our institution are well below population values. Use of vali-
ingful. (3) It is unclear what is meant by "a siqnificant
dated tools such as the SPS-6 may allow for more objective
difference when junior residents were compared to
analysis and decision making when planning for resident
seniors "-junior residents considered as a subgroup
education and workload.
had a significant difference before and after the inter-
vention? Junior residents had a significantly different
Presenteeism: the ability while on the job to produce quality
score frorn senior residents? Which group showed
work at maximum productivity.
improvement?
Decresased presenteeism: a state of decreased productivity
• Harms. None presented; qualitative study. Does not
and below-normal work quality related to health/workplace
state the lack of harm explicitly.
distracters )
• Conclusions. Presenteeism values for this group are
low. This is the only significant result. Fails to state
that other results were not significant and that the re-
sults are therefore negative. Discusses use of validated
/owa Orthop I 2009;2983-7. survey tool; however, this was not an objedive of t e
study.
h Reprinted with kind permission of the lowa Orthopedic Journal
244 © 2011 The Royal College of Physicians and Surgeons o' Cc~,,:;¿;
EXAMPLE 2 Annals of Internal Medicine checklist analysis of

this example abstract
Gender preferences in the choice of a pediatric den-
tal residency programo • Background. Does not explain background to study.
da Fonseca MA, 5tiers ML 15 there a concern that more men than women (or vice
versa) are applying for certain types of graduate train-
The goal of this study was to investigate whether men and ing in pediatric dentistry?
women applying for graduate training in pediatric dentistry Objective. Clearly stated in first sentence: gender
placed different emphasis on the same factors and program differences in factors and program characteristics in
characteristics upon making their final ranking decision. A making ranking decisions.
questionnaire was mailed to the first-year resident class in the • Design. Can be inferred qualitative study with ques-
United States in 2005 containing both multiple-choice and tionnaire.
open-ended questions covering six sections: 1) candidate's • Setting. Clear that this is a country-wide survey
background, 2) the application process, 3) program charac- (United States in 2005)
teristics, 4) nonclinical factors, 5) clinical factors, and 6) the • Patients. Participants were all first -year residents in
interview process. In sections three through six, respondents pediatric dentistry.
ranked factors and characteristics from "not important" or
"no influence" to "critical" The response rate was 69.2 • Intervention. Survey, with detailed explanation of
percent (180/260), with approximately 57.8 percent females questions, their format and the ranking system.
(104/180) and 61.4 percent non-Hispanic white respondents • Measurements. Not stated, presumably scores on the
(110/180). Statistically significant differences between gen- survey.
ders were as follows: 1) men were older (29.4 years versus • Results. Response rate and absolute number of
28.1, P < 0.05); 2) men applied to more programs (99 vs. respondents indicated. The ethnic background of the
8.1, P < 0.05); 3) women preferred programs affiliated with respondents is given, which was not an objective of
their own dental school (p = 0.046); 4) women preferred the study and seems to be irrelevant. Consistent with
university-based programs (p = 0049); 5) women preferred the objective, only statistically significant differences
programs that offered a high amount of patient care under between gender groups are reporteo. Actual mean
general anesthesia (p = O 040); and 6) women placed more rankings are not given; statistical significance is given.
importance on the salary/stipend amount offered by the pro- Measure of variability is not applicable, as this is a sur-
grams (p = O 045) vey of the entire population, not just a sample.
• Limitations. Not stated. Does the ethnic group re-
) Oent Educ 2009;73(9): 11 02-6.i sponse rate result mean the survey results should not
be generalized to all ethnic qroups?
• Conclusions. No conclusions are stated The authors
may have thought that these were clear from the
results, but there should be a statement about gender
differences and their practical significance. This would
have followed from the background, which is also miss-
ing. The authors may have wished to comment on the
borderline statistical significance of some of the results.
____ ._---_._-----
i Reprinted with kind permission 01 Journal 01 Dental Education. Copyright
2009 by the American Dental Education Association, www.jdentaled.org

•
•
en
•
~
e
.-
....,
~
o
c.
OJ
Q! •
246
© 2011 The Royal College 01 Physicians and Surgeons oí Caneca
28
Communicating your research
with slide and poster presentations
Jeffrey J. Perry, MD, MSc, CCFP(EM)
ILLUSTRATIVE CASE
A diagnostic imaging resident has completed a research project determining the sensitivity of computed tomography for
the diagnosis of subarachnoid hemorrhage. She has submitted her structured abstract to the Canadian Association of
Emergency Physicians (CAEP) conference and the Society for Academic Emergency Medicine (SAEM) meeting. The abstract
was accepted for an oral presentation by CAEP and as a poster presentation by SAEM. Now the resident needs to crea te
slide and poster presentations that clearly convey the importance, methods, results and potential impact of her work.
• You've submitted an abstract of researcher presem ro provide addirional inforrnarion.

your research ro a specialry conference (see ch. 27), and Alrhough this rype of presentation is more informal, ir is no
now you've been invited ro presenr your findings at the less challenging than a slide presentation. You willneed ro
meeting. This occasion is a valuable opporruniry for you ro atrract arrention ro your posrer by making ir visually appeal-
expand on the synopsis given in your abstraer, and a chance ing, easy ro read frorn a distance and self-explanatory. lf the
ro disseminare your findings, nerwork with colleagues with formar of the conference involves small presenrations, then
similar interests, and ger feedback rhar can be very use fui as a short oral presentation needs ro be prepared with only the
you prepare a manuscript for submission ro a journal or posrer available as a visual aid. Given the arnount of work
cominue with funher research.' Oral slide presentations that you have invested in yom research, you owe ir ro yom-
are usually lirnited ro 10 minutes, with an additional 5 self ro prepare an effectivc and professional-looking posrer.
minutes allotted for questions and rhe transitiori berween
presenters. Most conferences are very strict with these rime-
lines, and so ir is important ro know how ro ger your mes-
sage across efficiently and effectively. The essential
CHAPTER OBJECTIVES
ingredients of an effective oral presentation are: (1) organi-
zation of content: (2) clariry of slides; and (3) a well- After reading this chapter, you should be able to
• describe the important aspects of a research presentation
rehearsed presentation.
Poster presentations convey similar inforrnation but, of • describe how to prepare an effective oral slide
course, in a very diflerenr formar. Depending on rhe format presentation

• describe how to prepare an effective poster presentation
of the conference, a posrer can be presented one-on-one or
ro a small group, or ir may be displayed wirhout rhe
~-----~----------~----------~
KEY TERMS
Abstract Poster presentation
Presentation PowerPoint
Slide presentation
---- ----- ---------- _---------
© 2011 The Royal College of Physicians and Surgeons of Canada

247
Slide presentations
inrervenrion, followed by a slide describing the rn
ourcorneís), a slide on the sratisrical analyses perform
Preparing an effective text and, finally, a slide showing sample size calcularions a
In your text, use a point-form presentarion but don't com- assumptlons.
press the phrasing to rhe point where the meaning becomes
cryptic or ambiguous. Avoid acron)'ms and orher abbrevia- Results, The results of your srudy should be rhe focal poi
rions as much as possible unless rhey are srandard for your of the presentarion. Display rhem in a concise manner th
audience (e.g., "ED" will be readily recognized as "erner- is easy to read and inrerprer. Make sure thar the results a
genc)' deparrrnenr" at an Emergency Medicine confer- "trimrned down" so rhar only imporrant resulrs are presen
erice). Cirarions are rypically not given in the main text, ed and rhe audience does not get lost in the details and mi:
bur you can decide to make your key references available in the main findings. The data you include must support you
a Ioornore at rhe bottom of the slide or in a separare slide at cornrnents in the discussion and conclusion.
rhe end; rhese should be large enough ro be read easily.
Discussion: In this secrion, you can discuss issues rhat ar:
Structuring your presentation perrinent ro rhe research quesrion, how rhe resulrs compan
The basic strucrure of your slide presentarion wil! essentially wirh rhose of previous srudies, 01' your inrerpretation 01
mirror the elements of rhe abstraer you submirted ro rhe why the resulrs are what they are (see ch. 26). One strategy
conference (see ch. 27), as fol!ows: Tirle, Inrroducrion (back- is to develop a "story" rhat ries your findings ro existing
ground), Objecrives (research question), Methods (study data. 111is allows the listener to see how your work adds to
design, study population, study procedureís), outcorne mea- rhe current body of knowledge on your topic. Typically,
sures, analysis, and sample size calculations), Resulrs, and this concise attempr ro develop a hypothesis about rhe
Discussion (limitations and strengrhs, conclusion). irnplications of results wil! be the main thing that the audi-
en ce remembers.
Tirle slide. On rhis slide, give [he tirle of rhe presenrarion
along wirh rhe names of the investigators, the institution • Limitations and strengths. Present one or rwo of the
coordinating rhe study, rhe instirutions parricipating in the main limitations and strengrhs of your study on
study, and the fllnding agency (if applicable). If rhis infor- one slide; point out rhe potential irnpacr rhat any
rnation is plentiful (e.g., rhere are many co-investigators, or limitarions might have on your resulrs.
granring agencies), move rhe informarion abour funding • Conclusion. In your concluding slide, sumrnarize
sources to rhe second slide. your resulrs and what they mean. This slide should
succinctly reiterare the "story" you presented earlier,
Introduction. Your inrroducrion should consisr of rwo reminding the audience why your research mattcrs.
slides. The firsr slide rypical!y gives sorne background infor- You may suggesr direcrions for further research here
·- marion about rhe clinical problern, and rhe second pro-

vides rhe rationale for rhe srudy: rhar is, why ir was done.
as wel!, bur rhis should nor be rhe main conclusion.
This explanation needs ro be clear and concise.

Preparing your slides
Objectives. This slide should concisely srate your research Use the "less is more" principie in designing your slides.
question (i.e., the goal of srudy and specific objecrives/ Microsoft PowerPoint offers many srylization fearures
hypotheses). that, if overused, are more likely ro rnake your audience
wonder abour how you gor a cerrain cool effecr rhan ro
Methods. You will likely need roughly six slides ro describe keep thern Iocused on your lllcssage. To creare ernphasis.
rhe srudy merhods. These address the ioho and rhe hou: change rhe minimum number of elemenrs possible ro do
Typically, one slide covers the srudy design, locarion(s) and rhe job: thar is, you don'r need ro combine bold with italic,
rime frame. The nexr slide or rwo will describe the srudy underlining, capital Ietters, a new colour and larger . -pe o
population, including inclusion and exclusion crireria. draw attention ro a word. One or rwo of mese a tribu es
Then, a separare slide describes rhe srudy procedure(s) or will do. Use one rypeface, a limired number of izes and 2
248 © 2011 The Royal College o Physicians a o S_'¡;:c-s o" ::::-.a:c

28 Communica_ing yoor research with slide and poster presentations
consistent colour palerre throughout the presentation. A Exarnples of slide formats are given in Figures 28.1-28.6.
sans seriP rypeface such as Arial or Verdana is usual!y the
Figure 2 .1
best choice for small blocks of rext meant to be read ar a
Example of slide summarizing the study rationale
disrance. Because some rypefaces are more compan rhan
orhers, the rype size you need ro achieve legibiliry wil! vary;
using Arial, for exarnple, you might want a size 28 to 32 Rationale
font for text and a size 34 to 44 bolded font for ritles. Also,
choose widely used typefaces that are standard ro both Mac
and pe operating sysrems; not al! fonts display properly on • It is uncertain whether
emergency physicians'
borh platforms. Also remember thar "reverso rype" (white prescribing of antithrombotics
against black or a dark colour) can look attractive but if has improved
overused is difhcult to read. • We need to minimize
the chance of stroke after
transient ischemic attack
Tables and figures. Use tables and figures where possible ro
replace texr: they can compress a great deal of quantitacive • Anticoagulation reduces stroke risk
data into a smal! space, facilitare comparisons, and show in patients with atrial fibrillation by 64%
trends and effects at a glance. A rule of thurnb for tables is

to display percentages, rather than raw nurnbers, giving the
total number of patients or the relevant denorninator (rhe Figure 28.2
Example of slide summarizing the study methods
n) in the column heading
Ensure thar your rabies and figures are self-explanarory,
Inclusion criteria
containing al! of rhe information rhe viewer needs ro under-
stand what they are about and how ro interpret thern. Keep
graphs and other figures J.S simple as possible and ensure Alert patients > 15 years with
rhar legends and labels are legible. Three-dirnensional ren- non-traumatic acute headache
who have underqorie head CT
derings and orher advanced options offered by Exccl for
making charts can distraer the viewer from your message
• Alert: Glasgow coma score 15
rarher than enhancing ir. 111e judicious use of colour can
• Non-traumatic: no direct head
help to emphasize resulrs or make comparisons, but avoid
trauma in previous 7 days
red and green for rhese purposes (such as a red line on a
• Acute: maximal pain in < 1 hour and presents
green or orange background); up ro 10% of men and 1% of within 14 days
women in Nonh America have so me degree of red-green
colour blindness. Reduce visual "noise" in tables and
Figure 28.3
graphs, for example by using the minimum nurnber of grid- Example of slide with figure
lines necessary to guide the eye.2.3 --_._-----_._-
Ir is ofren helpíul to be able to re-use appropriate slides from Physician agreement with criteria
earlier presentations. This is very easily accomplished by select- for "clinically important" features
ing the insert rab in PowerPoint and choosing "Slides from ... 100
Other presenration" in the drop-down menu. 111iswill enable e 80
'ó
you ro browse your previous PowerPoint files, select the slides <11
E
60
<11
you wanr and insert rhem into the new presenration, You can
..~
40
en
eirher ler rhern be automatically reFormatted ro your new pre- 20
sentation format, or keep rhe original source formatring by Osteophyte Transverse Spinal process Compression
avulsion process < 25% body
selecting "keep source formatring."
ti Emergency physician Neurosurgeon
_.. J
a These typelaces are without (French: sans) the typically curved terminals
(serils) at the end of strokes, as in T versus T.
© 2011 The Royal College of Physicians and Surgeons 01 Canada

249
Figure 28.4 Preparing your delivery

Example of slide with flow diagram Pracrice your presemation. Alrhough you know rhe con-
¡--------------- tenr ofyour talk, to put it across ef}ectively you need a con-
Results: Flow of patients fidenr and polished delivery. Remember rhar you were
asked ro give rhis ralk because you are knowledgeable-so,
E'ligible patients
n = 1454 don't be intirnidared! In anticiparion of questions, you
Missed
.¡.- - - - - - - - - - - - + n =
430 might want ro insert one ro three extra slides at che end of I
rhe presenration, giving tables or figures with additional

Enrolied patients
n = 1024
-----~ No CT within 6 h dara that rime does not permit you to add ro the main pre-
n =711
sentation. If someone asks a quesrion to which these data
0' -Ó:
i are relevant, you can then display the appropriare slide and
CT < 6 hours from
discuss. If you are asked a reasonable question for which
_ _j
headache onset
n = 313
you do nor have an answer, rhen say "Good quesrion; 1 wil!
have ro look into this issue furrher" and move on!
Afrer loading your presentation OIl the compurer pro-
Figure 28.5 vided at the conference, verify that al! the slides appear cor- ,
Example of slide with atable recdy. There can be disrortions in sorne slides depending on ~
r--- ----------- ----- -----.--- rhe version ofPowerPoinr you used ro rnake your slides and ~
the version available at the conference. You sbould always ,
i Cohort characteristícs (N = 1024)
bring your presentation with yOl! on a USB key and/or a
Mean age, vears (range) 44.5 (16-92) laptop compurer in case there are technical problems load- ~
Female 60.5 % ing your presentarion. •
Worst he.adache 83.6 o There are diflerent rechniques for delivering an oral pre- •
Vomiting 30.4 o/" senration. You may wish tO use the flotes funcrion in Pow- •
Transient loss of conscíousness 6.4 %
erPoint ro prompt you; explore the "Presenters Tools" in ,
I CT < 6 hours 30.6%
the software to see what you feel comforrable with. (You
I l.umbar puncture 53.7%
wil! need ro make sure that at the presenrarion
will have a laptop compurer or a screen in from of you as
venue you
•
IL_. _ _ _j yOL!give your ralk.) Alrernarively, you mighr prefer ro read
your text ro ensure rhat all irnporrant informarion is includ-
Figure 28.6 ed wirhin che brief time allowed. This second rnerhod is
Example of slide showing 2 X 2 table quite appropriate for a beginner, but the following tips can
make ir more effecrive. Memorize your inrroduction and
Overall classification performance deliver ir in an engaging way while looking direcdy ar your
.for 83 cases of subarachnoid audience. Then, when you advance your slides, all eyes wil!
hemorrhage (SAH) (N 1024) =
o
e, SAH
move away from you and ro the slide screen (in the same
way a magician uses his wand ro distraer the audience frorn
what he is really doing), at which point you can refer ro
OJ Ves No
yOL!rscript, Be sure thar your scripr is in a large fom, and
o:: CT
positive
Ves 77 o carry a penlight. (Dimming (he room Jighring ro make che
No 6 961 slides more visible can make reading a challenge.) Clearlv
I
I note al! slide changes in rhe margin of your script: rhere's
¡ Sensitivity = 92.8% (95% el: 84.9% to 97.30/0)
I Specificity = 100.0% (95% C!: 99.6% to 100.0%) ¡
nothing worse than delivering an enrire ralk mar is Out 0--
L . _j sync with the slide projecred on rhe screen. Check ro rnake

sure you are speaking abour rhe correcr slide afrer each sli
change!
250 © 2011 The Royal College of Physicians ad S_-;~ ~ ;2:-a::c

28 Comtnunicet 9 jOJT 'eseerct: with s/ide and paster presentatians
No matrer which presentarion stl'ategy you choose, form a uniform border 011 rhe top, side and botrorn mar-
make sure that you practise the ralk out loud ar leasr five gins of rhe pos er for a polished and orderly efiect.
times, and ensure rhat you have tirned rhe delivery to less The advice given in rhe section 011slide presenrations on
rhan the 10 minutes allocared for the presentarion. the efFecrive use of fonts, colour, rables and figures applies
ro pos ter presenrarions. You will need to ensure thar the
Poster presentations fonts you choose are large and clear enough to be read at a
Key content requirements disrance of about 10 feet. Keep your rext to a minimurn, use
As wirh a slide presentarion, a poster requires rhe following a poinr-form sryle, use graphs and figures where possible
sections: Title, Inrrcduction, Objecrives, Methods, Results, insread of text, and ensure that these areself-explanarory.
Discussion, Lirnitarions, and Conclusions. 111e rables and figures will coritain the core informarion
111e rirle is rypically shown ar rhe top of rhe poster, along about your resulrs, and may be the only sections of the posr-
with rhe names of all aurhors and rhe organization rhar er that gain the viewer's atrention. 4.5 If photographs are used
coordinated the study. The Inrrcducrion should provide ro illustrare a clinical finding, or for visual appeal, ensure
one or rwo brief staternents rhat allow the viewer to under- that they are of a sufficienrly high enough resolution ro
stand the rarionale for and irnporrance of the study, The retain sharpness once the pos ter is enlarged. Remember
Objectives section should provide rhe specific research aims rhat clinical photographs require the patient's permission to
(i.e. goal and objectives/hypothesis) of rhe study: ir is essen- be used in this way, regardless of whether rhe patient is rec-
tial for the viewer to undersrand what quesrion rhe study ognizable.
arrernpred tO answer. The Merhods need to provide a brief Figure 28.7 displays an example of a previous poster pre-
description of how rhe srudy was conducred, including rhe sentation thar caprures many of these suggestions.
study design, srudy population, study procedure(s), analy-
Si5, and sample size (and how ir was dererrnined). The Making a poster using a PowerPoint slide. A poster can
resulrs secrion should describe rhe outcornes and findings be made easily using a single slide in Microsoft Powerf'oinr.
of the sratisrical analysis. 111is section should incorporate Before beginning, you wil! need ro know rhe dimensions of
tables and figures to present this data. The Discussion sec- rhe paper on which rhe poster will be printed and of rhe
tion is optional on a pos ter. So, if it is included, it should be poster board on which it will be rnounted. You will need ro
very shorr and empbasize why the srudy results are rhe same check the dimensions required by rhe conference. Select
as or difFerent frorn rhose of similar studies. 111e Limita- "Page Setup" under rhe "File" tab in Power Poinr. Select
tions secrion should point out any weaknesses of the study "Cusrorn" and selecr rhe dimensions you need. For most
and what efFect they may have on rhe results. The Conclu- horizontal set-ups, you can safely select 56 inches by 36
sion section needs to provide a summary of the significant inches. Under rhe "Insert" tab, selecr "Text Box" ro creare a
resulrs and the irnplications (for clinical practice or research) frame in which to enrer your text; righr-click in rhe box and
of these results, References are optional. select "Edir Text," and srart ryping )'our texto Use rhe for-
matting palette ro arrange your text as bullered lisrs where
Styling your poster presentation needed. The font size shoulcl be at leasr 32 for regular rext
Once you have determined the conrent and basic structure and approxirnarely 60 for headings. To add pictures and
of your posrer presentation, you will need to crea te a visual- graphics, select "Picture" under rhe "Insert" rab, various file
ly appealing formar thar helps the viewer navigate and rypes (e.g., jpeg, EPS, and PDF) can be imponed as picture
undersrand thc contenr easily. This is always more diflicult elcrnents. Ensure that any graphic files you impon have a
rhan ir sounds. There are many difFerent layout possibilities high enough resolution ro appear sharp when rhey are
for posters, but a few rules of thurnb are quite standard. Use printed at [he required size.
"wh ire space" efFectively and creatively ro make your poster Once the poster draft has been created and has all rhe
readable and arrractive, a presenration rhat appears too desired content and illusrrations, proofread it carefully, and
crowded or dense is nor appealing to read and might be rhen ask one or rwo colleagues ro proofreacl ir again. Now
skipped over, Arrange your rext and graphics in columns, that rhe poster is finalized, have ir professionally printed on
working from lefr to righr and top to bortorn: a four-col- one page and rnount ir 011your poster board. Professional
umn format is typical. Align the eclges of rhe colurnn to prinring will make [he poster more appealing and increase
© 2011 The Royal College 01 Physicians and Surgeons ot Canada 251

Reportinq
N ---1
Vl ::;
N m ::!1 ro
X lO :Al
DJ c::: ro
:3
\J
ro 1/1
ro
Q)
N -,
lmplementation O/ the Canadían C-5pine Rule I ro
O
-+-
CXl
~
n
::;
Gl
by Emergency Department Nurses I DJ
"'C
c:::
Q
lan Stiell MD; Catherine Clement RN; Jamie Brehaut PilO; Jerernv Grimshaw PhD; Annette O'Connor RN PhD; Jeffrey I ..•ro...•
O
VI
ro
Perry MD; George Wells PhD; Taryn Macl(enzie RN; Christine Beland RN; Pamela Sheehan RN; Barbara Davies RN PhD »
-o
"'C -,
Department of Emergency Medicine, Univers itv of Ottawa, Ottawa, ON, Canada ...•
ro :3
(D
---- ··--·1 VI
-,
Background ro
• Prolonged irnmobilization of trauma patients
Patient Characteristics
Age (years) [mean ± SO]
(N=1,608)
42.! 17
¡ Classification Performance of Nurses'
Interpretation of the Canadian C-Spine Rule ..•
::J
DJ
-----h
o
-,
adds to ED congestion and patient discomfort Range (years)
Gender - fernale (%)
16 - 95
55.5% Nurses' Interpretation
Cervical Spine Injury
~. ro
Objectives O 1/1
Ottawa Hospital Campus (%) ecn Pcsltive Ves No ::J Q._
Evaluate irnpact and safety of allowing nurses to Civic 48.5% J' Ves 24 812 ro
General 51.5% '.;': No O 766
::J
remove e-spine immobilization of trauma ,-+
patients Mechanism of injurv (%) lndeterrninate O Y'
Implementation of medical directive based on
Motor vehicle ccllisicn 59.6% Sensitivity (95% el) 100% (86-100%1 o
,-+
fall 23.4% 5pecificity (95% el) 48.5% 146-51%) :::y-
previously validated Canadian C-Spine Rule Other 8.1% ro
-,
(CeR) Bicvde cotnston 4.9% Nurse Comfort with Using CCR
Pedestrian struck 4.0%
::;
Methods " r.·~·,ln~
Uncomfortable 2.4%
ro
Time, injury to arrival EO, hours (median) 1.3 ·"""'1'1I1¡·,liu" Q)
• Prospective eohort study in 2 large hospital EDs Arrived by ambulance (%) 81.~% _~y_!>rv uocomtortabre 1.9% I ,-+
zr
• Enrolled alert and stable adult trauma patients e-spine radiography performed (%) 39.6% n
Q)
who presented with neck pain or were Clinicallv important e-spine injury (%) 1.0%
immobilized on an EMS backboard Clinically unimportant e-spine injury (%) 0.5% ro
ED triage nurses
by a CD and hands-on
had been trained
sessions
on the CCR
Oeveloped neurological déficit (%)
Stabihzing treatments
lnternal fixatlon
(%)
0.4%
1.3%
0.1%
I
I
Q)
::J
ro
RNs had to accurately evaluate 10 patients and Halo 0.2%
eomfortable 36.9% Very Comfortable 54.3% t ro
arace 0.1% _1/1
3 interobserver cases, and pass a written test . N"C-$I'lnll _____________. . J
e prior to being certified to clear the e-spine Rigld collar 0.9% '-'-'"~', .. ,¡
cu
N
O Evaluated safety and impact from study data
Time, ED arrival-disposition, hrs (median) 5.2 Conclusions ::J
o,
Admission to hospital (%) 3.9%
forms, imaging récords, and 30-day follow-up tnterobserver Agreement jor Criteria Training and empowering ED triage nurses to clear the -o
= -Canadian C-Spine Rule Positive Findings : of Caruidion C-Spine Rule (N=95) e-spine of stable trauma patients is safe and effective
ru
.n
This use of the CCR should improve patient flow in our
JII' nauua Value 95%(1 ~.
in 1,608 Study Patients crowded EDs, diminish unnecessary patient .-+
:1.. ~
Fintlings from History (%)
r"'I1',' High F;i$k ra-tor
discomfort, and íncrease ED nurse dedsion-making o
f.\.ge ~ 6S \,,?-,,'I'S 0.71 ::J
Dangerous mechanism 24.6% O(ln~er'J'J$ mecb arusr.i 0./2 0.57 - 0.87 Aclmowledgements ro
-,
Paresthesias in extremities 14.4% r-;:,r~s~;'e;i::-$ 052 0.7.8 - 0.77 1/1
Simple rear-end r\1VC 16.8%
• Thc wondcrful cooperation of the Ottawa Hospital ED
Anv to vv-Rirl; Factor
Ambulatorv .;Ji any time 36.0% nurses
ReJr¿n(l,'.I]\I( 03'1 0.66 - l.00
lmmediate onset oi neck paln 40.2% Uf,,'lght ~)CS1; 1(1", D.n 0.53 -1.00 Reference
. Findings fram Phvsical Examlnation (%) Amuutatorv o z; 0.50 - 0.95 StielllG, Clement CM, O'Connor A, Davies 8, Lec/air C,
Sitting position in ED 12.3%, Deteved oecf ~Í<,i,', o 8!.! 0.66 -1.00 Sl1eehan P, Clavet T, Beland e,
MacKenzie T, Wells GA.
Neck tenderness midline 31.8% ¡v¡ldlin€- tcnderne ss O.l~O 0.11- 0.69
Multicenter Prospectivc Validation 01 the Canadian C-
Abre to rotare neck 47.8% Abte ;:0 Rotate u 7:;; 0.46 -1.00
n, Spine Rule by ED Triage Nurses. CMAl2010;Aug
C-spine immobilization requlred 50.3% Overall RI,I.: Im.noburzunon 0.70 0.54 - 0.86
"1
C"I. -------------_._-- 10;18211 :1173-9
11'
\~
11.
~
28 Communicaring your research with slide and poster presentations
its impacr. The poster is best printed locally, so rhat you can Presenting your poster
ensure ir is correct befo re leaving for your conference. Ir is Pracrise giving an oral summary of your posrer. \Vhen
optirnal ro convert your PowerPoint cusrorn file ro a high- sornecne approaches rhe posrer and appears ro be interesr-
qualiry PDF print formar, so that nothirig will be altered ed, ask if you can give rhern rhe "rwo-rninure summary."
during the printing process, You may also be able to email Most people will gladly accept the offer, giving you an
rhe file direcdy ro the prinr shop. 111e cost of printing varies opportuniry ro engage rhe viewer as you describe your
but is rypically $115 ro $200 per poster. Group discounrs rarionale, objectives, rnerhods and results.
are ofren possible if multiple posrers from your departmenr
are being prepared. Once you receive your prinred póster.
Conclusion
ensure rhat you luye a solid (LIbe tO store the poster in wirh-
out damaging ir. Also purchase some Velero adhesive strips Creating a slide or poster presenration for a research meer-
or tabs ro srick on rhe back of rhe posrer so thar it can be ing can be a very rewarding experience. Ir allows you ro
displayed and subsequendy reused withour the damage presem and disserninare your research, nerwork wirh col-
caused by rape or pins. Many poster presemers offer a copy leagues with similar interests and gain valuable feedback
of the pos ter (a reduced version on legal-sized paper) ro rhat can help you improve your work befo re submirting a
those who may be interested in taking away a copy to review full manuscript for potential peer-reviewed publication. By
the rnerhodology, resulrs or references at a later date, carefully preparing a clear and concise presenrarion, you
will help your audience undersrand how rhe research was
conducted, what rhe results are, and the implications of rhe
findings. •
Acknowledgement
I thank Dr. lan G. Stiell, Chair of the Department of Emergency Medicine, University of Ottawa, for permission to use his
work to create Figures 28.3 and 28.7.
REFERENCES
1. Sherbinski LA, Stroup DR. Developing a poster for disseminating research findings. AA NA I 1992;60(6):567-72
2. Tufte ER. The visual display of quan tita tive information. 2nd ed. Cheshire (CT): Graphics Press; 2001
3. Lang TA. How to write, publish and present in health sciences. a guide for physicians and laboratory researchers. Philadeiphia
(PA) American College of Physicians; 2010.
4. Hardicre J, Devitt P, Coad J Ten steps to successful poster presentation. Sr} Nurs. 2007; 16(7):398-401.
5. Teaching Support Services, University of Guelph. Effective poster design: a step by step guide. University of Guelph . 2010.
Faulkes Z. DoctorZen.net. Better posters. A resource for improving poster presentations. Blog site. Available from: http://betterposters.
blogspot.com
• This lively blog site has regular postings on al! aspects of poster presentations.
Faulkes Z. DoctorZen.net. Smart posters. Blog post for 31 Mar 2011. Available from: http://betterposters.blogspot.com/2011/03/
smart-posters html
• A posting frorn the Doctor Zen blog site about using cel!phone-readable QR codes to link your audience to additional
information, resources, or a "virtual" presentation of your póster.

SUMMARY CHECKLlST
o Gather al! of the information you need to construct your presentation or poster, including any parameters assigned t
the conference or forum (e.g. font, slide number, size, etc).
O Select an appropriate template to use (e.g., one may be provided by your school or department).
O Create a draft presentation using the general outline of: Title, Introduction, Objectives, Methods, Results, Limitations
Discussion and Conclusion.
O Review your text to ensure it is an appropriate length. You should aim to use the briefest text that communicates your
essential ideas in each section. More than five lines of text on slide is typical!y crowded, for example.
O Ensure your fonts are large enough to be read in the room where the slide wil! be presented.
O Share your poster or slide set with col!eagues for feedback. Revise.
O Practice your presentation with col!eagues and revise on the basis of their feedback.
O Ensure you list any competing interests that authors may have (such as industry funding).
O Get your poster printed / submit your slides .
.-
254 © 2011 The Royal College oí Physicians a"lO _~ - (c;-¿c¿

29
How to write a health science paper
Grant Innes, MD, FRCPC
ILLUSTRATIVE CASE
Shortly after completing his residency, an emergency physician decides to launch his academic career by conducting his
first research study: a 1-year follow-up of patients who had presented to an emergency department with abdominal pain.
After convincing his colleagues to fill out data forms for 600 patients, he follows up each case personally by telephone
over an 18-month periodo He collates the data, painstakingly writes a manuscript and submits it for publication. This takes
a mountain of work, but he has no regrets: he has made a major contribution to the medicalliterature. A Nobel Prize is
unlikely, he thinks, but not out of the question. The journal's response arrives two weeks later: an outright rejection The
article is unsalvageable, the editor says. The science is bad and the writing is terrible. The doctor is stunned. As a physician,
he has never failed at anything. He had assumed his medical school training qualified him as a scientist and writer.
• Publication is a good thing tor surgica! procedures. Journal edirors can rapidly judge
healrh science researchers: ir leads ro credibiliry, visibiliry, quality, and one way ro have your work rejected is ro
prestige and promorion-alrhough not, in any direct way, submit a poorly writren manuscript. If you do nor
more money. The objective of this chapter is ro guide you have a strong writing background, a medical or scien-
rhrough the process of wriring a health science research tihc wriring course is a good investrnent that will save
repon and ro increase your chances of having your work )'ou rens or even huridreds of hours in manuscripr
published. A1rhough rhe specific requirernents for research preparation time-nor ro menrion reducing rhe risk
papers vary according ro rhe type-a case repon, for exarn- thar your subrnitred manuscriprs will be rejected.
pie, is structured differenrly from a sysrematic review-the
following rips will help you ro ser out on the right path no
rnarter whar caregory of repon you are wriring.
CHAPTER OBJECTIVES
After reading this chapter, you should be able to

Ten tips for writing a health science
• discuss key strategies in approaching the task of writing
manuscript
a research paper
1. Take a writing course. Medical school did not reach o list the components of an interventional research paper
you how ro wrire. \'{friters train as long as physicians • describe the submissions and peer review process for
do; hence, many physicians are about as qualitied ro biomedical manuscripts
write manuscripts as writers are ro perform invasive • list general tips for effective writing
KEY TERMS
Abstract Conflict of interest Limitations Peer review
Authorship Copyright Manuscript preparation Results
Competing interests Discussion Medical record reviews Systematic reviews
Conclusions Effective writing Methods Tables
Confidentiality Introduction Objective statement Title

2. Identify potentíal target journals and familiarize

rejecred ourrighr, or e!se senr back for repair before th
yourself with their areas of interest and the types of
edirors will consider ir or send ir for peer review.
articles they publish. Selecr rhose thar appear [O be
7. Be brief Recognize rhat, despire your own fascina
the best fir for your work, and decide on which one [O
tion wirh the ropic, nor everyone will nnd ir rivering
subrnir your arricle ro firsr. Don'r send a case repon or
Don't compound rhe problem by being long-winded
narrative review ro a journal rhar doesn'r publish rhese.
Say what you have ro say in the fewest words possible
When in doubr, send a query to rhe editor by email.
8. Do not repeat information in different parts of the
Ask if rhe journal would consider publishing an article
manuseript. Your readers are smarr. If rhey've seen ir
like the one you plan to write.
in the Introducrion, they don'r need ro see ir again in
3. Understand your potential readers and what they
the Discussion.
expeet. Edirors wanr arricies thar are inreresring, irn-
9. Revise repeatedIy. Mrer nnishing a draft of the rnan-
portanr and practice-changing; rherefore, strive for
uscripr, pLlt ir aside for a few days and review ir again.
breviry, clarity and high impacr.
Is it clear and easy ro read? Is ir well organized? Does
4. Craft a clear objeetive statement before you begin
ir Row logically? If you didn'r undersrand any pan of
writing. Why are you writing rhis paper? In a research
ir, that pan is badly writren. Revise your paper and
anide, rhe objecrive is, ideally, stared explicidy near rhe
read ir again. Then revise ir once more. ManyaLlthors
end of rhe Innoducríon. Review anicles and case re-
do nve to ten revisions before rhey have a manuscripr
porrs should also have objecrives. A clear understand-
they fee! is ready for submission.
ing of the objecrive is crirical: ir will help you dehne the
100 An'ange a.n internal peer review, Ask a colleague who
focus of the research and rhe scope of rhe anide. Every
is familiar wirh rhe publishing process ro read your draft
sentence in rhe manuscript should relare ro rhe objec-
and rnake suggestions before you submit ir.
tive statement. A clear objecrive will help you write
cleady and succincdy.
), Dl'aJt k, much oCthe papel' as yon can before yon Writing the sections within your
perform the research, Research grant applications re- manuscript
quire a well-denned research quesrion, a comprehen-
The guidelines be!ow focus on wriring up an interventional
sive Íirerarure search, a summary of exisring knowledge
trial, but rhe conceprs covered are also very relevanr ro writ-
in the research area, a jusrificarion for rhe proposed
ing up orher kinds of health science srudies. The sections
srudy, and a derailed descriprion of rhe merhods ro be
thar will be discussed in turn are rhe Title, Abstraer, Inrro-
used to address the srudy hyporhesis. You will need to
duction, Methods, Resulrs and Discussion.
complete rhese steps even if you are nor seeking grant
funding. This means thar, even before you begin ro Title
collect data, you will have drafted the Introducrion
Compose a Title that is as concise as possible while giving _
and Merhods, and you will have rnuch of rhe infor-
sufficient informarion ro enable rhose wirh an interesr in
marion necessary for rhe Discussion. (1hese compo-
your neld ro recognize that yOL![paper mighr be of value ro ~
nents of [he research paper will be discussed later in rhis
thern. This means using speciíic rerminology and giving a ~
chaprer.) In addirion, ir is he!pful ro draft the main data
clear sense of the quesrion rhar was addressed. Do nor use •
rabies a priori and ro populare rhern with sham data.
acronyms and other abbreviarions, and ensure rhar your
This exercise will help you plan the analysis and Discus-
ritle conrains terrns rhat wiU make your paper rerrievable in
sion, uncover problems or shonfalls you didn't rhink 0[,
a lirerarure search on your ropic area. Check rhe aurhor
and identify addirional data you may need to collecr.
guide!ines for your targer journal for any specific require-
6. Review the Uniform Requirements for Manuseripts
mcnrs; BMj, for exarnple, requires rhar rhe study method
Submitted to Biomedical joumals' and re-read the
be indicared in thc subritle of research anicles.
author instructions for your target journa1. Ensure
thar your anide is correcdy formarted to rhe journal's Abstract
requjrements. Editors rake a dim view of aurhors who
1he Abstraer collares rhe main elemems of your s udv,
fail to follow simple insrrucrions, and a manuscript
clarif)ring why ir was done (Introducrion or Background.
rhat does not conform ro a journal's formarring may be
how it was done (Merhods), whar was disco\'ered (R ulrs,
256
© 2011 The Royal College of Physicians anó S""~¿>O'3 e: :::¿_~
_9 How to write a hea/th science paper
how the data answered the research quesrion, and how they Methods
should be applied (Discussion or Interpretation)." Ir is 1he Methods secrion describes how rhe research was done,
arguably the most irnportant component of an anide, enabling readers ro assess irs qualiry and exrernal validiry
because ir may be rhe only pan rhar gers read. Many readers (i.e., wherher rhe resulrs can be generalized ro settings orher
will scan an abstraer ro determine wherher ro read rhe ani- rhan rhe site where the research was performed). The merh-
de ir summarizes. Ofren, readers who judge che abstraer ods should be described dearly enough that another inves-
worthy will inappropriarely accepr the informarion at face rigaror can replicare rhe work, but should provide only rhe
value without reading the rest of the anide. Chaprer 27 of necessary derail.l.3.4 If your study employed a complex but
chis guide describes in detail rhe components of effecrive previously validared methodology, ir might be reasonable
abstraer wriring. Some journals also ask for keywords ro ro provicle less derail here in preference ro referring readers
publish with rhe abstraer. If rhis is the case for your rarger ro previously published work, Conversely, if your srudy
journal, use MeSH rerms thar cover che main subjecr areas methods are novel, more extensive derail will be required so
(see ch. 7). rhar readers can undersrand (and replicare) whar was done.
Subheadings enhance organizarion and readabiliry.
Introduction Potential Merhods subheadings indude Design, Serring,
The Introduction is the hook that. grabs readers' interesr. Ir Parients, Inrervention, Study Procedures, Ourcornes and
should convince rhern that your anide is interesring and Data Analysis. You should describe rhe overall design, indi-
imporranr-a rnust read. Ir should also explain why you did cating whether the study was a randomized controlled rrial
this work, brieBy describe the imporrance of the general (RCT) , a crossover trial, a prospective or rerrospecrive
research area, specify rhe knowledge gap rhar rhis research will cohort study, a case-control study, a cross-secrional study, a
fil!, and clarify rhe research question, srudy objecrives and case series, a cost-efiectiveness analysis or some other varia-
hyporhesis. Introducrions ShOLJdbe short and sweet, running tion (see ch. 9). The setting subsection describes where the
ro about 300 words. Only rhe rnost irnportanr references research was performed, allowing readers ro determine
should be cited in rhis section. A good Introducrion has rhree whether the resulrs can be generalized ro rheir own pracrice.
paragraphs: rhe first describes the state of knowledge and the Parients and outcomes in inner-ciry rerriary care centres dif-
scope of the research, the second jusrifies the need for rhe study fer from those in suburban cornmuniry hospirals, and
(the knowledge gap being addressed); and the rhírd conrains patienrs in headache clinics differ from rhose in emergency
rhe studys objecrive staremenr and hypotheses.' If there is a departments. Uncler rhe patients subheading, describe- the
place for long-drawn-our discussions (and there probably population sarnpled (e.g., patients preseming ro an erner-
isnr), ir is in rhe Discussion secrion. gency departrnent wirh a primary cornplainr of chesr pain)
as well as che indusion and exdusion criteria used ro selecr
111e objective statement-rhar is, the staternent of your srudy parricipants.
research quesrion (see ch. 6)-warranrs special emphasis. If applicable, the experimental trearrnent or interuention
Often framed as a quesrion, this staternent is rhe most should be described, as well as the merhods for obtaining a
imponanr senrence in a scienrific paper and is a crirical pan comparison group (e.g., randomizarion, prc-post design).
of any research proposal, 1he objecrive defines the focus loe srudy prorocol or study procedures should describe how
and boundaries of the work, rhe required methodology, data were collected. Primary and secondary outcomes shoulcl
and the formar and scope of rhe wrire-up, Every concepr be defined, along with rhe techniques used to collecr and
discussed in the manuscripr should be relared ro ir. A clear measure these. The data analysis secrion should clariíy how
and focused objecrive will help you avoid the fatal Baw of major variables were analyzed, giving enough derail ro
rambling off on rangenrs or presenring exrraneous findings. enable a knowledgeable reader wirh access ro the original
The objecrive srarernenr also orienrs readers ro whar they dara ro verify rhe reponed calcularions and srarisrics. Main
should expecr ro find in the body of the article. Borh the outcornes should include estimares of precision, such as
main and secondary objecrives should be clear, and any 95% confidence inrervals and, if a hyporhesis is being test-
planned subgroup analyses should be described. ed, a P value. Ir is inappropriare, however, ro presem orily P
values, which tell reaclers only the likelihood rhar observed
outcorne differences occurred by chance, without indicar-

ing the precision of the estímate and the range of values presented and that calcularions and percentages are corree
consisrcnt with rhe observed result.? urprisingly, manuscripts subrnitred for publicarion [r
The Merhods section should also include a descriprion quenrly conrain dara and calculation errors in the Abstra
of the patient consent process, Research Ethics Board and the tables. Graphs should be used sparingly: rhar i
approvals and, if applicable, trial registration. If the Meth- only for imporrant study outcomes and only if rhe
ods secrion is roo exrensive, rhe necessary derails can be enhance understanding of rhe data. An essemial figure
moved into appendices or published in the online version however, is a study flow diagram ro show how rhe stud
of the journal. In designing a study and writing a rnanu- sample was arrived at; the CONSORT flow diagram, fo
scripr, aurhors should refer ro the appropriare reporring example, is the standard used ro report the nurnber o
guidelines, such as rhe CONSORT reporting standard for parienrs screened, eligible, missed, exduded, induded anc
RCTs.a4 See Textbox 29.1 [or a lisr of sorne o[ the more lost ro follow-up in a RCT. b
prominent reporting standards. Table 1 in your repon should sumrnarize the baseline
characreristics of rhe study popularion: that is, the relevam
atrributes of patients in the compararor groups prior ro rhe
TEXTBOX 29.1 GUIDELlNES FOR THE REPORTING OF
intervention or exposure of interest, 1his table rypically
HEALTH SClENCES RESEARCH
reports age, sex and other demographic predictors and
covariables thar could influence patient outcorne. For
Many evidence-based reporting guidelines have been devel-
example, in a study of myocardial iníarction ourcornes,
oped by various workinq qroups. The following list is a sample.
Some journals require compliance with certain guidelines; Table 1 might indude predicrors such as previous coronary
check the instructions for authors of your target journal to be artery disease, diabetes, hypertension, dyslipidemia, smok-
sure. For íurther information, lists of guide1ines and links to the
ing history, cardiac medications, NYHA (New York Heart
quidelinss themselves, see the EQUATOR.l\Jetwork* website at
vvvvvv.eq uato r-netvvork.org/h omel. Association) classihcation, viral signs OH arrival, and arrival
mode. Readers can use Table 1 ro determine wherher
r..:ONSORT Statement CO~Jsolidat¿d Standards of Reporting Trials (for patienrs in the two groups were similar at baselinc and I
randomized cC~Ti:rolled trials)
------- whether they resemble patients in their own setting. In an
fIIIOOSE Meta-analysis Of Observational Studies in
-------
Epidemiology observarional study, Table 1 should indude a sraristical
PRISfIIIA Preferred Reporting Items for Systematic reviews and
Meta-Analysis (This guideline replaees QUOROM:
analysis ro show wherher the rwo groups differee! signifi-
Quality of Reporting Of Meta-analyses.)
canrly wirh respecr ro important baseline predicrors. Con-
SQUIRE Standard íor Quality Improvement Reporting
Excellence versely, there is Iirrle value in derermining P values wirhin
STARD STAndards for the Reporting of Diagnostic accuracy Table 1 of a randomized trial: assuming an effective ran-
studies
domization process, any observed baseline differences
STROBE STrcngthening the Reporting of OBservational studies
berween groups rnusr have arisen by chanceo
_
in Epidemiology
------
Table 2 describes the patients after the intervemion or
. 'EQUATOR: Enhaneing the QUality And Transpareney Of health Researeh
exposure, highlighting differences in the primary outcorne .
Table 2 should display absolure and relative differences,
estimares of precision (e.g., 95% confidence inrervals) and
Results usuallya test o[ sraristical significance. Additional tables are I
The Results secrion summarizes the srudy findings in often necessary ro illusrrare differences in secondary out-
rables, figures and rext. Tables are the heart of the Resulrs comes or ro highlighr subgroup or posr-hoc analyses.
secrion, and main findings should be presemed in rabular Authors should realize that rabIes and figures consume
formo Tables and figures should be self-explanatory so that valuable page space; therefore, journal editors will limir I
readers do not have ro refer back [O the texr in order ro deci- rhese to the minimum number required ro ensure clariry,
pher thern. However, rables and figures should comple- The narrative component of rhe Results secrion should I
ment the text, nor duplicare ir. Before submission, authors be written afcer rhe tables are populared. Ir should be con-
should check rheir rables ro ensure that denominators are cise, presenting only important findings rhat relate ro the
aSee www.consort-statement.org/consort-statemenV b Seewww.consort-statement.org/consort-statemen flow-dreqrerr
258 © 2011 The RoyalCollege oí Physiciansand Sur~~ - 'Ca-¿::¿

29 How to write a health science paper
study objective. The texr should refer readers ro the relevanr tion summarize the main srudy findings, and indicare
cables, and al! tables and figures should be cited in the texr. whether and how they suppon the hypothesis. 111is is your
The rext should clarify, highlight or expand upon data pre- chance ro srare in a few senrences what the take-horne mes-
sented in the tables, bur should not duplicare it. sage is and why your study is imporrant. The second para-
The first paragraph in the Results section should provide a graph should discuss the results in the context of what they
brief summary of the nurnber and type of parienrs included add ro pre-existing knowledge and how they address rhe
in rhe final srudy sample, as wel! as the nurnber excluded and knowledge gap rhat rhe study was designed ro address. This
the reasons for exclusion. The second paragraph should paragraph or rhe nexr should review orher importanr srud-
describe the primary ourcomeís) as they relate to each study ies rhat suPPOrt or refute the main findings. Authors should
objective and hypothesis. Subsequent paragraphs should try ro clarify how and why their findings differ from earlier
highlight secondary ourcornes and planned subgroup analy- studies and suggest orher possiblc interprerations of the
ses. Each major outcorne should be compared descriprively, data. Irnportanrly, the Discussion should never introduce
using the appropriate measures of central tendency, such as data rhat did nor appear first in rhe Resulrs section.
means, medians or proporrions, and rhe applicable measures Take care not ro oversrare rhe imporrance of your find-
of variabiliry, such as range, variance, standard deviarion or ings or ro draw inferences that go beyond what rhe data jus-
interquartile range (see also ch. 24). For key outcornes, an tify. Moreover, rhe Discussion section should conclude
estimare of precision (e.g., confidence imerval) is valuable, with a paragraph or rwo describing rhe study's strengrhs and
and a P value is cusromary for each hypothesis test (see also limitations. (Sorne journals require a separate Limitations
ch. 25). More complex multivariable analyses or adjustments section.) The limitations of your study might relate ro rech-
may be necessary but are beyond the scope of this chapter. nical or data-collection problems, potencial biases, methcd-
When writing the Results section, it is important ro ological challenges and externa! validiry, Highlighr concerns
exclude inforrnation rhat is more appropriate for rhe Merh- that mighr orherwise be apparem only ro an investigator
ods or Discussion. Aurhors should also avoid rhe temptation who was direcdy involved in rhe study, This secrion should
ro present "interesting data" that are unrelated ro rhe study conclude wirh recommendarions for future work in your
objective, limiting rhernselves ro "just the facts." Opinions topic area. For more detail on writing the Discussion sec-
and interprerations should be moved ro rhe Discussion. tion, see chapter 26.
Discussion Conclusions
The Discussion section should summarize the main find- Your Conclnsions section should be brieí, consisting of a
ings, explain what the results mean, compare and conrrast couple of senrences or a shorr paragraph. Don'r reiterare
the findings previously reponed in rhe literarure, discuss argumems presented in the Discussion; insread, srare your
srudy strengths and limitations, and suggest directions for major findings and how rhese should be applied. The Con-
furure research. The Discussion section is rhe most difficult clusions should also highlight appropriare cavears about che
ro write; rhis is where authors frequendy go wrong by los- generalizabiliry of results, such as specilic settings or patient
ing focus, repeating information presented elsewhere in the groups ro which your conclusions do nor apply, Ir is a criri-
manuscript, drawing conclusions rhar are not justiíied by cal error ro make unqualified conclusions rhat are not justi-
rhe data, or drifting off on tangents that are irrelevant ro rhe fied by the data. However, simply calling attention ro the
srudy objective. As you draft rhe Discussion, keep a lirnit of need for more research is not an adequare conclusion.
approximately 750 words in mind. In your first draft, ir is
helpful ro use an outline and subheadings, which can be
Special cases
retained or excluded írorn rhe final manuscript as appropri-
ate. The following srructure is one possible approach ro 111efollowing sections describe particular aspects of writing
writing a clear discussion. up medica! record reviews and systematic reviews; more
The opening paragraph is rhe power positicn. Ir is the derailed informarion on how ro conduce these rypes of
only part of the discussion that many readers will read or studies are provided in chapters 12 and 15, respecrivel y.
remember. 111isparagraph should restare the research ques-

r
Medicar record reviews articles, ro apply val id inclusion and exclusion crireria, p
Medical records are informal descriprions of observarions, form a blinded assessrnent of source arricle qualiry, der
impressions and hunches. 111ey conrain mosrly narrarive mine rhe reliabiliry of this process by measuring (
descriprions of people and evenrs, and rhe rranslarion of agreement of rwo evaluarors, account for all srudies iden
rhese verbal descriprions inro hard quanrirarive dara is fied, included and exeluded, describe the merhod of cor
fraughr wirh error." Schwarrz and colleagues" demonsrrared bining srudy resulrs, discuss variarion wirhin and berwe.
poor agreemenr between informarion derived from rhe srudies, and draw appropriare conelusions. Authors shou
medical record and informarion garhered direcrly from (he ourline the limitarions of their review and suggesr areas f(
parienr. Medical records frequenrly lack crirical dara, furure research. The Cochrane Handbook for Systemar
ineluding chief presenring complainr, hisrorical and physi- Reviews ofInrervenrions is an excellenr reference for prc
cal findings and laborarory resulrs. Errors, inconsisrencies spective authors."
and ornissions in medical charrs are compounded when
informarion is exrracred during a scienrific invesrigarion.
Tips on good writing
When rwo absrracrors look for the same dara in rhe same
medical record, rheir resulrs ofren disagree. Common Remember rhar you are nor a Michael Crichron. YOL
sources of error in medical record reviews inelude missing haven't been signed to a íive-book deal, and no one is read-
charts, missing dara, conHicring dara, vague or illegible ing your article because they enjoyed the literary sryle of
charring, inconsisrenr rransformarion of descriprive dara )'our previous rwo. They are reading because )'our title
inro categorical data, and difíerences in rhe handling of caught their attention and your abstraer convinced rhem
arnbiguous or missing dara. In addirion, biased abstracrion rhar you had useful inforrnarion ro share, They wanr this
can occur if chart abstractors are aware of the study hypoth- informarion as guickly and easily as rhey can get ir. There-
eses 01 expecrations. To reduce error and increase reliabiliry, fore, alrhough good science and a well-organized rnanu-
studies involving chart review should inelude the following script are a rnust, eflective writing is equally important,
critica] elemenrs: No rnarter how imporranr )'our topic, readers (ineluding
reviewers and editors) are Íikely ro abandon rhe manuscript
• a descriprion of inelusion criteria if rhey encounter long, boring or obscure passages.
• definitions of che variables being analyzed Good scienrific wriring is clear and simple. Ir generall)'
• a descriprion of abstractor training prefers rhe acrive to the passive voice, uses strong and pre-
e rhe use of standardized abstracrion forms cise nouns and verbs, and eliminares all unnecessary words.?
• a descriprion of absrracror monitoring If )'0 u are a rypical physician wrirer, an editor will be able ro
• a descriprion of absrracror blinding ro study hypotheses reduce the word counr of your manuscript considerably
• a discussion of inter-rater reliabiliry wirhout eliminating imporranr informarion, in rhe process
making ir clearer and more readable. Bur substantive edir-
Systematic reviews ing consumes valuable time, and rejection is a simpler
Tradicional narrative review articles, in which rhe aurhor oprion. It is besr for the author to srreamline the article
sumrnarizes published lirerature in an unsrructured docu- befo re submission. Authors can improve their wriring qual-
ment, are subjecr ro several biases, and many journals will iry by remembering a few imporrant rips, examples of
no longer consider review articles unless they ernploy valid which are given in Texrbox 29.2 ar rhe end of rhe chaprer.
sysremaric review merhodology, as described in rhe PRIS- (Furrher resources on effecrive writing are given in Addi-
MA sraremenr (Preferred Reponing Irems for Systematic tional Resources.) These tips are as follows:
Reviews and Meta-Analyses).' BrieHy, rhese require authors
ro define a clear research quesrion, to evaluare earlier • \Xirire mosdy in rhe active voice using a subje r-verb-
reviews and justify the need for anorher one, ro develop a objecr senrence strucrure, This will help ro make your
sysrernaric search srraregy ro identity and selecr primary wriring elear and succinct.
Example: "The scoring systern was pilor-resred
by rhe dara absrracrors." (passive) \"5 "The data
c See www.prisrna-staternent.orq/ absrractors pilor-tesred [he scoring sysrern" (active).
260 © 2011 The Royal College 01 Physicians and Su·gec-.s o' C,-z:;¿
29 i-Iow to write a health science paper
Do nor forma (he article as you think ir might appear in

• Use paral!el structure by consuucting linked phrases
print, with page columns, different rypefaces, and figures
and sentences in a similar fashion.
Example: "Physicians idenriíied eligible parients; and rable incorporared into rhe text. Provide figures and
study drugs were given by nurses; radiologists tables ar rhe end of the file or in separare files, as rhe
performed image interprerarion" (non-parallel) journal's insrructions specify. Many journals prcfer thar
should become "Physicians idenrihed parients; authors use no special referencing software (such as
nurses gave study drugs; radiologists interpreted EndNore) and no rracked changes, small caps, superscriprs,
or special formarting) because these may conflict with
images" (parallel).
• Use logical progressions. \'(fhen combining ideas, journal sofrware .
proceed logica!ly from most ro leasr irnportant, largesr
The manuscript submission package
ro rhe smallest, or first ro last.
Example: "The srudy showed more patients with Mosr journal submissions should inelude the following
1
adequare blood pressure control, lower rates components.
of death, and fewer myocardial infarctions in
Cover letter. Provide a brief cover letter that describes rhe
the inrervention group" should beco me "The
study and irs findings in a senrence or rwo, specifies each
intervention group had [ower tates of death,
author's conrribution ro rhe work, and confirms rhar al!
myocardial infarction and hypertension."
coaurhors have provided their perrnission to publish rhe
• Eliminare words and phrases that add nothing. Many
manuscript. Similar work that rhe aurhors have published
aurhors believe rhat complicated and fl.owery words
elsewhere, which could be construed as duplicare publica-
add up to better writing. Nothing could be further
rían, shauld be reponed ro rhe editor along wirh a stare-
from the truth, Good writers use a spare sryle thar
menr of potential conflicts of interest and financial
relies on precise nouns and verbs." lf you need to
diselosures. lf you intend ro inelude previously published
qualify things with an adjective or adverb, this might
marerials (such as figures) in your anide, yau wiJl need ro
mean rhat you have the wrong noun (or verb). To
seek permission from the copyright holder and ro inelude
illusrrate: arhleres don't run quiddy; rhey sprint.
the appropriate documentation in your cover lettcr, Ar a
• Minimize rhc use of abbrcviations, especially
minimum, flag materials for which perrnissions are needed
acronyms (e.g., PAHO) and initialisms (e.g., TIA).
or pending so rhat, if your anide is accepted, rhis irnpor-
Use abbreviations only when they are well esrablished
tant task will not be averlooked. Many publishers have
or if it would be cumbersome ro spell out a terrn in
srrearnlined the permissions process through online servic-
full each time ir appears in your paper; in rhe larter
case, provide the terrn in full on first occurrence, e.g., es such as RightsLink.d Because copyright holders mar
require informarion abour your rarget journal (e.g., rype of
"rransienr ischemic auack (TIA)."
disrribution, whether the journal is for-profit or non-prof-
it), you may need ro consult wirh the journal editors before
Formatting your manuscript for camplering rhe permissions process. You wil! also need ro
submission determine who is responsible for covering any associated
Most journals conform to rhe manuscript preparation fees.

guidelines laid out in the Uniform Requirements for
Title page. The title page should includc the manuscript
Manuscripts Submitted to Biomedical [ournals, 1 and anides
riele, the authors' names, degrees and instirurional affilia-
thar adhere ro these guidelines wil! rarely be rejecred on rhe
tions, and rhe narne, mailing address, relephone number,
basis of sryle or format. Authors should also refer ro rhe
fax nurnber and email address oF rhe corresponding author.
specific instrucrioris provided by their rarger journal.
le should also indicare the nurnber of figures and rabies and
However, the following generally apply: double-space the
manuscripr, from tirle page ro references, and use a
consistent 12-poinr font. Number pages consecurively,
d Available through the Copyright Clearance Center at wwwcopyright.
beginning with the ritle page, and provide a running hcad.
corn
261
© 2011 The Royal College 01 Physicians and Surqeons 01 Canada
include a word COUll( for the abstraer as well as rhe main ous work in the area or rnethodological expertise. Many j(
rexr (excluding abstraer, tables and references). Relevanr nals allow authors to request specific reviewers or to iden
disdaimcrs should be reponed here, along wirh sources of reviewers who should not be solicired in light of possible F
Slippon in the form of funcling, study supplies, drugs or sonal or professional conílicts of inrerest.
deviees. Most journals requesr a shorr running head ar rhe Peer reviewers use structured review forms ro prov.
bortorn of rhe title page. A confiicr-of-inreresr disdosure cornments on the imporrance of the work, the qualiry
page should follow rhe tirle page. rhe merhods, (he validity of (he findings, rhe quality of t
manuscript and the priority for publication. Most journ:
Abstract. Structured abstracrs of 250-300 words are usu- invite reviewers ro indicate whether the manuscript shou
ally required for original research articles, sysrernatic be accepted withour revision, accepted after specific rev
reviews and rnera-analyses. Requirements for case reports sions, revised for resubmission and reconsideration, (
differ bur typically indude a brief unstructured abstraer rejected outright. Peer reviews hdp editors gauge rhe likel
that summarizes the problem or objcctive, the main points interest of rhe article ro readers, and oíten uncover subd
and rhe condusions. See chapter 27 for a detailed discus- Haws or concerns in the work, but reviewers do not rnal«
sion of abstracts. publication decisions.
Decision edirors collate the peer reviewers' comments
Main texto The cornponents of rhe main text of rhe anide and use rhern to inform their decision about the manu-
are discusscd earlier in rhis chapter. script. Depending on the size of the journal, rhis decision
may be made at an editorial committee meeting. After rhe
Figures aud tables, ]ournal editors ofren ask that figures decision, rhe editor composes a response letter indicaring
and rabies be subrnirted in separare digital files. Most have whether the manuscript is accepted, rejected, or requires
spccificaticns wirh respect ro rhe tormarting and permissi- revisión and resubmission. The response Íetter should SUD1-
ble file types of rabies and figures! If you are intending ro marize importanr concerns and requesred changes to the
reproduce or adapr figures. illusrrations, rabies 01' other manuscript. Because not all pecr review comrnents are
material from anorher publicarion, written pennission wil! appropriate, editors should provide some guidance regard-
need ro be sought Irorn the publisher or copyright holder ing how to respond ro these. Many journals provide rhe
ro use rhese irems. actual blinded peer review cornments along with rhe edi-
ror's response lerrer. Editorial response times vary among
journals and depend on the type of response. If rhe manu-
Understanding the peer review
process script is rejecred during its initial screening, authors may
receive an email response within days of submission. If rhe
Ir is irnporranr for you to have a basic knowledge of the manuscript is forwarded for peer review, response times of
peee review process, which is intensive, time-eonsuming 4-12 weeks are typical.
and oíten frustrating. The objective of peer review is to Many manuscripts are rejected outright (see Textbox
have peers as well as scientific and conrenr expens review 29.3 a( the end of the chapter), and few are accepred withou t
the submitted work and provide opinions regarding its revision. Most journals have esrablished criteria allowing
interesr, importance, quality and validity. authors ro appeal the rejection decision in cases where they
Subrnitred manuscripts are firsr logged and categorized, believe there was a fundamental misundersranding of rhe
and are rhen assigned ro an appropriare senior editor or deci- data or a potential conílict of interest during the editorial or
sien editor on the basis of their contento This editor will reject peer review process. Appeal criteria should be availabJe on
(or "intercepr") a significanr proporrion of manuscripts imme- rhe journal's website.
diarcly alter iniiial screening. Arricles rhat are judged tu be In mosr cases, aurhors should hope for a decision lerter
porentially suirable for the journal, and of suft1ciendy higb that requests revision and resubrnission. When this hap-
qualiry, are Iorwarded Ior blinded peer review by two 01' three pens ro you, address all of the queries and revision reques-
reviewers selecred on the basis of their content expertise, previ- made by the decision editor. If suggesred revisions \\;JI
improve the manuscript, make rhern without quesrion. 1-
e SeE:,lar example. the CMAJ guidelines far submitting tables, figures and you disagree with any speciíic change requested, ex lain
graahics al wwwcmaj.ca/aulhars/preparing.dtl#tables
262 © 2011 The Royal College oí Phys'cia'15 "rti s...--;;?:;-;:;:: r?:a::¿

29 Haw te write a health science pepet
yom reasons for doing so and come (O an agreemem with Criteria for authorship
the editor. In some cases, aurhor-editor disagreement arises To qualifv for authorship credit, a mernber of the research
frorn a misunderstanding of the paper by the editors or peer tearn should sarisfy each of rhree crireria: contriburions ro
reviewers, and rhis may reflect a poorly written or undear performing [he research (i.e., study design, data acquisition
manuscript. In some cases, edirors will ask for funher data or dara analysis): drafting or revising rhe anide; and
collecrion or re-analysis, which may or may not be feasible approving rhe final version for publication.' AlI lisred
depending on the accessibiliry of data. Complete the authors should qualify for authorship, and all people who
req uested changes as rapidly as possible (prornpr responses qualify should be listed. Each listed aurhor should feel
are viewed as evidence of a conscientious author) and for- cornfortable raklng responsibiliry for the srudy comento
ward the revised manuscript (O the decision editor with a Orher individuals who conrributed in some wa)' bur do not
brief explanarion of what changes were (or were not) made. rneet the crireria for authorship should be lisred in an
Frequently, aurhors will be asked (O do a second or third Acknowledgments section.
revision before acceptance. If the rnanuscripr concerns can
be addressed (O rhe satislaction of rhe decision editor or edi- Confliet of interest or eompeting interests
rorial committee, you will ultirnarely receive a letrer of Authors and their institutions, as well as editors and review-
acceprance and rhe manuscript will move into the publica- ers, have financial relationships, personal associarions and
rion pipeline. Depending on the timeliness of the subject, other interests rhat mighr influence rheir acrions. Financial
the arricle's prioriry raring, and the number of articles inrerests such as employment income, honoraria, consulr-
already in the pipeline, ir may still be an additional2 to 12 ing fees and stock ownership present the rnost obvious con-
momhs before your artide is published. Hicts, but personal relarionships, academic cornpeririori
Alter acceprance, the senior or decision editor will work and srrongly held beliefs can also compromise scientihc
wirh a copy editor to revise the accepred anide for clariry, and academic objectiviry, Authors rnust dedare al! poren-
breviry and journal sryle. Depending on the qualiry of the rial coníiicts of interest ar the time of article submission. A
initial submission, this somerimes means significant chang- standard coníiict of interest form is available from rhe
es (O the manuscripr and substancial reductions in word Inrernarional Comm irree of Medical Journal Edirors.?
count. Aurhors have rhe opportuniry (O review and approve journals frequendy have their own forms as pan of rheir
all revisions before publication, but the journal editor holds online submission process. Editors will publish rhis infor-
all rhe cards. Prior ro publication, aurhors are required (O mation if they feel thar ir is likely ro inAuence readers' inter-
sign rransfer-oí-copyrighr forms. Published manuscriprs pretarion of the work.
beco me rhe properry of the journal's copyrighr holder (like-
ly either rhe publisher or rhe sponsoring sociery), and may Data aeeess and publieation
not be published or disrribured elsewhere, even by the Dming the conducr of indusrry-funded srudies, investiga-
author, without permission. 1he exception to rhis is open rors should obtain assurances, in wriring, rhar rhey \ViII
access publication, in which the author retains copyrighr have full access (O rhe study darabase, will be able ro analyze
and readers are free to reprinr, disrribute or adapt rhe work, the data independenrly, and will be able ro use their own
with proper atrriburio n, in accordance with the panicular discretion, free of interlerence, in preparing anel publishing
Crearive Commons licence that the journal applies.' articles derived from rhe data. Authors should declare rhe
role of the study sponsor in study development, data collec-
tion and analysis, anel in writing the article-indueling the
Ethical considerations
absence of any such role. Edirors may choose nor (O publish
Various erhical consideration that rnusr be taken into an anide where there is any question abour rhe investiga-
aCCOUI1rin rhe conduct of health research are discussed in rors' access (O srudy data, or evidence suggesring inappro-
Chaprer 18 of this guide. 1he following secrions relate ro priare involvernenr of a porcntially biased sponsor in rhe
issues of particular relevance to che publishing process. design, analysis or study wri te-u p (see also ch. 18).
- _._-- __ ._------
See http//creativecommons.ca/index.php?p=explained 9 See www.icmje.org/sample_disclosure.pdl

The Research Guide: .11. primer for residents, other health care trainees, and practitioners
Confidentia!ity Ethical approval

111e confidentiality of people who participare In research Authors should inelude with their submissions a deelar
muse be scrupulously respected, No identifiers, including rion thar their work has received all necessar)' approv<
narnes, initials or recognizable phorographs, should be pub- from a Research Ethics Board and/or animal care and u.
lished withour me individual's wrirren informed consem. cornrnirtee.
Non-essential identifying details should be ornirted. Inforrned
consent, including a participant's review of me unpublished Trial registration
rnanuscripr, is also warranted if a case description is highly rec- lnvesrigators should register clinical trials wirh an appro
ognizable. When informed consenr has been obrained, mis priare clinical trials registry such as ClinicalTrials.gov a
should be srared in me submitted manuscript (see also ch. 18). soon as ethics board approval has been obtained. The tria
Authors also have confidentialiry rights. Reviewers, edi- registrarion number should be ineluded with the rnanu
rors and other journal sraff mernbers must not publiely dis- script sub miss ion.
cuss, present or disserninare srudy data before publication,
excepr by agreement with the authors. Also, journals edi-
tors should not diselose to anyone beyond their staff that Conclusion
they have received a given author's submission; aside from The natural outcorne of research should be the dissemina-
being a breach of the author's privacy, doing so can make tion of new informarion and the synrhesis of evidence-based
rhern open to undue influence. Reviewers should destroy knowledge. Publishing research articles in peer-reviewed
copies of manuscriprs alter reviewing them. Reviewer ano- journals remains one of the keysrones of knowledge transla-
nyrniry is variable: sorne journals offer reviewers the option tion in rhe health sciences, and is supporred by exrensive
o[ being anonymoLls, and if this is the reviewer's expressed databases for indexing and retrieval rhat enable researchers
preference at the time of review, journal editors cannot around the world ro benefir from one another's work. How-
reveal his 01' her identiry to anyone without perrnission. ever, as narural as rhe publication of research may seern, ir
requires a ser of skills rhat are by no rneans autornaric for
Ec!:'i:or¡al freedom everyone. This chapter has outlined the key considerations
JCllrnal editors should responsibly exercisc full control over the 'in drafi:ing and submitting rnanuscriprs for publicarion.
sclecrion of journal contenr, the review process, rhe selecriou Skill in research writing develops with pracrise, however,
and editing of rnanuscripts, and rhe timing of publication. and readers are encouraged ro seek guidance along the wa)'
]ournal owners, whose focus ma)' be commercial success rath- from rhe excellent wriring guidebooks and workshops thar
er than scientific imegriry, should neither interlerc in the edi- are available, man)' of which are railored to the exacting
torial process nor creare an environrnent thar unduly dernands of wri ring up health sciences research. •
inBuences editorial decision-rnaking, Editors should be free ro
express opinions or publish rnaterials that conflict with me
owners business goals.
o
e, CASE POSTSCRIPT
Q)
The ernergency physician decides to get a second opinion from a trusted colleague with a modest publication record. Unlike
o:: the journal editor, the colieague doesn't have dozens of other rnanuscripts to read every week, and so takes the time to point
out various ways in which both the organization and the writing style make it difficult to follow the methods or gain a clear
picture of the findings. Even the research question, she explains, is unclear After some dicussion. however. she is oerscacec
that the researcher's method was sound and his results worth reporting. The physician goes back to the drawing board, a .0
after several rounds of rewriting finally produces a manuscript that is clear, logical and concise. He resubmits the pape' ·0 Te
sarne journal editor, who, this time, agrees that it is worthy of being sent for peer review. A little older and m eh ·:ise-. e s
eagerly awaiting the result.
264 © 2011 The Royal College oí Physioans 2-0 S :;..~ =- ::3".32

9 How to write a health science paper
TEXTBOX 29.2: WRITING TECHNIQUES-SAMPLE - TEXTBOX 29.3: COMMON REASONS FOR REJECTION
SENTENCES FOR REPAIR
T e Ti-le is vague, misleading or uninterestinq.
The Abstract is too long, overly inclusive or disorqanized.
Resultsand conclusions reported in the Abstract differ
Prefer the active voice
In previous studies looking at the management of írorn those in the body of the rnanuscript These critical
inflammatory processes such as asthma, community- errors often occur because researchersforget to revise
acquired pneumonia and meningitis, important outcomes the Abstract along with the evolving manuscript
such as morbidity, mortality and lengths of hospital The Introduction is boring and does not "hook" the
stay have been shown to be affected positively by the reader.The study does not seem important, novel or
implementation of early interventions intriquinq. The authors include too much information
that should be incorporated in the Discussion_The
Better: Previous studies show that early interventions researchquestion, hypothesis and study objectives are
improve outcomes with respect to morbidity, mortality and not specified or are not clear.
hospitallengths of stay for inflammatory processes such as • The Methods do not follow the journal's required format
asthma, community-acquired pneumonia and meningitis. or are not clear enough to be reproduced; details are
missing or ornitted. There are major methodological
Keep structures parallel flaws; the dates of the study are not specified; ethics
Study subjects at the two sites had similar age and gender approval was not obtained. No sample-size calculation is
distributions; however, the inner-city sample included described. The study is underpowered.
larger proportions of African-Americans and Hispanics, The Resultsfail to display a patient flowchart, to
while whites were the predominant ethnic group at the compare baseline characteristics, or to address potential
community site. confounders. Data or patients are missing from the
analysis.
Better: Age and gender distributions were similar at the
• Outcomes are reported only as P values or odds ratios,
two sites; however, inner-city patients were mostly African-
denying readers the opportunity to look at patient
American and Hispanic, whereas community patients were
counts, rates or proportions. The authors failed to
predominantly white.
adjust for multiple comparisons or to analyze outcomes
by treatment received rather than intention to treat.
Don't waste words
For subjective observations, no reliability assessment is
Wiliiam Osler, one of the most notable medical teachers,
provided.
taught using an economy of words. Students marvelled at
Osler's teaching style, his clarity and the preciseness of his Numbers in tables don't add up. Text and tables are
words on hospital rounds. redundant Authors calculate too many P values or fail
to recognize the limits and meaning of P values. Authors
Better: William Osler was a remarkable teacher known for report statistical significance but fail to comment on
his clarity, precision and economy of words. clinical importance.
• The Discussion is unfocused, expansiveor tangential_
Use logical groupings The key results are not adequately discussedor
Our data show that, despite high levels of reported pain explained. Study implications and importance are
at discharge and high satisfaction with pain management, overstated or biased. Important findings are omitted;
subjects reported low rates of analgesic administration_ speculation is not identified as such; and limitations are
not described.
Better: Our data show that. despite low rates of analgesic
Referencesare outdated, or key papers are not
administration and high levels of pain at discharge, subjects
referenced.
reported high satisfaction with pain management
• The Conclusion does not answer the study question,
does not set limits for generalization, simply restates the
results, or merely calls for more study
© 2011 The Royal College of Pllysicians and Surgeons of Canada 265

REFERENCES
1. International Committee of Medical Journal Editors. Upda ed April 2010. Uniform requirements for manuscripts submitted to
biomedical joumals. Available from: www.icmje.org.
2. Foote M. Some concrete ideas about manuscript abstracts. Chest. 2006; 129(5); 1375-7.
3. Hopewell S, Clarke M, Moher D, Wager E, Middleton P,Altman DG, et al. CONSORTfor reporting randomised trials in journal
and conference abstracts. Lancet. 2008; 371 (9609):281-3.
4. Schulz KF,Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement updated guidelines for reporting parallel group
randomised trials. Ann Int Med. 2010;152(11 ):726-32.
5. Worster A, Rowe BH. Measures of association: an overview with examples from Canadian emergency medicine research. Can J
Emerg Med. 2001 ;3(3) 219-23.
6. Gilbert EH, Lowenstein SR, Koziol-McLain J, Barta DC, Steiner J Chart reviews in emergency medicine research: Where are the
methods? Ann Emerg Med. 1996;27(3):305-8.
7. Schwartz RJ,Boisoneau D, Jacobs LM. The quantity of cause-of-injury information documented on the medical record: an appeal
for injury prevention. Acad Emerg Med. 1995;2(2) 98-103.
8. Higgins JPT,Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.0.2 [updated September
2009J. The Cochrane Collaboration, 2009. Available from: wvvw.cochrane-handbook.org.
9. Strunk W, White EB. The e/ements of style 4th ed. Needham Heights (MA) Allyn & Bacon; 2000.
ADD!T10NAL RESOURCES
AMA Manual of Style Committee. AMA lVIanual of Style. 10th ed. New York (NY) American Medical Association; 2007.
Equator Network. Resources for authors. Available from: www.equator-network.orgAesource-centre/authors-of-research-reports/

autho rs-of-research-reports/#a uwrit
e A bibliography, with links, of resources that provide guidance on writing in the health sciences.
Gopen GD, Swan JA The science of scientific writing: if the reader is to grasp what the writer means, the writer must understand
what the reader needs. Am Scientist. 1990 Nov-Dec; 78:550-8.
• A classic article describing how scientific writing can be improved and made more effective.
lIes RL. Guidebook to better medical writing. Rev.ed. Olathe (KS): Island Press;2003.
Lang TA How to write. publlSh, and present in the health sciences: a guide for clinicians and laboratory researchers. Philadelphia
en (PA): American College of Physicians; 2009.
c:
u_
· A comprehensive guide on scientific writing by an expert in the fields of scientific writing and editing.
Lang TA, Secic M, editors. How to report statistics in medicine. 2nd ed. Philadelphia (PA): American College of Physicians; 2006.
• A comprehensive reference on how statistics should be reported in biomedical publications.
Tarshis B. How to be your own best editor. New York (NY): Three Rivers Pressand Random House; 1998.
Tufte ER. The visual display of qualitative information. 2nd ed. Cheshire (CT) Graphics Press;2001
• A conceptual exploration of how diagrams of various kinds can be used to convey complex data.
266 © 2011 The Royal College oí Physicians c-d S -;:::--oc ==:':a:.=

29 Haw to write a hea/th science paper
WRITING EXERCISES
1. Revise this sentence by eliminating unnecessary words.
Figure 3 shows that changes in emergency department volume in all o' the 3 study areas appear to follow the same
seasonal fluctuations, and that the ED volume peaks appear to occur during the summer months of May-August
2. Eliminate unnecessary words and phrases from these sentences.
AII personal identifiers were kept strictly confidential.
Thus, on average, the data showed that patients spent the majority of their time while in the emergency department
(57%) in moderate to severe pain.
3. Strip this sentence to the essentials and use the active voice.
Emergency departments across the country have often been acknowledged as suffering from profound levels of patient
overcrowding, which is the basis of long waiting times, and this challenging situation is not ameliorated
by the concurrent inefficient use of time and resources for any given clinical problem.
4. Edit this sentence using a logical progression.
Furthermore, the decrees required that manufacturers increase public and user awareness of the dangers of motorcycle
use through a media campaign, that warning labeis be attached to the vehicles and that a training program be provided
for those who purchase a motorcycle.
Possible answers
1. Figure 3 shows that emergency department volumes peaked during summer in the 3 study regions.
2a. AII personal identifiers were kept confidential.
2b. Patients spent 57% of their emergency department time in moderate to severe pain.
3. Emergency department overcrowding prolongs waiting times. Inefficient use of time and resources aggravates this
situation.
4. The decrees required that manufacturers attach warning labels to vehicles, provide training programs for purchasers, and
run media campaigns alerting the public to motorcycling hazards

SUMMARY CHECKlIST
o Prepare for your next publication by doing some personal development, e.g., a writing course, a discussion with a
writing mentor, or reviewing a text on effective writing for publication.
O Through their websites, familiarize yourself with your target journals of interest, their requirements, and their most
popular articles.
O Prepare yourself for the fact that many drafts are often needed befo re an article is suitable for publication.
O Consider your target audience for a particular study.
O Craft a clear goal for writing this publication.
O Draft as much of your article as you can before completing the research.
O Review the requirements for your selected target journal.
O Revise your manuscript with all of your data and appropriate references.
O Revise your manuscript again with a view to enhancing clarity, brevity and impact.
O Share your drafts with peers, col/eagues, members of your research team, your mentor, and an editor if you can.
Revise on the basis of their feedback
O Prepare your submission package.
O Submit your manuscript.
O Revise and respond lo the peer reviews of your draft. Resubmit.
tn
c:
.-
+J
~
O
a.
Q)
o::: 4
~
~
~
4
268 © 2011 The Royal College oí Physicia 521"' S;..C;2C'5::=:3-a::c

30
Knowledge translation
Monika Kastner, PhD
Sharon Straus, MD, MSc, FRCPC
lan D. Graham, PhD
Jacqueline Tetroe, MA
ILLUSTRATIVE CASE
A Geriatric Medicine rcsident is struck by the fact that, despite ample evidence demonstrating that routine influenza
vaccination is beneficial and safe, many staff working in nursing homes do not receive an annual flu shot. She would like
to explore strategies to improve vaccination rates in this group of care providers. She wonders what factors she should
take into account in designing and conducting her study to increase its impact. When she mentions this to her project
supervisors, they suggest that she first acquaint herself with the principies of "knowledge translation" and think about
how they might be applied to her research study.
• Knowledge translation (KT) has KT,3 McKibbon and colleagues identified more than 90
been defined by the Canadian Institutes of Health Research terms that refer ro the use of research findings. 1his pletho-
as "a dynamic and irerative process that includes synrhesis, ra of terms is a source of confusion and poses a challenge for
dissemination, exchange and ethically-sound application rhose trying ro idenrily relevant literature in the field ofKT
ofknowledge ro improve the health ofCanadians, provide Another source of confusion concerns the domains
more effective health services and produces and strengthen encornpassed by KT. For example, sorne organizarions use
rhe health care systern."' In basic terrns, KT is che process /mowledge translation synonymously wirh commercialization
of moving knowledge into acrion or clinical praccice. 1he
concept ofKT arose from rhe realization that it is no longer
CHAPTER OBJECTIVES
enough ro crea te, disrill or simply disseminate knowledge
through tradicional suategies such as publishing in peer- After reading this chapter, you should be able to:
reviewed publications and presenring at conferences. Rath- • define and discuss "knowledge translation";
er, KT aims ro foster rhe applicarion of high-qualiry • discuss "integrated knowledge translation" and how it
knowledge in informed clinical decision-rnaking. might provide guidance regarding the membership of a
An abundance of terrns are currently being used ro research team;
describe the process of KT. 1hese include implementation • discuss the "knowledge-to-action cycle" and its
scienee and research utilization (Unired Kingdom and relevance to the design, conduct and reporting of a
Europe); dissemination, diffusion, research use, knowledge research study; and
transfer and uptalee (Unired Stares): and knowledge transjer, • recognize that research studies can be designed to
knowledge exehange and /mowledge translation (Canadal.! In assess the effectiveness of various knowledge translation
the course of their work ro develop a search srraregy for strategies.
---_._-_._-- --~----
! KEY TERMS
Clinical decision making Knowledge inquiry Knowledge-to-action cycle
Integrated knowledge translation Knowledge synthesis Policy-making
Knowledge creation Knowledge tools/products Stakeholders
--------

The Research Guide: A primer for residents, other health care trarnees. and practitioners
or technology transftr. Although rhese processes have many sirn- In our illustrative case, the residenr will need to ensun
ilarities, the firsr rwo do not usually address me application of mar any imervention she develops in her study is guided b~
knowledge [O decision-making. Similarly, the terrn continuing appropriate, rigorous and relevanr research evidence. Sh(
medica! education (CME) is sometimes mistakenly used inrer- will also need ro consider how the research tearn should be
changeably with knowledge translation. AÍthough CME or conhgured, the most appropriate srudy design ro use, and
continuingprofessional developrnent interventions (e.g., audit what contextua] faerors need ro be taken inro account. For
ano feedback, journal clubs) can be considered srraregies for example, she might begin by recruiting research ream
knowledge implernentation, their targer audience-healrh members who can help identify the barriers to influenza
care professionals-is much smaller than that for KT. In con- vaceination (e.g., decision-rnakers concerned wirh influen-
rrast, KT strategies vary according ro the targered users (e.g., za control, vaccine administrarion, and vaccine uptake).
researchers, clinicians, policy-makers, public, industry) and This srep may involve conducting informal or formal quali-
the rype ofknowledge being translared (e.g., clinical, biomedi- tative interviews or focus groups with physicians and long-
cal, health services or policy relared)." terrn care sraff ro identify specific barriers ro vaccination
uptake. She could then conducr a workflow analysis and
discussion wirh staff at rhe long-rerm care faciliry ro derer-
Why is knowledge translation
mine which stakeholders should be involved in the devel-
important'?
opment of her srudy, and which srudy design mighr be
In health care, a failure ro use research evidence ro inform most suirable.1he selection of srudy design will also depend
clinical pracrice is evident across all sertings and decision- on available resourees and on wherher a proposed interven-
making groups: health care providers, patients, informal tion is ready for a pilor evaluarion, sueh as using an inrer- I
earegivers, managers and policy-rnakers. Practice audirs rupted rime-series design (a rnerhod more appropriare for
performed in a variery of serrings have shown that high- interventions that need further refinernent and when
qualiry evidenee is not used consisrently in pracrice." For resourees are Iimired), a randomized conrrolled trial (RCT)
example, despite high-quality evidenee that statins are involving sraff within one long-term care faeility (if rhe
effecüve in redueing the risk of rnortality and morbidiry in intervention is ready for a more formal evaluarion but
parienrs wirh a hisrory of srroke, these drugs are markedly smaller in scope), or a cluster RCT in whieh rhe unir of I
underprescribed.v" Conversely, anribiotics are, againsr the analysis is at the level of applicable long-rerm care facilities
evidenee on beneíits and harrns, overpreseribed in children (if more resources and a larger scope of evaluarion are feasi-
with upper respirarory rract syrnptoms.f The importance of ble and appropriare).
KT is also evident in patienr participation in decision-rnak- This example also represents the cOl1eept of integrated
ing. A synthesis of 14 studies showed that many patienrs knowledge translation (IKT), whieh involves active col-
(26%-95%) were dissatisfied wirh rhe inforrnation rhey laborarion and exchange among researchers and knowledge
reeeived from their physicians, and thar they want ro be end-users rhroughout rhe research process-rhat is, from
more involved in treatment decisions.? identifying rhe research quesrion ro collecting and inrer-
I!"""""
A failure ro use evidenee ro inform decision-making is prering rhe data, ro disseminaring and applying rhe
also evidenr in policy-making at higher levels. Lavis and results.P:" IKT is supporred by evidencc abour the collab-
colleagues'? studied 8 provincial health poliey-making pro- orative research proeess, frorn which ir is possible ro infer
cesses and found rhat only 4 of these had been informed by several key facrors for success that can be applied ro research
citable health science research. Similarly, it has been found that addresses a health or healrh sysrem issue." 1hese fac-
that World Health Organizarion policy-rnakers infrequenr- tors are as follows:
Iy use evidenee from sysrematic reviews.!' Dobbins and col-
leaguesl2 found rhar, although publie health guidelines in 8 a process ro develop a shared perspective, cornrnon
Omario were developed using sysrernatic reviews, the rec- language and eommon understanding about the
ornrnendarions were not adopted at che poliey level. problem or issue that the tearn will address;
Increasing recognirion of rhese deíicits in rranslaring • a plan for eollaborarion that explicitly describes roles
knowledge inro actiori has led ro efforts ro change behav- and responsibiliries and a commirment ro regularly
iour, pracrices and poliey. assess effeeriveness;
270 © 2011 The Royal College oí Physioans and Surgeons of Canada

30 Know/edge trans/ation
• a plan for che inclusion of tearn members who are mended pracri e), individual healrh care professionals (e.g.,
collaborarive, and variarions in knowledge, arritudes and ski lis in cricically
• a scracegy for ensuring that rrusting relationships appraising and using evidence from clinicalliteracure) and
among tearn members are maintained and that parienrs (e.g. low adherence ro recommendacions)-as well
conllicrs are resolved appropriately when they arise. as of how these dererrninanrs influence and inreracr with
one anorher. 1hus, KT stracegies thar work in one setting
may not be imrnediarely cransferable (O anorher, especially
What are the determinants of
wirhout adaptatiori (O local íactors. For example, findings
knowledge use?
from che geriacric resident's srudy may show rhar delivering
Mulciple factors influence che way research findings are an educacional intervention aimed at increasing flu aware-
applied by decision-rnakers. 1(,-20 lndeed, Cabana and col- ness is effective in improving vaccination rares when it is
leagues have identified more rhan 250 barriers (O physician delivered using an inreractive video presentation in che
adherence (O clinical pracrice guidelines.161he challenges lunchroom ofher local long-term care (LTC) facility. How-
faced by clinical decision-makers include che overwhelrn- ever, the same intervention rnighr not be effeccive in LTC
ing arnounr of research evidence currendy produced, barri- facilities where che workRow, environrnent (e.g., space,
ers to accessing thar evidence, a lack of time to read, and che equiprnent), or scaff arritudes and beliefs are different.
need for specific skills in appraising, underscanding and These potenrial barriers (O adaptation need ro be raken inro
applying research evidence. Twenty years ago, a general account in che design of inrervenrions imended for diverse
internist, for exarnple, would have needed ro read 17 arti- conrexts,
cles a day to keep abreasr of the primary clinical lirerarure
relevanr ro the field.21 Today, the challenge of staying cur-
The knowledge-to-action frarnework
rene is even more dauncing, given thar more rhan 1000
arricles are indexed daily in MEDLINE. Lack of ski 11 in the The many theories and frameworks for KT rhat abound
25 29
appraisal of published literature has been a challenge for all can be confusing. - Graham and colleagues have devel-
srakeholder groups: until recendy, rnost educational curri- oped a conceptual framework, terrned rhe knowledge-to-
cula did not include this competency as a formal compo- action (KTA) cycle, 2 that builds on cornmonaliries
nene. 19,22 It has also been found that the content of evidence identiíied in a review of more rhan 30 planned-action rheo-
resources does nor adequarely rneet rhe needs of end users. ries. The KTA cycle is rneanr ro be irerative and dynamic ro
For exarnple, although rhere are criteria to enhance rhe reflecc the complexities of rhe KT process, and idencifies
reponing of sysrernatic reviews.P rhese focus mosdy on the knowledge creation and the action cycle (knowledge applica-
ualidity of evidence rarher rhan its applicability. Glasziou rion) as the rwo main componems of moving knowledge
and colleagues" found that only 3 out of 25 systernaric into praccice. Although the model is drawn as a cycle (see
reviews published during a 1-ycar period in Euidence-Based Fig. 30.1),'0 che phases can be used in any sequence accord-
Medicine coneained a description of the inrervention thar ing to the stage of a project thar is che rnosr relevant for
was adequare (O enable clinicians (O apply the information moving knowledge forward. Integral to rhe model is rhe
(O clinical decision-making and practice; rhis was true even involvement of knowledge end-users (e.g., clinicians,
of"simple" inrerventioris such as medications. parients, policy-rnakers) in che process so rhat the knowl-
Alrhough improving rhe managemene of knowledge is edge and its subsequent implernenrarion are relevanr to
necessary, chis is not enough to ensure that knowledge will their needs and is more likely to be applied by thern.
be effectively translated. Challenges exist at differem levels
of the healrh care system, and ir is importanr to undersrand Knowledge creation. In che centre of che cycle is knowl-
rhe context in which knowledge is being cargeced. This edge creation, which can involve three srages of refine-
includes consideracion of determinants of knowledge ment. 1he firsc scage is lmowledge irtquiry, which
uprake and behavioural change at che level of che healrh represents rhe accumulated body offirst-generacion knowl-
care system (e.g., financial disincenrives), che health care edge and consists largely of a broad base of primary srudies
organizacion (e.g., lack of equipment), healrh care teams and inforrnarion. \'V'hen designing and conducting primary
(e.g., misalignmene of local standards of care with recorn- srudies, researchers should ensure rhar they select the most

The Research Guide: A primer for residents, other health care t ainees, and practitioners
appropriare srudy design ro answer rheir research ques- 4. Assessing barriers ro using the knowledge.
tions and consider rhe rnost appropriare knowledge end- 5. Tailoring, implemenring and moniroring
users in rhe process. 1he second srage, k.nowledge interventions related ro KT and resulring knowledg
synthesis, involves identifying, appraising and synthesiz- use.
ing informarion pcrrinenr ro rhe research quesrion. Ir 6. Evaluaring ourcomes or effecrs of using rhe
arremprs ro idenrify pa[rerns wirhin [he exisring body of knowledge.
research findings on a topic (e.g., rhrough sysremaric 7. Determining srraregies for ensuring sustained use of
rcviews and mera-analyses). Knowledge synrheses are the knowledge.
ofren rhe base unir ofKT acriviries: rhe roraliry of rhe evi-
dence is considered, rather than the resulrs of individual 1hese steps can occur sequentially or simulraneously,
srudies. Ir is also important ro consider the qualiry of rhe and the knowledge creation srages can influence the
evidence. For example, if we are considering a disease acrion sreps at any point in the cycle. Integral ro rhe
managemenr issue, ideatly we need evidence from a sys- framework is rhe need ro consider the various stakehold-
ternatic review of good-qualiry randomized trials. How- ers (e.g., parients, clinicians, managers and policy-mak-
ever, informarion on adverse events may nor be futly ers) who are the end users of rhe knowledge being
caprured in these srudies, in which case findings from implemented. Knowledge crearion and its action sreps
observarional srudies mighr also need ro be considered. can be seen as rwo separate components undertaken by
1he third srage in rhe cycle involves funher synrhcsizing different srakeholders, or as a unified process involving
and refining rhe besr available knowledge into knowl- the same ser of participanrs.
edge tools/prc ducts for decision-making, such as clini-
cal practicc guiddines, decision aids or rules (for both
How can the knowledqe-to-action
clinicians and parients), algorirhms, synopses such as
cycle be applied?
rhose published by rhe ACP [ournal CLub, and rhe "Prac-
rice Tools" series published by Canadian Family Physi- To illustrare each phase of rhe knowledge-ro-action cycle,
cían. Each srage can be railored according ro rhe needs of we will show how a local group (including researchers,
users, idenrified research q uesrions, and disseminarion physicians and patient advocates) might address rhe
srrategies, which can rhen facilirare knowledge uprake, As applicarion ofknowledge concerning falls prevention and
knowledge is refined rhrough each srage in the knowledge osteoporosis assessment and follow-up.
crcation process, ir beco mes more clear, the evidence
srronger, and the resulring knowledge potenrially more Identify, review, select k.nowledge. A group of clinicians
useful ro end users. and researchers is aware that systernaric reviews have
shown that osteoporosis medicarions such as bisphospho-
111c action cycle. 1he acrion cycle is focused on applying nares can reduce the risk offractures." Alrhough evidence
rhe knowledge that has been identified and refined dur- on the prevention of fatls is more conrroversial.V rhe
ing [he knowledge creation srage. The componenrs of rhis group is interested in invesrigating falls prevention and
cycle are derived from rheories of planned acrion that osteoporosis managemenr.
focus on deliberarcly causing change wirhin health care
sysrems and groups.25,26 Specifically, rhey involve map- Identify the problem. The group conducts a local audir
ping out a research plan for disseminaring knowledge and finds thar less than 40% of patients aged 65 years and
rhrough rhe following sreps: older who were admitted ro hospiral afrer a fall or wirh a
fracture were subsequently assessed for osreoporosis or
l. Idenrifying rhe problem. risk Ior falls.:n 'D1Cy fcel rhar somerhing should be done
2. Identilying, reviewing and selecting the knowledge about rhis gap between knowledge and pracrice.
ro implemenr (or identifying, reviewing and
selecring the knowledge ro irnplernent and then Adapt knowledge to the local context, 1he group recog-
idenrifYing wherher a problem exists). nizes that, ro be applied effecrively, evidence in an:
3. Adapring or cusromizing rhe knowledge ro the local form-e.g., knowledge syntheses, pacienr decisio aids,
conrexr. and clinical practice guidelines-needs ro be adaored ro
272 © 2011 The Royal College oí Physicians a-d S:.Jg_ es : r~z=¿

30 Knowledge translation
the local contexto For exarnple, rhe ADAPTE rool" offers a Assess barriers and facilitators to using the knowledge.
three-phase process whereby elinical, healrh service and The group also understands the imporrance of assessing
adrninistrarive decision-makers can adapt elinical pracrices porential barriers ro, and facilitatOrs of, rhe implernentarion
guidelines for local use." The goal of this tool is tO engage of rheir adapted tools. The Clinical Pracrice Guidelines
local srakeholders and preserve rhe integriry of rhe evidence Framework for Improvernent, a conceptual model for inves-
thar forms the basis for recommendarions. Methods such tigating barriers ro knowledge use in health care, initially
as ADAPTE help users idenrify porential local barriers, idemified 293 potencial barriers to physician guideline
avoid deviarions from the evidence base, and align evidence adherence." rhese were then extended to inelude íacilirarors
to local contexts, of knowledge use in elinical pracrice (i.e., factors rhat pro-
A useful resource in the conrext of oncology is the Can- more shared decision-making in elinical practice).!"
cer Guidclinc Adapration and Implernenrarion Projecr Among rhe barriers identihed by che group are (1) rhe
(CAN-IMPLEMENT) tool, which consists of a guide, a fragmentarion of healrh records that could be used to identi-
library science supplernenr, and a roolkit." 1his rool was fy patients ar risk, and (2) the distribution of rhe rarger popu-
developed as a practica] guide to adapting exisring guideline lation over a large region. Orher barriers are identihed ar rhe
recornmendations to local contexts, chus Iacilitating their level of rhe parient (e.g., lack of undersranding rhat osreopo-
implernenration. rosis and fractures are linked and that osteoporosis can be
In view of their own local conrexr, where many patienrs prevemed) and rhe provider (e.g., lack of time, and lack of
do not have a primary care physician or mighr not tend to access to bone mineral densiry resring and inrerpreration).
discuss falls and fracture risk with their doctor, the group
decides to creare rools ro help patienrs implernent reCOIl1- Select, tailor and implement interventions. Once factors
rnendarions for weighr-bearing exercise and intake of calci- rhat can impede or facilitare knowledge use have been iden-
um and vitarnin D. tified, KT intervenrions need to be selecred, railored to spe-
cific barriers for change and implemenred. 1he selection of
Figure 30.1
intervenrions is cornplex, since currenr evidence alone can-
The knowledge-to-action cycle.2.3°
Reproduced by kind permission of The Journal of Continuing not be used to guide irnplernenrers on rhe besr choice of
Education in the Health Professions intervention, Research to date has focused mainly on edu-
Monitor cacional programs, feedback and reminders,
knowledge
use
and has shown no more than a 10% absolure
irnprovernenr in selected outcomes (although
Select, tailor,
implement
- ---+-- rhis change can be elinically or economically
interventions , relevant). Active educational inrervenrions
Evaluate
/
KNOWlEDGE CREATION ,
/
outcomes (e.g., qualiry cireles for professionals), active
----~\
t
/
1\------
, ,
self-study rnarerials or websites (e.g., for dis-
," \\ Knowledge
,~
'.,
, 00,
I \
t t
\
inquiry ,s tan ce learning) are more likely ro induce
,,? change rhan passive educarional inrerventions,
t Synthesis
,
I.~
tb
I.c::-o,
patient-directed intervenrions (e.g., decision
t aids) can support qualiry improvement in some

Il.....
\
,
\
, Sustain
knowledge cases; and interventions rhat bring evidence ro
use
Adapt rhe poinr of care (e.g., reminders and dccision
knowledge to
local context support) are likely ro be effecrive in the arcas of
prevenrion and test ordering.-16.37
In our exarnple, the group developcd a
multi-cornponenr, nurse-led srraregy rhat
~ ~~n~f~;~o~l~ml
1
Identify, review,
select knowledge
J incorporared parienr educarion,
rnent, review of medications,
self- manage-
and assessment of
ACTION CYClE
homes for fall-relared risks.
(Application)
aSee www.g-i-n.netlactivitiesladaptation
© 2011 Tile Royal College 01 Physicians and Surgeons 01 Canada 273

The Researeh Guide: A primer for residents, other health care trainees. and praetitioners
Monitor knowledge use. After a KT intervention has knowledge use-rhar is, rhe degree ro which an innovario
been implemented, knowledge use should be monitored ro cominues w be used after inicial efforts ro secure irs adop
determine how and ro whar exrenr it has been disserninar- (ion have succeeded.Pé? Scant attenrion has been given ti
ed. This wil! give an indication of wherher adjusrrnenrs ro susrainabiliry, perhaps because many c1inical researcher
the KT plan are needed. How knowlcdgc upiake is mea- consider sysrem-Ievel change processes w be ourside rhei
sured depends on the user's perspecrive, and on how rhe realm of research. Another barrier is presenred by the shorr
knowledge is defined and applied. There are rwo main rerrn nature of funding opporrunities: follow-up rneasure-
types ofknowledge use, namely instrumental (the concrete rnents are rypical!y feasible for only 1 or 2 years, al!owing
application of knowledge and changes in behaviour or only shorter-terrn change processes ro be addressed.
practice) and conceptual (changes in understanding or atti- Al rhough susrainabili ry is rypically nor considered unril
tude that may inform decision-making bur do not change implernenration is under way, we recornmend rhat plans ro
practice). In our exarnple, rhe use of osteoporosis medica- ensure susrainabiliry be made as eady as possible in rhe
tions would be an example of instrumental or concrete KTA cycle, when rhe interventions for knowledge use are
knowledge use. being selecred and railored.
Evaluate outcomes or impacto 111e next step is ro deter- Conclusion

mine whether the knowledge use affects parient, provider Knowledge translarion suaregies and inrervenrions can
and sysrem ourcornes. This is a point in the cycle when close gap s berwcen evidence and practice in many ser-
studies can be designed and conducted [Q evaluare che rings and disciplines. Resources muse be focused on rhe
results of knowledge translarion. For exarnplc, Tu and col- use of effective KT straregies, as well as on srudying the
leagues conducted an evaluation that clearly docurnented effectiveness of untested KT inrerventioris. KT interven-
che effect of the BOPE study on ramipril prescribing." ticns must address all aspects of care, including access to
The group conducts a randomizec! trial LO determine and use of valid evidence, patient safery srraregies, and
wherher rheir KT srrategy is effective. The indicarors of organizational and sysrerns issues, However, we must
knowledgc use include appropriate osreoporosis manage- also be cautious of rhe assumption that all knowledge
rnent according ro guidelines (e.g., use of mcdicationsr." should be translared, for this depends on whether a
The trial also considers quality of life, parient satisfaction mature and valid base of evidence for the knowledge in
and fractures. Another outcorne is the strength of the col- quesrion exisrs. Moreover, healrh care systerns have lirni-
laborarion developed by the group, which grew ro indude rations and cannot accomplish cvery goal, no rnatrer how
represenratives from rhe provincial government, pharrna- desirable: researchers must work with srakeholders, ,
ceurical companies and insurance companies. induding patients, the public, clinicians and policy-
makers, ro establish an explicir process for prioritizing
Sustain knowledge use. An irnportant bur ofren over- activities related ro knowleclge translation. •
-_" looked phase of rhe KTA cycle is susrainabiliry of the
CASE POSTSCRIPT
To enhanee all aspects of her proposed study, espeeially its potential impaet on the influenza vaccination rates of staff in
long-term eare, the geriatries resident first eonducts a systematie review to identify which strategies have the potential to
improve vaecination rates. This represents the "knowledge creation" eomponent of the KTA framework. She then uses an
IKT (integrated knowledge translation) approaeh by working with her supervisor to recruit relevant knowledge end-users to
become part of the research team, and to determine how the findings of the systematic review could inform the development
of an intervention applicable in the eontext of LTC facilities. The inclusion of a long-term care nurse, a nursing home manager
and the local medical officer of health on the research team improved the study's question, design and eonduct, greatly
enhanees the practical applicability and reievanee of its results in her local eontext, and the potential of her interve ion o be
í
adapted to other settings.
274 © 2011 The Royal Colleqe oí Physicia s a d S~:g_"0"3 e" r¿-¿,:-¿

30 Know/edge trans/ation
REFERENCES
1. Canadian Institutes of Health Research (CIHR) definition of KT Available frorn: www.cihr-irsc.gc.ca/e/29418.html
2. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Los in knowledge translation: time for a rnap? J Contin
Educ Health Proi. 2006;26(1) 13-24.
3. McKibbon KA, Lokker e. Wilczynski NL, Ciliska O, Oobbins M, Oavis DA, et al. A cross-sectional study of the number
and frequency of terms used to refer to knowledge translation in a body of health literature in 2006: a Tower of Babel?
Implementation Science. 2010; 5 16.
4. Davis O, Evans M, Jadada A, et al. The case for knowledge translation: shortening the journey from evidence to effect BfvU.
2003;327 33-5.
5. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the
United States. N Engl J Med. 2003;348(2):2635-45
6. Majumdar SR, McAlister FA, Furberg CO. From knowledge to practice in chronic cardiovascular disease: a long and winding road.
J Am Coll Cardiol. 2004;43(10) 1738-42.
7. LaRosa Je. He J, Vupputuri S. Effect of statins on risk of coronary disease a meta-analysis of randomized controlled trials. JAMA.
1999;282(24) 2340-6
8. Arnold SR, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Oatabase Syst Rev.
2005 Oct 19;(4) CD003539
9. Kiesler DJ, Auerbach SM. Optimal matches of patient preferences for information, decision-making and interpersonal behavior:
evidence, models and interventions. Patient Educ Couns. 2006;61 (3):319-41.
10 Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in
public policymaking. Milbank Q 2002;80(1) 125-54
11. Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369(9576): 1883-9.
12. Oobbins M, Thomas H, O'Brien MA, Ouggan M. Use of systematic reviews in the development of new provincial public health
policies in Ontario. Int J Technol Assess Hea/th Care. 2004;20(4):399-404.
13. Graham ID, Tetroe J. How to translate health research knowledge into effective healthcare action. Healthc Q. 2007;10(3):20-2.
14. Denis JL, Lomas J. Convergent evolution: the academic and policy roots of collaborative research. J Health Serv Res Palie y. 2003;8
Suppl2 1-6.
15. Straus S, Tetroe J, Graham ID, editors. Knowledge translation in hea!th eare: moving from evidence to praetiee. Oxford Blackwell
Publishing; 2009. Chapter 5.1. Sustaining knowledge use. p. 240.
16. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don't physicians follow clinical practice guidelines7
A framework for improvement. JAMA. 1999;282(15): 1458-65.
17. Gravel K, Légaré F, Graham ID. Barriers and facilitators to implementing shared decision-making in clinical practice: a systematic
review of health professionals' perceptions. Implement Sci. 2006; 1: 16.
18. Légaré F, O'Connor AM, Graham ID, Saucier D, Coté L, Blais J, et al. Primary health care professionals' views on barriers and
facilitators to the implementation of the Ottawa Decision Support Framework in practice. Patient Educ Couns. 2006;63(3):380-90.
19. Milller M, Estabrooks CA, Myrick F Research utilisation and clinical nurse educators a systematic review. J Eval Clin Pract.
2006; 12(6) 639-55
20. Grimshaw JM, Eccles MP, Walker AE, Thomas RE. Changing physicians' behaviour what works and thoughts on getting more
things to work. J Contin Educ Health Prof 2002;22(4) 237-43.
21 Haynes RB. Where's the meat in clinical journals? [editorial] ACP J Club. 1993; 119 A22-3.
22. Lavis JN. Research, public policymaking, and knowledge-translation processes Canadian efforts to build bridges. J Contin Educ
Health Prof 2006;26(1) 37-45
23. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses the PRISMA
statement. BMI 2009;339 b2535.
24. Glasziou P, Meats E, Heneghan C. Shepperd S. Wllat is missing from descriptions of treatment in trials and reviews? BMI
2008;336(7659) 1472-4.
© 2011 The Royal College o; Physicians and 5urgeons 01 Canada 275

The Research Guide: A primer for residents, other healt (are rainees, and practitioners
25. Graham ID, Harrison MB, Logan J and the KT Theories Research Group. A review of planned change (knowledge translation
models, frameworks and theories. Presented at the Joanna Briggs Institute International Convention, Adelaide, Australia, No
28-30, 2005.
26. Graham ID, Tetroe J; KT Theories Research Group. Some theoretical underpinnings of knowledge translation. Acad Emerg M,
2007; 14(11):936-41.
27. Estabrooks CA, Thompson DS, Lovely JJ, Hofmeyer A. A guide to knowledge translation theory. J Contin Educ Hea/th Prof.
2006;26(1 ):25-36.
28. McDonald KM, Graham ID, Grimshaw J. Toward a theoretic basis for quality improvement interventions. In: Shojania KG,
McDonald KM, Wachter RM, Owens DK, editors. Closing the qua/ity gap: a critica/ ana/ysis of qua/ity improvement strategies.
Vol. 1: Series overview and methodology. Technical Review 9.1; 2004 AHRQ Report No. 04-0051-1. Rockville (MD): Agency for
Healthcare Research and Quality; 2004. Available from: www.ncbi.nlm.nih.govlbookshelf/br.fcgi?book=hstechrev&part=A265C
29. Wensing M, Bosch M, Foy R, van der Weijden, Eccles M, Grol R. Factors in theories on behaviour change to guide
imp/ementation and qua/ity improvement in hea/thcare. Nijmegen: Centre for Quality of Care Research; 2005.
30. Straus SE,Tetroe J, Graham 1. Defining knowledge translation. CMAJ. 2009; 181(3-4): 165-8.
31. Wells GA, Cranney A, Peterson J, Boucher M, Shea B, Robinson V, et al. Alendronate for the primary and secondary prevention
of osteoporotic fractures in postmenopausal women. Cochrane Oatabase 5yst Rev. 2008;(1):CD001155.
32. Gates S, Fisher JO, Cooke MW, Carter YH, Lamb SE. Multifactorial assessment and targeted intervention for preventing
falls and injuries among older people in community and emergency settings: systematic review and meta-analysis. BMJ.
2008;336(7636): 130-3.
33. Ciaschini PM, Straus SE,Dolovich LR, Goeree RA, Leung KM, Woods CR, et al. Community-based randomised controlled trial
evaluating falls and osteoporosis risk management strategies. Tria/s. 2008;9:62.
34. Fervers B, Burgers JS, Haugh Me, Latreille J, Mlika-Cabanne N, Paquet L, et al. Adaptation of clinical guidelines: literature review
and proposition for a framework and procedure. Int J Qua/ Hea/th Careo 2006;18(3):167-76.
35. Harrison MB, van den Hoek J; Canadian Guideline Adaptation Study Group. CAIV-IMPLEMENT guide/ine adaptation and
imp/ementation p/anning resource. Available from: www.cancerview.ca/idc/groupslpublic/documents/webcontentlcanimp_
toolkit.pdf
36. Grimshaw JM, Thomas RE,MacLcnnan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline
dissemination and implementation strategies. Hea/th Techno/ Assess. 2004;8(6):iii-iv, 1-72.
37. Straus SE,Tetroe, J, Graham ID, editors. Know/edge trans/ation in hea/th care: moving from evidence to practice. Oxford:
Blackwell Publishing; 2009. Chapter 3.5. Selecting KT interventions. p. 94-150.
38. Tu K, Mamdani MM, Jacka RM, Forde NJ, Rothwell DM, Tu JV.The striking effect of the Heart Outcomes Prevention Evaluation
(HOPE) on ramipril prescribing in Ontario. CMAJ. 2003;168(5):553-7.
39. Rogers EM. Oiffusion of innovations. 5th ed. New York: Free Press;2005. p. 429.
40. Straus Se,Tetroe J, Graham ID, editors. Know/edge trans/ation in hea/th care: moving from evidence to practice. Oxford:
Blackwell Publishing; 2009. Chapter 3.7. Sustaining knowledge use. p. 165-73.
SUMMARY CHECKLlST
o In the context of your study, describe why knowledge translation is relevant and important.
O List the stakeholders of your study.
O Using the knowledge-to-action cycle, develop a plan to enhance the adoption of any results of your wor
276 © 2011 The Royal College oí Physicians and _~ - ~a::G

31
Responsibility and integrity:
Disseminating research findings to the public
Robert L. Reid, MD, FRese
ILLUSTRATIVE CASE
A final-year resident in Internal Medicine has collaborated on a research project in which umbilical cord stem cells have
been induced to differentiate into what appear to be primitive renal glomeruli. To generate enthusiasm for his research
presentation at a national meeting, he titles it "Kidneys from a dish: advances in stem cell research." After the presentation
he is approached for a story by a local science reportero He is surprised and embarrassed the next day when his supervisor
calls him to ask what he was thinking when he spoke to the reponer. His presentation and interview resulted in headlines
across the country, and he was quoted as saying, "This breakthrough has opened the door for us to grow artificial kidneys
from stem cells." Apparently the phone at the hospital has been ringing constantly with calls from anxious parents of
children on dialysis wanting to know when their child can get a kidney. What went wrong?
11 Knowledge translation has been For their pan, journalists frequendy express concerns
defined by the Canadian Institutes of Healrh Researcb as "a abour health researchers and how they respond to legiti-
dynamic and iterative process that ineludes synthesis, dis- mate enquires abour healrh and scientific matrers (Textbox
sernination, exchange and erhically-sound applicarion of 31.2). Given rhese perceptions and misgivings on borh
knowledge ro improve the health of Canadians, provide sides, ir is not surprising rhat a mutual misrrusr has devel-
more effecrive healrh services and producrs and srrengrhen oped berween these rwo professional cultures.
rhe healrh care sysrem." I
CHAPTER OBJECTIVES
Successful dinician-researcbers are very busy people,
many of wbom feel thar rheir dury ro disseminate research After reading this chapter, you should be able to:
n.ndings ends wirh publishing their studies in peer-reviewed • describe the role of the media in the dissemination of
healrh science journals. Some tend ro avoid the lay media research
enrirely. 1here can be many reasons ro be media-shy (Texr- • describe sources of misunderstanding between
box 31.1), not rhe least of which is rhe difficulry of explain- researchers and the media
ing complex sciemific principies or new findings ro a • list occasions when it is appropriate for researchers to
reponer with little scienrific rraining. And rhen there are interact with the media
rhe horror srories of anides or broadcasrs in which rhe • cite common errors that have an adverse impact on
researcher's cornments are misquored or taken out of con- researchers' interactions with the media
rexr, or his or her work is ridiculed by an outspoken antago- • describe key strategies to deliver arr effective health
nisr in rhe name of"balance" or "debare."2-'1 science message
KEY TERMS
Absolute and relative risks Convenience sample Media interview
Bias Fraud Peer review
Competing interests Knowledge translation Pitfalls of reporting
Context Lay media Redundant publication
L _ ~ .~ __
. --~--------------- -- -------- ----- -
The Research Guide: A primer for resiclents, other heal h care trainees, and practitioners
TEXTBOX 31.1 HEALTH RESEARCHERS' NEGATIVE PERCEPTIONS OF JOURNALlSTS AND THE MEDIA
Journalists The media

•• Often lack health and scientific knowledge • Are profit-driven and highly competitive
e Are ill informed about critical appraisal and the principies • Prefer sensational articles to true ones ("If it bleeds it
of evidence-based medicine leads.")
• Prefer to report relative versus absolute risks and benefits • Concentrate on bad news and weird events
• Often fail to distinguish between association and • Have a bias toward controversy and conflict
causation • Believe "balance" is achieved when two opposing views
• May fail to put risks into context are presented without consideration of the validity of
• Report" latest news" from scientific meetings without those views
appreciating that these data have had only preliminary • Have a short time frame for comment
analysis and peer review • Distill complex issues into simple "Yes" or "No" answers
for the public
• Allow unadjudicated "Letters to the editor" by lay
public that frequently castigate health professionals and
misinform the public
Common errors in reporting scierrtif'ic

TEXT8QX 31.2 JOURNALlSTS' NEG.ClTIVE findings
PERC:¡;:PTtONS OF HEALTH RESEARCHcRS
This secrion will examine common errors rhat can lead ro
Heéil"th researcners misleading conclusions or interpretations being reponed
• vvork in a highly competitive "publish or perish" culture; ro the public. These errors can arise frorn rhe original
some go so far as 'lo fake their data research repon, its translation to rhe media rhrough news
e Sometimes try to use the media for self-promotion releases, an inaccurare representation of the work by the
~ Are often elitist and unapproachable (won't return calls) researcher in interviews, or a lack of understanding on rhe
•• Are poorly trained to be knowledge translators (don't part of the journalisr of rescarch practices and their limita-
speak in plain language) tions or the meaning of sratistical analyses. In their com-
• Give cornplex answers to simple questions munications with rhe media, researchers and rheir
•• Cannot agree: different experts often give conflicting communications staíf can be alert to these pitfalls of
answers and explanations reporting and present findings in a c1ear way that will help
• May have financial conflicts of interest (e.q., to averr misunderstanding.
.- e
pharmaceutical
Lack understanding
sponsorship or industry ties)
of the constraints of journalism (e.q..
TEXTBOX 31.3 REASONS FOR HEALTH EXPERTS
timelines and space limitations)
AND RESEARCHERS TO WORK WITH THE MEDIA
• To address important public health issues or respond to

As healrh researchers, we need ro improve the way we public concerns about health issues
rranslate knowledge arising from our research findings. • To respond to inaccurate or incomplete media coverage
Doing so involves developing skills in working effecrively of important issues
with ihe media. Thc sooner we rccognize thar the media rep- • To provide information during a health crisis (medica: on
resenr a critica! bridge to rhe public, and ro the policy-makers shortage, SARS, H1N 1, etc.)
who support our research, rhe betrer off we will be. Although • To discuss new research findings after here has G2-c;' a~
ir is irnporranr ro be aware of me porential pitfalls of comrnu- opportunity for peer review through formal prese w-~-
nicaring research findings through the lay media, ir is also and publication
irnportanr ro appreciate me potential beneíirs (Textbox 31.3).
: ,..-----
278 © 2011 The Royal Colieqe oí Physicians ano S'-J'geo-,s '-.:::.c...::
37 Responsibility and integrity
• To draw the attention of funding agencies and policy-

Example: A newspaper reponed rhat a medication reduces
rhe risk of hearr artack by 50%. Your patient, a 40-year-old
makers to a promising line of research in order to
man wirh a farnily hisrory ofheart disease, asks to be started
facilitate grant support
on rhe medicarion. You read the research paper on which
• To raise public awareness about a research centre and its
programs, attracting potential research participants and
the news irern was based and discover rhat, given his age
and his few personal risk factors for heart disease, your
those wishing to make donations to support the centre's
patienr would need ro take rhe medicine for 10 years to
research
achieve a reducrion in myocardial infarction risk from
• To respond to a request for expert opinion about a new
111000 ro 0.5/1000. (You advise instead that he increase
research finding
his physical activiryl)
Example: The Resulrs section of the first repon from the

Extrapolating results from convenience samples to Women's Healrh Initiative announced thar women wbo
the general population used combined estrogen/progestin bormone therapy had a
26% increase in the risk for brease cancel'. '; Not surprisingly,
Researchers ofren choose a readily accessible research targer
many media outlets used rhis worrisome statistic in subse-
population without considering, or making it dear, thar
que m repons. In fact, rhe published data actually showed
chis convenience sample does not necessarily represem rhe
that no increase in brease cancer risk had been observed in
true population at large and thar rhe results might not,
che 75% of women who had never used horrnone therapy
therelore, be wide!y applicable.
befo re study entry, and that among those women who had
received horrnone therapy in rhe past, brease cancer was
Example: A health researcher decides to study the rates of
detected in 30/10 000 of those who received placebo versus
Chlamydia infection in rhe community by examining the
available test results of an infertility clinic. Of 100 swabs, 8
38/10000 who received hormone rherapy, 1he "absolute"
increase was rhus 8/10 000 users per year, giving an attrib-
are positivefor Chlamydia. The researcher reports rhat
utable risk of 0.08%-a leve! classified as "rare" by rhe
Chlamydia is on the rise in the local community because a
ó
public healrh survey hve years earlier found only a 2% posi- World Health Organization.
tive rate in women arrending for annual Pap smears. Clear-
Failing to put risks and benefits in context.
ly, the convenience oE using results that were readily
available needed to be balanced with caurion about extrap-
"By underestimating common risks iohile exaggerating exotic
olation to a differem popularion: no condusions about a
ones toe end up protecting ourselues against the unlileely perils
rise or fal! in the prevalence of Chlamydia would be war-
u/hile fa ili ng to take precautions against those most likely to do
ranred. 1his error is repeated in media repons on the us in."
-Larry Laudan, The Book ofRis/e/
research.
A number of factors determine bow che general public per-

Reporting risks and benefits reported as relative ceives and assesses risk. The perception of risk varies accord-
risk rather than as absolute or attributable risks ing to whether the risk is natural or arcificial (exposure to
sun is perceived as Jess worrisome than exposure ro asbes-
A common marketing tool is to repon a benefit as a per-
ros), volunrary or involuntary (smoking is seen as less wor-
centage (or relative) change because this ofren magnifies
risome than exposure to pesricides or nuclear power
rhe apparem effect. Journal arricles ofren repon absolute
plants), offers significant benefirs (bearr surgery has risks,
and relative risks, and aurhors and edirors need to take
bur rhe potencial benchr is rhoughr to outweigb rhe risk for
care that rhese rwo categories are disti nct and clear. Simi-
most parients), or has affected a personal acquaintance
larly, media repons frequently fail ro make rhis distinction,
(having a relative with osreoporosis or Alzheimer's disease
reporting relarive risks and benefits as if they were absolute
engenders increased worry about rhese conditions). Finally,
or artributable, thus presenring an exaggerated and even
and probably most imponantly, risk is perceived relarive ro
misleading picrure of the research results. S tl
other risks of daily living-that is, in context. -
279
. .
- --- -- --- - - -
The Research Guide: A primer for residents, other health care rainees. and practitioners
Example: The risk of breast cancer in postrnenopausal wherher Canadians should receive the seasonal intiuenzs
women using combined hormone therapy for 5 years is vaccine. The results of the implicared studies were
comparable ro rhe risk resulting from early menarche, ultimarely published in April 2010,!5 afrer che HINI
delayed menopause, postmenopausal obesiry, failure ro pandemic had subsided.
exercise, and daily ingestion of alcohol." Ir is less rhan the
risk thar resulrs from delayed nrsr pregnancy and failure ro Advocating the use of certain medications,
breasrfeed. When rhe risks are considered in this conrext, procedures or products without acknowledging
many women are less fearful than rhey mighr otherwise be vested interests
of using short-rerrn hormone rherapy for the relief of dis- The issue of ves red interesrs arises in various contexrs of
tressing vasomotor symptoms. knowledge rranslarion. Medical journals require that nnan-
cial and other competing interests be declared, and con-
Failing to properly examine correlations to remporary guidelines for continuing medical educarion
determine whether a finding represents
require presenrers to disclose their indusrry ties. Ir is hardly
association, or causation.
surprising, however, that medical experts are called on ro
This is perhaps one of rhe rnost common rnistakes in media act as advisors or consulrants for pharmaceurical or device
reporting. In evaluating causation, the Bradford-Hill crite- manufacruring companics. Indeed, such collaborations are
ria 1) should be used. These stipulare that the case for a caus- needed. In Canada, a lack of collaborarion berween aca-
al associarion is strengthened by a dernonsrration of demic researchers and indusrry, particularly in phase 1 tri-
consisrency, rhe strengrh of rhe association, speciíiciry, a als, has led ro the loss of opportunities for researchers and
temporal relationship, biological plausibility, coherence, a their insriturions.!" The acknowledgement of a competing
dese-response relationship, and experimentarion, interest should nor be seen as an indication that bias or
undue influence necessarily exists; however, to ajen the
Example: The rising incidence of autism has occurred over media audience ro the potential for bias, competing inrer-
the sarne time frame as the increasing use of vaccines con- ests-borh for rhe researchers whose findings are reponed,
taining the preservative thimerosal. The resulting specula- and for any experts asked to comment on that research-
tion that the rwo mighr be causally linked has led ro fear should be stated transparently. Further, the potential
and rhe under-urilizaiion of vaccines, resulring in recent effects of industry conílict of in terest have been document-
ourbreaks of measles and mumps. ed in at least rwo srudies, one nnding thar 98% of papers
based on industry-sponsored studies reAecred favourably
Reporting (non peer-reviewed) preliminary on the indusrry's produces" and anorher concluding rhar
findings without subsequent confirmation
industry-íunded studies were 8 times less likely ro reach
journalisrs, eager to give their readership
tific news, describe new breakthroughs
rhe larest scien-
described in rhe
conclusions unfavourable ro rheir drugs rhan were inde-
,
._ program of a healrh scicnce meeting. Unfonunarely,
pendently funded srudies." This issue was highligbted
recenrly by the conrroversy caused by revelations rhat the t
rhese nndings have not undergone the scruriny of peer increased cardiovascular risks caused by rhe use of selecrive •
review and publicarion in a scientiíic journal, the meth- cyclo-oxygenase-2 (COX-2) inhibitors were deliberarely 4
ods may not be matute and reproducible, and rhe sug- concealed by the rnanufacrurer." ~
gested results may never pan out. Untortunately, this
misleads rhe general public, raising false hope abour new Portraying a visionary interpretation of ~
tests and treatrnents. . preliminary findings a state-of-the-art
The health sciences need visionaries: individuals who are
•
~
Example: In Septcmber 2009, in rhe rnidst of the HINl forward-looking and can picture exciring new appl icarions
inAuenza pandernic, rhe media reponed on preliminary for emerging technologies. 1his is the basis for research and •
~
research suggesting that rhe seasonal Au shot may pur clinical experimentation. However, when such individuals
people at greater risk for getting pandemic Hu." Alrhough ralk ro reporters they must clearly identify when their ideas •
they had flor been validared ar [he rime, these reports are conjectural and when they are backed by sound scien- ~
resulted in increased uncenainty and anxiery about ritic evidence.

37 Responsibi/ity and integrity
Example: See the illusrrarive case at the beginning of rhis

TEXTBOX 31,4 EXAMPLES OF FALSIFICATION OF
chaprer!
RESEARCH FINDINGS
Publishing duplicate or fraudulant data 1. Jon Sudbe and 13 coauthors from Denmark reported in
There are numerous examples of siruations in which a sci- The Laneet that nonsteroidal anti-inflammatory drugs
entist has published a case series involving a new trearrnenr reduced the risk of oral caneers. This "findinq" was based
and has had a second publicarion at a later dar e in which entirely on fabricated data on the lives and lifestyles of 900
rhe original cases were reponed again. Unless this is explic- people." A retraction was published by the journal's editor
idy sta red in rhe merhods, ir gives the impression thar the in 2006 when the data were discovered to be false.
merhod is more mature and valid rhan is justified by rhe Penalty: Sudbe's authorization as physician and dentist
clinical experience. There are also a surprising nurnber of was revoked by the Norwegian Board of Health, his PhD
cases of research repores wherein data were íabricared or fal- was revoked by the University of 0510, and his papers were
sitied. Media accounts of the original repores garnered retracted from journals, including The Laneet.
worldwide atrenrion because of the srriking claims made by
the investigarors. Although subsequent revelations that rhe 2. Malcolm Pearce (then associate editor of the British
repores were Iraudulent have ser the record srraight, such Journal oi Obste tries and Gynaecolagy [BJOGJ) authored
cases underrnine rhe credibility of scientists in general and twa papers in ane issue of the BJOG25.26 In one article the
heighren public skepticism about rhe role and value of claim vvas made that an ectopic pregnancy was moved
research. from the fallapian tube into the uterus, resulting in survival
In 1997 an organizarion called COPE (Cornrnirree on of the offspring. The other was a randamized clinical trial
Publication Ethics) was founded by medical editors in rhe involving a new treatment to prevent recurrent miscarriage
United Kingdom in response ro growing anxiery about the in women with polycystic ovary syndrome. Both reports
integrity of research submissions. The members-includ- were later proven to have been falsified.
ing rhose of rhe British Medica! [ournal, Cut and The Lan-
Penalty: Pearce vvas found guilty of serious professional
cet-advise on cases of suspecred redundant publication
misconduct and struck from the medical register. Geoffrey
or fraud brought ro thern by other edirors. Of 212 issues
Chamberlain (then editor of the BJOG and President of the
discussed by COPE berween 1997 and 2004,58 involved
Royal College of Obstetrics and Gynaecology in the United
undeclared duplicare or redundant publicarion, 26 involved
Kingdom, and whose name appeared on one of the
authorship issues, 25 involved a lack of ethics approval, 22
papers (a "qift" authorship), resigned as editor of BJOG
involved rhe absence or inadequacy of inforrned consent,
and president of the College. The medical school identified
and 19 involved fabricarion or Ealsincarion.20,21
other publications by Pearce to be dubious, and a total of
The implicarions of research fraud are enorrnous.iv" les
six publications were retracted, including two in BMI
vicrims include other researchers, who ma)' base current or
Euture research on hndings rhat are impossible ro replicare,
funding agencies that invest valuable resources on Iraudu- 3. Woo-Suk Hwang was acclaimed as a national hero in
lent research, pharmaceutical companies thar may invest his native South Korea after he published a report in the
millions into developing a promising medicarion when journal Scienee describing the first claning of human
adverse side effecrs are hidden, and, most imporrandy, the embryonic stem cells.? Other researchers could not
public who may receive inrcrventions wirh lirtle or no ben- replicate his work, and the findings vvere subsequently
efir and undisclosed risks. shown to have been falsified.
Penalty: Hwang was dismissed by Seoul National University
Examples: See Texrbox 31.4. and expelled from the Korean Society for Molecular and
Cellular Biology; his licence on embryonic stem cell research
was revoked by the South Korean government, he was
convicted of fraud and embezzlement, and his papers were
retracted from Scienee.

¡he Research Guide: A primer for residents, other health care trsinees, and practitioners
Giving an effective media interview

ro expand faciliries ro allow better local care for elderly peo-
If your professional role exposes you to the media, rhe rnosr
pie in your community. At last night's hospital board rneer- I
important preparation you can make is ro take a media
ing a community member pLlt forward a rnorion to spend
training course. A one-day training program for Canadian
che money on a much-needed facelífr for rhe hospital,
residenrs in Obste[rics and Gynecology was shown ro sig-
which would delay rhe expansion of the geriatrics deparr.
nificantly improve residenrs' ability ro deliver medical mes-
sages effectively. 28 rnenr for three years. A reporter calls to get your viewpoin t.
You prepare your bottom-line message before calling the
When asked for a media interview, remember ro respecr
reporter: "While there is no doubt rhat a cosmetic facelife of
the reporter's [imelines. You do have [he Opportunity ro
rhe whole facility is needed, our resources are limited and
control the interview, bur only ifyou come prepared. To do
every year that we delay rhe upgrade to our geriatric facili-
this you should understand rhe reporrer's angle on the story
ties more families with ageing parenrs in this community •
and determine who else will be interviewed, to see what
will have ro travel great distances ro get the care rhar should ~
opposing or contradictory viewpoinrs may be aired. Your
be available locally."
audience is rhe general public-nor rhe reporter-and so
During rhe interview the reporter asks, "Aren't you jusr
you should speak in shorr, manageable
will be dear ro non-scientisrs.
"sound bites" thar
Be sure to express your views
opposing the overa11hospital facelift because you are a geri-
atrician who wanrs nicer facilities to work in? Shouldn'r all
•
••
in a \Vay that is symparheric ro rhe concerns of rhe general
hospital staffbenefit rrorn renovation dollars?" ~
publico For example, don'r say: "The cutbacks in operating
At this point you need ro take the hear ofFyourself, focus 41
room rime are upserring rhe cardiac surgeons." This sounds
on the perspective of rhe general public, and "bridge" to "
as if rhe rnain problern with curbacks is lost incorne for doc-
wrs-a view that will gain little sympathy from the publico
your borrorn lineo You reply: "The debate is nor abour who e
is going to enjoy the renovations more. Rarher, the real
A more appropriate concern ro express would be: "Cut-
backs in operating room time are raising fears of longer
issue is rhar if we do not upgrade our geriatric facilities, 'I!
many families wil! experience significanr disruption to their •
waiting lisrs and che possibility of needless dearhs while
1ives whcn referrals to other centres are needed ro get basic ••
people wair for life-saving cardiac surgery."
care for their ageing parents-care rhar should be available el
The telephone interview right here in our community!"
You then add a catchy sound-bite: ''As geriarricians our "
If you have a receprionisr, ir may be advisable for him or
goal is not just to add years ro 1ife;we realize rhe imporrance •
her to fie1d calls from rhe media so thar, rather than agree-
of adding life ro years!" •
ing ro speak ro a reponer immediately, you will have a lirde
rime ro prepare. Train your receptionist to ask for rhe rea-
son for rhe interview, rhe "angle" for rhe story, who else will
The television interview
A 1ive television interview is generally a "safer" bet rhan a
••
be interviewed, and the deadline. If rhe reporter is seeking
prerecorded interview, from which selected cornrnenrs can
commenrs abour a newly published artide , have him or
be edited to suit a reporrer's agenda. In a live interview it is
her fax a copy ro your offlce. This wili give you a chance to
read rhe article and seek advice from other experrs so thar
your critique
"bottom-line
seek Opportunities
is well-informed and rhorough.
message" in advance of rhe interview
Prepare a
ro bridge your answer ro that point. If

and
critical ro prepare as you would for a relephone interview:
know rhe agenda and have a prepared,
sage. Practice bridging to your message from a variery of
angles. Remember
bottom-line
to dress neatly and professionally, 100k

at rhe camera, don'r fidget, and do not respond wirh hostil-
mes-
.--
you are asked whether you have anyrhing else ro add at rhe
iry, as rhis can make you look nervous or "guilry as
end of rhe interview, rerurn ro your key poinr. Media peo-
charged."
ple also know rhar a carchy phrase or quoration rhar sup-
You should avoid saying "No co mrnen r" when asked
pons rheir viewpoinr will ofren fearure prorninenrlv in the
riext day's paper. about something of a sensitive nature: chis ma)' give che
appearance of a cover-up. For example, a newborn baby
disappears during your shifr on call ar che ho piral. The
Examp¿e: Imagine you are rhe head ofGeriatrics at a region-
reporter asks: "Is ir true rhar rhe hospital doesrú _' ow
al hospital and you have been lobbying rhe administrarion
where baby X was taken Iasr nighr?" Racher than say -_ -o
282
© 2011 The Royal College of Physicia~s an S:J'g._=o-s -, C¿--==e
31 Responsibi/ity and integrity
cornmenr," you could srare: "\'V'hile 1 am not at liberty to ricularly irnportanr when an article lacks balance or per-
cornrnenr on a specific situarion, 1 can assure you that if a spective. Provide references so thar your posirion is well
baby has gone missing ar our hospital there will be a thor- subsrantiared. Rarely will rhis cause the reponer to write a
ough invesrigation and appropriare action taken." rerracrion or correcrion. However, he or she may appreciate
Ir is en rirely appropriare to decline to answer "what if" your experrise in rhe subjecr area and cal! you in advance of
questions (e.g., "If your hospital burned down, whar would a furure report on a similar topic.
happen to cardiac surgery?") by stating thar you are nor pre- If you read or see an excellent repon on a healrh topic,
pared to address hyporherica! issues: "Let's stick to the real rhis mighr be an excellent opporruniry ro build rappon
problern today [bridge], which is .... [botrorn line mes- wirh rhe reponer by sending a short congratulatory email
sage]." Ir is also appropriate to decline to answer questions on the qualiry or balance of the srory. Posirive feedback can
of a personal nature (e.g., "Would you allow yom daughter be a facwr in how journalists interact wirh you and the
to have an abortion?") by responding, "What 1 would or medical profession in rhe fmure.
wouldn't do is not rhe issue. What needs to be remembered
[bridge] is rhe fact that ... [botrorn-line message]."
Conclusion
In a recorded interview, ir is appropriare to "stop and
reload" if you srurnble on an answer: "Let me rephrase rhat As healrh care researchers, we need ro appreciare our viral
...". The edi ror will usual!y only use a very short sound bi te responsibiliry ro share our discoveries with colleagues in the
after what may seem like an hour of actual raping. health sciences, with the public, and with policy-makers,
Don'r ramble on in an inrerview Await a specific ques- who may use these findings to improve healrh careo Healrh
tion and stop after giving a succinct answer. Sorne reporters care researchers are held in high regard: this respect carries
willlook at you and nod or say "Uh-huh" as a way ro get with it a responsibility ro ensure that the messages we del iv-
you to keep talking, but don't be lured inro saying more er are well reasoned and balanced. Alrhough presentations
rhan you intended. at scienrific meerings and publications in peer-reviewed
journals provide a microscope under which rhe qualiry of
research is examined, history has shown thar these filrers
Follow-up
alone are not foolproof. We must be vigilanr in our scruriny
If you see an article with faulry health informarion in your of others' research claims and have rhe integriry ro acknowl-
local newspaper, you may wish to write to rhe editor of rhe edge the potential and real shortcomings in our own
paper wirh a correction. You need to sound professional in research. The media playa powerful role in public educa-
both the tone and rhe conrent of your letrer. tion. As an example, ir was an article in Reader's Digesr'?
that did more to promote internal mammary arrery liga-
Example: "In the interest of clariry and accuracy 1wanred to rion for coronary artery disease in the 1950s rhan rhe rech-
ser the record srraight abour rhe number of knee replace- nical publicarion thar firsr described rhe proposed surgical
rnents rhat we do and our cornplicarion rate ... " technique.F:" Healrh practitioners and researchers need ro
Unforrunarely, lerrers of this narure often incite disgrun- understand how media work and how ro deliver a health
ded parierits ro write with rheir horror stories of bungled science message effecrively. This chaprer has highlighrecl rhe
knee operarions, and so forrh, and so it's wise to reserve opporrunities and perils inherenr in health researchers'
your lctters ro the editor for exrrernely irnporrant points. interactions wirh the media and should be seen as a srep-
Alternacively, you can send a letter or email direcdy ro the ping-srone toward becoming a more capable and confident
reponer to help hirn or her get the facts srraight. This is par- cornrnunicaror. •

REFERENCES
1. Canadian Institutes of Health Research (CIHR) definition of KT.A ailable from: www.cihr-irsc.gc.ca/e/29418.html
2. Pribble JM, Goldstein KM, Fowler EF,Greenberg MJ, Noel SK, Howell JD. Medical news for the public to use? What's on local TV
news. Am J Manag Cere. 2006; 12(3): 170-6.
3. Moti SE,Timpe EM, Eichner SF.Evaluation of accuracy of health studies reported in mass media. J Am Pharm Assoc.
2003;45(6):720-5.
4. Zuckerman D. Hype in health reporting: "checkbook science" buys distortion of medicalnews. Int J Health Serv. 2003;33(2):383-9.
5. Rossouw JE,Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, et al. Risks and benefits of estrogen plus
progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial.
JAMA 2002;288(3):321-33.
6. Council of International Organizations of Medical Sciences. Guidelines for preparing core c!inical-safety data on drugs. 2nd ed.
Geneva: CIOMS; 1998.
7. Laudan L The boak ot risks: fascinating facts about the chances we take everyday. New York: John Wiley; 1995.
8. Maore RA, Derry S, McQuay HJ, Paling J. What do we know about communicating risk? A brief review and suggestion
for contextualising serious, but rare, risk and the example of cox-2 selective and non-selective NSAIDs. Arthritis Res Ther.
2008; 10(1):R20.
9. Gigerenzer G, Gaissmaier W, Kurz-Milcke E, Schwartz LM, Woloshin S. Knowing your chances: what health stats really mean. Sci
Am fVlind. 2009; 1 April 44-51.
10. Sandman PM. The Peter Sandman risk communication website, Available at: www.psandman.com
11. Weinstein ND, Sandman PM, Hallman WI< Testing a visual display to explain small probabilities Risk Anal. 1994; 14(6):895-7
12. Singletary SE. Rating the risk factors for breast cancer. Ann Surg. 2003;237(4)474-82
13. Hill AB. The environment and disease: Association or causation. Proc R Soc Med. 1965;58:295-300.
14. CBC news. Seasonal flu shot may increase H1 Nl risk. 2009; Sept 23. Available from: wwvv.cbc.ca/news/health/story/2009/09/23/
f! u-shots-h 1n l-seasonal. html
•
15. Skowronski DM, De Serres G, Crowcroft NS, Janjua NZ, Boulianne N, Hottes TS, et al. Association between the 2008-09 seasonal
influenza vaccine and pandemic H1N 1 illness during Spring-Summer 2009: four observational studies from Canada. PLoS Med.
201 0;7(4):e1 000258.
••
16. Silversides A. Clinical trials: the muddled Canadian landscape. CMAl 2009; 180(1):20-2
17. Rachon PA, Gurwitz JH, Simms RW, Fortin PR,Felson DT, Minaker KL, et al. A study of manufacturer-supported trials of
•
nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med. 1994; 154(2): 157-63.
18. Campbell EG, Lauis KS, Blumenthal MD. Looking a gift horse in the mouth. JAMA 1998;279(13):995-9
.- 19. Faunce T, Townsend R, McEwan A The Vioxx pharmaceutical scandal: Peterson v Merke Sharpe & Dohme. J Law Med.
2010;18(1 ):38-49.
••
20. Mary C. Tackling fraud in medical research and scientific communication: A report of the lecture by Dr Frank Wells at the 28th
EMWA Conference. The Write Stuff [Journal of the European Medical Writers Association] 2009; 18:28-9. Available from: www.
avicenne-sciences.com/IMG/pdf/Fraud.pdf
21. Fraud Advisory Panel. Fraud in Research: Is it new or just not true? Occasional Paper 01/07. Available from www.
f raudadvisorypanel. org/newsite/pdf _show php ?id= 71
22. Smith R. Research misconduct: the poisoning of the well. J R Soc Med. 2006;99(5):232-7
23. Jaffer U, Cameron A. Deceit and fraud in medical research. Int J Surg. 2006;4(2) 122-6.
24. Sudbe J, Lee JJ, Lippman SM, Mork J, Sagen S, Flatner N, et al. Non-steroidal anti-inflammatory drugs and the risk of oral cáncer:
a nested case-control study Lancet. 2005;366(9494):1359-66 Retraction in: Hartan R. Lancet. 2006;367(9508):382
25. Pearce JM, Manyonda IT,Chamberlain GVP Term delivery after intrauterine relocation of an ectopic pregnancy. Br J Obstet
Gynaecol. 1994; 101(8):716-7 Retraction in: Br J Obstet Gynaecol. 1995;102(11):853.
284 © 2011 The Royal College 01 Physicians and Su'geors:J' r:c--~

37 Responsibility and integrity
26. Pearce JM, Hamid RI. Randomised controlled trial of the use of human chorionic gonadotrophin in recurrent miscarriage
associated with polycystic ovaries. Br J Obstet Gynaecol. 1994; 10 1(1):685-8. Retraction in: Br J Obstet Gynaecol.
1995; 102(11 ):853
27. Hwang WS, Ryu YJ, Park JH, Park ES,Lee EG, Koo JM, et al. Evidence of a pluripotent human embryonic stem cellline derived
from a cloned blastocyst. Science. 2004;303(5664): 1669-74. Retraction in Kennedy D. Science. 2006;311 (5759):335.
28. TessierJ, Hahn P,Reid RL. Media training for residents in Obstetrics and Gynecology: Improving communication ski 115 and
advocacy for women's health. J Soc Ob Gyn Can. 2001 ;23(6):495-500
29. Ratcliff J. New surgery for aíling hearts. Reader's Digest. 1957;71 70-3.
30. Kitchell JR, Glover Rp,Kyle R. Bilateral internal mammary artery ligation for angina pectoris preliminary clinical considerations.
Am J Cardiol. 1958; 1(1)46-50
31. Glantz SA. Primer of biostatistics. 6th ed. New York: McGraw-Hill Medical Publication Division; 2005:448.
SUMMARY CHECKLlST
o Thinking about your study, list the potential benefits of sharing the findings in the lay media. List any pitfalls.
O Prepare a list of key messages arising from your study that you would like to disseminate.
O List the forms of media that you would like to use (e.g., web page, blog, Twitter, radio, TV, newspapers).
O If you have arranged a media interview, prepare and practise the delivery of your key messages.
O List key findings from your study that could be misunderstood if taken out of context. Prepal-e explanations that convey
these findings simply and dearly
~
ro
'"C
O
~
Fí~

•
,
•
,•
,
~
•
I
•
•••
,
,
••
•
•
111
286 © 2011 The Royal Coliege 01 Physicians ano Surgeo- ':-22

32
So, you've finished your first research project
now what?
Ross Upshur, MD, MA, MSc, CCFP, FRCPC
ILLUSTRATIVE CASE
After completing a research project exploring whether the introduction of a computer-based reminder system actually
increased the provision of selected preventive services in a General Internal Medicine ambulatory clinic, a third-year Internal
Medicine resident considers what, if any, further research experiences she should pursue during the remainder of her
residency training. She also wonders whether it would be possible to pursue clinical research as a component of her career
after the completion of specialty training and, if so, what additional research training she might require.
11 SO. you've just completed your first Few descriprions of medica! research careers accounr
healrh sciences research projecr. Now whar? Including for rhe wide range of projecrs that a healrh professional can
research acriviries as one componenr of your career as a become involved with, or rhe various degrees of involve-
healrh professional is an excellent way ro keep your inter- menr rhat are possible. Each rype of engagemenr enrails a
esrs diversiíied, ro contribute ro knowledge generarion in differenr leve! of skill developmem (which can include srudy
your field and, perhaps mosr importantly, ro engage in a design, statistics, research ethics, darabase construction and
wide range of collaborarions on interesring issues relevant managemem, and manuscripr and granr writing), time
ro your pracrice. Many healrh professionals finish rheir cornrnitment, responsibiliries and rewards. 1he fol!owing
rraining wi th a desire ro acq uire clinical masrery and ro be a secrions describe six variants of research involvement.
researcher, but feel rhar ir is impossible ro combine rhese
rwo parhs, especial!y in view of rhe rime involved not only
in conducring research bur also in acquiring rhe rnerhod-
CHAPTER OBJECTIVES
ological experrise necessary ro do so. Balancing rnulriple
cornpeting obligarions is a recognized difficulry in rhe tran- After reading this chapter, you should be able to
sirion from supervised ro independenr pracrice. 1his chap- • discuss various paths to further involvement in research
ter makes rhe case rhar al! healrh professionals should play after the completion of your first research project
sorne role in research in rheir field, and thar ir is possible ro • describe the merits of including research activities as a
participare in research ar differenr levels, and ar any poinr component of your professional career
in one's career.
KEY TERMS
Co-investigator Mentor
Collaborator Principal investigator
Critical appraisal skills Recruiter
Medical research careers

gram proposal ro a peer-reviewed publication. Ir truly Conclusion

is an accomplishment worth celebrating. Even if most There are many paths ro engagemem in the research pro-
of the world ignores your paper, family members are cess. Increasingly, rhe more 1 do research the more 1 think it
bound ro be pleased ro receive a copy of a paper with should be inregral part ro every health professional's acrivi-
your name listed among the authors. And- more ries. In this chapter 1 have tried ro point ro sorne of the ways
seriously-publishing in the peer-reviewed literarure in which healrh professionals can integrare research into
esrablishes your credibiliry and may lead ro invitations their day-ro-day practice. 1 hope all readers who are in rhe
as a peer reviewer or to participate on expen advisory midst of or have already cornplered their healrh training
boards locally and imernationally. will find ways ro become involved in research in sorne way
• Last but by no means least, your research may in rheir professionallives. •
contribute ro rhe developmem of best pracrices and
the irnprovement of patient outcomes.
REFERÉNCES
1. Royal College of Physicians and Surgeons of Canada. Specific standards of accreditation for c/inica/ investigator programs.
Ottawa: The College; 2009. Available from: http://rcpsc.medical.org/residency/accreditation/ssas/CIP _e.pdf
2. College of Family Physiciansof Canada. C/inica/ Scho/ar Program. Mississauga (ON): The College; 2009. Available from: www.cfpc.
ca/local/files/Education/Clinician%20Scholar%20Program%20-%20Final%20January%202009%20(2).pdf
ADD!TIONAL RESOURCES
Hayward AR. Making clinical research a robust career path. Acad Med. 2009;84(4):409-10.
• This brief editorial describes initiatives to enhance the attractiveness of research to clinicians.
Jarvis P The practitioner-researcher: deve/oping theory from practice. New York (NY): Jossey-BassPublishers; 1999.
• Although not focused on medical practice, this book contains a comprehensive account of what is required to do research in
practice-based locations.
Melnick A. Transitioning from fellowship to a physician-scientist career track. Hemetoioqy Am Soc Hemato/ Educ Program. 2008: 16-22.
• This paper contains excellent practical advice on getting a research career started in academic settings.
Teo AR. The development of clinical research training: past history and current trends in the United States. Acad Med.
2009;84(4):433-8
• This paper highlights the development of clinical research career paths in the United States and outlines current trends .
•••••
-
290 © 2011 The Royal College of Physicians and Surgeons of Canace
32 So, you've fi ed your fist research project ", now what?
SUMMARY CHECKLlST
o Thinking about your experiences with research and your career interests, describe the role you would like research to play
in your career. Describe how you expect this to change over time,
O Develop your strategy for a successful role in research, as outlined above. Be sure to list:
- experience you wish to have
- training you wish to obtain
mentors you would like to work with
- levels of involvement in proposed studies

Reviewers
Katherine Boydell, PhD Corinne Hohl, MD, FRCPC

Department of Psychiatry Department of Emergency Medicine
University of Toronto University of British Columbia
Toronto, Ontario Vancouver, British Columbia
Antoinette Colacone, BSc, CCRA Tanya Horsley, PhD

Emergency Multidisciplinary Research Unit-FMRU Centre for Learning in Practice
Sir Mortimer B. Davis-Jewish General Hospital Royal College of Physicians and Surgeons of Canada
Montreal, Quebec Ottawa, Ontario
Donald Cole, MD, MSc, FRCPC Thomas A. Lang, MA

Department of Public Health Sciences Tom Lang Communications and Training International
Dalla Lana School of Public Health Kirkland, Washington
University of Toronto
Toronto, Ontario Trevor Langhan, MD, FRCPC
Division of Emergency Medicine
Scott Compton, PhD University of Calgary
Dept of Emergency Medicine Calgary, Alberta
New Jersey Medical School- The University Hospital
Newark, New Jersey A. Curtis Lee, PhD
Education and Evaluation and Analysis Unit
David C. Cone, MD Royal College of Physicians and Surgeons of Canada
Department of Emergency Medicine Ottawa, Ontario
Yale University
New Haven, Connecticut Jacques S. Lee, MD, MSc, FRCPC
Department of Emergency Medicine and
Neil Drummond, BA, MFPHM (UK), PhD Clinical Epidemiology Unit
Department of Family Medicine and Community Health Sunnybrook Health Sciences Centre
University of Calgary Ioronto. Ontario
Calgary, Alberta
Marilyn MacDonald, RN, PhD
Nancy Feeley, RN, PhD School of Nursing
School of Nursing Dalhousie University
McGi11 University Halifax, Nova Scotia
Montreal, Quebec
Jessie McGowan, BMus, MUS, PhD
Dionne Gesink, PhD Departments of Medicine and Family Medicine
Dalla Lana School of Public Health University of Ottawa
University of Toronto Ottawa,Ontario
Toronto, Ontario
292 © 2011 The Royal College 01 Physicians an SurgE c= Ca.-.a::z

Reviewers
Andrew McRae, MD, FRCPC David L. Streiner, PhD, CPsych

LHSC-University Hospital Department of Psychiatry and Behavioural Neurosciences
University of Western Ontario McMaster University, Hamilton, Ontario
London, Ontario Department of Psychiatry
University of Toronto
Diana Petitti, MD, MPH, FACPM Toronto, Ontario
Department of Biomedicallnformatics
Arizona State University Lehana Thabane, BSc, MSc, PhD
Phoenix, Arizona Department of Clinical Epidemiology & Biostatistics
McMaster University
Robert L. Reid, MD, FRCSC Hamilton, Ontario
Division of Reproductive Endocrinology and Infertility
Department of Obstetrics and Gynecology Ross E.G. Upshur, MD, MA, MSc, CCFp, FRCPC
Kingston General Hospital, Queen's University Department of Family and Community Medicine and
Kingston, Ontario Sunnybrook Health Sciences Centre, University of Toronto
Toronto, Ontario
Norm Rosenblum, MD
Division of Nephrology David R. Urbach, MD, MSc, FACS, FRCSC
The Hospital for Sick Children, University of Toronto Department of Surgery
Ioronto, Ontario Toronto General Hospital, University of Toronto
Toronto, Ontario
Howard M. Smith, MS
Medical Writing Robert L. Wears, MD
INC Research Department of Emergency Medicine
Richmond, Virginia University of Florida Health Science Center Jacksonville
Jacksonville, Florida
Julie Spence, MD, MSc, FRCPC
Department of Emergency Medicine Andrew Worster, MD, MSc, CCFP(EM), FCFP
S1. Michael's Hospital, University of Ioronto Division of Emergency Medicine
Toronto, Ontario McMaster University
Hamilton, Ontario
Sharon E. Straus, MD, MSc, FRCPC
Li Ka Shing Knowledge Institute
S1. Michael's Hospital, University of Toronto
Toronto, Ontario

Index
Note: (f) after a page reference denotes a figure, (t) atable.
abstracts "satisficing" bias, 73

for case reports/series, 237-38, 243 social desirability bias, 73
checklists for, 244-45 systernic bias, 60
conference abstracts, 234-35 withdrawal bias, 136
functions of, 233-34 See also selection bias
IMRAD structure, 234 bibliographic darabases, 42, 44-46(t)
informative vs descriptive, 237, 241-42 binary (dichotomous) data, 52-53
journal abstracts, 235-37, 244-45, 262 biological variation, 206-7, 215
optimizing for electronic retrieval, 238 BIOSIS database, 42, 44(t)
revising, 238 biostatistics. See statistics
structured/ unsrructured, 237, 241-42 blinding, 62, 65,112-13,172
for systernaric reviews, 237-38, 242-43 Boolean operators, 43
See also slide presentations budge~,21, 156-57, 158(D
academic mentors. See mentors
administrarive database research, 97-100
allocation concealment, 62, 65, 112-13, 172 CanMEDS framework, 11
alpha (a level), 216, 220-22 case-control studies, 59-60, 64(t), 65
ancestry searches, 111 case reports and series, 58, 237-38, 243
ANOVA (analysis ofvariance), 217-18 case selection (charr reviews), 91-95
association, 230, 231,280 case srudies, 120-21
authorship of papers, 18, 189,263 caregorical data, 52-53
autonomy (respect for persons), 142 causal relationships, 230, 231, 280
central tendancy, 198-99
chart reviews, 91-95
bcfore-after (pre-post) studies, 63 CINAHL database, 44(t), 47
beneficcnce (concern for welfare), 142 citation indexes, 42
beta (~ level), 222 citation software, 46
bias clinical irnportance, 216-17, 231
co-intervention bias, 136 clinical trial regisrries, 112, 148-49, 264
contarninarion bias, 136 co-inrervention bias, 136
end-aversion,73 co-investigarors, 21,137,189,289
inforrnation bias, 60, 65-66, 136 ce-variables, 52
insrrument bias, 136 Cochrane Library, 42, 44(t), 110
intervention bias, 136 coding errors, 99
measurement bias, 136 Cohen's d, 107
in measurement scales, 73 Cohen's kappa (K), 94
non-respondent bias, 136 cohort studies, 60-62, 64(t)
non-response bias, 93 collaboration in research projects, 157,288-89
publicarion bias, 111-12, 148,229 cornpetencies in health care, 11
recall bias, 60, 136 competing interests, 148,263,280
referral bias, 136 complex sampling design, 85, 86
response bias, 73, 87 confidence intervals
risk ofbias rabies, 112-13 in descriptive statisrics, 82, 207-9
294 © 2011 The Royal College 01Physicians and 5_°:;''''0"'5 c:" ~-a::G
Index
in hyporhesis resting, 221, 222 quesrion formats, 182

staristical significance and, 82, 207-9, 221, 228 selecrion bias in data extracrion, 93
confidenrialiry/privacy, 146, 148,264 srared in research prorocols, 134, 135
conflicts of interest, 148,263,280 survey modes, 83, 179
confounding, 66 in systernatic reviews, 113-16
consrrucr validity, 76-77 testing in pilot studies, 172
constructs, hyporhetical (latent variables), 71-72 using medical records, 92, 179
See al so measuremenr scales using volunreers, 157
contarnination bias, 136 validiry, reliabiliry, practicaliry, 179
conrinuous data, 53, 204 data-sharing agreemenrs, 99
correlarion, 218-19, 230 data variables
correlarion coefficienrs, 74 co-variables, 52
criterion validarion, 76 definirion and role, 51, 52, 54
Cronbach's 0., 75 outcorne, 52,135
cross-over srudies, 63 predictor,52
cross-secrional studies, 59, 64(t) databases, adminisrrative, 97-100
crystallizarion, 124 databases, bibliographic, 42, 43, 44-46(t), 47
dependent variables, 52
descriptive statistics
DARE darabase, 42, 45(r) biological variatiori, 206-7, 215
data confidence intervals, 82, 207-9
absrracrion, 113 dichoromous data, 204-5
in adrninisrrative databases, 97-101 discrete data, 53, 205-6
analysis in systernaric reviews, 113-16 frequency distribution, 198-99,214
caregorical data, 52-53 measures of central tendency, 198-99
conrinuous data, 53,204 measures of variabiliry (dispersion), 200-1, 202
dara cleaning, 180, 183 measuring/inrerpreting data variabiliry, 206-9
discrere data, 53,205-6 nominal data, 204-5
hererogeneiry, 113 normal (Gaussian) disrribution, 201
interval data, 53 ordinal data, 53, 205
levels of rneasurement (rypes of data), 52-53 precision, 82, 87, 206-9
nominal data, 52,204-5 randorn variariori, 206-7, 215
oprimizing data qualiry, 93 skewed distributions, 201-4
ordinal data, 53, 205 standard deviatiori, 200-1, 203, 209
quantirative data, 53-54 standard error, 207, 209
data analysis (qualirative), 123-24 statistical tesring and reponing, 204
data analysis (quantitarive). See descriprive statistics; descriptive studies, 58, 64(t)
hyporhesis resring (inferential statistics) design effecr, 86
data cleaning, 180, 183 dichoromous (binary) data, 52-53
data collection di fferential misclassificarion, 65
from adminisrrative databases, 98-100 discrete data, 53, 205-6
conrexrualizi »g. 180 dispersion, 200-1,202
demographic informaríon, 180 double enrry, 183
designing rhe form, 180-82
idenrificarion of respondents, 180, 190
privacy/confidenrialiry issues, 146, 180,264 ecologic fallacy, 59
in qualitarive research, 121-23 ecologic studies, 59

lndex
effecr modificarion, 66 informal funding, 159-60

electronic health records (EHRs), 98 managing srudy finances, 190
e!ectronic medical récords (EMRs), 98 for pilor studies, 175
Embase database, 42, 43, 45(t), 47 See also granring agencies
Emtree subject headings, 42, 43, 45(t)
end-aversion bias, 73
erhical considerations Gaussian disrriburion, 201
in chart reviews, 94 generalizabiliry (exrernal validiry), 64, 82-83,164
in rhe FINER crireria, 1, 38 Google Scholar, 42
in program evaluations, 107 granring agencies
in publishing papers, 263-64 formal research fundi ng, 157-59, 160-62
in qualitative research, 125 granr applications, 21, 37
research prorocols, 137 insritutional research administration and, 149-50
role of study sponsors, 263 research protocols, 131-38
erhics rcview research questions and, 37
clinical trial regisrrarion, 148-49, 264 See also funding for research
conliicrs of interesr, 148 grey lirerarure, 47, 111-12
Institucional research adrninistrarion,
issues considered, 144-48
148, 149-50 grounded theory, 124
,
~
necessary for publicarion, 264

pilot studies, 175
•
,•
health datasets, 97
prorocol amendmenrs, 148, 188 healrh professions educarion research
reasons for, 141-42 categories of studies, 104-6
Research Erhics Board (REB), 143-44 conducting a projecr, 106-7 f
standards for healrh sciences research, 142-43,
studies requiring review, 144
ethnography, 120
150 spectrum of education,
"hedges" search strategy, 47
heterogenei ry, 113
103-4
,
f
evidence tables, 113-15 human resources for research projecrs, 155-57 .-

Excel software, 182-83, 184(t) hyporhesis generation, 38
•
exclusion crireria, 164
experimental studies, 62, 64(r)
hyporhesis testing (inferenrial statistics)
alpha (a level), 216, 220-22
•
externa] validiry (gcneralizabiliry), 64, 82-83,164 ANOVA (analysis ofvariance), 217-18
beta (~ leve!), 222
chi-squared test, 219
facevalidiry, 182 clinical imponance, 216-17
feasibiliry studies. See pilot srudies
fie!d observarion, 121
confidence inrervals, 221, 222
correlation, 218-19, 230
•
FINER crireria, 1, 38 description, 38-39, 214-16, 228
finire population correcrion, 85, 86 non-pararnetric statistical tests, 218, 219 ••
focus groups, 121-22 null hypothesis, 215-16, 227-28
foresr plot, 113, 115(f) one-railed resrs, 220
framing, 181
frequency disrriburion, 198-99
P value (sce Pvalue)
paired t test, 217
•
funding for research pararnetric sratistical tests, 218, 219
budgering, 21,156-57, 158(r), 190, 191(r) regression, 219
derermining resource requirements, 155-56 sarnple size calcularion, 222-23
formal funding, 157-59, 160-62 statistical power, 222
296 © 2011 The Royal College of Physicians and S~-;2C-.s:;; Ca-a=c

lndex
statistical significance (see sratistical significan ce) elinical decision-making and, 269-70
test statistic, 215 description, 269-70, 277
rwo-tailed tests, 220 dererrninants ofknowledge use, 271
Type 1 and II errors, 220-21, 222 irnportance, 270-71

inregrated knowledge translation, 270-71
un paired t test, 218
hypotherical constructs (Íatent variables), 71-72 iterative loop, 26, 27(f)
See also measuremenr scales knowledge creation, 271-72, 273(f)

knowledge-ro-action cyele, 271-74
research question, 39
immersion/ crystallization, 124 role of research nerworks, 25
imputation,93 See also media

IMRAD structure for papers, 125,234 Kruskal-Wallis test, 219
in-depth inrerviews, 121-22

incidence/surveillance studies, 58
latent variables (hypothetical constructs), 71-72
inelusion criteria, 163-64
independent variables, 52 librarians, 41,42,46-47
iníerential sratistics. See hypothesis testing (inferential Likert scales, 182
staristics) Íirerarure review
information bias, 60, 65-66,136 ancestry searches, 111
information technology persorinel Boolean operarors, 43
on research teams, 189 clinical trial registries, 112, 148-49
informed consent, 142, 145-46, 147(t) grey literature, 47,111-12
instrument bias, 136 hand searching, 47, 111
inter-observer agreement, 94 PRISMA guidelines, 47
inrer-rater reliabiliry/agreement, 74,113 purpose, 39, 41

internal consistency, 74-75 in research protocols, 133
search suategies, 42-43, 46, 47
interna] validiry
defi n ing study population and, 164 snowball searching, 47
sources, 41, 42, 44-46(r)
deiinition, 65
threats ro, 190, 193 steps,41
interquartile range (IQR), 200, 202 storing searches and resulrs, 46-47
inrerrupted time series analyses, 63 longitudinal surveys, 59
interval data, 53
intervenrion bias, 136
interviews, ín-depth, 121-22 management of rhe srudy
intra-class correlation coefhcienr, 74 amendmentofprotocol, 148, 188
intra-rarer agreemem, 113 data securiry, 189-90
iterarive loop, 26, 27(f) finances, 190, 191 (t)

moniroring protocol compliance, 190
moniroring srudy progress, 190, 193
journal clubs, 20,157,288 research diary, 193

rhreats ro internal validiry, 190, 193
jusrice principie, 142, 143
time management, 188, 190, 192(t)
workingwith research tearn, 18-19, 188-89
Kirkpatrick's four-level framework, 106-7 Mann-Whirney U test, 219

knowledge rranslation margin of error in surveys, 82,87
action cyele, 272-74 mean, 199,202,203-4

297
Index
measurement bias, 136 null hyporhesis, 215-16, 227-28

measurement scales observational srudies, 61, 229-30, 231
assessing healrh/behaviour, 71-72 odds mio, 60, 230
checking irerns, 73, 84 ordinal data, 53, 205
construct deíinition, 72 ourcorne variables, 52,135
developing iterns for, 72-73, 84 ourliers, 202
Likert scales, 182
lisrs of, 78, 79
reliability of, 73-75 Pvalue
utility of, 77-78 alpha (a level) , 216,220-22
validity of, 76-77 description, 216-17, 227-28
media null hypothesis, 215-16, 227-28
benefirs of using, 278-79 rwo- and one-railed tests, 220
interviews, 282-83 paired t test, 217
journalists' and researchers' perceptions of each parameter estimation, 38
orher, 277-78 parametric staristical tests, 204
lerters to the editor, 283
pitfalls in reporting scienrific findings, 278-81
participant
participant
observation,
rerention,
120
165-66 ,
I
median, 199, 202, 203-4

medical record reviews (MRR), 91-95, 259-60
participant selection/ recruitment
assessing in pilot studies, 171-72
•
MEDLINE, 42, 43, 45(t), 46, 47, 238 defining study population, 163-64
mentors determining number of subjecrs needed, 166
as advisors, 31 identihcarion of participants, ] 80, 190
irnporrance of, 29, 288-89 inclusion/ exclusion cri teria, 163-64
opportunities afForded, 30-31 logistical/feasibility issues, 166
prorection from academic attacks, 31-32 losses to follow-up, 165-66
provision of resourccs, 29-30 recruiting, 165
selecring, 32 review by erhics boards, 144, 145
MeSH subject headings, 43, 238 reviewed in research protocols, 134
•
rnera-analyses,
missing values, 93
109, 110 screening for study participants,
selection bias, 164-65
164-65
•
mode, 199 so urce population, 163
MOOSE Sraternenr, 116 study sample, 134, 163-64
for surveys, 83-84
target popularion, 163
N-of-l srudies, 63 threats to interna! validiry, 193
narrati ve reviews, 109 See also privacy/confidentialiry, sampling
natural experirnenrs, 63 pearl-growing strategy, 46, 111
needs assessrnenr, 104 Pearson product-rnoment correlation coefficient, 74,
negative predictive value (NPV), 76 218-19
nominal data, 52,204-5 Pediatric Ernergency Resea rch of Canada (PERC), 25-27
non-difterenrial misclassificarion, 66
nori-parametric statisrical tests
percentile, 199,200
PICOT approach, 1,39-40,57, 111
••
use, 204 pilor studies
non-respondent bias, 136 description, 169-70
non-response bias, 93 determining sarnplc size, 174
298 © 2011 The Royal Colleqe oí Physioans a~á SVgeoct5 o' Ca~
Index
ethics board approval, 175 trial registration, 264

funding, 175 Sce also abstracrs; wriring a paper
purposes, 170-74, 176 PubMed (MEDLINE), 42, 43, 238
reporting/publishing, 175
using data in full study, 175-76
placebo effect, 63 qualitative research
placebos, 62 collecting data, 121-23
point estimares, 83, 87 data analysis, 123-24
populations description, 119-20
interna] validiry and study populations, 164 ensuring qualiry, 124-25
source population, 163 ethical considcraticns, 125
target population, 163 rnethodologies, 120-21
vulnerable popularions, 142 reBexiviry and, 124
See also participant selection/recruirment wriring rhe paper, 125-27
positive predictive value (PPV), 76 quantirarive data, 53-54
pos ter presenrations quantitative research, 58-64, 91-95,104-8
content requiremenrs, 251 See also descriptive statistics; pilot srudies, research
presenring rhe poster, 247, 253 design
sryle and formar, 251, 252(f) quasi-experimental studies, 63
using PowerPoinr, 251, 253 quesrionnaires
PowerPoinr, 248-50, 251, 253 analysis, 182-83, 184(t)
pre-studies. See pilot studies in cross-sectional studies, 59
predictor variables, 52 designing the form, 180-82
prevalence studies, 58 ensuring validiry, 82
principal investigators, 137, 187,289 face validiry, relevan ce, clariry, 182
PRISMA guidelines, 47, 116 in focus groups, 122
privacy/confidenrialiry, 146, 148,264 information bias and, 65
program evaluations ofhealrh professions educarion, question formats, 182
105-6 See also data collection; measurement scales; sllrveys
psychornerrics, 73
publication bias, 1J 1-12, 148, 229
publicizing research. See knowledge translation; media; RACI managemem tool, 20-21
publishing random error/variation, 64-65
confidenrialiry of participanrs, 264 randorn sampling, 82, 84-85
conflict of inrerest, 263 random variariori, 206-7, 215
criteria for aurhorship, 263 randomized conuolled rrials (RCTs)
editorial freedom, 264 allocarion concealrnent, 62, 65,112-3,172
erhical considerations, 263-64 blinding in, 62, 65, 172
erhics approval, 264 cornparability of study groups, 62
formarring subrnissions, 261 pilot studies for, 172
peer review process, 262-63 range, 200
pilor srudies, 175 ratio data, 53
redundam publicarion or fraud, 281 recall bias, 60, 136
rejection, 21-22, 262-63, 265 recruiting study participanrs, See participanr selection/
role of study sponsor, 263 recruitmen t
submission package, 261-62 red undancy (saruration), 123
© 2011 The Royal College of Physicians and Surgeons of Cenada 299

reference software, 46 srudy biases, 60, 65--66, 73, 87
re fenal bias, 136 research disciplines, 27
reflexivity, 124 research environment, 27
registration of clinical trials, 148-49,264 Research Ethics Board, 143-48
regression analysis, 61,230 See also erhics review
reliabiliry research in residency
administering tests, 77 comrnon pitfalls, 5
assessing measurernent scales, 73-75 core competency, 11
in data collection, 179 grant applications, 21, 37
inter-rater reliabiliry, 74 interdisciplinary, 19
test-rerest reliability, 74 parallels with clinical practice, 19-20
represenrive surveys, 82 prospective, 13
research RACI management tool, 20-21
as co-invesrigarors, 21,137,189,289 reasons for, 12, 13,27
as collaborators, 288-89 rerrospective, 12-13
as critical consumers, 288 road map, 1-5
grant applicarions, 21 time trame, 22
insritutional adrninistrarion, 148,149-50 research nerworks, 25, 27,157
interdisciplinary, 19,26-27 research outlines, 2,156,175
as principal invesrigators, 137, 187 research prorocols
as recruirers, 288 arnendrnenrs, 148, 188
rewards of a research career, 289-90 components, 132-38
time trame, 21-22 descriprion, 131
rescarch coordinarors, 166, 188 ethical considerations, 137
research design ethics rcview (see ethics review)
before-aíter (pre-post) srudies, 63 moniroring compliance wirh, 190
case-control studies, 59-60, 64(r) for systernaric reviews, 111
case srudies, 120-21 tips for writing, 131-32, 136
cohorr studies, 60-62, 64(r) research questions
conFounding, 66 ethical review and, 144
cross-over studies, 63 FINER criteria, 1,38
cross-sectional srudies, 59, 64(r) formulating, 38-40
descriptive studies, 58, 64(t) PICOT approach, 1,39-40,57,111
design effect, 86 refining, 37-38
erhnography, 120 research ideas, 35-38
experimental studies, 62, 64(t) in research prorocols, 133
Feasibility of 156 in sysremic reviews, 111
generalizability (external validity), 64, 82-83 research supervisors
interna] validity, 64-65 aurhorship of papers, 18, 189
interrupted time series analyses, 63 collaborarive supervisory tearns, 16-17
N -of-1 studies, 63 cornmunication with, 18
natural expcriments, 63 conAicr resolurion, 19
observational studies, 61 establishing norms (rules), 18-19
qualirarive, 121-23 funding record/assisrance, 156-57
quantirative, 58-64 informallearning, 20
quasi-experimental studies, 63 project or process Focused, 1 5-16
srepped wedge, 62 selecrion, 1, 16-17
300 © 2011 Tile Royal College 01 Physicians and S_-g~:)""'5 e'

Index
work management, 18 in data extracrion, 93

See also mentors descriprion, 134
research reams derecrion in research protocol, 134-35
advantages, 16-17,25,27 screening study panicipants, 164-65
authorship of papers, 18, 189 in survey parricipants, 87
collaborative, 16-17 in sysremic reviews, 111-13
communication, 18-19, 189 selection of panicipants. See parricipanr selection/
interdisciplinary, 19 recruitment
norms for conducr, 18-19 sensitiviry analyses, 112
pe~onnclon, 188-89 simple random sampling, 84-85, 86, 93
RACr management tool, 20-21 slide presentations
ream skills, 17, 19 preparating the slides, 248-49
working efiecrively wirh, 188-89 presentation srrategies, 250-51
resources and research feasibiliry, 156 slide formats, 249-50(f)
response bias, 73, 87 rabIes and figures, 249-50
response rates, 83-84, 87 time limitations, 247
retaining srudy parricipanrs, 165-66 SMART research objectives, 134
risk snowball searching, 47
absolure vs relarive, 279 social desirability bias, 73
contexrand, 279-80 source population, 163
erhical review of, 145 Spearman rank correlation, 218-19
minimal risk, 145 spectrum of health professions education, 103, 104(f)
standard deviation, 200-201, 203, 209
standard error, 207, 209
sample size statistical significance
calcularing, 135, 137, 166 alpha (a level) , 216, 220-21
esrimaring from pilot srudies, 173-74 confidence inrervals, 82, 207-9, 221, 228
in medical record reviews, 92 deíinition, 216
numbers needed for specified precision, 213, 222-23 "negative" studies, 229
for pilot studies, 174 null hypothesis, 215-16, 227-28
in qualitative research, 123, 124-25 odds ratio, 60, 230
staristica] power and, 222-23 Pvalue, 216-17, 220-22, 227-28
in surveys, 82, 85-86 subgroup analysis, 229
sampling vs clinical imporrance, 216-17, 231
for medical record reviews, 93 statisrical software, 87
purposive, 122 starisricians
in qualirarive research, 122-23, 124-25 in adrninistrative data research, 99-100
redundancy, 123 data interpretation, 135, 189,228
for surveys, 84-85 determiningsamplesize, 135, 137, 166, 189
See also panicipam selection/recruirrnent on research tearns, 189
sampling frame, 83 in survey design, 85, 86, 87-88
"satisficing" bias, 73 statistics
scales. See rneasurernenr scales descriptive (see descriptive statistics)
Scholar Role (CanMEDS framework), 11 design effecr, 86
Scopus database, 42 impuration, 93
selecrion bias inferenrial (see hypothesis tesring
in case-control studies, 65 [inferen tial sratisricsj)

Index
margin of error in surveys, 82, 87 validiry

missing values, 93 construct, 76-77
precision, 82, 87,206-9 in dara colIection, 179
See also sratistical significance; statisricians erhical review and, 145
stepped wedge research design, 62 externa], 64
stratiíication, 61 face validi ry, 182
study samples, 163-64 interna! validiry, 65, 164, 190, 193
subject headings, 42-43 of measurement scales, 76-77
surveys in questionnaires, 82
conducting, 86-87 of surveys, 82
definition, 82 variabiliry (dispersion) of data, 198
designing, 82-86 variables
resources, 9, 87-90 dara,51,52,54,135
samples and sampling, 83-86 definition in research prorocols, 135
survey mode, 83-84 latenr, 71-72
valid surveys, 82 variance, 87, 201
sysrernatic reviews volunteers in research projecrs, 157
data collection/analysis, 113-16 vulnerable populations, 142
foresr plor, 113, 115(f)
literature search information, 47
purposes of, 1 10 Web ofScience database, 42
research question/prorocol, 111 WiIcoxon signed-rank test, 219
risk ofbias tables, 112-13 withdrawal bias, 136
sarnple selection, 111-12 writing a paper
selection bias, 111-13 abstraer (see abstracts)
types of reviews, 109--10 conclusion section, 234, 259
writing paper on, 259-60 copyright issues, 261,263
systernic bias, 60 discussion section, 127,230,259
formatting for submission, 261
IMRAD structure, 125,234
tables and figures in slide presentations, 249-50 introduction, 125,234,257
target population, 163 on medical record reviews, 259-60
test-rerest reliabiliry, 74 merhods section, 125, 127,234,257-58
time management peer review process, 256, 262-63
irnporrance to career, 29 qualitative papers, 125-27
RACI tool, 20-21 resulrs section, 127,234,258-59
in research study, 188, 190, 192(t) rules of effective writing, 235, 260-61, 265
Tri-Council Policy Sraternenr, 142-43, 150 submission package, 261-62
triai registration, 112, 148-49,264 on systernaric reviews, 259-60
rrial runs. See pilot studies tables, 258-59
rrue zero point, 53 tips for writing a paper, 255-56
ti de, 256-57
utiliry of measurement scales, 77-78 See also publishing
302 © 2011 The Royal College 01 Physicians and Surgeons 01 Canada I

Notes

Guia para Investigación Residentes

Uploaded by

Document Informationclick to expand document information

Document Informationclick to expand document information

Copyright:

Available Formats

Guia para Investigación Residentes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Guia para Investigación Residentes

Uploaded by

Copyright:

Available Formats

J:4t~() «:..

e Research Gu· e ~/('~('~~

trainees, and practitioners

Royal College of Physicians and Surgeons of Canada

774 Echo Orive TEL 613-730-8177

Royal College of Physicians and Surgeons of Canada

Reprinted with an index February 2012 in Ottawa, Ontario, Canada.

How to reference this document:

--------------- - - - -- .- -----_. ------------~~~--

The Research Guide: A primer for residents,

Bart J. Harvey, MD, PhD, MEd, FACPM, FRCPC

iv © 2011 The Royal College of Physicians and Surgeons of Canad

© 2011 The Royal College 01 Physicians and Surgeons 01 Canada

Stefan Grzybowski, MD, FCFp, MCISc

Philip M. Hahn, MSc

Bart J. Harvey, MD, PhD, MEd, FACPM, FRCPC

June C. Carroll, MD, CCFp, FCFP Grant Innes, MD, FRCPC

Scott Garrtson, MD Lorie A. Kloda, MLlS, AHIp, PhD (candidate)

vi © 2011 The Royal College 01 Physicians and Surgeons 01 Ca-a.::a

Thomas A. Lang, MA David L. Streiner, PhD, CPsych

© 2011 The Royal College 01 Physicians and Surgeons 01 Canada vii

5. Develop a research outllne."

• .- • Conduct a thorough literature search (ch. 7).

2 © 2011 The Royal College 01 Physicians and Surgeons e

your timetable for starting and finishing the

• Be prepared for some hard work on preparing your

b See the PSI Foundation website at www.psifoundation.org/

© 2011 The Royal College 01 Physicians and Surgeons 01 Canada 3

take-home points are: individual who has, overall, contributed th

figures. that authorship requires: (1) substantial

Develop a systematic approach to the introduc- contributions to conception and design, or

and incorporate useful feedback. critically for important intellectual content;

4 © 2011 The Royal College 01 Physicians and Surgeons 01 Cal

TEN COMMON RESEARCH PITFALLS

1. Not establishing a focused, answerable question.

TIPS FOR STAYING ON TRACK

© 2011 The Royal College of Physicians and Surgeons of Canada 5

6 © 2011 The Royal College of Physicians and Surqeons 01 Canada

© 2011 The Royal College of Physicians and Surgeons of Canada 7

10 © 2011 The Royal College 01 Physicians and Surgeons 01 Canad

© 2011 The Royal College of Physicians and Surgeons of Canada 11

© 2011 The Royal College 01 Physicians and Surgeons 01 Can,

Albert Szent-Gyórgyi (1893-1986)

© 2011 The Royal College of Physicians and Surgeons of Canada 13

._ Searching her department's

o Describe at least three reasons to get involved in the world of research.

14 © 2011 The Royal College 01 Physicians and Surgeons oí Canada

Choosing your topic: What comes

© 2011 The Royal College of Physicians and Surgeons of Canada 15

that area will be the rnost imporrant aspecr of your search.

.- regardless of the topic, you are seeking a supervisor who will

16 © 2011 The Royal College 01 Physicians and Surgeons 01 Canada

members. Whatever the rearn's cornposition,

© 2011 The Royal College 01 Pl1ysicians and Surgeons 01 Canada 17

Milestones for management, to discuss progress, de!iverables and timelines? Or do they

© 2011 The Royal College 01 Physicians and Surgeons 01 Canada

© 2011 The Royal College 01 Physicians and Surgeons 01 Cana da 19