research methods

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
& reporting
Improving the reporting of pragmatic trials: an extension of
the CONSORT statement
Merrick Zwarenstein,1 2 3 Shaun Treweek,4 5 Joel J Gagnier5 6 Douglas G Altman, 7 Sean Tunis,8 9 10 Brian Haynes,11
Andrew D Oxman,5 David Moher,12 13 for the CONSORT and Pragmatic Trials in Healthcare (Practihc) groups
1
Health Services Sciences, Pragmatic trials are designed to inform decisions about practice, but poor reporting can reduce
Sunnybrook Hospital, Toronto,
Ontario, Canada their usefulness. The CONSORT and Practihc groups describe modifications to the CONSORT
Institute for Clinical Evaluative
guidelines to help readers assess the applicability of the results
2

Sciences, Toronto Department of

Health Policy, Management and
Evaluation, University of Toronto,
Toronto Abstract
3
Division of International Health Background
(IHCAR), Karolinska Institute, The CONSORT statement is intended to improve reporting of randomised controlled trials and focuses on minimising the risk
Stockholm, Sweden
of bias (internal validity). The applicability of a trial’s results (generalisability or external validity) is also important, particularly
4
Clinical and Population Sciences
for pragmatic trials. A pragmatic trial (a term first used in 1967 by Schwartz and Lellouch) can be broadly defined as a
and Education, University of
Dundee, Dundee randomised controlled trial whose purpose is to inform decisions about practice. This extension of the CONSORT statement is
5
Norwegian Knowledge Centre for intended to improve the reporting of such trials and focuses on applicability.
the Health Services, Oslo, Norway Methods
6
Faculty of Medicine, University At two, two-day meetings held in Toronto in 2005 and 2008, we reviewed the CONSORT statement and its extensions, the
of Toronto literature on pragmatic trials and applicability, and our experiences in conducting pragmatic trials.
7
Centre for Statistics in Medicine,
University of Oxford, Oxford Recommendations
8
Center for Medical Technology We recommend extending eight CONSORT checklist items for reporting of pragmatic trials: the background, participants,
Policy, Baltimore, MD, USA interventions, outcomes, sample size, blinding, participant flow, and generalisability of the findings. These extensions are
9
Division of General Internal presented, along with illustrative examples of reporting, and an explanation of each extension. Adherence to these reporting
Medicine, Johns Hopkins School of criteria will make it easier for decision makers to judge how applicable the results of randomised controlled trials are to
Medicine, Baltimore, MD their own conditions. Empirical studies are needed to ascertain the usefulness and comprehensiveness of these CONSORT
10
Center for Healthcare Policy, checklist item extensions. In the meantime we recommend that those who support, conduct, and report pragmatic trials
Stanford University School of should use this extension of the CONSORT statement to facilitate the use of trial results in decisions about health care.
Medicine, Palo Alto, CA, USA
11
Department of Clinical
Epidemiology and Biostatistics Randomised controlled trials are used to assess the ben‑ more, a trial may be valid and useful in the healthcare
and Department of Medicine, efits and harms of interventions in health care. If con‑ setting in which it was conducted but have limited appli‑
McMaster University Faculty of
Health Sciences, Hamilton, ON, ducted properly, they minimise the risk of bias (threats cability (also known as generalisability or external valid‑
Canada to internal validity), particularly selection bias.1 2 There is, ity) beyond this because of differences between the trial
12
Clinical Epidemiology Program, however, considerable evidence that trials are not always setting and other settings to which its results are to be
Ottawa Health Research Institute,
Ottawa, Canada
well reported,3 4 and this can be associated with bias, extrapolated.
13
Department of Epidemiology and such as selective reporting of outcomes.5 Schwartz and Lellouch6 coined the terms “pragmatic”
Community Medicine, Faculty of The usefulness of a trial report also depends on the to describe trials designed to help choose between
Medicine, University of Ottawa, clarity with which it details the relevance of its interven‑ options for care, and “explanatory” to describe trials
Ottawa
Correspondence to: M
tions, participants, outcomes, and design to the clinical, designed to test causal research hypotheses—for exam‑
Zwarenstein merrick. health service, or policy question it examines. Further‑ ple, that an intervention causes a particular biological
zwarenstein@ices.on.ca

Cite this as: BMJ Table 1 | Key differences between trials with explanatory and pragmatic attitudes, adapted from a table presented at the 2008 Society
2008;337:a2390 for Clinical Trials meeting by Marion Campbell, University of Aberdeen
doi: 10.1136/bmj.a2390
Question Efficacy—can the intervention work? Effectiveness—does the intervention work when used in
normal practice?
Setting Well resourced, “ideal” setting Normal practice
Participants Highly selected. Poorly adherent participants and those with Little or no selection beyond the clinical indication of interest
conditions which might dilute the effect are often excluded
Intervention Strictly enforced and adherence is monitored closely Applied flexibly as it would be in normal practice
Outcomes Often short term surrogates or process measures Directly relevant to participants, funders, communities, and
healthcare practitioners
Relevance to practice Indirect—little effort made to match design of trial to decision Direct—trial is designed to meet needs of those making
making needs of those in usual setting in which intervention decisions about treatment options in setting in which
will be implemented intervention will be implemented

BMJ | 20 December 2008 | Volume 337 1

RESEARCH METHODS & REPORTING

Table 2 | Comparison of trial that was highly explanatory in attitude with trial that was highly pragmatic
Highly explanatory attitude (NASCET7) Highly pragmatic attitude (Thomas et al8)

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
Question Among patients with symptomatic 70-99% stenosis of carotid artery can carotid endarterectomy plus Does a short course of acupuncture delivered by a qualified
best medical therapy reduce outcomes of major stroke or death over next two years compared with best acupuncturist reduce pain in patients with persistent non-
medical therapy alone? specific low-back pain?
Setting Volunteer academic and specialist hospitals with multidisciplinary neurological-neurosurgical teams General practice and private acupuncture clinics in UK
and high procedure volumes with low mortality in US and Canada
Participants Symptomatic patients stratified for carotid stenosis severity, with primary interest in severe carotid Anyone aged 18-65 with non-specific low back pain of 4-52
stenosis (high risk) group, who were thought to be most likely to respond to endarterectomy. Exclusions weeks’ duration who were judged to be suitable by their
included mental incompetence and another illness likely to cause death within 5 years. Patients also general practitioner. There were some exclusion criteria, eg
were temporarily ineligible if they had any of seven transient medical conditions (eg, uncontrolled those with spinal disease
hypertension or diabetes)
Intervention Endarterectomy had to be carried out (rather than stenting or some other operation), but the surgeon Acupuncturists determined the content and number of
was given leeway in how it was performed. Surgeons had to be approved by an expert panel, and were treatments according to patients’ needs
restricted to those who had performed at least 50 carotid endarterectomies in the past 24 months with
a postoperative complication rate (stroke or death within 30 days) of less than 6%. Centre compliance
with the study protocol was monitored, with the chief investigator visiting in the case of deficiencies
Outcomes The primary outcome was time to ipsilateral stroke, the outcome most likely to be affected by carotid Primary outcome was bodily pain as measured by SF-36.
endarterectomy. Secondary outcomes: all strokes, major strokes, and mortality Secondary outcomes included use of pain killers and patient
satisfaction
Relevance to practice Indirect—patients and clinicians are highly selected and it isn’t clear how widely applicable the results are Direct—general practitioners and patients can immediately
use the trial results in their decision making

change. Table 1 shows some key differences between about practice.

explanatory and pragmatic trials. Table 2 compares a
trial that was highly explanatory in attitude7 with one CONSORT initiative
that was highly pragmatic.8 There is a continuum rather The original CONSORT statement (www.consort-
than a dichotomy between explanatory and pragmatic statement.org), last revised in 2001, was developed
trials. In fact, Schwartz and Lellouch characterised by clinical trialists, methodologists, and medical jour‑
pragmatism as an attitude to trial design rather than a nal editors to help improve the reporting of parallel
characteristic of the trial itself. The pragmatic attitude (two) group randomised trials.39 The objective of the
favours design choices that maximise applicability of statement is to enable readers to critically appraise
the trial’s results to usual care settings, rely on unargu‑ and interpret trials by providing authors with guid‑
ably important outcomes such as mortality and severe ance about how to improve the clarity, accuracy,
morbidity, and are tested in a wide range of partici‑ and transparency of their trial reports. It consists of
pants.9‑11 As Schwartz and Lellouch wrote: “Most trials a 22-item checklist and a diagram, detailing the flow
done hitherto have adopted the explanatory approach of participants through the trial. It is a living docu‑
without question; the pragmatic approach would often ment that is updated as needed, incorporating new
have been more justifiable.”6 evidence.40 The guidelines have been endorsed by
Calls have been made for more pragmatic trials in more than 300 journals,41 and by several editorial
general,6 12 13 and in relation to specific clinical prob‑ groups, including the International Committee of
lems.14‑16 Articles have been published discussing the Medical Journal Editors.42 The CONSORT state‑
characteristics and value of pragmatic trials17‑35 or pro‑ ment has been translated into several languages.43
posing improvements in the design and conduct of Since its original publication in 1996 the quality of
these trials.36‑38 Patients, advocacy groups, clinicians, reports of controlled trials has improved.44
systematic reviewers, funders, and policymakers want The CONSORT recommendations are intention‑
to use the results of randomised controlled trials. As ally generic, and necessarily do not consider in detail
such, a clear description of the design and execution all types of trials. Extensions of the CONSORT state‑
of the trial, the intervention and comparator, and the ment have been developed for non-inferiority and
setting in which health care is provided may simplify equivalence,45 cluster randomised designs,46 report‑
their decision on the likely benefits, harms, and costs ing of abstracts,47 data on harms,48 trials of herbal
to be expected when implementing the intervention interventions,49 and of non-pharmacological interven‑
in their own situation. There is, however, no accepted tions,50 51 but not yet for the reporting of pragmatic
standard to guide reporting on the aspects of design and trials, although some issues pertaining to pragmatic
conduct of trials that affect their usefulness for decision trials were discussed in the CONSORT explanation
making, particularly considerations that would affect and elaboration paper.4
the applicability of the results.
We propose here guidance for reporting pragmatic Methods
trials, as a specific extension of the CONSORT state‑ In January 2005 and in March 2008, we held two-
ment. Our aim is to identify information which, if day meetings in Toronto, Canada, to discuss ways
included in reports of pragmatic trials, will help users to increase the contribution of randomised control‑
determine whether the results are applicable to their led trials to healthcare decision making, focusing on
own situation and whether the intervention might be pragmatic trials. Participants included people with
feasible and acceptable. Reporting this information is experience in clinical care, commissioning research,
crucial for any trial that is intended to inform decisions healthcare financing, developing clinical practice guide‑

2 BMJ | December 2008 | Volume 337

 research methods & reporting

Table 3 | Checklist of items for reporting pragmatic trials

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
Section Item Standard CONSORT description Extension for pragmatic trials
Title and abstract 1 How participants were allocated to interventions (eg, “random allocation,”
“randomised,” or “randomly assigned”)
Introduction
Background 2 Scientific background and explanation of rationale Describe the health or health service problem that the intervention
is intended to address and other interventions that may commonly
be aimed at this problem
Methods
Participants 3 Eligibility criteria for participants; settings and locations where the data were Eligibility criteria should be explicitly framed to show the degree to
collected which they include typical participants and/or, where applicable,
typical providers (eg, nurses), institutions (eg, hospitals),
communities (or localities eg, towns) and settings of care (eg,
different healthcare financing systems)
Interventions 4 Precise details of the interventions intended for each group and how and Describe extra resources added to (or resources removed from)
when they were actually administered usual settings in order to implement intervention. Indicate
if efforts were made to standardise the intervention or if the
intervention and its delivery were allowed to vary between
participants, practitioners, or study sites
Describe the comparator in similar detail to the intervention
Objectives 5 Specific objectives and hypotheses
Outcomes 6 Clearly defined primary and secondary outcome measures and, when Explain why the chosen outcomes and, when relevant, the length
applicable, any methods used to enhance the quality of measurements (eg, of follow-up are considered important to those who will use the
multiple observations, training of assessors) results of the trial
Sample size 7 How sample size was determined; explanation of any interim analyses and If calculated using the smallest difference considered important
stopping rules when applicable by the target decision maker audience (the minimally important
difference) then report where this difference was obtained
Randomisation—sequence 8 Method used to generate the random allocation sequence, including details
generation of any restriction (eg, blocking, stratification)
Randomisation—allocation 9 Method used to implement the random allocation sequence (eg, numbered
concealment containers or central telephone), clarifying whether the sequence was
concealed until interventions were assigned
Randomisation— 10 Who generated the allocation sequence, who enrolled participants, and who
implementation assigned participants to their groups
Blinding (masking) 11 Whether participants, those administering the interventions, and those If blinding was not done, or was not possible, explain why
assessing the outcomes were blinded to group assignment
Statistical methods 12 Statistical methods used to compare groups for primary outcomes; methods
for additional analyses, such as subgroup analyses and adjusted analyses
Results
Participant flow 13 Flow of participants through each stage (a diagram is strongly The number of participants or units approached to take part in
recommended)—specifically, for each group, report the numbers of the trial, the number which were eligible, and reasons for non-
participants randomly assigned, receiving intended treatment, completing participation should be reported
the study protocol, and analysed for the primary outcome; describe
deviations from planned study protocol, together with reasons
Recruitment 14 Dates defining the periods of recruitment and follow-up
Baseline data 15 Baseline demographic and clinical characteristics of each group
Numbers analysed 16 Number of participants (denominator) in each group included in each
analysis and whether analysis was by “intention-to-treat”; state the results in
absolute numbers when feasible (eg, 10/20, not 50%)
Outcomes and estimation 17 For each primary and secondary outcome, a summary of results for each
group and the estimated effect size and its precision (eg, 95% CI)
Ancillary analyses 18 Address multiplicity by reporting any other analyses performed, including
subgroup analyses and adjusted analyses, indicating which are prespecified
and which are exploratory
Adverse events 19 All important adverse events or side effects in each intervention group
Discussion
Interpretation 20 Interpretation of the results, taking into account study hypotheses, sources
of potential bias or imprecision, and the dangers associated with multiplicity
of analyses and outcomes
Generalisability 21 Generalisability (external validity) of the trial findings Describe key aspects of the setting which determined the trial
results. Discuss possible differences in other settings where
clinical traditions, health service organisation, staffing, or
resources may vary from those of the trial
Overall evidence 22 General interpretation of the results in the context of current evidence

lines, and trial methodology and reporting. Twenty four ing some of those invited to the meeting but unable to
people participated in 2005 and 42 in 2008, including attend. After several revisions the writing group pro‑
members of the CONSORT and Pragmatic Trials in duced a draft summary paper. At the 2008 meeting
Healthcare (Practihc) groups.52 the draft was discussed and modified. It was circulated
After the 2005 meeting a draft revised checklist for to the CONSORT group for feedback, modified, and
the extension was circulated to a writing group, includ‑ submitted for publication.

BMJ | 20 December 2008 | Volume 337 3

RESEARCH METHODS & REPORTING

Recommendations for reporting pragmatic trials mention the intervention under investigation and the
Meeting participants agreed that no items needed to be usual alternative(s) in relevant settings. To help place

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
added to the CONSORT checklist and that the flow dia‑ the trial in the context of other settings authors should
gram did not need modification. However, participants explain key features that make the intervention feasible
felt that eight items (2-4, 6, 7, 11, 13, and 21) needed in their trial setting and elsewhere (such as, the wide‑
additional text specific to the reporting of pragmatic trials spread availability of the trial drug, the availability of
(see table 3). Although participants discussed additional trained staff to deliver the intervention, electronic data‑
text for item 1 of the checklist (title/abstract), principally bases that can identify eligible patients).
adding the word pragmatic to the title or abstract, we
decided against making this recommendation because Item 3: methods; participants
it may reinforce the misconception that there is a Eligibility criteria for participants and the settings and the loca-
dichotomy between pragmatic and explanatory trials tions where the data were collected
rather than a continuum. We elected not to extend item Extension for pragmatic trials: Eligibility criteria should
5 (objectives), although we would encourage trialists to be explicitly framed to show the degree to which they
report the purpose of the trial in relation to the decisions include typical participants and, where applicable, typi‑
that it is intended to inform and in which settings; we cal providers (eg, nurses), institutions (eg, hospitals), com‑
have included this recommendation in connection with munities (or localities eg, towns) and settings of care (eg,
the extension of item 2 (background). different healthcare financing systems).
For each of the eight items we present the standard Examples—“The study population included all National
CONSORT text and additional guidance, an example Health System physicians in the Northern Region of
of good reporting for the item, and an explanation of Portugal except for those not involved in any clinical
the issues. The selection of examples is illustrative for a activity (eg, administrators, laboratory analysis); those
specific item and should not be interpreted as a marker of working in substance abuse and rehabilitation cent‑
quality for other aspects of those trial reports. The sugges‑ ers or specialty hospitals (because they cover multiple
tions in this paper should be seen as additional to the gen‑ geographical areas); and those working at the regional
eral guidance in the main CONSORT e­ xplanatory paper pharmacosurveillance center or any department having
and where relevant, other CONSORT ­guidance. a specific voluntary ADR reporting program.”55
“Our study took place in the three public hospitals
Item 2: introduction; background (totalling 850 beds) in southern Adelaide, Australia, with
Scientific background and explanation of rationale a regional population of about 350 000. In Australia,
Extension for pragmatic trials: Describe the health or entry to long term care (nursing home) can occur only
health service problem that the intervention is intended after an independent clinical assessment by the aged
to address, and other interventions that may commonly care assessment team (ACAT), who determine level of
be aimed at this problem. dependency.”56
Example (a): Describe the health or health service problem Explanation—Treatments may perform better when
which the intervention is intended to address—“Although inter‑ evaluated among selected, highly adherent patients with
ventions such as telephone or postal reminders from severe but not intractable disease and few comorbidities.
pharmacists improve compliance their effect on clinical Reports of these restricted trials may be of limited appli‑
outcome is not known. We investigated whether periodic cability. Excessively stringent inclusion and exclusion
telephone counselling by a pharmacist . . . reduced mor‑ criteria reduce the applicability of the results and may
tality in patients” receiving polypharmacy.53 result in safety concerns,57 so the method of recruitment
Explanation—Users of pragmatic trial reports seek to should be completely described. This stringency seems
solve a health or health service problem in a particular to be reducing over time but remains a problem.58
setting. The problem at which the intervention is tar‑ In some trials the unit of randomisation and interven‑
geted should thus be described. This enables readers tion might be healthcare practitioners, communities, or
to understand whether the problem confronting them healthcare institutions such as clinics (that is, cluster ran‑
is similar to the one described in the trial report, and domised pragmatic trials). In these trials volunteer institu‑
thus whether the study is relevant to them. Ideally, the tions may be atypically well resourced or experienced,
report should state that the trial is pragmatic in attitude successful innovators. Since the feasibility and success of
(and why) and explain the purpose of the trial in relation‑ an intervention may depend on attributes of the health‑
ship to the decisions that it is intended to inform and in care system and setting, reporting this information ena‑
which settings. bles readers to assess the relevance and applicability of
Example (b): Describe other interventions that may commonly the results in their own, possibly different, settings.
be aimed at this problem—“Sublingual buprenorphine is
increasingly being prescribed by General Practitioners Item 4: methods; interventions
for opiate detoxification, despite limited clinical and Precise details of the interventions intended for each group and
research evidence. Comparing methadone, dihydroco‑ how and when they were actually administered.
deine and buprenorphine it is important to note several Extension for pragmatic trials: Describe extra resources
factors which may impact upon prescribing and use of added to (or resources removed from) usual settings in
these agents”.54 order to implement the intervention. Indicate if efforts
Explanation—The background of the trial report should were made to standardise the intervention or if the inter‑

4 BMJ | December 2008 | Volume 337

 research methods & reporting

vention and its delivery were allowed to vary between care from the healthcare team and completed all out‑
participants, practitioners or study sites. Describe the come measures on the same time frame as the interven‑

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
comparator in similar detail to the intervention. tion group. After randomisation, this group received a
Example: (a) Describe extra resources added to (or resources two page leaflet entitled “Exercise after cancer diagno‑
removed from) usual settings in order to implement the inter- sis,” which provided safe guidelines. After the six month
vention—“The hospitals and a private long term care follow-up, these women were helped to construct their
provider developed and ran the off-site transitional care own personalised exercise plan and invited to join a local
facility, which was 5-25 km from the study hospitals. general practice exercise referral scheme.”63
The private provider supplied accommodation, cater‑ Explanation—In a randomised controlled trial the effects
ing, cleaning, nursing (5.0 full time equivalents in 24 of the intervention are always related to a comparator.
hours), and career staff (10.0 full time equivalents in 24 To increase applicability, and feasibility, pragmatic trials
hours) while the hospitals provided the allied health often compare new interventions to usual care. The cho‑
staff (4.4 full time equivalents), medical staff, and a tran‑ sen comparator should be described in sufficient detail
sitional care nurse coordinator (1.0 full time equivalent). for readers to assess whether the incremental benefits or
The whole team assessed all patients on admission to harms reported are likely to apply in their own setting,
the transitional care unit and had weekly case confer‑ where usual care may be more, or less, effective.
ences. Specialist medical staff visited the site for the
case conferences and reviewed all admissions. On-call Item 6: methods; outcomes
medical care was available 24 hours a day.”56 Clearly defined primary and secondary outcome measures, and,
Explanation—If the extra resources to deliver the inter‑ when applicable, any methods used to enhance the quality of
vention are not described, readers cannot judge the fea‑ measurements (eg, multiple observations, training of assessors)
sibility of the intervention in their own setting. When Extension for pragmatic trials: Explain why the chosen
relevant, authors should report details (experience, train‑ outcomes and, when relevant, the length of follow-up are
ing etc) of those who delivered the intervention51 and its considered important to those who will use the results
frequency and intensity. If multicomponent interventions of the trial.
are being evaluated, details of the different components Example—“The patient-based outcomes used in the
should be described. evaluation were selected on the basis of empirical evi‑
Example: (b) Indicate if efforts were made to standardise the dence from consumers about the most important out‑
intervention or if the intervention and its delivery were allowed comes from SDM [shared decision making] and risk
to vary between participants, practitioners or study sites—“Two communication.”64
trained leaders introduced a structured sequence of top‑ The total number of days off work in the year after
ics using a collaborative approach. All leaders had run inclusion was calculated for each patient. Days off were
at least one previous group. Throughout the 12 week defined as days 100% compensated by the NIA [National
programme leaders received three hours of supervision Insurance Administration]. Thus, days on ASL [Active
each week from a certified trainer.”59 Sick Leave] were considered as days absent. After a full
Explanation—In explanatory trials the intervention is year of sick leave, administrative proceedings are initi‑
standardised, and thus the results may not apply under ated to transfer the beneficiary to other measures of reha‑
usual conditions of care where no such standardisation is bilitation or disability pension within the NIA system.
enforced. Pragmatic trials are conducted in typical care One year of absence was therefore a proxy measure for
settings, and so care may vary between similar partici‑ long-term disability.”65
pants, by chance, by practitioner preference, and accord‑ Explanation—The primary outcome(s)66 in pragmatic
ing to institutional policies.60 For pragmatic trials, efforts trials are chosen to be relevant to the participants and
that may reduce this natural variation in the intervention key decision makers at whom the trial is aimed. The
and its delivery should be described. However, if reduc‑ length of follow-up should be appropriate to the decision
ing variation in a care process or shifting practice patterns the trial is designed to inform. If the target decision mak‑
is itself the main purpose of the intervention, this should ers are patients and their clinicians, the primary outcome
be explicit in the title, abstract, and introduction. is likely to be a health outcome, while trials aimed at
Regardless of the extent to which the intervention was policymakers and institutional leaders may focus on a
standardised, pragmatic trials should describe the inter‑ process or system efficiency or equity outcome. Explic‑
vention in sufficient detail that it would be possible for itly indicating that the chosen outcome is important to
someone to replicate it, or include a reference or link to decision makers, and specifying the decision makers to
a detailed description of the intervention. Unfortunately, whom it was important will assist other readers to decide
this information is often lacking in reports of trials.61 whether the results are relevant to them.
Examples: (c) Describe the comparator in similar detail
to the intervention—“Standard advice was given as for Item 7: methods; sample size
the naproxen group. Participants were provided with How sample size was determined; when applicable, explanation
co­codamol for additional pain relief and an informa‑ of any interim analyses and stopping rules
tion leaflet about “tennis elbow” based on the Arthritis Extension for pragmatic trials: If calculated using the
Research Campaign publication but omitting specific smallest difference considered important by the target
treatment recommendations.”62 decision maker audience (the minimally important differ‑
“Women assigned to the control group received usual ence) then report where this difference was obtained.

BMJ | 20 December 2008 | Volume 337 5

RESEARCH METHODS & REPORTING

Example—“There were no previous data using the main ipation should be reported.
outcome measure on which to base the sample size cal‑ Example—“These practices ascertained 3392 registered

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
culation, and therefore the sample size was calculated patients with Parkinson’s disease; 3124 were eligible for
on the number of days with URTI [upper respiratory study of whom 1859 (59.5%) agreed to participate (fig 3).
tract infection]. It was decided, in line with other rigor‑ Twenty-three patients died during recruitment, leaving
ous pragmatic studies that the smallest difference worth 1836 patients when the intervention began. Seventeen
detecting was a 20% reduction in number of days with of the 1836 patients were not traced at the NHS cen‑
URTI.”67 tral registry and are therefore not included in mortality
Explanation—The minimally important difference analyses”.69
(MID) is the size of a change in the primary outcome Explanation—The more similar the participants, prac‑
which would be important to the key decision making titioners, or other units of intervention or randomisation
audience. The MID may differ between settings, conse‑ are to those in usual care, the more likely that the results
quently readers need to know what MID was considered of the trial will be applicable to usual care. Consequently
important in the trial setting, and by whom, to contrast the text and/or the trial flow diagram should mention, if
with their own expectations. known, the number of participants or units approached
to take part in the trial, the number whom were eligible,
Item 11: methods; blinding (masking) and reasons for non-participation. Although this informa‑
Whether participants, those administering the interventions, and tion is requested in the CONSORT statement, the need
those assessing the outcomes were blinded to group assignment for it is greater when reporting a pragmatic trial.
Extension for pragmatic trials: If blinding was not done,
or was not possible, explain why. Item 21: generalisability (applicability, external validity)
Example—“Randomisation was done by telephone to Generalisability of the trial findings
an interactive voice response system. We entered and Extension for pragmatic trials: Describe key aspects of
managed all data in an anonymised format; we held the setting which determined the trial results. Discuss
data on patient contacts and other administrative data possible differences in other settings where clinical tradi‑
in a separate database. The study was a pragmatic, ran‑ tions, health service organisation, staffing, or resources
domised, prospective, open trial. In exercise studies, may vary from those of the trial.
blinding the participants to allocation is not possible. Examples—“The intervention was tailored to the spe‑
We took steps to blind the evaluation of outcomes by cific study population and may not be as effective in a
having questionnaire responses in sealed envelopes and different group. The positive results may reflect in part
ensuring that outcome measures were taken by research‑ unique aspects of the Portuguese health care system or
ers who were not involved in exercise classes.”63 the regional physician culture. Willingness to report
Explanation—In explanatory trials blinding68 prevents adverse drug reactions may be less in countries in which
belief in the effectiveness of the intervention (by par‑ there is greater concern about malpractice liability.”55
ticipant, clinician and/or assessor) from confounding the “The incentive for implementing the clinical path‑
causal link between the intervention and the primary way will be different for a single-payer third-party
outcome. In pragmatic trials, as in the real world deliv‑ system, as exists in Canada, in which costs of the
ery of care, blinding of participants and clinicians may pathway and offsetting hospital costs are realized by
be impossible. Belief (or disbelief) in the intervention, the same payer, than for a multiple payer system as
extra enthusiasm and effort (or less), and optimism (or exists in the United States, in which hospital cost off‑
pessimism) in the self-assessment of outcomes may thus sets will be realized by the hospital and not the nurs‑
add to (or detract from) the effects of an intervention. ing home payer.”70
Pragmatic trials may incorporate these factors into the Explanation—The usefulness of the trial report
estimate of effectiveness, rendering the findings more is critically dependent on how applicable the trial
applicable to usual care settings. Authors should specu‑ and its results are and how feasible the intervention
late on the effect of any suspected modifying factors, would be. The authors are well placed to suggest how
such as belief in the intervention, in the discussion (item feasible the intervention might be, which aspects of
20). Moreover, in pragmatic trials, it is still desirable and their setting were essential to achieve the trial result,
often possible to blind the assessor or obtain an objective and how that result might differ in other settings. The
source of data for evaluation of outcomes. applicability of the study result could be encapsulated
here by reference to the setting (is it a usual care set‑
Item 13: results; participant flow ting), the participants and providers (how selected
Flow of participants through each stage (a diagram is strongly were they), intensity of intervention and follow up
recommended). Specifically, for each group report the numbers (how much like usual care was this), adherence to
of participants randomly assigned, receiving intended treatment, the intervention and whether efforts were made to
completing the study protocol, and analyzed for the primary standardise its delivery, the use of intention to treat
outcome. Describe protocol deviations from study as planned, analysis, and the amount of loss to follow up. Feasi‑
together with reasons bility can be encapsulated by reference to economic,
Extension for pragmatic trials: The number of partici‑ political, and logistic barriers to implementation and
pants or units approached to take part in the trial, the by the range of settings and societies in which these
number which were eligible and reasons for non-partic‑ barriers would be low.

6 BMJ | December 2008 | Volume 337

 research methods & reporting

Discussion Needham, Dave Sackett, Kevin Thorpe, and the Practihc and CONSORT
As demand rises for more pragmatic trials to inform real groups for their contributions.

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
world choices,13 so too does the need to ensure that the Funding: The Practihc group was supported by the European Commission’s
5th Framework INCO programme, contract ICA4-CT-2001-10019. The 2005
results are clearly reported. Readers need to be able to Toronto meeting was supported by the Canadian Institutes for Health Research
evaluate the validity of the results, the extent to which grant number FRN 63095. The 2008 Toronto meeting was supported by
they are applicable to their settings, and the feasibility the UK Medical Research Council, the Centre for Health Services Sciences at
of the tested interventions. The existing CONSORT Sunnybrook Hospital, Toronto, Canada, the Centre for Medical Technology
Policy, Balitimore, Maryland, USA and the National Institute for Health and Clinical
statement applies fully and directly to pragmatic trials. Excellence, UK. DM is supported by a University of Ottawa research chair; much
Here we have proposed extensions for eight items in the of his contribution to this paper was while he was with the Children’s Hospital of
statement to make more explicit the important attributes Eastern Ontario Research Institute. DA is supported by Cancer Research UK.
of pragmatic trials and thus to ease the task of users in Competing interests: None declared.
assessing feasibility, relevance, and likely effects of the Provenance and peer review: Not commissioned; externally peer reviewed.
intervention in their own setting. 1 Altman DG, Bland JM. Statistics notes. Treatment allocation in controlled
We reached consensus that the trial results are likely trials: why randomise? BMJ 1999;318:1209.
2 Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L, Naylor CD,
to be more widely applicable if the participants, com‑ et al. Users’ guides to the medical literature: XXV. Evidence-based
munities, practitioners, or institutions were not narrowly medicine: principles for applying the users’ guides to patient care. JAMA
selected; if the intervention was implemented without 2000;284:1290-6.
3 Chan AW, Altman DG. Epidemiology and reporting of randomised trials
intense efforts to standardise it; if the comparator group published in PubMed journals. Lancet 2005;365:1159-62.
received care or other interventions already widely used; 4 Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne DG, et
al. The revised CONSORT statement for reporting randomized trials:
and if the outcomes studied were of importance to the explanation and elaboration. Ann Intern Med 2001;134:663-94.
relevant decision makers. The intervention needs to be 5 Gluud LL. Bias in clinical intervention research. Am J Epidemiol
precisely described if readers are to be able to assess its 2006;163:493-501.
6 Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in
feasibility. therapeutical trials. J Chronic Dis 1967;20:637-48.
The multiplicity and independence of the elements 7 North American Symptomatic Carotid Endarterectomy Trial
Collaborators. Beneficial effect of carotid endarterectomy in
constituting the design of pragmatic trials guarantee symptomatic patients with high-grade carotid stenosis. N Engl J Med
that pragmatism is not an all or none attribute; rather, 1991;325:445-53.
it might be best conceived as a continuum along sev‑ 8 Thomas KJ, MacPherson H, Thorpe L, Brazier J, Fitter M, Campbell
MJ, et al. Randomised controlled trial of a short course of traditional
eral dimensions. For example, a randomised trial could acupuncture compared with usual care for persistent non-specific low
have broad inclusion criteria for participants but rely back pain. BMJ 2006;333:623-8.
9 Coca SG, Krumholz HM, Garg AX, Parikh CR. Underrepresentation of
primarily on a short term, physiological outcome rather renal disease in randomized controlled trials of cardiovascular disease.
than one that is more meaningful to the participants. JAMA 2006;296:1377-84.
10 Lee PY, Alexander KP, Hammill BG, Pasquali SK, Peterson ED.
Alternatively, a trial might include a wide range of par‑ Representation of elderly persons and women in published randomized
ticipants, meaningfully assess the effect, but evaluate an trials of acute coronary syndromes. JAMA 2001;286:708-13.
intervention that is enforced or tightly monitored and 11 Zarin DA, Young JL, West JC. Challenges to evidence-based medicine:
a comparison of patients and treatments in randomized controlled
thus not widely feasible. Other permutations probably trials with patients and treatments in a practice research network. Soc
exist. It is not the case that more pragmatic is always Psychiatry Psychiatr Epidemiol 2005;40:27-35.
12 Liberati A. The relationship between clinical trials and clinical
better; a trial’s design should be such that the results will practice: the risks of underestimating its complexity. Stat Med
meet the needs of the intended users. A trial intended to 1994;13:1485-91.
inform a research decision about the biological effect of 13 Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the
value of clinical research for decision making in clinical and health
a new drug is likely to be more explanatory in design. policy. JAMA 2003;290:1624-32.
At a later date, a trial of that same drug aimed at helping 14 Marson A, Kadir Z, Chadwick D. Large pragmatic randomised studies of
new antiepileptic drugs are needed. BMJ 1997;314:1764.
patients, practitioners, or policymakers to decide whether 15 Williams H. Fungal infections of skin and nails of feet. Pragmatic clinical
it should be prescribed is likely to be more pragmatic in trial is now needed. BMJ 1999;319:1070-1.
design. To help display this multidimensionality, we have 16 Lieberman JA, Stroup TS. Guest editors’ introduction: what can large
pragmatic clinical trials do for public mental health care? Schizophr Bull
developed of a tool, primarily intended to be used in 2003;29:1-6.
designing a trial, for characterising where it will stand 17 Roland M, Torgerson DJ. What are pragmatic trials? BMJ 1998;316:285.
18 Rothwell PM. External validity of randomised controlled trials: “to whom
along the pragmatic-explanatory continuum in relation do the results of this trial apply?” Lancet 2005;365:82-93.
to each design decision.71 19 Macpherson H. Pragmatic clinical trials. Complement Ther Med
2004;12:136-40.
We hope that these reporting guidelines will help edi‑ 20 Godwin M, Ruhland L, Casson I, MacDonald S, Delva D, Birtwhistle R,
tors, reviewers, trialists, and policy makers in reporting, et al. Pragmatic controlled clinical trials in primary care: the struggle
reviewing, and using pragmatic trials. Journals that have between external and internal validity. BMC Med Res Methodol
2003;3:28.
endorsed the CONSORT statement could also support 21 Hotopf M, Churchill R, Lewis G. Pragmatic randomised controlled trials in
CONSORT for pragmatic trials, by including refer‑ psychiatry. Br J Psychiatry 1999;175:217-23.
22 Engel CC. Explanatory and pragmatic perspectives regarding
ence to this extension paper in the journal’s instructions idiopathic physical symptoms and related syndromes. CNS Spectr
to authors. We also invite editorial groups to consider 2006;11:225-32.
endorsing the CONSORT extension for pragmatic trials 23 Whittaker K, Sutton C, Burton C. Pragmatic randomised controlled
trials in parenting research: the issue of intention to treat. J Epidemiol
and encourage authors to adhere to it. Up to date ver‑ Community Health 2006;60:858-64.
sions of all CONSORT guidelines can be found on the 24 Coutinho ES, Huf G, Bloch KV. [Pragmatic clinical trials: an option
in the construction of health-related evidence]. Cad Saude Publica
CONSORT website (www.consort-statement.org). 2003;19:1189-93.
We thank Eduardo Bergel, Curt Furberg, Jeremy Grimshaw, Jan Hux, Andreas 25 Ernst E, Canter PH. Limitations of “pragmatic” trials. Postgrad Med J
Laupacis, John Lavis, Simon Lewin, Carl Lombard, Malcolm Maclure, Dale 2005;81:203.

BMJ | 20 December 2008 | Volume 337 7

RESEARCH METHODS & REPORTING

26 Vallve C. [A critical review of the pragmatic clinical trial]. Med Clin (Barc) 52 Pragmatic Randomized Controlled Trials in HealthCare. www.
2003;121:384-8. practihc.org.

BMJ: first published as 10.1136/bmj.a2390 on 11 November 2008. Downloaded from http://www.bmj.com/ on 10 April 2024 by guest. Protected by copyright.
27 Thompson SG, Barber JA. How should cost data in pragmatic 53 Wu JY, Leung WY, Chang S, Lee B, Zee B, Tong PC, et al. Effectiveness
randomised trials be analysed? BMJ 2000;320:1197-200. of telephone counselling by a pharmacist in reducing mortality in
28 Silverman WA. Sample size, representativeness, and credibility in patients receiving polypharmacy: randomised controlled trial. BMJ
pragmatic neonatal trials. Am J Perinatol 1987;4:129-30. 2006;333:522.
29 Andersen B. [Principles of medical statistics. XII. Causal or pragmatic 54 Wright NM, Sheard L, Tompkins CN, Adams CE, Allgar VL, Oldham NS.
clinical research?]. Nord Med 1971;86:1434-6. Buprenorphine versus dihydrocodeine for opiate detoxification in
30 Flamant R. [From the explanatory to the pragmatic]. Bull Cancer primary care: a randomised controlled trial. BMC Fam Pract 2007;8:3.
1987;74:169-76. 55 Figueiras A, Herdeiro MT, Polonia J, Gestal-Otero JJ. An educational
31 Eschwege E, Bouvenot G. [Explanatory or pragmatic trials, the dualism]. intervention to improve physician reporting of adverse drug reactions: a
Rev Med Interne 1994;15:357-61. cluster-randomized controlled trial. JAMA 2006;296:1086-93.
32 Califf RM, Woodlief LH. Pragmatic and mechanistic trials. Eur Heart J 56 Crotty M, Whitehead CH, Wundke R, Giles LC, Ben-Tovim D, Phillips PA.
1997;18:367-70. Transitional care facility for elderly people in hospital awaiting a long
33 Resch K. Pragmatic randomised controlled trials for complex therapies. term care bed: randomised controlled trial. BMJ 2005;331:1110.
Forsch Komplementarmed 1998;5(suppl S1):136-9. 57 Juurlink DN, Mamdani MM, Lee DS, Kopp A, Austin PC, Laupacis A, et al.
34 Charlton BG. Understanding randomized controlled trials: explanatory Rates of hyperkalemia after publication of the randomized aldactone
or pragmatic? Fam Pract 1994;11:243-4. evaluation study. N Engl J Med 2004;351:543-51.
35 Algra A, van der Graff Y. [Roaming through methodology. XII. Pragmatic 58 Van Spall HG, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized
and pathophysiologic trials: a question of goal formulation]. Ned controlled trials published in high-impact general medical journals: a
Tijdschr Geneeskd 1999 6;143:514-7. systematic sampling review. JAMA 2007;297:1233-40.
36 Green LW, Glasgow RE. Evaluating the relevance, generalization, and 59 Hutchings J, Gardner F, Bywater T, Daley D, Whitaker C, Jones K, et al.
applicability of research: issues in external validation and translation Parenting intervention in Sure Start services for children at risk of
methodology. Eval Health Prof 2006;29:126-53. developing conduct disorder: pragmatic randomised controlled trial.
37 Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA. Practical BMJ 2007;334:678.
clinical trials for translating research to practice: design and 60 Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can
measurement recommendations. Med Care 2005;43:551-7. a randomised controlled trial be? BMJ 2004;328:1561-3.
38 Tansella M, Thornicroft G, Barbui C, Cipriani A, Saraceno B. Seven 61 Schroter S, Glasziou PP, Heneghan CJ. Quality of descriptions of
criteria for improving effectiveness trials in psychiatry. Psychol Med treatments: a review of published randomised controlled trials. BMJ (in
2006;36:711-20. press).
39 CONSORT Group. CONSORT statement 2007. www.consort-statement. 62 Hay EM, Paterson SM, Lewis M, Hosie G, Croft P. Pragmatic randomised
org/?o=1004. controlled trial of local corticosteroid injection and naproxen for
40 Moher D, Schulz KF, Altman DG. The CONSORT statement: revised treatment of lateral epicondylitis of elbow in primary care. BMJ
recommendations for improving the quality of reports of parallel-group 1999;319:964-8.
randomised trials. Lancet 2001;357:1191-4. 63 Mutrie N, Campbell AM, Whyte F, McConnachie A, Emslie C, Lee L, et
41 CONSORT Group. CONSORT endorsers—journals. 2007. www.consort- al. Benefits of supervised group exercise programme for women being
statement.org/index.aspx?o=1096.
treated for early stage breast cancer: pragmatic randomised controlled
42 Davidoff F. News from the International Committee of Medical Journal
trial. BMJ 2007;334:517.
Editors. Ann Intern Med 2000;133:229-31.
64 Edwards A, Elwyn G, Hood K, Atwell C, Robling M, Houston H, et al.
43 CONSORT Group. CONSORT translations. 2007. www.consort-statement.
Patient-based outcome results from a cluster randomized trial of shared
org/?o=1216.
decision making skill development and use of risk communication aids
44 Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does
in general practice. Fam Pract 2004;21:347-54.
the CONSORT checklist improve the quality of reports of randomised
65 Scheel IB, Hagen KB, Herrin J, Carling C, Oxman AD. Blind faith? The
controlled trials? A systematic review. Med J Aust 2006;185:263-7.
effects of promoting active sick leave for back pain patients. A cluster-
45 Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ. Reporting of
randomised trial. Spine 2002;27:2734-40.
noninferiority and equivalence randomized trials: an extension of the
66 Roland M, Torgerson D. Understanding controlled trials: what outcomes
CONSORT statement. JAMA 2006;295:1152-60.
46 Campbell MK, Elbourne DR, Altman DG. CONSORT statement: extension should be measured? BMJ 1998;317:1075.
to cluster randomised trials. BMJ 2004;328:702-8. 67 Steinsbekk A, Fonnebo V, Lewith G, Bentzen N. Homeopathic care for the
47 Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. prevention of upper respiratory tract infections in children: a pragmatic,
CONSORT for reporting randomised trials in journal and conferences randomised, controlled trial comparing individualised homeopathic
abstracts. Lancet 2008;371:281-3. care and waiting-list controls. Complement Ther Med 2005;13:231-8.
48 Ioannidis JP, Evans SJ, Gotzsche PC, O’Neill RT, Altman DG, Schulz K, et 68 Schulz KF, Altman DG, Moher D. Blinding is better than masking. BMJ
al. Better reporting of harms in randomized trials: an extension of the 2007:334:918.
CONSORT statement. Ann Intern Med 2004;141:781-8. 69 Jarman B, Hurwitz B, Cook A, Bajekal M, Lee A. Effects of community
49 Gagnier JJ, Boon H, Rochon P, Moher D, Barnes J, Bombardier C. based nurses specialising in Parkinson’s disease on health outcome
Reporting randomized, controlled trials of herbal interventions: an and costs: randomised controlled trial. BMJ 2002;324:1072-5.
elaborated CONSORT statement. Ann Intern Med 2006;144:364-7. 70 Loeb M, Carusone SC, Goeree R, Walter SD, Brazil K, Krueger P, et
50 Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, the CONSORT al. Effect of a clinical pathway to reduce hospitalizations in nursing
Group. Reporting of nonpharmacological treatment interventions: an home residents with pneumonia: a randomized controlled trial. JAMA
extension of the CONSORT statement. Ann Intern Med 2008;w60-66. 2006;295:2503-10.
51 Boutron I, Moher D, Altman DG, Schulz KF, Ravaud P, the CONSORT 71 Thorpe KE, Zwarenstein M, Oxman A, Treweek S, Furberg CD, Altman DG,
Group. Extending the CONSORT statement to trials reporting et al. A proposal for graphing randomized controlled trials within the
nonpharmacological treatments: extension and elaboration. Ann Intern pragmatic-explanatory continuum: PRECIS. J Clin Epidemiol (in press).
Med 2008;148:295-309. Accepted: 2 October 2008

8 BMJ | December 2008 | Volume 337