ARTICLE

Improving Adherence to Otitis Media Guidelines With

Clinical Decision Support and Physician Feedback
AUTHORS: Christopher B. Forrest, MD, PhD,a,b Alexander G. WHAT’S KNOWN ON THIS SUBJECT: Expectations are high that
Fiks, MD, MSCE,a L. Charles Bailey, MD, PhD,a Russell electronic health record–based clinical decision support and
Localio, PhD,c Robert W. Grundmeier, MD,a Thomas performance feedback will improve adherence to guidelines by
Richards, MS,d Dean J. Karavite, MS,a Lisa Elden, MD,e and delivering relevant and actionable information to clinicians. Few
Evaline A. Alessandrini, MD, MSCEf studies have evaluated these assertions or examined the
Departments of aPediatrics, and eSurgery, Children’s Hospital of combined effects of decision support and feedback.
Philadelphia, Philadelphia, Pennsylvania; bLeonard Davis Institute
of Health Economics, and cDepartment of Biostatics and
Epidemiology, School of Medicine, University of Pennsylvania,
WHAT THIS STUDY ADDS: Clinical decision support customized to
Philadelphia, Pennsylvania; dCenter for Population Health a patient’s history and presentation and performance feedback
Information Technology, Johns Hopkins Bloomberg School of are both effective for improving adherence to guidelines for otitis
Public Health, Baltimore, Maryland; and fDepartment of media. However, the combination of the 2 interventions is no
Pediatrics, Cincinnati Children’s Hospital Medical Center, better than either delivered alone.
Cincinnati, Ohio
KEY WORDS
otitis media, electronic health record, clinical decision support,
physician performance feedback, guidelines
ABBREVIATIONS
AOM—acute otitis media abstract
CDS—clinical decision support
OBJECTIVE: To assess the effects of electronic health record–based
EHR—electronic health record
OM—otitis media clinical decision support (CDS) and physician performance feedback
OME—otitis media with effusion on adherence to guidelines for acute otitis media (AOM) and otitis
Dr Forrest led the design, conduct, and analysis of the study and media with effusion (OME).
was the primary author for the manuscript. Dr Fiks provided
conceptual input into all aspects of the study, wrote sections of
METHODS: We conducted a factorial-design cluster randomized trial with
the manuscript, and approved the final version. Dr Bailey led the primary care practices (n = 24) as the unit of randomization and visits as
programming team that developed the otitis media episode of the unit of analysis. Between December 2007 and September 2010, data
care grouping software, provided conceptual input into all
were collected from 139 305 otitis media visits made by 55 779 children
aspects of the study, critically edited the manuscript, and
approved the final version. Dr Localio conducted the statistical aged 2 months to 12 years. When activated, the CDS system provided
analyses for the study, wrote sections of the manuscript, and guideline-based recommendations individualized to the patient’s history
approved the final version. Dr Grundmeier designed and led the and presentation. Monthly physician feedback reported adherence to
development of the decision support system, wrote sections of
the manuscript, provided input into all aspects of the study, guideline-based care, changes over time, and comparisons to others
critically edited the manuscript, and approved the final version. in the practice and network.
Mr Richards provided substantive intellectual input into the
design and analysis of the guideline adherence metrics, helped
RESULTS: Comprehensive care (all recommended guidelines were ad-
to program the episode-of-care software and clinical decision hered to) was accomplished for 15% of AOM and 5% of OME visits dur-
support system, critically edited the manuscript, and approved ing the baseline period. The increase from baseline to intervention
the final version. Mr Karavite created the user interface for the
periods in adherence to guidelines was larger for CDS compared with
clinical decision support system, provided intellectual input into
the conduct of the study and data analysis, critically edited the non-CDS visits for comprehensive care, pain treatment, adequate
manuscript, and approved the final version. Dr Elden co-led diagnostic evaluation for OME, and amoxicillin as first-line therapy
development of the local translation of the otitis media for AOM. Although performance feedback was associated with
guidelines, provided intellectual input into all aspects of the
study, critically edited the manuscript, and approved the final improved antibiotic prescribing for AOM and pain treatment, the
version. Dr Alessandrini co-led the conduct of the study, joint effects of CDS and feedback on guideline adherence were not
recruited practices into the study, co-led the development of additive. There was marked variation in use of the CDS system,
guideline adherence metrics, wrote sections of the manuscript,
and approved the final version.
ranging from 5% to 45% visits across practices.
This trial has been registered at www.clinicaltrials.gov CONCLUSIONS: Clinical decision support and performance feedback
(identifier NCT00581711). are both effective strategies for improving adherence to otitis media
(Continued on last page) guidelines. However, combining the 2 interventions is no better than
either delivered alone. Pediatrics 2013;131:e1071–e1081

PEDIATRICS Volume 131, Number 4, April 2013 e1071

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 Otitis media (OM) is a hallmark disease a robust CDS intervention that provides Philadelphia’s primary care network
of pediatrics with 80% of cases oc- patient-specific information to reduce located in southeastern Pennsylvania,
curring among children.1 It is the third diagnostic uncertainty and treatment southern New Jersey, and Delaware.
most common reason for a pediatric variability. Participation in the study was volun-
visit, behind preventive care and acute The approach of summarizing clinical tary. Members of the study team visited
upper respiratory infections,2 and the practice over a specified interval has practices to explain the purpose of the
most common condition pediatricians been extensively used to deliver per- project, to discuss the expectations of
refer to specialists.3 Primary care4 formance information to clinicians. participants, and to increase aware-
and subspecialist physicians5 have Nonetheless, there is a need to de- ness of the newly created, local OM
difficulty diagnosing OM, which con- termine the best strategies for de- guidelines. Of 27 eligible practices, 24
tributes to variability in incidence livering feedback to maximize its agreed to participate. The study pro-
rates6 and overuse of antibiotics.7 It is effects. Previous research has shown tocol was approved by the Children’s
the most common reason for pre- that feedback is most effective when Hospital of Philadelphia’s Institutional
scribing antibiotics in the United States8 rates of adherence to practice guide- Review Board, which waived the need
and is responsible for approximately lines are low,17,18 the information is for informed consent from patients
half the antibiotic prescriptions for directly useful for care,19 and practi- and physicians for practical data col-
children aged 3 to 60 months.9 Despite tioners are motivated to change.20 Lit- lection purposes.
nationally promulgated guidelines10,11 tle is known regarding the effects of Local Adaptation of National
that recommend judicious use of anti- physician performance feedback that Guidelines
microbial agents, physicians have been is generated from the rich clinical data
Published guidelines are not always
reluctant to alter their rates of pre- located in EHRs. Moreover, no previous
accompanied by strong evidence or
scribing antibiotics for OM.12 studies have examined the combined
clearly defined diagnostic or thera-
The proliferation of electronic health and potentially additive effects of per-
peutic choices. Thus, we formed a panel
records (EHRs) in hospitals and physi- formance feedback in combination
of primary care physicians, otolar-
cians’ offices and inclusion of clinical with prospectively delivered CDS in-
yngologists, and outcomes researchers
decision support (CDS) as a criterion dividualized to a patient’s history and
who reviewed the nationally recognized
for meaningful use incentives under presentation.
OM guidelines,10,11 adapted them ac-
the Health Information Technology for We postulated that an EHR-based in- cording to the local context, and mod-
Economic and Clinical Health Act13 tervention that provided patient- ified them using the Guidelines Element
have heightened expectations that EHR- specific recommendations in real- Markup methodology21,22 such that lo-
based CDS and performance feedback time and physician feedback that cal recommendations could be pro-
will improve adherence to guidelines. summarized adherence to guideline- grammed as CDS. When choice or
This is because EHRs have the ability to based care would improve the di- ambiguity existed in certain guideline
deliver relevant, patient-specific, and agnosis and treatment of OM compared recommendations, we relied on our
actionable information to clinicians.14 with either intervention delivered expert panel to apply local consensus
Demonstrating the full benefit of CDS alone. To test this hypothesis, we con- to make recommendations deliverable
interventions in a complex health care ducted a factorial-design cluster ran- unambiguously via CDS. Clinical judg-
environment will take rigorous evalu- domized trial in a primary care ment was not eliminated, however,
ation methods, such as multipractice practice–based research network to because use of the tool was optional
cluster randomized trials.15 evaluate the independent and joint and not programmed as a hard stop
effects of an EHR-based CDS system within the EHR.
There has been only 1 published study of
and performance feedback on adher-
the impact of an OM CDS tool.16 The study Several adaptations to the guidelines
ence to OM guidelines.
was done at a single academic teaching were made. First, we required pain
site that facilitated access to a generic METHODS management in cases of only moderate
set of guideline recommendations. Given or severe pain because the guidelines
the paucity of efforts to apply CDS to OM Study Setting did not specify pain severity. Second, for
care, the national burden of this disease, The study was done in the Pediatric patients allergic to non–type I penicil-
the variability of care, and the large Research Consortium, a practice-based lin, the guidelines list cephalosporins
opportunity for improved antimicrobial research network that includes all as the preferred option, but our panel
management, OM is a natural target for practices in the Children’s Hospital of recommended inclusion of macrolides

e1072 FORREST et al

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 ARTICLE

as well. Third, for severe disease in of randomization was the practice, which Two practices were lost to follow-up.
penicillin-allergic patients, guidelines minimized contamination between study One randomized to CDS and feedback
recommend only ceftriaxone, but our groups because physicians in the same left the hospital system month 13 of the
experts felt that oral agents were an practice often comanage patients. The 4 study period, and 1 randomized to the
acceptable alternative. Fourth, local con- groups were CDS with feedback (n = 8 CDS-only group asked to leave the study
sensus endorsed the use of a narrower- practices), CDS only (n = 8 practices), month 13 of the study period.
spectrum antibiotic, amoxicillin, in- feedback only (n = 4 practices), and
stead of amoxicillin-clavulanic acid as usual care (n = 4 practices). This de- CDS System
first line even in severe disease. This sign facilitated estimation of the in- Ample evidence indicates that the
recommendation was based on local dependent and joint effects of CDS and implementation of EHR-based decision
practice and resistance patterns and physician feedback on adherence to support should be integrated as seam-
was discussed, along with all the rec- OM guidelines. lessly as possible into existing workflow
ommendations, with clinicians during The study was done in 3 phases (Fig 1). In processes.23,24 CDS systems that pro-
the training sessions we held with Phase 1 (baseline period lasting 12 vide personalized recommendations
each practice before the launch of the months from December 2007 to November are more effective than those that offer
CDS system. Finally, after extensive 2008), no practices were exposed to ei- generic information.25,26 Building on
consultation with our expert panel and ther intervention. In Phase 2 (CDS only these specifications, we designed, pro-
review of aggregate local data as well period lasting 11 months from Decem- totyped, pilot tested, and then imple-
as individual chart review, we de- ber 2008 to October 2009), 16 practices mented the study’s clinical decision
termined that the overprescribing of were exposed to the CDS intervention support system to bring OM guidelines
antibiotics, not the overuse of watchful and 8 others served as non-CDS con- to clinicians at the point of care with
waiting, was the overwhelming prob- trols. In Phase 3 (CDS and feedback pe- minimal effort. The CDS system was
lem in our setting. Our panel unani- riod lasting 10 months from November programmed using a web service that
mously agreed that decision support 2009 to August 2010), 8 of the CDS was not part of the EHR. It appeared as
should be directed at encouraging practices were exposed to Feedback, 8 part of the EpicCare EHR visit navigator
watchful waiting. other CDS practices were not exposed to and gathered data from and returned
feedback, 4 non-CDS practices were ex- information through the EHR. (Details of
Cluster Randomized Factorial posed to feedback, and 4 non-CDS the design of the CDS system are avail-
Design practices were not exposed to feed- able from the authors on request.)
A cluster randomized factorial design back (“usual care”), which yielded the The CDS system had 3 components that
was used for this study (Fig 1). The unit four-group factorial design. used a full range of CDS strategies

FIGURE 1
Study design. A 4-group factorial design was used for this cluster randomized trial. An “R” in the figure indicates randomization of practices. The study began
with a baseline phase lasting 12 months followed by randomization of practices to clinical decision support or usual care. This CDS-only period lasted for 11
months when a second randomization was done allocating practices to feedback. The second randomization produced the 4 groups that enable evaluation of
the individual and combined effects of decision support and feedback.

PEDIATRICS Volume 131, Number 4, April 2013 e1073

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 including data entry tools; cognitive Identification of OM Visits and Performance Feedback
aids that gather, organize, and display Episodes of Care Beginning month 24 of the study period,
information; and workflow aids such OM visits were identified using struc- we provided individualized feedback on
as facilitated order entry to expedite tured (ie, diagnostic codes) and un- a monthly basis to physicians in 12 of
clinical care (shown in Supplemental structured (ie, free text) data from the practices (Fig 1). The reports
Appendix 1). The first appeared pas- the EHR. Telephone encounters were showed physicians their adherence to
sively for all visits with an ear-related excluded because the CDS tool would OM guidelines for the month, change
problem; it provided a visual display of not be available during these con- over time, and comparisons with oth-
previous OM encounters including of- versations. Extensive programming ers in their practice and the full net-
fice visits, audiograms, subspecialist was required to search text from the work. A member of the project study
referrals, and past antibiotic history EHR for combinations of words that team delivered reports to each prac-
regardless of indication. These data would reflect diagnosis, physical ex- tice’s office manager, who distributed
were grouped into episodes of related amination, and history because them to practicing physicians. (An ex-
events as described below, and a search of only structured information, ample of one of the feedback reports is
results were overlaid on the display. as in diagnostics codes, proved in- shown in Supplemental Appendix 2.)
Although the CDS system customized adequate as a means of capturing all
guideline-based recommendations to visits accurately and completely. Statistical Analysis
the patient according to past history
For some adherence to guideline met- We examined adherence to guideline
and current presentation, we also
rics, we needed to determine when the recommendations for OM pain man-
provided hyperlinks to the complete
first visit for an OM episode began. To do agement, diagnostic documentation,
locally adapted clinical practice guide-
this, we developed OM episode of care and medication management (Table 1).
lines for acute OM (AOM) and OM with
algorithms as follows. A visit was All measures operationalized the local
effusion (OME).
identified as possibly related to OM if the adaptation to national guidelines. As-
Clinicians actively chose to use the EHR data for that visit contained a coded sessment of the effects of the CDS
second component, an OM-specific diagnosis of OM or an ear-related system was done for all metrics shown
data-gathering tool for recording the problem, a reason for visit that con- in Table 1 except 2: pain assessed and
history of present illness and the tained an ear- or hearing-related term, avoidance of antihistamines and de-
physical examination. or a prescription for an antibiotic congestants, which were achieved for
The final component, also triggered by commonly used to treat OM. Eligible nearly 100% of all OM visits. A subset
the clinician, returned guideline- visits were further characterized as of these indicators (pain treatment,
based recommendations for treat- related to AOM or OME based on In- amoxicillin as first-line therapy for
ment using data collected by the first ternational Classification of Diseases, AOM, and high-dose amoxicillin for
2 components; generated patient- Ninth Revision codes, descriptions of AOM) were evaluated to determine the
specific orders for diagnosis, treat- the reason for visit, and specific ear effects of feedback. Our local panel of
ment of pain, antibiotics or watchful examination findings documented by OM experts selected these quality
waiting as appropriate, and referral; the clinician (eg, presence of middle indicators because they were judged
wrote a progress note; and provided ear effusion). Visits with a documented most likely to change in response to
patient-specific discharge instruc- bilateral normal ear examination (ie, feedback.
tions. no findings of inflammation or effu- The analysis used the “as randomized”
We extensively validated all aspects of sion) were excluded from additional (intention-to-treat) principle in which
the CDS system by randomly selecting consideration. Visits categorized as each site and each clinician within-site
100 EHR records for review by 2 clini- AOM or OME were extended into epi- was assumed to use the intervention
cians, who iteratively compared the sodes by including all visits of the ap- arm to which the site and its clini-
computer algorithms to implicit clinical propriate type (AOM vs OME) meeting cians were assigned regardless of ac-
review. We focused on each element the inclusion criteria and falling within tual use. As is consistent with sound
of the decision support and revised a defined interval (45 days for OME; 14 conventional practice for a cluster-
algorithms to address inconsistencies. days for AOM) of another included visit. randomized design, we performed an
These validation studies continued until We validated visit selection and episode analysis with the unit of randomiza-
all clinically meaningful errors were of care algorithms using the same ap- tion as the cluster (ie, practice). Cor-
addressed. proach as for the CDS. relation of visits within practices was

e1074 FORREST et al

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 ARTICLE

TABLE 1 Adherence to OM Guidelines Metric Definitions for Children 2 Months to 12 Years of Age
Metric name: Guideline Recommendation Adherence to Guideline Metric Definition

Denominator Numerator
All OM
Pain assessed: Pain should be assessed for • OM visit • Pain score recorded
all patients with OM.
Pain treated: Moderate to severe pain should • OM visit • Analgesic prescribed or analgesic recommended
be treated with analgesics for all patients • Pain score $4 in the patient instructions
with OM.
AOM
AOM adequate diagnostic evaluation: Clinicians • AOM visit • Effusion or poor tympanic membrane movement
should identify signs of a middle-ear effusion documented
and evaluate for signs and symptoms of • Middle ear inflammation documented—defined
middle-ear inflammation. as tympanic membrane inflammation recorded
in examination, pain documented in progress
note, or nonzero pain score
Amoxicillin as first-line therapy for AOM: • AOM visit • Amoxicillin prescribed
Amoxicillin is the treatment of first choice • First visit in AOM episode
for AOM. • Oral antibiotic prescribed
• No co-occurring pneumonia, sinusitis,
conjunctivitis, or pharyngitis
• No allergy to penicillin
AOM treatment of penicillin-allergic patients: • AOM visit • If type I allergy, oral macrolide prescribed and
Appropriate antibiotics should be prescribed • First visit in AOM episode oral cephalosporin not prescribed or if non–type
for penicillin-allergic patients treated for AOM. • Oral antibiotic prescribed I allergy, either an oral cephalosporin or
• No co-occurring pneumonia, sinusitis, macrolide prescribed
conjunctivitis, or pharyngitis
• Either type I (hives/urticaria or anaphylaxis)
or non–type I penicillin allergy
High-dose amoxicillin for AOM: When amoxicillin • AOM visit • Patient wt recorded on day of visit
is used for AOM, a high dosage should be • First visit in AOM episode • If patient wt ,25 kg, amoxicillin dose is
prescribed (80–90 mg/kg per day). • Oral antibiotic prescribed $60 mg/kg/day or if patient wt $25 kg,
• No co-occurring pneumonia, sinusitis, amoxicillin dose is $1500 mg/day
conjunctivitis, or pharyngitis
• Amoxicillin prescribed at visit
AOM watchful waiting: Watchful waiting can be • AOM visit • Oral antibiotic not prescribed
done in a child with uncomplicated AOM. • First visit in AOM episode
• No co-occurring pneumonia, sinusitis,
conjunctivitis, or pharyngitis
• Age 6–24 mo: not severe illness
(temperature ,39°C and pain score 0–3)
and uncertain diagnosis (criteria for
adequate documentation not met)
• Age $25 mo to 12 y: not severe illness
(temperature ,39°C and pain score 0-3)
or uncertain diagnosis (criteria for
adequate documentation not met)
OME
OME adequate diagnostic evaluation: Clinicians • OME visit • Laterality of effusion documented
should document OME with pneumatic • Diagnostic testing for effusion done
otoscopy or tympanometry and record
laterality of effusion.
Avoidance of decongestants and antihistamines • OME visit • No decongestant or antihistamine or
for OME: Decongestants or antihistamines • OME is only diagnosis cough/cold preparation prescribed
should not be used because they are ineffective
in the treatment of OME.
OME watchful waiting: Clinicians should manage • OME visit • No oral antibiotic prescribed
OME with watchful waiting. • First visit in the episode
• No co-occurring pneumonia, sinusitis,
conjunctivitis, or pharyngitis

PEDIATRICS Volume 131, Number 4, April 2013 e1075

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 addressed using both mixed effects 24–33]) 4 treatment groups, and ad- practice patterns reflective of attend-
and marginal models. We chose the justed effects for patient age, gender, ing physicians), (2) in which a patient
latter for presentation of results be- presence of a comorbidity that complaint or physician diagnosis sug-
cause b estimates and P values were increases the risk of OM (ie, immuno- gested OM but the physical examina-
nearly identical between the 2, and deficiencies, developmental disorders, tion indicated that the OM was
marginal models have fewer assump- blindness, hearing loss, cleft lip/ resolved, and (3) in which otitis externa
tions about the distribution of random palate), and number of diagnosis co-occurred. Among the resulting 139
effects and the correlation structure of codes as a measure of visit-level clini- 305 visits, 64% were for AOM, 17% OME,
the model.27 We found that 2-level mixed cal complexity. For presentation of 6% both AOM and OME. The rest (13%)
effects models, clustering on physicians data, we computed the predictive were unclassified because they had
and practices, in many cases would not margins, which estimate the probabil- nonspecific diagnosis codes and non-
converge, and when they did, neither ity of the outcome standardized to specific ear examination information.
parameter nor SE estimates were sub- the characteristics of patients in the The 139 305 OM visits were distributed
stantively different from marginal mod- sample and thus reflect adjusted across the 4 groups as follows: CDS and
els that clustered visits within practices. probabilities and their differences. feedback, 45 843 visits (range 3292–
Moreover, we attempted confirmatory Our measure of intervention effect 8128 per practice); CDS only, 38 215 visits
analysis using other algorithms (most contrasted the absolute percentage (range 1955–8053 per practice); feed-
notably using mixed-effects models with change between 2 time periods for the back only, 32 214 visits (range 4600–15
practice as a random intercept and time intervention group versus control 363 per practice); and usual care, 23 033
as a random slope). To do mixed-effects group (in other words, a standardized visits (range 3186–7362 per practice).
models correctly requires careful atten- difference-in-difference estimate) and
Baseline practice and visit character-
tion to the choice of the algorithm, be- used 0.05 as the critical value.
istics of the study sample are shown in
cause simplistic methods that are often Table 2. On average, practices employed
used (eg, penalized quasi-likelihood RESULTS 7 physicians (range 6–9 across study
[PQL]) are well known to lead to bias. For There were 1 222 283 visits made to the groups) and managed 182 OM visits per
that reason, we attempted to implement 24 study practices during the 33-month month. The most notable difference
adaptive quadrature, a computer-intensive study period (Fig 2). To form the study across study groups is the substantially
method, but experienced convergence sample, we excluded visits (1) made higher number of visits per practice in
problems in some cases, perhaps in with pediatric residents (to capture the feedback-only group, which was due
part due to the small sample sizes at
the individual physician level. As a con-
sequence of testing these alternative
approaches, we feel that our analysis is
both sound and robust.
To estimate the contrasts of interest,
the analysis implemented a marginal
model with a logit link function and
computed robust SE estimates for pa-
tient visits nested within practices us-
ing the survey algorithms implemented
in Stata v11.1 and SAS v9.2. These
algorithms are equivalent to using
generalized estimating equations with
working independence correlation
structures but permit the analysis of
large data sets. The models included 3
periods (before intervention [months
1–12], CDS-only intervention phase
[months 13–23], and the CDS plus FIGURE 2
feedback intervention phase [months OM visit sample.

e1076 FORREST et al

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 ARTICLE

to inclusion of a practice with nearly (Table 4). The difference-in-difference groups. The CDS and feedback group
double the number of OM visits (n = 15 intervention effects, contrasting the had smaller increases in adherence to
363) as the other practices. change in the CDS group to the guidelines than the feedback only
There was marked variation for several change in the non-CDS group, are group (Table 5).
of the adherence to guideline measures shown in the fifth column of Table 4.
across the study groups during the Compared with the non-CDS group, DISCUSSION
baseline period (Table 3). The pro- the CDS group had a significant in-
This article describes the results from
portion of visits in which all guideline crease in adherence to guidelines
a cluster-randomized trial of the effects
adherence metrics were achieved between the 2 time periods for com-
of CDS and physician performance
(“comprehensive care”), given a mini- prehensive care (AOM and OME), pain feedback on adherence to guidelines
mum of $3 opportunities, was also treated, amoxicillin as first-line ther- for OM care. We developed a CDS system
variable, ranging across study groups apy for AOM, and OME adequate di- for this study that was prospectively
from 10% to 21% for AOM and 2% to 9% agnostic evaluation. applied at the point of care, seamlessly
for OME. In the feedback period of the study integrated into EHR workflow, provided
Across both follow-up periods (months (Fig 1), we provided individualized patient-specific recommendations, and
13–33), the average practice-level use physician feedback reports to half the generated a progress note and patient
of the CDS tool was 17% (range 5%– practices, equally balanced between instructions. Physician feedback sum-
45% across practices) of eligible visits. CDS and non-CDS groups. The contrasts marized the past month’s adherence
We analyzed the effects of the CDS of interest were the effects during time to key guideline recommendations
intervention by contrasting rates of 3 (feedback period) versus time 2 (CDS and contrasted an individual physi-
adherence to guidelines during the CDS- only period; Table 5). There was no cian’s performance with others in the
only versus baseline periods (Fig 1). difference in adherence to guideline practice and across the full primary
Among CDS practices, rates of adher- indicators for the CDS only versus care network. Our findings suggest
ence to guidelines rose for 7 of the 10 usual care groups. The feedback only that both types of interventions can
measures evaluated, whereas the only group had significantly greater improve adherence to guidelines.
metric that significantly increased for increases in guideline adherence than However, our hypothesis that the
the non-CDS practices was pain treated both the usual care and CDS only combination of CDS and feedback

TABLE 2 Practice and Visit Characteristics by Study Group During the Baseline Period (Months 1–12).
Total Sample Study Group

CDS and Feedback CDS Only Feedback Only Usual Care

Practice Characteristics
No. primary care practices 24 8 8 4 4
No. primary care physicians 182 56 66 25 35
Female physicians, % 68 67 74 61 71
Physicians/practice, mean 7.1 7.0 8.3 6.2 8.8
OM visits/practice, mean 2185 2196 1771 3029 2145
Visit Characteristics, %
Patient age
2–12 mo 23.7 24.6 23.0 24.4 22.3
13–36 mo 36.1 36.1 37.2 35.0 35.9
37–60 mo 17.9 17.6 17.7 18.3 18.3
61–155 mo 22.2 21.7 22.0 22.3 23.5
Female 46.7 47.5 46.4 46.1 46.6
Comorbidity that increases risk of OM 2.9 3.0 3.4 2.0 2.9
Diagnostic codes/visits
1 50.9 39.2 53.7 62.8 53.3
2 35.7 42.6 34.2 28.5 34.2
3+ 13.4 18.2 12.1 9.7 12.1
OM type
AOM 64.6 59.7 67.7 65.6 69.1
OME 15.2 19.2 13.5 13.3 12.3
AOM and OME 6.1 6.6 6.3 6.3 4.7
Unspecified OM 14.1 14.5 12.5 15.3 13.9

PEDIATRICS Volume 131, Number 4, April 2013 e1077

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 would be superior to either alone was There was no effect of CDS on overuse of or if withholding treatment is perceived
not supported. The effects of feedback antibiotics for either AOM or OME. Sim- as decreasing productivity.30
were larger than those for the CDS ilar to other studies, our findings in- Because the feedback period lasted just
system, and feedback was more bene- dicate that CDS systems can encourage 10 months, this study does not tell us
ficial in practices that had not been physicians to take or alter an action, how long a program of provider feed-
exposed to CDS than those that had such as changing the selection of an back can influence physician behavior
been using it. Other studies have found antibiotic or modifying choice of ther- change, what happens when feedback
that when feedback is coupled with apy,28,29 but have much weaker effects is removed, or how long the feedback
other interventions, its effect is re- on withholding treatment, especially in must persist to achieve optimal effect.
duced.17,18 the context of strong clinician beliefs29 Future work should examine the effects

TABLE 3 Adherence to Guidelines Metrics by Study Group During the Baseline Period (Months 1–12)
Adherence to Guidelines Metrics Total Sample Study Group

CDS and Feedback CDS Only Feedback Only CDS and Feedback
All OM

% adhering to guideline-recommended care (no. qualifying visits)

Pain assessed 96 94 98 95 98
(52 429) (17 568) (14 164) (12 117) (8580)
Pain treated 32 33 27 34 35
(8800) (2582) (2419) (1795) (2004)

AOM

% adhering to guideline-recommended care (no. qualifying visits)

AOM diagnostic evaluation 45 39 46 48 49

(37 103) (11 647) (10 480) (8647) (6329)
Amoxicillin as first-line therapy for AOM 71 68 77 70 64
(26 696) (7852) (7775) (6579) (4490)
High-dose amoxicillin for AOM 49 65 42 30 63
(18 984) (5387) (6038) (4634) (2925)
AOM treatment of penicillin-allergic patients 79 80 75 78 82
(1873) (612) (433) (445) (383)
AOM watchful waiting 6 6 7 4 6
(20 830) (6425) (6005) (5058) (3342)

OME

% adhering to guideline-recommended care (no. qualifying visits)

OME diagnostic evaluation 7 3 7 15 8

(7952) (3367) (1911) (1616) (1058)
OME watchful waiting 88 93 87 78 87
(2721) (1162) (698) (488) (373)
Avoidance of decongestants and antihistamines for OME 98 98 99 98 100
(465) (134) (135) (153) (43)

Summary measures for adherence to guidelines

Mean no. of adherence to guideline opportunities 3.0 2.9 3.1 3.0 3.0
per visit
Mean no. (%) adherence to guideline opportunities 2.0 1.8 2.0 1.9 2.1
achieved per visit
(66.7) (62.1) (64.5) (63.3) (70.0)
Comprehensive care: adherence to guideline opportunities
achieved for visits with $3 opportunities, % (no. visits)
AOM 14.7 15.8 13.2 10.4 21.4
(30 063) (8962) (8575) (7300) (5226)
OME 4.9 2.3 5.3 9.0 6.6
(3331) (1400) (832) (600) (499)
Either AOM or OME 13.7 14.0 12.5 10.3 20.1
(33 394) (10 362) (9407) (7900) (5725)

e1078 FORREST et al

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 ARTICLE

TABLE 4 Standardized Effects of CDS on Adherence to Guidelines for AOM and OME During CDS-only Follow-up Period (Time 2, Months 13–23) Versus
Baseline Period (Time 1, Months 1–12)
Adherence to guidelines metrics Sample size CDS Practices Non-CDS Practices CDS Practices – Non-CDS
Practices

Time 2 – Time 1 Time 2 – Time 1 Time 2 – Time 1

Percentage difference (P value)

Comprehensive care
AOM 56 141 5.3 (,.001) 0.9 (.417) 4.4 (.006)
OME 6705 1.5 (.148) 21.0 (.084) 2.5 (.032)
Pain treated 16 289 11.4 (,.001) 5.7 (,.001) 5.7 (.030)
AOM diagnostic evaluation 69 419 3.8 (.035) 20.9 (.786) 4.6 (.214)
Amoxicillin as first-line therapy for AOM 49 821 5.0 (,.001) 0.9 (.115) 4.1 (.001)
High-dose amoxicillin for AOM 36 136 9.7 (,.001) 3.5 (.329) 6.2 (.142)
AOM treatment of penicillin-allergic patients 3581 21.2 (.371) 20.9 (.782) 20.3 (.937)
AOM watchful waiting 39 098 ,0.1 (.811) 20.7 (.207) 0.8 (.211)
OME diagnostic evaluation 16 116 3.2 (.045) 21.8 (.058) 5.0 (.008)
OME watchful waiting 5560 –3.2 (.033) 21.3 (.223) 21.8 (.323)

of feedback over longer periods to time and the reduced power of even this concerning findings from this study is
measure lags after onset and at- large study among 24 practices to de- the overall low adherence to guidelines
tenuation over time. We are also un- tect statistically significant intervention for OM across most of the metrics. Com-
able to explain the mechanisms for effects. One approach for reducing prehensive care (ie, all recommended
the large feedback effect, and it is interpractice variation in quality is to guidelines for a visit were adhered to)
possible that feedback could be less standardize care, which minimizes was accomplished for only 15% of visits
effective for improving adherence to unnecessary variation. Once this has with AOM and 5% with OME during the
guidelines for the metrics that were been accomplished, new interventions baseline period of observation. An-
not included in the physician feedback such as CDS systems might produce other finding of note is that clinicians
reports. more uniform effects across practices chose watchful waiting for only 6% of
The unit of randomization for this study and thus be more readily shown to be eligible AOM visits. It appears that, at
was the physician practice, a design effective. This combination of quality least in our sample, the tremendous
often required to avoid the potential of improvement coupled with research public emphasis on judicious use of
contamination of the intervention ef- that occurs within a community of antimicrobial therapy to reduce the
fects if randomization had been done physicians and patients organized to risk of drug-resistant microbes35 has
at the levels of physician, patient, or advance patient health is the bedrock of had little impact on physician pre-
visit. The downside of practice-level learning health systems.31–33 scribing.
randomization, however, was large Previous research has found that the In summary, using the EHR to provide
design effects (data not shown) arising promulgation of national guidelines real-time decision support and physi-
from high interpractice variation in for AOM and OME did not have substan- cian performance feedback are both
quality at baseline and change over tive impacts on quality.34 One of the effective strategies for improving

TABLE 5 Standardized Effects of Provider Feedback With and Without CDS on Adherence to Guidelines for AOM During 10-month CDS and Feedback
Period (Time 3, Months 24–33) Versus 11-month CDS-only Period (Time 2, Months 13–23)
Adherence to guideline metrics Sample size Feedback vs Usual Care Practices Feedback vs CDS-Only vs CDS and Feedback vs
CDS-Only Practices Usual Care Practices Usual Care Practices

Time 3 – Time 2 Time 3 – Time 2 Time 3 – Time 2 Time 3 – Time 2

Percentage Difference (P value)

Comprehensive care, AOM only 48 782 5.4 (.070) 8.9 (.005) 23.5 (.284) 1.8 (.586)
Pain treated 13 724 16.0 (.001) 16.7 (,.001) 20.7 (.880) 20.5 (.936)
Amoxicillin as first-line therapy for AOM 43 242 6.3 (,.001) 5.4 (,.001) 0.9 (.485) 5.8 (.012)
High-dose amoxicillin for AOM 32 386 12.9 (.007) 15.6 (.004) 22.7 (.653) 4.4 (.335)

PEDIATRICS Volume 131, Number 4, April 2013 e1079

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 adherence to guidelines for otitis additive effects. Modest utilization attention to overcoming barriers
media care. In a direct comparison of of the CDS tool in this study may to physician adoption of CDS sys-
the two interventions, feedback pro- have attenuated its potential effects. tems must be addressed as they
duced stronger effects on guideline Future work will be done to charac- are disseminated to encourage
adherence than the CDS system, and terize physician and practice level meaningful use of health information
the two interventions did not have adoption of the CDS tool. Careful technology.

REFERENCES
1. Culpepper L, Froom J, Bartelds AI, et al. 11. American Academy of Pediatrics Sub- 21. Shiffman RN, Michel G, Essaihi A, Marcy TW.
Acute otitis media in adults: a report from committee on Management of Acute Otitis Using a guideline-centered approach for
the International Primary Care Network. Media. Diagnosis and management of acute the design of a clinical decision support
J Am Board Fam Pract. 1993;6(4):333–339 otitis media. Pediatrics. 2004;113(5):1451– system to promote smoking cessation.
2. Hing E, Cherry DK, Woodwell DA. National 1465 Stud Health Technol Inform. 2004;101:152–
Ambulatory Medical Care Survey: 2004 12. Coco A, Vernacchio L, Horst M, Anderson A. 156
Summary. Hyattsville, MD: National Center Management of acute otitis media after 22. Shiffman RN, Agrawal A, Deshpande AM,
for Health Statistics; 2006 publication of the 2004 AAP and AAFP clin- Gershkovich P. An approach to guideline
3. Forrest CB, Glade GB, Baker AE, Bocian AB, ical practice guideline. Pediatrics. 2010;125 implementation with GEM. Stud Health
Kang M, Starfield B. The pediatric primary- (2):214–220 Technol Inform. 2001;84(Pt 1):271–275
specialty care interface: how pediatricians 13. Blumenthal D, Tavenner M. The “meaningful 23. Ash JS, Bates DW. Factors and forces af-
refer children and adolescents to specialty use” regulation for electronic health fecting EHR system adoption: report of
care. Arch Pediatr Adolesc Med. 1999;153 records. N Engl J Med. 2010;363(6):501–504 a 2004 ACMI discussion. J Am Med Inform
(7):705–714 14. Osheroff JA, Pifer EA, Teich JM, Sittig DF, Assoc. 2005;12(1):8–12
4. Froom J, Culpepper L, Grob P, et al. Diagnosis Jenders RA. Improving Outcomes With 24. Stead WW. Rethinking electronic health
and antibiotic treatment of acute otitis me- Clinical Decision Support: An Implementer’s records to better achieve quality and safety
dia: report from International Primary Care Guide. Chicago, IL: Healthcare Information goals. Annu Rev Med. 2007;58:35–47
Network. BMJ. 1990;300(6724):582–586 and Management Systems Society; 2005 25. Kawamoto K, Houlihan CA, Balas EA, Lobach
5. Adeli M, Bender MJ, Sheridan MJ, Schwartz 15. Chuang JH, Hripcsak G, Heitjan DF. Design DF. Improving clinical practice using
RH. Antibiotics for simple upper res- and analysis of controlled trials in natu- clinical decision support systems: a sys-
piratory tract infections: a survey of aca- rally clustered environments: implications tematic review of trials to identify
demic, pediatric, and adult clinical for medical informatics. J Am Med Inform features critical to success. BMJ. 2005;330
allergists. Ann Allergy Asthma Immunol. Assoc. 2002;9(3):230–238 (7494):765
2008;100(4):377–383 16. Christakis DA, Zimmerman FJ, Wright JA, 26. Bates DW, Kuperman GJ, Wang S, et al.
6. González-Vallejo C, Sorum PC, Stewart TR, Garrison MM, Rivara FP, Davis RL. A ran- Ten commandments for effective clinical
Chessare JB, Mumpower JL. Physicians’ domized controlled trial of point-of-care decision support: making the practice
diagnostic judgments and treatment deci- evidence to improve the antibiotic pre- of evidence-based medicine a reality.
sions for acute otitis media in children. scribing practices for otitis media in chil- J Am Med Inform Assoc. 2003;10(6):
Med Decis Making. 1998;18(2):149–162 dren. Pediatrics. 2001;107(2). Available at: 523–530
7. Lyon JL, Ashton A, Turner B, Magill M. Var- www.pediatrics.org/cgi/content/full/107/2/: 27. Fitzmaurice GM, Laird NM. Applied Longi-
iation in the diagnosis of upper respiratory E15 tudinal Analysis. New York, NY: John Wiley
tract infections and otitis media in an ur- 17. Jamtvedt G, Young JM, Kristoffersen DT, & Sons; 2004
gent medical care practice. Arch Fam Med. O’Brien MA, Oxman AD. Audit and feedback: 28. Bates DW, Teich JM, Lee J, et al. The impact
1998;7(3):249–254 effects on professional practice and health of computerized physician order entry on
8. McCaig LF, Hughes JM. Trends in antimi- care outcomes. Cochrane Database Syst medication error prevention. J Am Med
crobial drug prescribing among office- Rev. 2006;(2):CD000259 Inform Assoc. 1999;6(4):313–321
based physicians in the United States. 18. Jamtvedt G, Young JM, Kristoffersen DT, 29. Teich JM, Merchia PR, Schmiz JL, Kuperman
JAMA. 1995;273(3):214–219 O’Brien MA, Oxman AD. Does telling people GJ, Spurr CD, Bates DW. Effects of comput-
9. Finkelstein JA, Metlay JP, Davis RL, Rifas- what they have been doing change what erized physician order entry on prescribing
Shiman SL, Dowell SF, Platt R. Antimicrobial they do? A systematic review of the effects practices. Arch Intern Med. 2000;160(18):
use in defined populations of infants and of audit and feedback. Qual Saf Health 2741–2747
young children. Arch Pediatr Adolesc Med. Care. 2006;15(6):433–436 30. Lee F, Teich JM, Spurr CD, Bates DW.
2000;154(4):395–400 19. Kanouse DE, Jacoby I. When does information Implementation of physician order entry:
10. American Academy of Family Physicians; change practitioners’ behavior? Int J Technol user satisfaction and self-reported usage
American Academy of Otolaryngology-Head Assess Health Care. 1988;4(1):27–33 patterns. J Am Med Inform Assoc. 1996;3
and Neck Surgery; American Academy of 20. Mugford M, Banfield P, O’Hanlon M. Effects (1):42–55
Pediatrics Subcommittee on Otitis Media of feedback of information on clinical 31. Etheredge LM. A rapid-learning health sys-
With Effusion. Otitis media with effusion. practice: a review. BMJ. 1991;303(6799): tem. Health Aff (Millwood). 2007;26(2):
Pediatrics. 2004;113(5):1412–1429 398–402 w107–w118

e1080 FORREST et al

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user
 ARTICLE

32. Olsen L, Aisner D, McGinnis JM. The 34. Vernacchio L, Vezina RM, Mitchell AA. 35. Interagency Task Force on Antimicrobial
Learning Healthcare System. Washington, Knowledge and practices relating to the Resistance. A Public Health Action Plan to
DC: National Academy Press; 2007 2004 acute otitis media clinical practice Combat Antimicrobial Resistance. Wash-
33. Slutsky JR. Moving closer to a rapid- guideline: a survey of practicing physi- ington, DC: Health and Human Services/
learning health care system. Health Aff cians. Pediatr Infect Dis J. 2006;25(5):385– Office of the Assistant Secretary for Pre-
(Millwood). 2007;26(2):w122–w124 389 paredness and Response; 2011

(Continued from first page)

www.pediatrics.org/cgi/doi/10.1542/peds.2012-1988
doi:10.1542/peds.2012-1988
Accepted for publication Dec 4, 2012
Address correspondence to Christopher B Forrest, MD, PhD, Children’s Hospital of Philadelphia, 34th St and Civic Center Blvd, Philadelphia, PA 19104. E-mail:
forrestc@email.chop.edu
PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).
Copyright © 2013 by the American Academy of Pediatrics
FINANCIAL DISCLOSURE: Drs Grundmeier, Fiks, and Bailey are coinventors of the otitis media clinical decision support software used in this intervention. They
have received no income for this technology and no licensing agreement exists. All results were verified by a statistician (Dr Localio) who has no conflict of
interests for this project. The other authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was supported by grant R18HS017042 from the Agency for Healthcare Research and Quality.

PEDIATRICS Volume 131, Number 4, April 2013 e1081

Downloaded from http://publications.aap.org/pediatrics/article-pdf/131/4/e1071/1100263/peds_2012-1988.pdf

by Dalhousie University-Dal 11762 user