Hill et al.

Implementation Science (2020) 15:23

https://doi.org/10.1186/s13012-020-0975-2

SYSTEMATIC REVIEW Open Access

The effectiveness of continuous quality

improvement for developing professional
practice and improving health care
outcomes: a systematic review
James E. Hill1, Anne-Marie Stephani1, Paul Sapple2 and Andrew J. Clegg1*

Abstract
Background: Efforts to improve the quality, safety, and efficiency of health care provision have often focused on
changing approaches to the way services are organized and delivered. Continuous quality improvement (CQI), an
approach used extensively in industrial and manufacturing sectors, has been used in the health sector. Despite the
attention given to CQI, uncertainties remain as to its effectiveness given the complex and diverse nature of health
systems. This review assesses the effectiveness of CQI across different health care settings, investigating the
importance of different components of the approach.
Methods: We searched 11 electronic databases: MEDLINE, CINAHL, EMBASE, AMED, Academic Search Complete,
HMIC, Web of Science, PsycINFO, Cochrane Central Register of Controlled Trials, LISTA, and NHS EED to February
2019. Also, we searched reference lists of included studies and systematic reviews, as well as checking published
protocols for linked papers. We selected randomized controlled trials (RCTs) within health care settings involving
teams of health professionals, evaluating the effectiveness of CQI. Comparators included current usual practice or
different strategies to manage organizational change. Outcomes were health care professional performance or
patient outcomes. Studies were published in English.
Results: Twenty-eight RCTs assessed the effectiveness of different approaches to CQI with a non-CQI comparator in
various settings, with interventions differing in terms of the approaches used, their duration, meetings held, people
involved, and training provided. All RCTs were considered at risk of bias, undermining their results. Findings
suggested that the benefits of CQI compared to a non-CQI comparator on clinical process, patient, and other
outcomes were limited, with less than half of RCTs showing any effect. Where benefits were evident, it was usually
on clinical process measures, with the model used (i.e., Plan-Do-Study-Act, Model of Improvement), the meeting
type (i.e., involving leaders discussing implementation) and their frequency (i.e., weekly) having an effect. None
considered socio-economic health inequalities.
(Continued on next page)

* Correspondence: AClegg3@uclan.ac.uk
1
Faculty of Health and Wellbeing, University of Central Lancashire (UCLan),
Preston, Lancashire PR1 2HE, UK
Full list of author information is available at the end of the article

© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if
changes were made. The images or other third party material in this article are included in the article's Creative Commons
licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons
licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain
permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the
data made available in this article, unless otherwise stated in a credit line to the data.
Hill et al. Implementation Science (2020) 15:23 Page 2 of 14

(Continued from previous page)

Conclusions: Current evidence suggests the benefits of CQI in improving health care are uncertain, reflecting both
the poor quality of evaluations and the complexities of health services themselves. Further mixed-methods
evaluations are needed to understand how the health service can use this proven approach.
Trial registration: Protocol registered on PROSPERO (CRD42018088309).
Keywords: Continuous quality improvement, Systematic review, Health care, Clinical process, Patient-based
outcomes, RCTs

from industry (e.g., total quality management), it is

Contributions to the literature
evident that the core features shared by the different
 The paper presents the first systematic review of the methods have evolved [17–19]. A review of the char-
effectiveness of continuous quality improvement (CQI) acteristics of CQI in health care [20] identified three
compared to non-CQI approaches on improving the quality, essential elements, which are systematic data-guided
activities, iterative development and testing process,
safety and efficiency in any health care sector;
and designing with local conditions in mind [20].
 It assesses the importance of the health care setting, the CQI
Despite some uncertainty around the characteristics
model used and key components of the different
of CQI [21], several approaches encompass the funda-
approaches used on changing clinical process and patient- mental principles and have been used in health, such
based outcomes; as Lean Management, Six Sigma, Plan-Do-Study-Act
 The review examines the consideration given to socio- (PDSA) cycles, and Root Cause Analysis [20].
economic health inequalities in improving health care Several systematic reviews have assessed the use of dif-
through CQI. ferent approaches to help improve quality in health care,
focusing on descriptions of the methods used and
highlighting the differences in components included
[22–32]. Those assessing CQI were in specific popula-
Background tions or clinical settings, considering their application
Improving the quality and safety of health care is a [29], effectiveness [31], and the barriers and facilitators
priority of governments, health care workers, and the to the implementation of CQI [28, 30]. None compared
public [1, 2], with efforts often focused on investment the effectiveness of CQI across a range of health settings,
in changes to the way health care is organized and assessed the benefits of specific components, or consid-
delivered (system-level quality improvements) [3, 4]. ered the actual impact of the factors that may influence
While there are many different approaches that may the effects of CQI. Given these limitations, we systemat-
be taken, continuous quality improvement (CQI) has ically reviewed the evidence to assess the effectiveness of
received considerable attention within health care [5] different approaches to CQI for developing professional
as a way to enhance the quality of care and reduce practice and improving health care outcomes in any
costs [6–9]. The use of CQI in health care has health care setting. We aimed to examine the impact of
evolved since the 1990s, using quality control tech- the various components encompassed in, and that affect
niques and management theories employed in the in- the application of, the different approaches, which may
dustrial and manufacturing sectors [10–14]. In its act as facilitators or barriers to change. These compo-
earliest form, CQI was based on five main principles, nents were based upon previously identified common
specifically: a focus on organizational process and sys- features within CQI [20, 33] and criteria used to evaluate
tems, rather than on individuals within the system; quality improvement interventions [34]. Also, we
the use of statistically and methodologically robust intended to consider the influence of socio-economic
structured problem-solving approaches; the use of health inequalities on the effectiveness, and the imple-
multi-disciplinary team working; empowerment of mentation, of the approaches to CQI in improving
employees to help identify problems and action im- health care. The importance of socio-economic inequal-
provement opportunities; and, a focus on “customers” ities in determining health, and the use of health and so-
(i.e., public) through an emphasis on creating the best cial care services, is widely recognized [35]. Increasingly,
possible patient experience and outcomes [13, 15, 16]. efforts are focusing on incorporating consideration of
As the use of CQI has grown in health care, and new health inequalities in developing health and social care
approaches to quality improvement have emerged services to address the widening health gap [36].
Hill et al. Implementation Science (2020) 15:23 Page 3 of 14

Methods pain, health-related quality of life, mortality). Abstracts

Searches and conference proceedings were only considered if
Our systematic review followed recognized guidance and enough detail of their methodology and results were
reporting standards (see Additional file 1 for PRISMA published. Study selection occurred through two stages.
checklist) [37, 38], with the methods outlined in a re- First, two reviewers independently screened the titles
search protocol registered on PROSPERO (CRD4201 and abstracts of papers from the searches, using criteria
8088309). We identified studies through searches of 11 specified prior to screening (Table 1). Discrepancies
electronic databases, specifically MEDLINE (via Ovid), were discussed between reviewers, with arbitration by a
CINAHL, EMBASE, AMED, Academic Search third independent reviewer where required. Second,
Complete, HMIC, Web of Science, PsycINFO, Cochrane manuscripts of studies appearing to meet the selection
Central Register of Controlled Trials, LISTA, and NHS criteria at title and abstract screening were retrieved.
EED (see Additional file 2 for example of search strat- These were then screened using the same process as that
egy). All databases were searched from their inception to for assessing titles and abstracts.
23 February 2019 and were limited to studies published
in English. Additional references were identified through Data extraction and study quality assessment
screening reference lists of all included studies and rele- Data was extracted using a pre-piloted form by one re-
vant systematic reviews. Linked companion publications viewer and checked by a second reviewer. Disagreements
were identified through checks of published study were discussed between reviewers and, if consensus was
protocols. not achieved, arbitration was carried out by a third re-
viewer. When further information was required, at-
Study selection tempts were made to contact the authors for
Studies were eligible if they were randomized controlled clarification. We extracted data on the characteristics of
trials (RCTs) within any health care setting involving the CQI intervention that have previously been identi-
teams of health professionals, evaluating the effective- fied as important [20, 21, 38], including its scope; inclu-
ness of CQI (Table 1). Recognized features of CQI had sion of factors considered key components of CQI, i.e.,
to be present, including systematic data-guided activities, systematic data-guided activities, iterative development
involvement of iterative development and testing, and a and testing process, and designing with local conditions
focus on a process or system rather than at an individual in mind [20, 33]; and the use of important features of
patient level [20]. Comparators could include different quality improvement in the implementation strategy
CQI strategies, current usual practice without an inter- (planned and actually implemented) [34]. Risk of bias
vention to manage organizational change, or other non- was assessed using the Cochrane Collaboration tool by
CQI interventions to manage organizational change. one reviewer, with decisions checked by a second re-
Studies had to assess measures of health care profes- viewer [38]. Decisions on the key criteria of random se-
sional performance (e.g., adherence to recommended quence generation, allocation concealment, and blinding
practice or process of care) or patient outcomes (e.g., of patients and outcome assessment were also checked

Table 1 Study selection criteria

Category Inclusion criteria Exclusion criteria
Participants Teams of health professionals responsible for improving the health of their Groups that do not include health professionals
populations and/or providing patient care in any health care setting or that are conducted in a non-health
care/non-public health setting or that only
involve students.
Intervention CQI that includes Interventions targeting the improvement of
(i) use of measurement and data analysis to assess and review the effect of administrative, management, or other processes
changes; not directly related to clinical care.
(ii) review and analysis of a process or system used to deliver clinical care to
identify sources of variation and areas for improvement;
(iii) an iterative procedure within a continuous process; and
(iv) a structured process improvement method or problem-solving approach
that is used to plan and test changes to the work process.
Comparison Current usual practice (non-active control), different CQI strategies, or other
non-CQI interventions to manage organizational change.
Outcome Any objective measure of health care professional performance (e.g.,
adherence to recommended practice or process of care) or patient
outcome (e.g., pain, health-related quality of life, function, mortality).
Study design RCTs
Hill et al. Implementation Science (2020) 15:23 Page 4 of 14

using a semi-automated process through RobotReviewer group analyses planned to focus on studies assessing the
[39]. This involved uploading study text to, and checks health setting, the CQI approach, key components of
being made against the criteria by, RobotReviewer. CQI that were previously identified as common across
Where differences occurred, these were checked, justi- models, and assessed in studies (i.e., type and frequency
fied and alterations made when required. Any disagree- of both training and meetings) and socio-economic
ments were discussed, with arbitration by a third health inequalities. The synthesis was presented as the
reviewer, if consensus was not reached. number and proportion of studies in each group, with
the narrative focusing on those RCTs finding no statisti-
Data synthesis cally significant difference between the CQI intervention
The synthesis focused on those studies which compared and the comparator and those RCTs showing a statisti-
a CQI intervention with a non-CQI intervention that cally significant benefit from CQI in half or more of the
was considered either current usual practice (i.e., with- outcomes assessed. This approach was used as the RCTs
out an intervention to manage organizational change) or rarely identified their primary outcome measures, and it
another non-CQI intervention to manage change, allow- was felt that showing an effect on over half or more out-
ing an assessment of the comparative benefits of the comes would limit the opportunity for selective report-
addition of CQI and limiting the effects of heterogeneity. ing of specific outcomes where benefit was shown.
Studies were synthesized through a narrative synthesis Meta-analyses were not produced due to heterogeneity
with a tabulation of results of included studies. Out- in the studies, particularly in the interventions and out-
comes were separated into three groups, specifically clin- comes assessed.
ical process outcomes, patient outcomes, and other
outcomes. All outcomes were then categorized into five Results
groups based on the ratio of outcomes demonstrating a Our search strategy identified 7518 papers which,
statistically significant difference at the 5% significance after duplicate removal, resulted in 6998 papers for
level on the summary measures presented (i.e., risk ra- inspection. Screening of titles and abstracts excluded
tios or mean difference with 95% confidence intervals) 6718 records (Fig. 1). Manuscripts for 280 papers
(Table 2). Differences were based on either the change were screened, with 44 studies presented in 72 papers
from baseline to end of study (first data point after inter- included in the review. Some 27 additional link pa-
vention) for CQI compared to that for control (differ- pers were identified through checking study protocols
ence within difference) or a comparison of CQI versus and snowball sampling. Although 44 RCTs met the
control at the end of the study with no statistically sig- selection criteria, the results presented are for 28
nificant difference at baseline (baseline versus end of RCTs comparing CQI with other non-CQI interven-
study). If both approaches were presented, the results tions, whether considered current usual practice (i.e.,
from difference within difference were used. Where usual care, normal practice, delayed intervention, or
baseline values were not compared statistically, a visual waiting list (19 RCTs)), a new management interven-
inspection was carried out to assess equivalence. Sub- tion without a CQI component (7 RCTs) or where no
description was provided of the comparator (2 RCTs).
Table 2 Categorisation of outcome measures
Study characteristics
Proportion of outcomes in Definition
studies showing All 28 included studies were cluster RCTs. Most RCTs
comparative benefit from were carried out in high-income countries, with 15 in
CQI
the USA [40–54], two in the Netherlands [55, 56], two
No outcomes No outcomes demonstrated a statistically in Canada [57, 58], two in the UK [59, 60], and one each
significant difference between interventions
in any study. in Sweden [61] and Spain [62]. Four RCTs were under-
Under half of outcomes Less than half of the outcomes in studies
taken in the middle- or low-income countries, specific-
showed a statistically significant benefit ally in India [63], Mexico [64], Nigeria [65], and Malawi
from CQI versus its comparator. [66]. Another RCT was conducted across multiple Afri-
Half of outcomes Half of the outcomes in studies showed can countries [67]. The clinical setting for the RCTs was
a statistically significant benefit from CQI mainly in primary (i.e., general practice) (13 RCTs) [40,
versus its comparator.
41, 43, 44, 48–53, 58, 62, 64] or secondary care (i.e., hos-
More than half of More than half of the outcomes in studies pitals) (10 RCTs) [45, 54, 55, 59–61, 63, 66–68]. The
outcomes showed a statistically significant benefit
from CQI versus its comparator. remaining five RCTs were set in substance misuse clinics
All outcomes All outcomes in the studies showed a
[69], community outreach [65], social services, and social
statistically significant benefit from CQI care [47, 57] or tertiary care [56]. Most RCTs were pub-
versus its comparator. lished recently, with 19 RCTs published since 2010 [40,
Hill et al. Implementation Science (2020) 15:23 Page 5 of 14

Fig. 1 PRISMA Flow Diagram

41, 46–49, 51, 52, 54–58, 60, 63–67] and only 9 RCTs frequency of their team meetings, whether weekly (3
before 2010 [43–45, 50, 53, 59, 61, 62, 68]. The RCTs RCTs) [48, 49, 60], fortnightly (1 RCT) [44], monthly (10
varied in the duration of the intervention, with 15 RCTs RCTs) [41, 46, 47, 53, 54, 58, 59, 63, 65, 66] or quarterly
lasting 52 weeks or less [40, 43–45, 47–49, 52, 54, 57, or less frequently (2 RCTs) [55, 57]. The remaining 12
60–62, 65, 67], 11 RCTs more than 52 weeks [41, 42, 46, RCTs did not indicate the schedule of meetings [40, 43,
51, 53, 55, 56, 58, 59, 64, 66]. Two RCTs used a stepped 45, 50–52, 56, 61, 62, 64, 67, 68]. Duration of the meet-
wedge design resulting in variation in intervention dur- ings was rarely stated, with 7 RCTs reporting meetings
ation [50, 63]. Multi-disciplinary teams (MDT) were that lasted either under 10 min [49], 40 to 70 min [48],
used in 19 RCTs [43–46, 49, 53, 55–61, 64–68, 70], with 60 to 120 min [51, 53, 65], or 90 to 180 min [46, 57].
8 RCTs not adequately describing membership of their The other 21 RCTs did not describe duration of meet-
teams [40, 47, 48, 50–52, 54, 63]. One RCT explicitly ings [40, 43–45, 47, 50, 52, 54–56, 58–64, 66–68, 70].
stated that they did not use an MDT approach [62]. The total number of meetings held also varied. Al-
PDSA was the CQI model most frequently used, with 12 though 9 RCTs did not describe the number of meet-
RCTs using this approach [40, 43, 45, 46, 48, 50, 54, 57, ings held [40, 45, 50, 61–64, 67, 68], 19 RCTs
58, 63, 67, 70] and 7 RCTs using an adaptation of PDSA reported that they held either 1 to 4 [57], 5 to 9 [51,
(the Model of Improvement (MoI)) [44, 55, 60, 61, 64– 54–56], 10 to 14 [43, 46, 52, 70], 15 to 20 [58], or
66]. One RCT used root cause analysis [47]. Eight RCTs more than 20 meetings [44, 47–49, 53, 59, 60, 65,
used a range of undefined CQI approaches [49, 51–53, 66]. Seventeen RCTs involved meetings that included
56, 59, 62, 68]. organizational leaders as participants and discussed
Important characteristics of approaches to CQI were the implementation of the CQI [44, 46, 48, 49, 51,
infrequently reported. Only 16 RCTs described the 53–55, 57–61, 63, 65, 66, 70]. In contrast, five RCTs
Hill et al. Implementation Science (2020) 15:23 Page 6 of 14

involved organizational leaders in meetings but did

not make it clear if the implementation of the CQI
was discussed [40, 43, 47, 52, 56]. Six RCTs did not
describe the nature of the meetings [45, 50, 64, 67,
68, 71].
Training, often thought fundamental to implement-
ing CQI, was described in 24 RCTs [40, 44–54, 56,
57, 60–68, 70]. Fifteen RCTs used “in-person” training
(i.e., meet for face to face training) [44, 46, 48–52,
54, 61, 62, 64–67, 70], eight RCTs used “in-person
plus” training with the addition of other supporting
elements (e.g., tele-/video-conferencing [40, 45], web-
based materials [57, 60], handouts/manuals [53, 72]
or combinations of support [56, 68]). One RCT used
web-based training [47]. Duration of training ranged
from 1–3 h [48, 56, 57, 64, 70], 4–8 h [49, 51], 9–16
h [45, 68], and over 16 h [44, 53, 60, 65]. Duration of
training was not described in 15 RCTs [40, 43, 46,
47, 50, 52, 54, 55, 58, 59, 61–63, 66, 67].

Risk of bias
Assessment of the risk of bias showed that the
reliability of the results was uncertain due to the
variability in the methodological rigor of the RCTs (Fig.
2). As such, findings should be interpreted with caution.
Of the 28 RCTs, 26 RCTs had at least four criteria
judged unclear or at high risk of bias [40, 43–56, 59–
61, 63–68, 70, 71], with only 2 RCTs having five or
more criteria judged low risk [57, 58].

Effectiveness of CQI versus a non-CQI comparator

Of the 28 RCTs that compared CQI with a non-CQI
intervention, 24 RCTs reported clinical process outcomes
[40, 43–45, 48–55, 58, 60, 61, 63–68, 70, 71, 73], 17 RCTs
reported patient outcomes [40, 43–47, 50, 54–59, 61, 63,
64, 66], and 3 RCTs reported other outcomes [46, 50, 65]
(Tables 3, 4, and 5). The benefits that resulted from using
CQI interventions over those provided by non-CQI com-
parators were limited. Over half of the RCTs reported
no statistically significant difference between the in-
terventions in their effect on any of the outcome
measures assessed (clinical process 54.2% (13 RCTs)
[43, 44, 50–52, 55, 58, 63–68]; patient 64.7% (11
RCTs) [40, 43, 44, 46, 55–59, 63, 66]; other 100% (3
RCTs) [46, 50, 65]). Improvements were reported.
Some 29.2% of RCTs (7 RCTs [48, 49, 54, 57, 61, 62,
70]) assessing clinical process measures found a statis-
tically significant comparative benefit from CQI on
half or more of the outcomes. In contrast, 17.7% (3
RCTs [50, 61, 64]) and no RCTs found a beneficial
effect on half or more of patient and other outcomes,
respectively. The two RCTs at low risk of bias re-
ported no difference between the interventions in
Fig. 2 Risk of bias for included studies
terms of their effects on patient outcomes [57, 58];
Hill et al. Implementation Science (2020) 15:23 Page 7 of 14

Table 3 RCTs evaluating the effects of CQI compared to non-CQI interventions on clinical process outcomes
Sub-group Number Number (%) of RCTs reporting a statistically significant difference on different proportions of
of clinical process outcomes
studies
All outcomes Over half of outcomes Half of Under half of No outcomes One or more
outcomes outcomes outcomes
All studies 24 4 (16.7%) 2 (8.3%) 1 (4.2%) 4 (16.7%) 13 (54.2%) 11 (45.8%)
Clinical background
Primary Care 13 4 (30.8%) 0 (0%) 0 (0%) 2 (15.4%) 7 (53.8%) 6 (46.2%)
Secondary Care 9 0 (0%) 1 (11.1%) 1 (11.1%) 2 (22.2%) 5 (55.6%) 4 (44.4%)
Tertiary Care 0 0 0 0 0 0 0
Social Care 1 0 (0%) 1 (100.0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%)
Other 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Primary quality improvement model
Plan-Do-Study-Act 11 2 (18.2%) 1 (9.1%) 1 (9.1%) 2 (18.2%) 5 (45.5%) 6 (54.5%)
Model for Improvement 7 0 (0%) 1 (14.3%) 0 (0%) 1 (14.3%) 5 (71.4%) 2 (28.6%)
Root cause analysis 0 0 0 0 0 0 0
Other 6 2 (33.3%) 0 (0%) 0 (0%) 1 (16.7%) 3 (50.0%) 3 (50.0%)
Training type
Web-based 0 0 0 0 0 0 0
In person 14 4 (28.6%) 1 (7.1%) 1 (7.1%) 0 (0%) 8 (57.1%) 6 (42.9%)
In person plus 7 0 (0%) 1 (14.3%) 0 (0%) 4 (57.1%) 2 (28.6%) 5 (71.4%)
Not described 3 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3 (100.0%) 0 (0%)
Training duration
1-3 hours 4 2 (50.0%) 1 (25.0%) 0 (0%) 0 (0%) 1 (25.0%) 3 (75.0%)
4-8 hours 2 1 (50.0%) 0 (0%) 0 (0%) 0 (0%) 1 (50.0%) 1 (50.0%)
9-16 hours 2 0 (0%) 0 (0%) 0 (0%) 1 (50.0%) 1 (50.0%) 1 (50.0%)
>16 hours 4 0 (0%) 0 (0%) 0 (0%) 2 (50.0%) 2 (50.0%) 2 (50.0%)
Not described 12 1 (8.3%) 1 (8.3%) 1 (8.3%) 1 (8.3%) 8 (66.7%) 4 (33.3%)
Meetings
Participant leader, implementation 15 3 (20.0%) 2 (13.3%) 1 (6.7%) 2 (13.3%) 7 (46.7%) 8 (53.3%)
discussed
Participant leader, unclear implementation 3 0 (0%) 0 (0%) 0 (0%) 1 (33.3%) 2 (66.7%) 1 (33.3%)
discussed
Not described 6 1 (16.7%) 0 (0%) 0 (0%) 1 (16.7%) 4 (66.7%) 2 (33.3%)
Meeting schedule
Once a week or more 3 2 (66.7%) 0 (0%) 0 (0%) 1 (33.3%) 0 (0%) 3 (100.0%)
Fortnightly 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Monthly 7 1 (14.3%) 0 (0%) 1 (14.3%) 1 (14.3%) 4 (57.1%) 3 (42.9%)
Quarterly or less frequent 2 0 (0%) 1 (50.0%) 0 (0%) 0 (0%) 1 (50.0%) 1 (50.0%)
Not described 11 1 (9.1%) 1 (9.1%) 0 (0%) 2 (18.2%) 7 (63.6%) 4 (36.4%)
Range of year of publication
2010–2020 16 3 (18.8%) 1 (6.3%) 1 (6.3%) 2 (12.5%) 9 (56.3%) 7 (43.7%)
2000–2009 6 1 (16.7%) 1 (16.7%) 0 (0%) 1 (16.7%) 3 (50.0%) 3 (50.0%)
1990–1999 2 0 (0%) 0 (0%) 0 (0%) 1 (50.0%) 1 (50.0%) 1 (50.0%)

however, one RCT showed a statistically significant Sub-group analyses

benefit from the CQI intervention compared to non- Findings were similar in the sub-group analyses that in-
CQI comparator on clinical process measures [57]. vestigated the influence of the health setting, type of
Hill et al. Implementation Science (2020) 15:23 Page 8 of 14

Table 4 RCTs evaluating the effects of CQI compared to non-CQI Interventions on patient outcome measures
Subgroup Number Number (%) of RCTs reporting a statistically significant difference on different proportions of
of patient outcomes
studies
All outcomes Over half of Half of outcomes Under half of No outcomes One or more
outcomes outcomes outcomes
All studies 17 0 (0%) 2 (11.8%) 1 (5.9%) 3 (17.6%) 11 (64.7%) 6 (35.3%)
Clinical background
Primary Care 6 0 (0%) 1 (16.7%) 1 (16.7%) 0 (0%) 4 (66.7%) 2 (33.3%)
Secondary Care 7 0 (0%) 1 (14.3%) 0 (0%) 2 (28.6%) 4 (57.1%) 3 (42.9%)
Tertiary Care 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Social Care 2 0 (0%) 0 (0%) 0 (0%) 1 (50.0%) 1 (50.0%) 1 (50.0%)
Other 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Primary quality improvement model
Plan-Do-Study-Act 9 0 (0%) 0 (0%) 1 (11.1%) 2 (22.2%) 6 (66.7%) 3 (33.3%)
Model for Improvement 5 0 (0%) 2 (40.0%) 0 (0%) 0 (0%) 3 (60.0%) 2 (40.0%)
Root cause analysis 1 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%) 1 (100.0%)
Other 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Training type
Web-based 1 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%) 1 (100.0%)
In person 7 0 (0%) 2 (28.6%) 1 (14.3%) 1 (14.3%) 3 (42.9%) 4 (57.1%)
In person plus 5 0 (0%) 0 (0%) 0 (0%) 1 (20.0%) 4 (80.0%) 1 (20.0%)
Not described 4 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4 (100.0%) 0 (0%)
Training duration
1–3 hours 3 0 (0%) 1 (33.3%) 0 (0%) 0 (0%) 2 (66.7%) 1 (33.3%)
4–8 hours 0 0 0 0 0 0 0
9–16 hours 1 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%) 1 (100.0%)
>16 hours 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100%) 0 (0%)
Not described 12 0 (0%) 1 (8.3%) 1 (8.3%) 2 (16.7%) 8 (66.7%) 4 (33.3%)
Meetings
Participant leader, implementation discussed 10 0 (0%) 1 (10.0%) 0 (0%) 1 (10.0%) 8 (80.0%) 2 (20.0%)
Participant leader, unclear implementation 4 0 (0%) 0 (0%) 0 (0%) 1 (25.0%) 3 (75.0%) 1 (25.0%)
discussed
Not described 3 1 (33.3%) 1 (33.3%) 0 (0%) 1 (33.3%) 0 (0%) 3 (100.0%)
Meeting schedule
Once a week or more 0 0 0 0 0 0 0
Fortnightly 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Monthly 7 0 (0%) 0 (0%) 0 (0%) 2 (28.6%) 5 (71.4%) 2 (28.6%)
Quarterly or less frequent 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Not described 7 0 (0%) 2 (28.6%) 1 (14.3%) 1 (14.3%) 3 (42.9%) 4 (57.1%)
Range of year of publication
2010–2020 11 0 (0%) 1 (9.1%) 0 (0%) 2 (18.2%) 8 (72.7%) 3 (27.3%)
2000–2009 5 0 (0%) 1 (20.0%) 1 (20.0%) 1 (20.0%) 2 (40.0%) 3 (60.0%)
1990–1999 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)

CQI model used, and the influence of specific core fea- significant benefit from CQI compared to the non-CQI
tures of the CQI approach (e.g., type and duration of comparator on all the outcomes assessed. For the out-
training, type and schedule of meetings). In most sub- comes defined as “other,” this included all three RCTs
groups, over 50% of RCTs reported no statistically finding no statistically significant effect [46, 50, 65].
Hill et al. Implementation Science (2020) 15:23 Page 9 of 14

Table 5 RCTs evaluating the effects of CQI compared to non-CQI interventions on other outcome measures
Subgroup Number of Number (%) of RCTs reporting a statistically significant difference on different proportions of
studies other outcomes
All outcomes Over half of Half of Under half of No outcomes One or more
outcomes outcomes outcomes outcomes
All studies 3 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3 (100.0%) 0 (0%)
Clinical background
Primary Care 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Secondary Care 0 0 0 0 0 0 0
Tertiary Care 0 0 0 0 0 0 0
Social Care 0 0 0 0 0 0 0
Other 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Primary quality improvement model
Plan-Do-Study-Act 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Model for Improvement 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Root cause analysis 0 0 0 0 0 0 0
Other 0 0 0 0 0 0 0
Training type
Web-based 0 0 0 0 0 0 0
In person 3 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3 (100.0%) 0 (0%)
In person plus 0 0 0 0 0 0 0
Not described 0 0 0 0 0 0 0
Training duration
1–3 hours 0 0 0 0 0 0 0
4–8 hours 0 0 0 0 0 0 0
9–16 hours 0 0 0 0 0 0 0
>16 hours 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Not described 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Meetings
Participant leader, implementation 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
discussed
Participant leader, unclear implementation 0 0 0 0 0 0 0
discussed
Not described 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Meeting schedule
Once a week or more 0 0 0 0 0 0 0
Fortnightly 0 0 0 0 0 0 0
Monthly 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
Quarterly or less frequent 0 0 0 0 0 0 0
Not described 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
Range of year of publication
2010–2020 2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 2 (100.0%) 0 (0%)
2000–2009 1 0 (0%) 0 (0%) 0 (0%) 0 (0%) 1 (100.0%) 0 (0%)
1990–1999 0 0 0 0 0 0 0

There were some exceptions where more than 50% of of training (i.e., clinical process outcomes: in-person plus
RCTs reported a statistically significant benefit from training; patient outcomes: in-person training), as well as
CQI. These were limited to the effects of specific types types and frequencies of meetings (i.e., patient outcomes:
Hill et al. Implementation Science (2020) 15:23 Page 10 of 14

not described) on the clinical process and patient out- Meeting type and frequency
comes. Benefits from the use of CQI compared to non- The type of meeting and their frequency appear to have
CQI comparators were evident, although these varied some influence on the effectiveness of CQI. When it was
between the different sub-groups and outcomes clear that meetings involved a discussion of the imple-
considered. mentation of the improvement initiatives, a higher pro-
portion of RCTs (40% (6 RCTs)) [48, 49, 54, 57, 61, 70]
Care setting found a statistically significant benefit on half or more of
In terms of the setting of care, CQI appeared marginally the clinical process outcomes reported compared to
more effective in primary care than in secondary care. when it was not discussed (0%). Where patient outcomes
Over 30% of RCTs in primary care reported a statisti- were assessed, meetings that were not described had a
cally significant improvement in half or more of the clin- statistically significant beneficial effect on half or more
ical process (4 RCTs) [48, 49, 70, 71] and patient outcomes in more RCTs than other types of meetings
outcomes (2 RCTs) [50, 64] compared to less than 23% (66.6% (2 RCTs)) [50, 64]. The effects of meeting fre-
for secondary care for clinical process (2 RCTs) [54, 61] quency were less clear. Meetings that were at least
and patient outcomes (1 RCT) [61]. The effectiveness of weekly (66.7% (2 RCTs)) [48, 49] and meetings that were
CQI in other settings (i.e., tertiary care, social care, or monthly (28.6% (2 RCTs)) [54, 70], appeared to be more
other) was less clear given the limited evidence available effective than other meeting schedules in producing sta-
[46, 47, 56, 57, 65]. tistically significant improvements in half or more out-
comes in RCTs assessing clinical process measures. In
contrast, meetings that did not describe their frequency
CQI models had greater influence on RCTs reporting patient out-
PDSA and the MoI were the main CQI models used. Al- comes (42.9% (3 RCTs)) [50, 61, 64].
though PDSA appeared more effective than MoI in im-
proving half or more of clinical process outcomes in Range of year of publication
RCTs (36.4% (4 RCTs) [48, 54, 57, 70] versus 14.3% (1 The majority of RCTs were published from 2010 to
RCT) [61], respectively), the reverse was found for pa- 2020. There appeared to be no consistent improvement
tient outcomes (11.1% (1 RCT) [50] versus 40% (2 RCTs) in the effectiveness of CQI over time for all outcomes.
[61, 64], respectively). Other unspecified models of CQI Similar effects were reported when RCTs published be-
also appeared effective in impacting on half or more of tween 2000 and 2009 (33.3% (2 RCTs)) [61, 71] were
the clinical process outcomes in 33.3% of RCTs (2 compared with those between 2010 and 2020 (31.4% (5
RCTs) [49, 71]. RCTs)) [48, 49, 54, 57, 70] in producing statistically sig-
nificant improvements in half or more clinical process
Training type and duration outcomes. For patient outcomes, a difference was evi-
In-person training was used most frequently and had the dent with fewer RCTs reporting a statistically significant
largest influence on outcomes, leading to statistically sig- improvement in half or more outcomes between 2010
nificant improvements in half or more of outcomes in and 2020 (9.1% (1 RCT)) [50] than 2000 and 2009 (40%
42.8% (6 RCTs) [48, 49, 54, 61, 70, 71] and 42.9% (3 (2 RCTs)) [50, 61].
RCTs) [50, 61, 64] of RCTs assessing clinical process
and patient outcomes respectively. Person plus training Discussion
with additional elements was beneficial in half or more Increasingly the provision of health and social care has
of outcomes in 14.3% (1 RCT) [57] of RCTs assessing been shaped by the challenges of a growing demand for
clinical process outcomes. Although a range of training services, pressures on available funding and a continued
durations were used, shorter training durations appeared drive for efficiency [1]. Different approaches have been
more effective. Training sessions of 1 to 3 h and 4 to 8 h adopted in an attempt to maintain the comprehensive-
were beneficial in improving half or more outcomes in ness and quality of care, and to tackle inequity in
75% (3 RCTs [48, 57, 70] and 50% (1 RCT [64], respect- provision of services [74]. Recently, attention has shifted
ively) of RCTs assessing clinical process outcomes. Simi- to improving services by developing the capabilities and
larly, training lasting 1 to 3 h was shown to be beneficial capacity of organizations through building their know-
for 33.3% (1 RCT) [64] of RCTs assessing patient out- ledge, skills, and infrastructure [74]. The focus on
comes. Training where the duration was not described system-level quality improvement has resulted in CQI
had some beneficial effects on half or more of outcomes methods being identified, and increasingly used, as an
in 24.9% (3 RCTs) [54, 61, 71] and 16.6% (2 RCTs) [50, approach to enhance the quality of care and reduce costs
61] of RCTs assessing clinical process and patient out- [3–9, 72]. Despite its effectiveness within industrial and
comes, respectively. manufacturing sectors, it remains unclear whether CQI
Hill et al. Implementation Science (2020) 15:23 Page 11 of 14

could be successfully employed in the health care sector. incomplete or inconsistent reporting of the details of the
In systematically reviewing the evidence comparing the approach taken [22, 26, 32]. Although the evidence base
use of CQI with non-CQI interventions in health care, it has grown in recent years, there has been no discernible
was apparent that, regardless of the growth in evidence change in the effectiveness of CQI within the health care
in the last 10 years, the results were largely equivocal. setting. This may be due to several factors; however, its
Although this appears to perpetuate much of the uncer- likely to reflect the fact that studies undertaken are het-
tainty, we identified elements of CQI that may prove erogeneous in nature through the approaches to CQI
beneficial in improving outcomes and possible reasons used, populations studied, and outcomes reported.
for our findings that may inform further research. Socio-economic health inequalities were not reported in
Our findings appear to concur with those of previous any RCTs, which is not uncommon outside public health
systematic reviews on developing professional practice research, appearing to reflect their primary focus on the
and improving health care outcomes [26, 31], whether health condition and not the other underlying determi-
showing limited benefit [31], the influence of different nants of population health.
components [26, 31], and/or reasons for the continuing The limited effects of CQI initiatives may reflect sev-
uncertainties [22, 26, 28, 31, 32]. Where CQI appeared eral factors. First, health and social care organizations,
effective, collaboration and communication between both nationally and locally, are complex organizations
health care professionals appeared important. We found which may lack the necessary structure, resources, and
that meetings helped to facilitate the implementation of resolve to operationalize CQI initiatives effectively and
CQI, particularly when meetings were led by participant consistently [81, 82]. Given the opportunity for ap-
leaders, who were an integral part of multidisciplinary proaches to CQI to be adapted to local conditions, there
teams, focusing implementation of initiatives through is a chance for variation in their implementation. This
cooperative working. If these meetings were held fre- may reduce the inherent strengths of the CQI approach,
quently, such as weekly rather than monthly, this limit its effectiveness and make it more difficult to re-
seemed to improve the effectiveness of the CQI ap- search. Second, CQI initiatives are often implemented
proach taken. The importance of direct communication over a short period, restricting the opportunity to affect
was re-enforced through the benefits reported for CQI the different outcome measures assessed in the RCTs,
initiatives that used person focused face to face training, particularly patient-based outcomes. Third, recognition
which appeared more effective than other forms of train- of the importance of different components used in CQI
ing (e.g., web-based training or combinations of training (e.g., audit, feedback, meetings, and training), has re-
methods) and were thought to help improve competence sulted in their adoption as part of standard management
and motivation [75]. Others have found similar effects practice. Consequently, they are increasingly part of dif-
through different forms of interaction between those in- ferent management interventions that are compared in
volved in CQI [28, 30, 76]. Audit and feedback have trials, effectively controlling for their effects. Fourth,
been recognized as important facilitators when imple- identifying the reasons underlying the effectiveness of
menting CQI, with increased intensity of support more specific approaches to CQI has proven difficult to clarify.
effective in helping to incorporate improvements into This may reflect their frequent adaptation during im-
practice [28, 30, 76]. The impact of collaboration and ac- plementation and that details of the approach were
tive communication may help to explain the apparent often incompletely reported [22, 26, 32]. Although a
benefits from the use of CQI in primary care, where pragmatic approach to the use of CQI may be neces-
team structures reflect those used in operationalizing sary in practice, adherence to the core components
CQI methods [77, 78] and such initiatives are incentiv- and more complete reporting of the different models
ized through other mechanisms (e.g., Quality and Out- used in trials would help to distinguish which models
comes Framework) [79]. Despite several different and elements are most effective [22, 26, 28, 31, 32].
approaches to CQI, we identified that PDSA and MoI Fifth, the limited evidence identified and its poor
were the models most frequently used, showing benefit quality may result in uncertainty in the findings. The
on clinical process and patient outcomes in a third of unclear or high risk of bias reported for most RCTs
trials respectively. PDSA was previously reported to be may reflect either the inherent challenges in conduct-
an effective approach in improving health outcomes ing RCTs of CQI initiatives (e.g., blinding in cluster
[32]. The rationale for the use of PDSA and MoI, and RCTs) or a lack of understanding of the importance
the reasons for their effectiveness in specific situations, of ensuring, and reporting, the rigor used in imple-
has proven difficult to clarify. This may reflect the fre- menting the study methodology. It may be that the
quent adaptation of CQI models during implementation use of RCTs for evaluating CQI is undermined by the
rendering the differences unclear [80], that models often challenges faced and other approaches could
have overlapping features [17] and frequently there is compliment such experimental studies [82].
Hill et al. Implementation Science (2020) 15:23 Page 12 of 14

The systematic review had certain strengths, including Conclusion

the following: it was produced following a registered re- CQI is an important and proven approach to improving
search protocol by independent researchers, clearly de- the quality and efficiency of industrial processes, which
scribing the methods followed; identified evidence has drawn considerable and growing attention in health
through comprehensive searches of electronic databases, care. Evaluations of its use in health have been inad-
reference checking and citation checks; selected studies, equate, causing uncertainty as to its benefits. It is evident
extracted data, and assessed risk of bias using standard that in certain situations, it has had significant effects on
pre-piloted forms and processes; and involved public ad- improving the provision of health care, although these
visors in commenting on the research protocol and final were limited. Further independent research is required
report. Also, it had certain limitations, such as searches to clarify what approaches to CQI may be employed to
could have been extended to other sources; inclusion improve the quality and efficiency of service provision.
criteria were limited to RCTs which, although the gold
standard for assessing effectiveness through limiting po- Supplementary information
tential confounding, may restrict the opportunity to as- Supplementary information accompanies this paper at https://doi.org/10.
1186/s13012-020-0975-2.
sess more real-world evidence provided by other
comparative study designs; comparisons were with non- Additional file 1. PRISMA 2009 Checklist
CQI approaches, removing the opportunity to directly Additional file 2. Search strategy used for MEDLINE (via Ovid) from
compare between different CQI approaches; details of database inception to 23 February 2019
the studies were limited in the publications and further
evidence was not obtained from study authors; extrac- Abbreviations
tion of data and assessment of risk of bias were under- AMED: Allied and Complimentary Medicine Database; CINAHL: Current
Nursing and Allied Health Literature; CQI: Continuous quality improvement;
taken by a single reviewer with information checked by a HMIC: Healthcare Management Information Consortium; LISTA: Library
second reviewer, providing the opportunity for error; the Information Science and Technology Abstracts; MDT: Multi-disciplinary team;
synthesis categorized the evidence, limiting the extent of MOI: Model of Improvement; n: Number; NHS EED: NHS Economic Evaluation
Database; PDSA: Plan, Do, Study, Act; PRISMA: Preferred Reporting Items for
data presented from each RCT; and a meta-analysis was Systematic Reviews and Meta-Analyses; RCT: Randomized controlled trial
not undertaken.
Further research into the effectiveness of CQI inter- Acknowledgements
We would like to thank William Morton and Shamim Khan who contributed
ventions in health and social care would be beneficial. to the systematic review as public advisors for the NIHR ARC NWC. Also, we
A systematic review comparing different CQI models would like to thank Janet Reed who conducted the literature searches for
and other active comparisons may help to identify the the systematic review.
elements of these approaches that are useful to orga- Authors’ contributions
nizations. It could include experimental and non- JR contributed to the development of the research protocol; undertook
experimental comparative studies and look at the spe- study selection, data extraction, and assessment of risk of bias; synthesized
the results; and made a major contribution to writing of the manuscript.
cific influence of potentially important moderators AMS contributed to the development of the research protocol; undertook
(e.g., training methods/type and focus of meeting). If study selection, data extraction, and assessment of risk of bias; and assisted
further RCTs are going to be undertaken it is import- with the synthesis of the evidence. PS contributed as a public advisor to the
development of the research protocol and to the reporting and
ant that they take a mixed-method approach, as it is interpretation of the findings of the systematic review. AC developed the
currently unclear within the literature exactly which research protocol, contributed to the study selection, data extraction, and
moderators are important. Any RCTs should be con- assessment of risk of bias; contributed to the synthesis of the evidence; and
led the writing of the manuscript. All authors read and approved the final
ducted by independent researchers that assess out- manuscript.
comes over a longer period, as this would help to
clarify if the benefits could be realized in terms of Funding
The study is funded, and the co-authors (AC, JH) are part-funded, by the Na-
clinical process or organizational outcomes and, more
tional Institute for Health Research (NIHR) Applied Research Collaboration
importantly, for patient-related outcomes. The RCTs North West Coast (ARC NWC). The views expressed are those of the authors
could specifically compare the different key compo- and not necessarily those of the NIHR or Department of Health and Social
Care.
nents that have been identified as core to the differ-
ent approaches to CQI. Any RCT that is undertaken Availability of data and materials
should report against a standard set of outcomes, Data sharing is not applicable to this article as no datasets were generated
provide full descriptions of all elements of the CQI or analyzed during the current study. All original data synthesised can be
obtained from the selected studies.
process, and consider health inequalities. It has been
evident that the quality of the evidence and the qual- Ethics approval and consent to participate
ity of its reporting is poor, preventing a full under- Not applicable

standing of the findings and the context in which Consent for publication
they have been attained. This should be addressed. Not applicable
Hill et al. Implementation Science (2020) 15:23 Page 13 of 14

Competing interests 25. Cheung YY, Riblet NBV, Osunkoya TO. Use of iterative cycles in quality
The authors declare that they have no competing interests improvement projects in imaging; systematic review. J Am Coll Radiol. 2018;
15(11):1587–602.
Author details 26. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and
1
Faculty of Health and Wellbeing, University of Central Lancashire (UCLan), feedback: effects on professional practice and health care outcomes.
Preston, Lancashire PR1 2HE, UK. 2Warrington, UK. Cochrane Database Syst Rev. 2006;(2):CD000259.
27. Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP.
Received: 14 October 2019 Accepted: 19 February 2020 Evidence for the impact of quality improvement collaboratives: systematic
review. BMJ. 2008;336(7659):1491–4.
28. Gardner K, Sibthorpe B, Chan M, Sargent G, Dowden M, McAullay D.
Implementation of continuous quality improvement in aboriginal and
References
Torres Strait islander primary health care in Australia: a scoping systematic
1. NHS England. NHS five year forward view. England: NHS; 2014.
review. BMC Health Serv Res. 2018;18(1):541.
2. Academy of Medical Royal Colleges. Quality improvement–training for
29. Nunes JW, Seagull FJ, Rao P, Segal JH, Mani NS, Heung M. Continuous
better outcomes. 2016.
quality improvement in nephrology: a systematic review. BMC Nephrol.
3. Ham C, Berwick D, Dixon J. Improving quality in the English NHS a strategy
2016;17(1):190.
for action. London: The Kings fund; 2016.
30. Candas B, Jobin G, Dube C, Tousignant M, Abdeljelil AB, Grenier S, et al.
4. Schneider A. How quality improvement in health care can help to achieve
Barriers and facilitators to implementing continuous quality improvement
the millennium development goals. World Health Organization. 2006;84(4):
programs in colonoscopy services: a mixed methods systematic review.
257–336.
Endosc Int Open. 2016;4(2):E118–33.
5. Ferlie EB, Shortell SM. Improving the quality of health care in the United
31. Vecchi S, Agabiti N, Mitrova S, Cacciani L, Amato L, Davoli M, et al. Audit
Kingdom and the United States: a framework for change. Milbank Q. 2001;
and feedback, and continuous quality improvement strategies to improve
79(2):281–315.
the quality of care for type 2 diabetes: a systematic review of literature.
6. Cox S, Wilcock P, Young J. Improving the repeat prescribing process in a
Epidemiol Prev. 2016;40(3–4):215–23.
busy general practice. A study using continuous quality improvement
32. Knudsen SV, Laursen HVB, Johnsen SP, Bartels PD, Ehlers LH, Mainz J. Can
methodology. Qual Health Care. 1999;8(2):119.
quality improvement improve the quality of care? A systematic review of
7. Rokoske FS, Schenck AP, Hanson LC. The potential use of autopsy for
reported effects and methodological rigor in plan-do-study-act projects.
continuous quality improvement in hospice and palliative care. Medscape J
BMC Health Serv Res. 2019;19(1):683.
Med. 2008;10(12):289.
8. Manyazewal T, Mekonnen A, Demelew T, Mengestu S, Abdu Y, Mammo D, 33. O'Neill SM, Hempel S, Lim YW, Danz MS, Foy R, Suttorp MJ, et al. Identifying
et al. Improving immunization capacity in Ethiopia through continuous continuous quality improvement publications: what makes an improvement
quality improvement interventions: a prospective quasi-experimental study. intervention ‘CQI’? BMJ Qual Saf. 2011;20(12):1011–9.
Infect Dis Poverty. 2018;7(1):119. 34. Hempel S, Shekelle PG, Liu JL, Sherwood Danz M, Foy R, Lim YW, et al.
9. Werth GR, Connelly DP. Continuous quality improvement and medical Development of the Quality Improvement Minimum Quality Criteria Set (QI-
informatics: the convergent synergy. Proc Annu Symp Comput Appl Med MQCS): a tool for critical appraisal of quality improvement intervention
Care. 1992:631–5. publications. BMJ Qual Saf. 2015;24(12):796–804.
10. Barney M. Motorola, Inc. Forum Mag. 2002;1(3):13–6. 35. Whitehead M, Bambra C, Barr B, Bowles J, Caulfield R, Doran T, et al. Due
11. Deming WE. The new economics. Cambridge: MIT Press; 2000. north: report of the inquiry on health equity for the north. 2014.
12. Griffin P, Nembhard H, DeFlitch C, Bastian N, Kang H, Muñoz D. Healthcare 36. NIHR CLAHRC North West Coast. NIHR CLAHRC North West Coast Health
systems engineering. Hoboken: Wiley; 2016. Inequalities Assessment Toolkit (HIAT). 2017. Available from: http://www.hiat.
13. Shortell SM, O'Brien JL, Carman JM, Foster RW, Hughes EF, Boerstler H, et al. org.uk/index.html.
Assessing the impact of continuous quality improvement/total quality 37. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred
management: concept versus implementation. Health Serv Res. 1995;30(2): reporting items for systematic reviews and meta-analyses: the PRISMA
377–401. statement. BMJ. 2009;339:b2535.
14. Suneja A, Suneja C. Lean doctors. Milwaukee: ASQ Quality Press; 2010. 38. Higgins J, Green S. Cochrane handbook for systematic reviews of
15. Boerstler H, Foster RW, O'Connor EJ, O'Brien JL, Shortell SM, Carman JM, et interventions: Wiley; 2011.
al. Implementation of total quality management: conventional wisdom 39. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for
versus reality. Hosp Health Serv Adm. 1996;41(2):143–59. automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2015;
16. Shortell SM, Bennett CL, Byck GR. Assessing the impact of continuous 23(1):193–201.
quality improvement on clinical practice: what it will take to accelerate 40. Chaney EF, Rubenstein LV, Liu CF, Yano EM, Bolkan C, Lee M, et al.
progress. Milbank Q. 1998;76(4):593–624. Implementing collaborative care for depression treatment in primary care: a
17. McLaughlin C, Kaluzny A. Continuous quality improvement in health care: cluster randomized evaluation of a quality improvement practice redesign.
theory, implementation and application. Gaithersburg: Aspen Publishers; Implement Sci. 2011;6:121.
2006. 41. Coronado GD, Vollmer WM, Petrik A, Aguirre J, Kapka T, DeVoe J, et al.
18. Wallin L, Bostrom AM, Wikblad K, Ewald U. Sustainability in changing Strategies and opportunities to STOP colon cancer in priority populations:
clinical practice promotes evidence-based nursing care. J Adv Nurs. pragmatic pilot study design and outcomes. BMC Cancer. 2014;14(1):55.
2003;41(5):509–18. 42. Filardo G, Nicewander D, Herrin J, Edwards J, Galimbertti P, Tietze M, et al. A
19. Larson JS, Muller A. Managing the quality of healthcare. J Health Health hospital-randomized controlled trial of a formal quality improvement
Serv Adm. 2002;25(3):261–80. educational program in rural and small community Texas hospitals: one
20. Rubenstein L, Khodyakov D, Hempel S, Danz M, Salem-Schatz S, Foy R, et al. year results. Int J Qual Health Care. 2009;21(4):225–32.
How can we recognize continuous quality improvement? Int J Qual Health 43. Goldberg HI, Wagner EH, Fihn SD, Martin DP, Horowitz CR, Christensen DB,
Care. 2014;26(1):6–15. et al. A randomized controlled trial of CQI teams and academic detailing:
21. Zhan L, Finch L. Accelerated education in nursing : challenges, strategies, can they alter compliance with guidelines? Jt Comm J Qual Improv. 1998;
and future directions. New York: Springer; 2012. 24(3):130–42.
22. Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic 44. Homer CJ, Forbes P, Horvitz L, Peterson LE, Wypij D, Heinrich P. Impact of a
review of the application of the plan–do–study–act method to improve quality improvement program on care and outcomes for children with
quality in healthcare. BMJ Qual Saf. 2014;23(4):290. asthma. Arch Pediatr Adolesc Med. 2005;159(5):464–9.
23. Card AJ, Ward J, Clarkson PJ. Successful risk assessment may not always lead 45. Horbar JD, Carpenter JH, Buzas J, Soll RF, Suresh G, Bracken MB, et al.
to successful risk control: a systematic literature review of risk control after Collaborative quality improvement to promote evidence based surfactant
root cause analysis. J Healthc Risk Manag. 2012;31(3):6–12. for preterm infants: a cluster randomised trial. BMJ. 2004;329(7473):1004.
24. Deblois S, Lepanto L. Lean and six sigma in acute care: a systematic review 46. Hunter SB, Rutter CM, Ober AJ, Booth MS. Building capacity for continuous
of reviews. Int J Health Care Qual Assur. 2016;29(2):192–208. quality improvement (CQI): a pilot study. J Subst Abus Treat. 2017;81:44–52.
Hill et al. Implementation Science (2020) 15:23 Page 14 of 14

47. Kane RL, Huckfeldt P, Tappen R, Engstrom G, Rojido C, Newman D, et al. mortality in three districts of Malawi: MaiKhanda, a cluster randomized
Effects of an intervention to reduce hospitalizations from nursing homes: a controlled effectiveness trial. Int Health. 2013;5(3):180–95.
randomized implementation trial of the INTERACT program. JAMA Intern 67. Sherr K, Gimbel S, Rustagi A, Nduati R, Cuembelo F, Farquhar C, et al.
Med. 2017;177(9):1257–64. Systems analysis and improvement to optimize pMTCT (SAIA): a cluster
48. Meropol SB, Schiltz NK, Sattar A, Stange KC, Nevar AH, Davey C, et al. randomized trial. Implement Sci. 2014;9:55.
Practice-tailored facilitation to improve pediatric preventive care delivery: a 68. Filardo G, Nicewander D, Hamilton C, Herrin J, Galimbretti P, Tietze M, et al.
randomized trial. Pediatrics. 2014;133(6):e1664–75. A hospital-randomized controlled trial of an educational quality
49. Noël PH, Romero RL, Robertson M, Parchman ML. Key activities used by improvement intervention in rural and small community hospitals in Texas
community based primary care practices to improve the quality of diabetes following implementation of information technology. Am J Med Qual. 2007;
care in response to practice facilitation. Qual Prim Care. 2014;22(4):211–9. 22(6):418–27.
50. Rubenstein LV, Meredith LS, Parker LE, Gordon NP, Hickey SC, Oken C, et al. 69. Hunt P, Hunter SB, Levan D. Continuous quality improvement in substance
Impacts of evidence-based quality improvement on depression in primary abuse treatment facilities: how much does it cost? J Subst Abus Treat. 2017;
care: a randomized experiment. J Gen Intern Med. 2006;21(10):1027–35. 77:133–40.
51. Ruhe MC, Bobiak SN, Litaker D, Carter CA, Wu L, Schroeder C, et al. 70. Coronado GD, Vollmer WM, Petrik A, Taplin SH, Burdick TE, Meenan RT, et al.
Appreciative inquiry for quality improvement in primary care practices. Qual Strategies and opportunities to STOP colon cancer in priority populations:
Manag Health Care. 2011;20(1):37–48. design of a cluster-randomized pragmatic trial. Contemp Clin Trials. 2014;
52. Shaw EK, Ohman-Strickland PA, Piasecki A, Hudson SV, Ferrante JM, 38(2):344–9.
McDaniel RR Jr, et al. Effects of facilitated team meetings and learning 71. Canovas J, Hernandez P, Botella J. Effectiveness of internal quality assurance
collaboratives on colorectal cancer screening rates in primary care practices: programmes in improving clinical practice and reducing costs. J Eval Clin
a cluster randomized trial. Ann Fam Med. 2013;11(3):220–8 S1-8. Pract. 2009;15:813–9.
53. Solberg LI, Kottke TE, Brekke ML, Calomeni CA, Conn SA, Davidson G. Using 72. Care Quality Commission. Quality improvement in hospital trusts: sharing
continuous quality improvement to increase preventive services in clinical learning from trust on a journey of QI. 2018.
practice—going beyond guidelines. Prev Med. 1996;25(3):259–67. 73. Kellams A, Parker MG, Geller NL, Moon RY, Colson ER, Drake E, et al.
54. Solomon DH, Losina E, Lu B, Zak A, Corrigan C, Lee SB, et al. TodaysBaby quality improvement: safe sleep teaching and role modeling in
Implementation of treat-to-target in rheumatoid arthritis through a learning 8 US maternity units. Pediatrics. 2017;140(5).
collaborative: results of a randomized controlled trial. Arthritis Rheum. 2017; 74. Jones B, Horton T, Warburton W. The improvement journey. Why
69(7):1374–80. organisation wide improvement in health care matters, and how to get
55. Dirks M, Niessen LW, van Wijngaarden JD, Koudstaal PJ, Franke CL, van started; 2019.
Oostenbrugge RJ, et al. Promoting thrombolysis in acute ischemic stroke. 75. Boonyasai R, Windish D, Chakraborti C, Feldman L, Rubin H, Bass E.
Stroke. 2011;42(5):1325–30. Effectiveness of teaching quality improvement to clinians: a systematic
56. van der Veer SN, de Vos ML, van der Voort PH, Peek N, Abu-Hanna A, review. JAMA. 2007;298(9):1023–37.
Westert GP, et al. Effect of a multifaceted performance feedback strategy on 76. Flottorp S, Jamtvedt G, Gibis B, McKee M. Using audit and feedback to
length of stay compared with benchmark reports alone: a cluster health professionals to improve the quality and safetty of health care, Policy
randomized trial in intensive care. Crit Care Med. 2013;41(8):1893–904. summary prepared for the Belgian EU Presidency Conference on Investing
57. Kennedy C, Ioannidis G, Thabane L, Adachi JD, Marr S, Giangregorio LM, et in Europe’s health workforce of tomorrow: scope for innovation and
al. Successful knowledge translation intervention in long-term care: final collaboration (La Hulpe, 9–10 September 2010): European Obervatory on
results from the vitamin D and osteoporosis study (ViDOS) pilot cluster Health Systems and Policies; 2010. Contract No.: Policy Summary 3.
randomized controlled trial. Trials. 2015;16:214. 77. Geboers H, Grol R, Bosch W, Hoogen H, Mokkink H, Montfort P, et al. A
58. Liddy C, Hogg W, Singh J, Taljaard M, Russell G, Deri Armstrong C, et al. A model for continuous quality improvement in small scale practice. Qual
real-world stepped wedge cluster randomized trial of practice facilitation to Health Care. 1999;8(1):43–8.
improve cardiovascular care. Implement Sci. 2015;10:150. 78. Solberg LI, Kottke TE, Brekke ML. Will primary care clinics organize
59. Curran E, Harper P, Loveday H, Gilmour H, Jones S, Benneyan J, et al. Results themselves to improve the delivery of preventive services? A randomized
of a multicentre randomised controlled trial of statistical process control controlled trial. Prev Med. 1998;27(4):623–31.
charts and structured diagnostic tools to reduce ward-acquired meticillin- 79. Dawda P, Jenkins R, Varnam R. Quality improvement in general practice. An
resistant Staphylococcus aureus: the CHART project. J Hosp Infect. 2008; inquiry into the quality of general practice in England. Discussion paper.
70(2):127–35. 2010.
80. Boaden R, Harvey G, Moxham C, Proudlove N. Quality improvement: theory
60. Power M, Tyrrell PJ, Rudd AG, Tully MP, Dalton D, Marshall M, et al. Did a
and practice in healthcare.: NHS Institute for Innovation and Improvement;
quality improvement collaborative make stroke care better? A cluster
2008.
randomized trial. Implement Sci. 2014;9(1):40.
81. Braithwaite J, Churruca K, Long J, Ellis L, Herkes J. When complexity science
61. Carlhed R, Bojestig M, Wallentin L, Lindström G, Peterson A, Åberg C, et al.
meets implementation science: a theoretical and empirical analysis of
Improved adherence to Swedish national guidelines for acute myocardial
systems change. BMC Med. 2018;16.
infarction: the Quality Improvement in Coronary Care (QUICC) study. Am
82. Lipsitz L. Understanding Helth care as a complex system. The foundation of
Heart J. 2006;152(6):1175–81.
unintended consequences. JAMA. 2012;308:243.
62. Gascón Cánovas JJ, Saturno Hernández PJ, Antón Botella JJ. Effectiveness of
internal quality assurance programmes in improving clinical practice and
reducing costs. J Eval Clin Pract. 2009;15(5):813–9. Publisher’s Note
63. Huffman MD, Mohanan PP, Devarajan R, Baldridge AS, Kondal D, Zhao L, et Springer Nature remains neutral with regard to jurisdictional claims in
al. Effect of a quality improvement intervention on clinical outcomes in published maps and institutional affiliations.
patients in India with acute myocardial infarction: the ACS QUIK
randomized clinical trial. JAMA. 2018;319(6):567–78.
64. Barceló A, Cafiero E, de Boer M, Mesa AE, Lopez MG, Jiménez RA, et al.
Using collaborative learning to improve diabetes care and outcomes: the
VIDA project. Prim Care Diabetes. 2010;4(3):145–53.
65. Oyeledun B, Phillips A, Oronsaye F, Alo OD, Shaffer N, Osibo B, et al. The
effect of a continuous quality improvement intervention on retention-in-
care at 6 months postpartum in a PMTCT program in northern Nigeria:
results of a cluster randomized controlled study. J Acquir Immune Defic
Syndr. 2017;75(Suppl 2):S156–S64.
66. Colbourn T, Nambiar B, Bondo A, Makwenda C, Tsetekani E, Makonda-Ridley
A, et al. Effects of quality improvement in health facilities and community
mobilization through women's groups on maternal, neonatal and perinatal