Systematic Reviews: What, Why, and How

Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Cochrane graphic illustration of the systematic review process
What is a systematic review?

• Uses transparent procedures to find, evaluate and

synthesize the results of independent studies.
• Procedures are explicitly defined in advance, to
assure that the process is transparent and can be
replicated.
• This process is also designed to minimize bias.
Unique characteristics of a systematic review

• A systematic review must have:

– Clear inclusion and exclusion criteria
– Explicit search strategy
– Systematic coding and analysis of included studies
– Meta-analysis (where possible)
What is a meta-analysis?

• Optional component of a systematic review

– A statistical analysis of results from individual studies
• Increase power
• Improve estimates of the size of the effect
• Resolve uncertainty when reports disagree
What is a traditional review?

• Uses informal, unsystematic and subjective methods

to collect, interpret, and summarize information.
– Searching, quality appraisal and data synthesis are often
not documented.
– Since scientific methods are not routinely used to identify,
assess and synthesize information, assumptions cannot be
easily verified.
– Greater risk of author and selection bias
Comparison of traditional and systematic reviews

Traditional review Systematic review

• Subjective • Objective
• Broad question • Narrow question
• Methods unclear • Methods clearly defined
• Studies appraised
Evidence based pyramid

Reviews
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Information overload
EBP and clinical practice guidelines

• Healthcare decisions should be informed by the best

available research evidence.
– Many guidelines rely on previously published systematic
reviews
– Practice guideline quality is dependent on rigorous
systematic review methods and high quality primary
studies
• Primary studies
• Randomized controlled trials
• Systematic reviews
• Evidence based practice and clinical practice guidelines
Known and unknown areas of study

• SRs can show which treatments and prevention

methods have been proven to work - and what
remains unknown.
• SRs are important for pointing to areas where more
research is needed.
• Systematic reviews are the basis for what is often
called evidence-based medicine or health care.
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
 Stages of a systematic review

Develop a focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as necessary
 Stages of a systematic review

Develop a focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
 Research Question

• A clearly defined, focused systematic review begins

with a well formulated research question.
Research Question

• The research question guides the author in working

through many stages of the systematic review
process
– Defining inclusion and exclusion criteria
– Searching the literature
– Selecting studies
– Extracting data
– Analyzing and presenting results
FINER Criteria

• A research question should be:

– Feasible
– Interesting
– Novel
– Ethical
– Relevant
PICO Framework

• A well-established format for structuring research

questions is known by the acronym PICO.
– Patient or Population
– Intervention or Indicator
– Comparator or Control
– Outcome
PICO
• Patients • Patients undergoing upper abdominal surgery
• Intervention • Prophylactic physical therapy
• Comparison • No prophylactic physical therapy
• Outcome • Prevent post operative pulmonary complications
Types of clinical research questions

• Many types of research questions can be expressed

using PICO components.
– Therapy
– Diagnosis
– Prognosis
– Etiology / Harm
– Clinical Prediction Guides
Stages of a systematic review

Develop A focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
Inclusion and exclusion criteria

• One of the features that distinguish a systematic

review from a traditional review is the pre-
specification of inclusion and exclusion criteria.
Inclusion criteria

• Inclusion criteria are a combination of

• Aspects of the research question
• Population
• Intervention
• Comparison
• Outcome
• Study type
• Randomized controlled trials
• Observational studies
Stages of a systematic review

Develop A focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
Search the literature

• The goal of the literature search is to discover all

studies that meet the inclusion criteria
– Search comprehensively
• Terminology
• Databases
– Search for grey literature
• Not commercially published
– Search for unpublished studies
• Reduce risk of publication bias
Recommended databases

• The Cochrane Collaboration recommends searching

the following databases (at minimum):
– PubMed
– EMBASE
– Cochrane Central Register of Controlled Trials
Supplementary databases

• Interdisciplinary databases
– Scopus
– Web of Science
• Specialized databases
– CINAHL Plus
– PEDro: Physiotherapy Evidence Database
– PsycINFO
Search strategy

• The search strategy should be designed to identify

the maximum number of studies relevant to the
research question.
– The search strategy should be systematic, transparent and
reproducible.
– Database specific controlled vocabulary terms and all
relevant text words should be included in the search
strategy.
Supplementary searching techniques

• Search cited and citing references

– Scopus
– Web of Science
Supplementary searching techniques

• Hand search selected journals and conference

proceedings
• Conduct author searches for recent articles written
by topic experts
Unpublished and grey literature

• The inclusion of unpublished and grey literature may

minimize the potential effects of publication bias.
– Publication bias
• Occurs when the outcome of an experiment or research study
influences the decision about whether—or how quickly—the
manuscript may be published
What is grey literature?

• Grey literature refers to academic, business,

government or industry print or electronic literature
that is not controlled by commercial publishers.
– Conference proceedings
– Research reports
– Government reports
– Dissertations, theses
– Research monographs
– Organization websites
Sources of grey literature

• Conference proceedings
– EMBASE
– Scopus
– Web of Science
– Google
Sources of unpublished literature

• Clinical trials
– ClinicalTrials.gov
– Centerwatch.com
– EU Clinical Trials Register
– ISRCTN Registry
– OpenTrials
– WHO International Clinical Trials Registry Platform
Compile search results

• Compile search results using reference management

software (EndNote or Mendeley)
• Remove duplicate records
Document search process

• Document the search process

– Databases
– Dates searched
– Search strategies
– Limits (date ranges, publication types, language
restrictions)
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search The Literature
Select studies
Assess Study Quality
Extract Data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Select studies

• Identification of studies meeting inclusion criteria

should be done independently by two review
authors.
– Review titles and abstracts of retrieved citations.
– Review full text of studies which are found to meet the
inclusion criteria.
– Keep a record of reasons for inclusion or exclusion.
Tools for sorting

• Tools for sorting

– Reference management software
• EndNote
– New web application
• Rayyan
– Excel
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess study quality
Extract Data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Assess study quality

• Following the full text review, assess the selected

studies for risk of bias and study quality.
Tools for assessing risk of bias

• Tools for assessing risk of bias

– ROBIS: Risk of Bias in Systematic Reviews
– ROBINS-I tool: Risk of Bias in Non-randomized studies of
Interventions
– RoB 2.0 tool (revised tool for Risk of Bias in randomized
trials
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Extract data

• Extract reported findings from selected studies using

a data extraction form.
– Extraction forms and approaches should be determined by
the needs of the specific review.
– At least two review authors should independently extract
data from study reports.
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract Data
Analyze and present results
Interpret Results And Draw Conclusions
Update As Needed
Analyze and present results

• The findings from individual studies are aggregated

to produce a type of evidence synthesis appropriate
to the type of data within the review.
– Narrative synthesis – findings are summarized and
explained in words
– Quantitative/statistical synthesis – data from individual
studies are combined statistically and then summarized
(meta‐analysis)
Tables and Figures

• Tables and figures are used to present included

studies and their findings in a systematic and clear
format.
– Flow diagram
– Summary of findings table
– Forest plot
PRISMA Flow diagram

• PRISMA Flow diagram

– Depicts the flow of information through the different phases
of a systematic review
– Documents the number of studies that remain after each
stage of the selection process
– Maps the number of studies identified, included and
excluded, and the reasons for being excluded
– PRISMA Flow Diagram Generator
PRISMA Flow Diagram
Summary of findings table

• Summary of findings table

– Provides key information concerning the quality of evidence
– Depicts the magnitude of effect of the interventions
– Illustrates the sum of available data on all important
outcomes for a given comparison

Cochrane Handbook
Summary of findings table using GRADE methodology
Forest Plot

• Forest Plot
– A graphical display designed to illustrate the relative
strength of treatment effects in multiple quantitative
scientific studies addressing the same question (meta-
analysis)
 Forest Plot

Names of fictional studies on left; odds ratios and confidence intervals on right; odds ratios
(squares proportional to weights used in meta-analysis); summary measure (center line of
diamond); associated confidence intervals (lateral tips of diamond); solid vertical line of no
effect
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract Data
Analyze And Present Results
Interpret results and draw conclusions
Update As Needed
Interpret results and draw conclusions

• Statement of findings, discussion and conclusions

– Information on all important outcomes, including adverse
outcomes
– Quality of evidence for each outcome
– How values and preferences may bear on balance of
benefits, harms, and costs of interventions
THANK YOU