Scribd Logo
Upload

Welcome to Scribd!

  • 33 views

    Diy RCT Appraisal

    Uploaded by

    imperiallight
    This document discusses how to critically appraise randomized controlled trials. It provides a checklist of factors to consider when evaluating validity, including recruitment, allocation, maintenance of comparable groups, blinding of measurements, and accounting for placebo effects, chance and sample size. Statistical values like p-values and confidence intervals are also important to consider.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Diy RCT Appraisal

    Uploaded by

    imperiallight
    33 views43 pages
    This document discusses how to critically appraise randomized controlled trials. It provides a checklist of factors to consider when evaluating validity, including recruitment, allocation, maintenance of comparable groups, blinding of measurements, and accounting for placebo effects, chance and sample size. Statistical values like p-values and confidence intervals are also important to consider.

    Original Description:

    Diy Rct Appraisal

    Original Title

    Diy Rct Appraisal

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document discusses how to critically appraise randomized controlled trials. It provides a checklist of factors to consider when evaluating validity, including recruitment, allocation, maintenance of comparable groups, blinding of measurements, and accounting for placebo effects, chance and sample size. Statistical values like p-values and confidence intervals are also important to consider.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    33 views43 pages

    Diy RCT Appraisal

    Uploaded by

    imperiallight
    This document discusses how to critically appraise randomized controlled trials. It provides a checklist of factors to consider when evaluating validity, including recruitment, allocation, maintenance of comparable groups, blinding of measurements, and accounting for placebo effects, chance and sample size. Statistical values like p-values and confidence intervals are also important to consider.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 43

    Rapid Critical Appraisal

    of Controlled Trials
    Carl Heneghan
    Dept of Primary Health Care
    University of Oxford

    November 23rd 2009

    Five steps in EBM


    1. Formulate an answerable question
    2. Track down the best evidence
    3. Critically appraise the evidence for:

    Relevance
    Validity
    Impact (size of the benefit)
    Applicability

    4. Integrate with clinical expertise and patient values


    5. Evaluate our effectiveness and efficiency

    keep a record; improve the process

    Searching for critical appraisal checklists randomized controlled trials .


    11,100 articles (0.40 seconds)
    1.
    2.
    3.
    4.
    5.
    6.
    7.
    8.
    9.
    10.
    11.
    12.
    13.
    14.
    15.
    16.
    17.
    18.
    19.
    20.

    A CHECKLIST FOR APPRAISING RANDOMIZED CONTROLLED TRIALS


    Was the objective of the trial sufficiently described?
    Was a satisfactory statement given of the diagnostic criteria for entry to the trial?
    Were concurrent controls used (as opposed to historical controls)?
    Were the treatments well defined?
    Was random allocation to treatments used?
    Was the potential degree of blindness used?
    Was there a satisfactory statement of criteria for outcome measures? Was a primary outcome measure identified?
    Were the outcome measures appropriate?
    Was a pre-study calculation of required sample size reported?
    Was the duration of post-treatment follow-up stated?
    Were the treatment and control groups comparable in relevant measures?
    Were a high proportion of the subjects followed up?
    Were the drop-outs described by treatment and control groups?
    Were the side-effects of treatment reported?
    How were the ethical issues dealt with?
    Was there a statement adequately describing or referencing all statistical procedures used?
    What tests were used to compare the outcome in test and control patients?
    Were 95% confidence intervals given for the main results?
    Were any additional analyses done to see whether baseline characteristics (prognostic factors) influenced the outcomes
    observed?
    Were the conclusions drawn from the statistical analyses justified?

    Clinical Question
    In people who take long-haul flights does
    wearing graduated compression stockings
    prevent DVT?

    Page 71 and 95 in your books

    Causes of an Effect in a controlled trial


    Who would consider wearing stockings on a
    long haul flight?

    QUESTION:
    Participants
    Intervention Group (IG)
    & Comparison Group
    (CG)

    Outcome

    VALIDITY

    VALIDITY

    QUESTION:
    Participants
    Intervention Group (IG)
    & Comparison Group
    (CG)

    Outcome

    +
    -

    I
    G

    C
    G

    + A

    VALIDITY

    QUESTION:
    Participants

    Recruitment

    Intervention Group (IG)


    & Comparison Group
    (CG)

    Outcome

    +
    -

    I
    G

    C
    G

    + A

    VALIDITY

    QUESTION:

    Recruitment

    Participants
    Intervention Group (IG)
    & Comparison Group
    (CG)

    Outcome

    +
    -

    I
    G

    C
    G

    + A

    Allocation
    concealment
    comparable groups

    VALIDITY

    QUESTION:

    Recruitment

    Participants
    Intervention Group (IG)
    & Comparison Group
    (CG)

    Outcome

    +
    -

    I
    G

    C
    G

    + A

    Allocation
    concealment
    comparable groups
    Maintenance
    treated equally
    compliant

    VALIDITY

    QUESTION:

    Recruitment
    Participants
    Intervention Group (IG)
    & Comparison Group
    (CG)

    Outcome

    +
    -

    I
    G

    C
    G

    + A

    Allocation concealment
    comparable groups
    Maintenance
    Treated equally
    compliant

    Measurements
    blind? OR
    objective?

    Appraisal checklist - RAMMbo


    Study biases
    1.
    Recruitment

    Who did the subjects represent?


    2.
    Allocation
    Was the assignment to treatments randomised?
    Were the groups similar at the trials start?
    Maintenance

    Were the groups treated equally?

    Were outcomes ascertained & analysed for most patients?


    Measurements

    Were patients and clinicians blinded to treatment? OR

    Were measurements objective & standardised?

    3.

    4.

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993


    Page-95

    How were the patients recruited?

    Scurr et al, Lancet 2001; 357:1485-89

    Randomization
    Volunteers were
    randomized by
    sealed envelope
    to one of two
    groups.

    Envelopes

    Passengers were randomly allocated to one of two groups: one


    group wore class-I below-knee graduated elastic compression
    stockings, the other group did not.

    Scurr et al, Lancet 2001; 357:1485-89

    Take out the envelopes


    Sign the back

    You have now consented to the trial


    Please open your envelopes now

    Blue Bunnies
    Argued with your
    boss
    Been to New
    York

    Pink Bunnies

    Ensuring Allocation Concealment


    BEST most valid technique
     Central computer
    randomization

    DOUBTFUL
     Envelopes, etc

    NOT RANDOMIZED
     Date of birth, alternate days, etc

    Were the groups similar at the trials start?


    By chance a greater
    proportion of women
    were included in the
    stocking group

    Page 96

    Appraisal checklist - RAMMbo


    Study biases
    1.
    Recruitment

    Who did the subjects represent?


    2.
    Allocation
    Was the assignment to treatments randomised?
    Were the groups similar at the trials start?
    Maintenance

    Were outcomes ascertained & analysed for most patients?

    Were the groups treated equally?


    Measurements

    Were patients and clinicians blinded to treatment? OR

    Were measurements objective & standardised?

    3.

    4.

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993

    Effects of non-equal treatment

    Apart from actual intervention - groups should


    receive identical care!
    Trial of Vitamin E in pre-term infants (1949)
    Vit E "prevented" retrolental fibroplasia

    Rx: Give placebo in an identical regime, and a standard protocol

    Equal treatment in DVT study?


    Table 3: All drugs
    taken by volunteers
    who attended for
    examination before
    and after air travel*

    Bottom page 96 (1487)

    Follow-up in DVT study?

    200 of 231 analyzed

    27 were unable to attend for


    subsequent ultrasound
    2 were excluded from analysis
    because they were upgraded to
    business class
    2 were excluded from analysis
    because they were taking
    anticoagulants

    (87%)

    See figure on page 96

    Scurr et al, Lancet 2001; 357:1485-89

    Losses-to-follow-up
    How many is too many?
    5-and-20 rule of thumb
    5% probably leads to little bias
    >20% poses serious threats to validity
    Depends on outcome event rate and
    comparative loss rates in the groups
    Loss to follow-up rate should not exceed outcome
    event rate and should not be differential

    How important are the losses?


    Equally distributed?
    Stocking group: 6 men, 9 women - 15
    No stocking group: 7 men, 9 women - 16

    Similar characteristics?
    No information provided

    Intention-to-Treat Principle
    Maintaining the randomization
    Principle:
    Once a patient is randomized, s/he should be
    analyzed in the group randomized to - even if they
    discontinue, never receive treatment, or crossover.

    Exception: If patient is found on BLIND reassessment


    to be ineligible based on pre-randomization criteria.

    Appraisal checklist
    Study biases
    1.
    Recruitment

    Who did the subjects represent?


    2.
    Allocation
    Was the assignment to treatments randomised?
    Were the groups similar at the trials start?
    Maintainence

    Were outcomes ascertained & analysed for most patients?

    Were the groups treated equally?


    Measurements

    Were patients and clinicians blinded to treatment? OR

    Were measurements objective & standardised?

    3.

    4.

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993

    Measures in DVT study?


    Blood was taken from all participants before travel
    All participants had US once before travel (30 had US twice)
    All participants were seen within 48 hr of return flight, were
    interviewed and completed a questionnaire, had repeat US

    Scurr et al, Lancet 2001; 357:1485-89

    Measurement Bias minimizing differential error


    Blinding Who?
    Participants?
    Investigators?
    Outcome assessors?
    Analysts?
    Most important to use "blinded"
    outcome assessors when outcome
    is not objective!
    Papers should report WHO was
    blinded and HOW it was done
    Schulz and Grimes. Lancet, 2002

    Evaluation
    Most passengers removed their
    stockings on completion of their
    journey. The nurse removed the
    stockings of those passengers who
    had continued to wear them. A
    further duplex examination was
    then undertaken with the
    technician unaware of the group to
    which the volunteer had been
    randomized.

    Appraisal checklist
    Study biases
    Recruitment

    1.

    Allocation

    2.

    Was the assignment to treatments randomised?


    Were the groups similar at the trials start?
    Maintainence

    3.

    Were the groups treated equally?


    Were outcomes ascertained & analysed for most patients?
    Measurements

    4.

    5.
    6.
    7.

    Who did the subjects represent?

    Were patients and clinicians blinded to treatment? OR


    Were measurements objective & standardised?
    Placebo Effect
    Chance
    Real Effect

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993

    Placebo effect
    Trial in patients with chronic severe itching
    60

    No treatment

    50
    40

    Trimeprazine
    tartrate
    Cyproheptadine
    HCL

    30

    Itching score

    20
    10
    0

    Treatment vs no treatment for itching

    Placebo effect
    Trial in patients with chronic severe itching
    60

    No treatment

    50
    40

    Trimeprazine
    tartrate
    Cyproheptadine
    HCL

    Placebo

    30

    Itching score

    20
    10
    0

    Treatment vs no treatment vs placebo for itching

    Placebo effect - attributable to the expectation that


    the treatment will have an effect

    Appraisal checklist
    Study biases
    Recruitment

    1.

    Allocation

    2.

    Was the assignment to treatments randomised?


    Were the groups similar at the trials start?
    Maintainence

    3.

    Were the groups treated equally?


    Were outcomes ascertained & analysed for most patients?
    Measurements

    4.

    5.
    6.
    7.

    Who did the subjects represent?

    Were patients and clinicians blinded to treatment? OR


    Were measurements objective & standardised?
    Placebo Effect
    Chance
    Real Effect

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993

    Two methods of assessing the role of chance


    P-values (Hypothesis Testing)
    use statistical test to examine the null hypothesis
    associated with p values - if p<0.05 then result is
    statistically significant

    Confidence Intervals (Estimation)


    estimates the range of values that is likely to include the
    true value

    P-values (Hypothesis Testing) - in DVT study


    Incidence of DVT
    Stocking group - 0
    No Stocking group - 0.12

    Risk difference = 0.12 - 0 = 0.12 (P=0.001)


    The probability that this result would only
    occur by chance is

    1 in 1000 statistically significant

    Confidence Intervals (Estimation)


    Incidence of DVT

    Stocking group = 0
    No Stocking group = 0.12
    Risk difference = (0.12 0) = 0.12 (95% CI, 0.058 - 0.20)
    The true value could be as low as 0.058 or as high as
    0.20 - but is probably closer to 0.12
    Since the CI does not include the no effect value
    of 0 the result is statistically significant

    Appraisal checklist
    Study biases
    Recruitment

    1.

    Allocation

    2.

    Was the assignment to treatments randomised?


    Were the groups similar at the trials start?
    Maintainence

    3.

    Were the groups treated equally?


    Were outcomes ascertained & analysed for most patients?
    Measurements

    4.

    5.
    6.
    7.

    Who did the subjects represent?

    Were patients and clinicians blinded to treatment? OR


    Were measurements objective & standardised?
    Placebo Effect
    Chance
    Real Effect

    Study statistics (p-values & confidence intervals)

    Guyatt. JAMA, 1993

    Causes of an Effect in a controlled trial


    Who would now consider wearing stockings
    on a long haul flight?

    M Clarke, S Hopewell, E Juszczak, A Eisinga, M Kjeldstrm


    Compression stockings for preventing deep vein thrombosis in airline passengers
    Cochrane Database of Systematic Reviews 2006 Issue 4

    10 RCTs (n = 2856)
    Seven included low or medium risk (n = 1548) and two included high risk
    participants (n = 1273).
    All flights > seven hours.
    Fifty of 2,637 participants in the trials of wearing stockings on both legs had a
    symptomless DVT; three wore stockings, 47 did not

    (OR 0.10, 95% CI 0.04 to 0.25, P < 0.00001).

    No deaths, pulmonary emboli or symptomatic DVTs were reported.


    Wearing stockings had a significant impact in reducing oedema (based on six
    trials).
    No significant adverse effects were reported.

    M Clarke, S Hopewell, E Juszczak, A Eisinga, M Kjeldstrm


    Compression stockings for preventing deep vein thrombosis in airline passengers
    Cochrane Database of Systematic Reviews 2006 Issue 4

    Thank you
    Small groups

    You might also like