Understanding and Quantifying

EFFECT SIZES EFFECT SIZES

Karabi Nandy, Ph.d.
Assistant Adjunct Professor
Translational Sciences Section, School of Nursing , g
Department of Biostatistics, School of Public Health,
University of California Los Angeles (UCLA)
Objective Objective
Effect size comes up in the context of
sample size calculations for proposals
reporting results from pilot studies.
By the end of this talk, you will be able to calculate,
interpret and report effect sizes in your work.
4/9/2012 EffectSize 2
Outline Outline
Why are Effect Sizes (ES) important?
Types of Effect Sizes
Quantifying Magnitudes of Effect Sizes
Calculating Effect Sizes Calculating Effect Sizes
Use of Softwares for Calculating Effect Sizes
Specific Guidelines for Reporting Effect Sizes
4/9/2012 EffectSize 3
Definition Definition
Effect - A change or changed state occurring as a direct
result of action by somebody or something (Encarta,
2009) 2009)
Size - The degree of something in terms of how big or
small it is
'Effect size' is simply a way of quantifying the size of the Effect size is simply a way of quantifying the size of the
difference between two groups.
4/9/2012 EffectSize 4
It is particularly valuable for quantifying the
effectiveness of a particular intervention relative to effectiveness of a particular intervention, relative to
some comparison. It allows us to move beyond the
simplistic, 'Does it work or not?' to the far more
sophisticated 'How well does it work How well does it work in a range of sophisticated, How well does it work How well does it work in a range of
contexts?
4/9/2012 EffectSize 5
Why is Effect Size Important Why is Effect Size Important
Knowing the magnitude of an effect allows us to
ascertain the practical significance of statistical
significance significance
Can always reach statistical significance if there is a large
enough sample size, unless the effect size is 0.
Even a large effect may not be statistically significant if Even a large effect may not be statistically significant if
the sample size is too small.
4/9/2012 EffectSize 6
Why is Effect Size Important Why is Effect Size Important
Practical Significance
Even a statistically significant treatment difference may
not be practically important if the effect size is too small not be practically important if the effect size is too small.
However, there could still be practical importance even for
small effect sizes, especially in cases where cost and ease
make it easy to be implemented on a large scale. y p g
4/9/2012 EffectSize 7
Why is Effect Size Important Why is Effect Size Important
Sample Size Calculation for Studies
ES l di t l i l i l l ti f ES plays a direct role in sample size calculations for any
study. It is connected to the power of a test, the level of
significance and sample size (n).

or
Given any 3 quantities (power ES n) we can find the 4
th
Given any 3 quantities (power, ES, , n), we can find the 4
th
.
4/9/2012 EffectSize 8
Why is Effect Size Important Why is Effect Size Important
Meta-Analysis
pooling information from many studies to verify results of p g y y
past research and inform future studies.
ES i t d i h t d d th fi di l d ES is computed in each study and the findings are pooled
together to draw overall inferences.
4/9/2012 EffectSize 9
Types Types of Effect of Effect Sizes Sizes
Mean Differences between Groups
Eff t Si C h d Effect Size: Cohens d
Correlation/Regression
Effect Size: Pearsons r and R
2
Effect Size: Cohens f
2
Contingency tables Contingency tables
Effect Size: Odds Ratio or Relative Risk (association
between binary variables)
4/9/2012 EffectSize 10
Types Types of Effect of Effect Sizes Sizes
ANOVA or GLMs
Effect Size: Eta-squared
Effect Size: Omega squared Effect Size: Omega squared
Effect Size: Intraclass correlation (rater equality)
Chi-square tests
Effect Size: Phi (2 binary variables) Effect Size: Phi (2 binary variables)
Effect Size: Cramers Phi or V (categorical variables)
4/9/2012 EffectSize 11
Cohens d Cohens d
2 1
2
2 2
2
1 1
pooled
pooled
2 1
) 1 ( ) 1 (
where
n n
s n s n
s
s
x x
d

Standardizes ES of the difference between two means

2 1 pooled
Used for two sample independent t-tests
d ranges from - to +
interpretation: the difference between the mean values interpretation: the difference between the mean values
is d standard deviations, Cohen (1988)
4/9/2012 EffectSize 12
ES Example 1 ES Example 1
ES = 0 00 means that the average treatment ES 0.00 means that the average treatment
participant outperformed 50% of the control
i i participants
4/9/2012 EffectSize 13
ES Example 2 ES Example 2
ES = 0.85 means that the average treatment participant g p p
outperformed 80%of the control participants
4/9/2012 EffectSize 14
General Guidelines General Guidelines
In general, 0.20 is a small effect size, 0.50 is a moderate
effect size and 0.80 is a large effect size (Cohen, 1992)
d standardized Percentage of d- standardized Percentage of
mean difference variance explained
Small .20 1%
M d t 50 10% Moderate .50 10%
Large .80 25%
4/9/2012 EffectSize 15
Cohens d Cohens d
Special Cases
For small sample sizes use Hedges G
) 1 ( ) 1 (
h
2
2 2
2
1 1 2 1
s n s n x x
d
For unequal group variances, use Glasss
2
) ( ) (
where
2 1
2 2 1 1
pooled
pooled
2 1


n n
s
s
d
q g p
uses sample sd of the control group only so that effect sizes
would not differ under equal means and unequal variances would not differ under equal means and unequal variances
4/9/2012 EffectSize 16
Pearsons Pearsons r r
used in the context of correlationmeasuring association
between 2 variables between 2 variables
Interpretation: For every 1-unit standard deviation change
in x, there is a r-unit standard deviation change in y
4/9/2012 EffectSize 17
Pearsons Pearsons R R
2 2
used in the context of regressionmeasuring how well a
regression line fits to a given data regression line fits to a given data
R - linear association between 2 continuous variables
R
2
(Coefficient of Determination) proportion of shared
variability between 2 or more variables
Interpretation: R
2
*100% is percent variance of the
outcome y that can be explained by the linear regression y p y g
model (i.e. indicates how well the linear regression line fits
the data))
4/9/2012 EffectSize 18
Odds Ratio Odds Ratio
Used in the context of binary/categorical outcomes
Odds of being in one group (eg. success) relative to the odds of
being in a different group (eg. failure) g g p ( g )
OR ranges from 0 to
OR>1 indicates an increase in odds relative to the reference group
OR < 1 indicates a decrease in odds relative to the reference group
4/9/2012 EffectSize 19
Relative Risk Relative Risk
RR measures the risk of an event relative to an independent
variable variable
For small probabilities, the relative risk is approximately equal to
the odds ratio
Interpretation: If RR > 1 then the risk of disease X among p g
smokers is RR times the risk of disease X among non-smokers
(vice versa if RR < 1) (vice versa if RR < 1)
4/9/2012 EffectSize 20
Eta Eta--Squared ( Squared (
22
) )
and partial Eta and partial Eta Squared ( Squared (
22
)) and partial Eta and partial Eta--Squared ( Squared (
pp
22
))
Used with ANOVA family and GLMs
Measures the degree of association in the sample
St d di Eff t Si f th h d i b t Standardizes Effect Sizes of the shared variance between a
continuous outcome and categorical predictors
Partial eta-squared is the proportion of the total variability
attributable to a given factor attributable to a given factor.
4/9/2012 EffectSize 21
Eta Eta--Squared ( Squared (
22
) )
and partial Eta and partial Eta--Squared ( Squared (
22
))
Interpretation:
2
*100% is percent of the variance in y
and partial Eta and partial Eta--Squared ( Squared (
pp
))
explained by the variance in x (similar to the R
2
interpretation for linear regression (Dattalo, 2008)). p g ( , ))

2
is biased and on average overestimates the variance
explained in the population, but decreases as the sample size
gets larger.
Caution: these effect sizes depend on the number and
magnitude of the other effects magnitude of the other effects
4/9/2012 EffectSize 22
Cohens Cohens ff
22
Used in multiple linear regression, R
2
=
2
Standardized effect size is the proportion of explained
variance over unexplained variance variance over unexplained variance
Estimate is biased and overestimates the effect size for
ANOVA (unbiased estimate is Omega-Squared)
4/9/2012 EffectSize 23
Omega Omega- -Squared Squared
Estimates the proportion of variance in the population
that is explained by the treatment
is always smaller than or since Omega is always smaller than or since Omega
measures the population variance and Eta measures the
sample variance
4/9/2012 EffectSize 24
Intraclass Intraclass Correlation Correlation
ICC is used to measure inter-rater reliability for two or more
raters. It may also be used to assess test-retest reliability. ICC may
be conceptualized as the ratio of between-groups variance to total
variance.
Can be used in ANOVA
Similar interpretation to Omega-Squared
4/9/2012 EffectSize 25
Phi Phi
n
2

Used for crosstabs or for chi-square tests (equality of
proportions or tests of independence between 2 binary p p p y
variables) = 0 indicates independence
Phi are related to correlation and Cohens d (for 2 binary
variables)
Interpreted like Pearsons r and R
2
4/9/2012 EffectSize 26
Cramers Cramers Phi or V Phi or V
CramersPhi(CramersV)canbeusedwithcategoricalvariables
withmorethan2categories(m>2)(RxCtables) g ( ) ( )
k i (R C) ;k=min(R,C)
measurestheintercorrelationofthevariables,butisbiased
sinceitincreaseswiththenumberofcells.IncreaseinRandC
willindicateastrongassociation,whichisjustanartifactofthe
typeofvariableused.
4/9/2012 EffectSize 27
Magnitude of Effect Summary Table Magnitude of Effect Summary Table
Effect Size Small Medium Large
r 0.10 0.30 0.50
r
2
0.01 0.09 0.25

2
0 01 0 06 0 14
2
0.01 0.06 0.14
R
2
0.01 0.06 0.14
Cohens d 0.20 0.50 0.80
/ Cramers V 0.10 0.30 0.50
Cohens f
2
0.02 0.15 0.35
OR 1.44 2.47 4.25
4/9/2012 EffectSize
OR 1.44 2.47 4.25
28
Effect Size Conversions Effect Size Conversions
Effect Size Converted to Cohens d
Correlation
2
1
2
r
r
d

Chi-Square
df =1 df =1
df > 1
N
d
2
4

Odds Ratio (Chinn, 2000)

4/9/2012 EffectSize 29
Software Software
Calculate effect size
http://www.uccs.edu/~faculty/lbecker/
http://fac lt assar ed /lo r /ne cs html http://faculty.vassar.edu/lowry/newcs.html
Statistical softwares such as SAS, R, STATA, SPSS S s c so w es suc s S S, , S , S SS
can calculate most of the standard Effect Sizes
4/9/2012 EffectSize 30
Calculating Cohens d Calculating Cohens d online calculator online calculator
http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 31
Calculating Cohens d Calculating Cohens d online calculator online calculator
http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 32
Calculating Cramers Phi Calculating Cramers Phi online calculator online calculator
http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 33
Calculating Cramers Phi Calculating Cramers Phi online calculator online calculator
http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 34
Calculating Cramers Phi Calculating Cramers Phi online calculator online calculator
http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/~faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/ http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 35
A (not A (not--so so--great) Alternative: great) Alternative:
Calculating Cohens d using Calculating Cohens d using GPower GPower g g g g
Step 1:
Choose t-
test from
drop down
Step 2:
Choose
means
fromdrop
Step 3:
Choose
drop down
menu
from drop
down menu
Solution:
Calculated
effect size
Choose
Sensitivity
from the
drop down
menu
Step 4: Based on
4/9/2012 EffectSize
menu
data from pilot study
36
Calculating SS given Cohens d Calculating SS given Cohens d -- GPower GPower
Abetter use of Gpower is to calculate sample sizes A better use of Gpower is to calculate sample sizes.
4/9/2012 EffectSize 37
Reporting Guidelines and Trends Reporting Guidelines and Trends
Reporting effect sizes has three important benefits (APA,
1999): 1999):
Meta-analysis
Informing subsequent research
Interpretation and evaluation of results within the context
of related literature
4/9/2012 EffectSize 38
Reporting Guidelines and Trends Reporting Guidelines and Trends
What to report (APA, 2010):
Type of effect size
Value of the effect size (in original units such as lbs or Value of the effect size (in original units, such as lbs., or
mean differences on a scale, and/or the effect size statistic)
Interpretation of the effect size
Practical significance of the effect size Practical significance of the effect size
4/9/2012 EffectSize 39
References References
Chinn, S.(2000). A simple method for converting an odds ratio to effect
i f i t l i St ti ti i M di i 19 3127 3131 size for use in meta-analysis. Statistics in Medicine, 19, 3127-3131.
Cohen, J.(1992). A power primer. Psychological Bulletin, 112(1), 155-
159.
Cohen J (1988) Statistical power analysis for the behavioral sciences Cohen, J. (1988) Statistical power analysis for the behavioral sciences .
New Jersey: Lawrenced Erlbaum Associates, Inc. Publishers. pp 283-286.
4/9/2012 EffectSize 40
References References
Dunst Carl J et al (2004) Guidelines for Calculating Effect Sizes for Dunst, Carl J. et al. (2004) Guidelines for Calculating Effect Sizes for
Practice-Based Research Syntheses. Centerscope 3(1)
http://courseweb.unt.edu/gknezek/06spring/5610/centerscopevol3no1.pdf
A presentation on effect size:
http://www.family.umaryland.edu/ryc_research_and_evaluation/publication
product files/selected presentations/presentation files/pdfs/effect%20size _product_files/selected_presentations/presentation_files/pdfs/effect%20size
%20and%20intervention%20research.pdf
Lee B.R., Bright C.L., Svobo da, D., Fakunmoju, S. and Barth R.P.
Outcomes of group care for youth: A review of comparative literature.
Research on Social Work Practice
4/9/2012 EffectSize 41