CPSS slides handout 2 per page
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop Karen Grace-Martin THE ANALYSIS FACTOR
Workshop Outline 1. 2. 3. 4. 5.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples
1 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
A Review of the Concepts • • • • •
The Hypothesis Test Rejection Regions Type I error Type II error Power
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
2 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value •
One sample t test
H0: µ = 25 H1: µ ≠ 25
•
Two sample t test
H0: µ1 − µ2 = 0 H1: µ1 − µ2 ≠ 0
•
Regression Coefficient
H0 : β = 0 H1 : β ≠ 0
•
One-way ANOVA
H0: Var(µ) = 0 H1: Var(µ) ≠ 0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
5
H0
20
25
30
35
40
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
X
6
3 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
H0
20
25
30
35
Reject H0
X
40
Reject H0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
7
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
8
4 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
9
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
10
5 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
11
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
12
6 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: How Power Relates to Sample Size Karen Grace-Martin THE ANALYSIS FACTOR
1. One sample t test
2. Two sample t test
t=
3. Regression Coefficient
t=
4. One-way ANOVA
X −µ
t=
s2
n
( X 1 − X 2 ) − ( µ1 − µ 2 ) s 2pooled ( 1 + 1 ) n1 n2
b−β s2
n
SSB F=
SSE
k −1 n−k
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
7 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
H0
t=
H1
1−α
X −µ s2
n
1−β
α/2
Power
α/2
β
X
25
37
Reject H0
Reject H0 -1.8
0
1.8
3.6
5.4
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
t=
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
X −µ s2
n
4
8 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
t=
X −µ s2
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
5
t=
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
n
X −µ s2
n
6
9 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
To increase power: 1. Increase alpha 2. Conduct a one-tailed test 3. Increase the effect size 4. Decrease random error 5. Increase sample size
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
7
All else being equal: Lower alpha 2 tails Smaller effect size Higher Power Larger variance
larger n larger n larger n larger n larger n
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
8
10 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Estimating Sample Sizes Prospectively Karen Grace-Martin THE ANALYSIS FACTOR
Why Estimate Sample Sizes?: 1. Many published studies have low power 2. Avoid wasting time and money pursuing the impossible 3. Reduce risk and wasting resources 4. Funding agencies and many committees require it
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
11 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
The Steps to Estimating a Sample Size: 1. Specify a hypothesis test 2. Specify the significance level of the test 3. Specify the smallest effect size of scientific interest 4. Estimate the values of other parameters 5. Specify the intended power of the test 6. Plug all information into the appropriate equation 7. Solve for n Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
z=
X −µ
σ2
n
E 1 − β = P (Z > zα / 2 + ) σ n
za = value of z that has an area under the curve of a to the right of z
z1− β = zα / 2 +
σ n
σ = standard deviation of X n = sample size
E
n=
( z1−α / 2 + z1− β ) 2 σ 2
E = effect size, difference between µ1 and µ0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
E2
4
12 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Issues in Power Analysis Karen Grace-Martin The Analysis Factor
Post-hoc Analyses Observed Power: Calculate power assuming the observed sample statistics equal the true parameter values Confidence Intervals: use the width to indicate likelihood of the null Equivalence Testing: The null states that the effect size is greater than a defined value Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
13 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Standardized Effect Sizes Correlation Coefficient r Standardized Regression Coefficient β Coefficient of Determination R2 Cohen’s d =
x1 − x2 sp
.2 = small .5 = medium .8 = large Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
Points to Remember 1. 2. 3. 4. 5. 6. 7. 8.
DesignAnalysisSample Size Sample size is an educated guess Identify the study’s primary outcomes Use the smallest meaningful effect size Increase the ‘real’ sample size to reflect loss to follow up, expected response rate, lack of compliance, etc. If you have a really tricky sampling situation, get some input from a statistician There is no point in doing power analysis after the study is done Use Equivalence Testing to show no effect Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
14 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Examples Karen Grace-Martin The Analysis Factor
Power and Sample Size Software PiFace G*Power 3 StudySize 2.0 Power and Precision http://www.power-analysis.com/software_overview.htm
nQuery http://www.statsol.ie/index.php?pageID=2 Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
15 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Example 1 Question: does a high protein diet decrease calories consumed? Study design: participants randomized into one of two groups (high and low protein diet) Outcome: total kcalories consumed over 7 days Analysis: 2 sample t-test Want to detect: a change of at least 1500 kcal Known: from past studies, the standard deviation varies between 1400 and 2200 kcals Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
Example 2 Question: Do people living in agricultural areas have higher levels of pesticides in their houses ? Study design: observational study of 4 groups Outcome: detectable levels of pesticides swabbed from flooring. Expect values ~ 25 for controls Analysis: One way ANOVA Want to detect: a difference of at least 8 µg/sq ft Known: from past studies, the standard deviation ranges from 9 to 15 µg/sq ft Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
16 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Example 3 Question:Does television exposure affect infants’ language development? Study design: Observational study, control for age, gender Outcome: the number of spoken words infants hear in a 24 hour period Analysis: Multiple regression analysis Want to detect: a 5% minimum decrease, or 500 words less than 11,000 for each additional hour of TV Known: from past studies, the standard deviation of Y varies between 900-1200 and of TV watching is .7 hours Example: http://www.newscientist.com/article/dn17227-tvwatching-totsmiss-out-on-vital-chat.html?DCMP=OTC-rss&nsref=online-news Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
5
Example 4 Question: Does toxoplasmosis infection affect accident rates and does Rh protein have a protective effect? Study design: Observational study Outcome: Odds of having a traffic accident Analysis: Logistic regression analysis Want to detect: And odds ratio of 2.0 or above Known: from past studies, the probability of accident at mean levels of taxo ranges from .05 to .15 Example: http://www.biomedcentral.com/1471-2334/9/72 Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
6
17 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
Calculating Power and Sample Size Workshop Karen Grace-Martin THE ANALYSIS FACTOR
Workshop Outline 1. 2. 3. 4. 5.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples
1 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
A Review of the Concepts • • • • •
The Hypothesis Test Rejection Regions Type I error Type II error Power
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
2 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value •
One sample t test
H0: µ = 25 H1: µ ≠ 25
•
Two sample t test
H0: µ1 − µ2 = 0 H1: µ1 − µ2 ≠ 0
•
Regression Coefficient
H0 : β = 0 H1 : β ≠ 0
•
One-way ANOVA
H0: Var(µ) = 0 H1: Var(µ) ≠ 0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
5
H0
20
25
30
35
40
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
X
6
3 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
H0
20
25
30
35
Reject H0
X
40
Reject H0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
7
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
8
4 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
9
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
10
5 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
11
Truth H0 True Decision
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
12
6 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: How Power Relates to Sample Size Karen Grace-Martin THE ANALYSIS FACTOR
1. One sample t test
2. Two sample t test
t=
3. Regression Coefficient
t=
4. One-way ANOVA
X −µ
t=
s2
n
( X 1 − X 2 ) − ( µ1 − µ 2 ) s 2pooled ( 1 + 1 ) n1 n2
b−β s2
n
SSB F=
SSE
k −1 n−k
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
7 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
H0
t=
H1
1−α
X −µ s2
n
1−β
α/2
Power
α/2
β
X
25
37
Reject H0
Reject H0 -1.8
0
1.8
3.6
5.4
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
t=
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
X −µ s2
n
4
8 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
t=
X −µ s2
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
5
t=
Calculating Power and Sample Size Workshop Copyright http://TheAnalysisFactor.com
n
X −µ s2
n
6
9 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
To increase power: 1. Increase alpha 2. Conduct a one-tailed test 3. Increase the effect size 4. Decrease random error 5. Increase sample size
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
7
All else being equal: Lower alpha 2 tails Smaller effect size Higher Power Larger variance
larger n larger n larger n larger n larger n
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
8
10 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Estimating Sample Sizes Prospectively Karen Grace-Martin THE ANALYSIS FACTOR
Why Estimate Sample Sizes?: 1. Many published studies have low power 2. Avoid wasting time and money pursuing the impossible 3. Reduce risk and wasting resources 4. Funding agencies and many committees require it
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
11 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
The Steps to Estimating a Sample Size: 1. Specify a hypothesis test 2. Specify the significance level of the test 3. Specify the smallest effect size of scientific interest 4. Estimate the values of other parameters 5. Specify the intended power of the test 6. Plug all information into the appropriate equation 7. Solve for n Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
z=
X −µ
σ2
n
E 1 − β = P (Z > zα / 2 + ) σ n
za = value of z that has an area under the curve of a to the right of z
z1− β = zα / 2 +
σ n
σ = standard deviation of X n = sample size
E
n=
( z1−α / 2 + z1− β ) 2 σ 2
E = effect size, difference between µ1 and µ0
Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
E2
4
12 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Issues in Power Analysis Karen Grace-Martin The Analysis Factor
Post-hoc Analyses Observed Power: Calculate power assuming the observed sample statistics equal the true parameter values Confidence Intervals: use the width to indicate likelihood of the null Equivalence Testing: The null states that the effect size is greater than a defined value Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
13 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Standardized Effect Sizes Correlation Coefficient r Standardized Regression Coefficient β Coefficient of Determination R2 Cohen’s d =
x1 − x2 sp
.2 = small .5 = medium .8 = large Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
Points to Remember 1. 2. 3. 4. 5. 6. 7. 8.
DesignAnalysisSample Size Sample size is an educated guess Identify the study’s primary outcomes Use the smallest meaningful effect size Increase the ‘real’ sample size to reflect loss to follow up, expected response rate, lack of compliance, etc. If you have a really tricky sampling situation, get some input from a statistician There is no point in doing power analysis after the study is done Use Equivalence Testing to show no effect Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
14 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Calculating Power and Sample Size Workshop: Examples Karen Grace-Martin The Analysis Factor
Power and Sample Size Software PiFace G*Power 3 StudySize 2.0 Power and Precision http://www.power-analysis.com/software_overview.htm
nQuery http://www.statsol.ie/index.php?pageID=2 Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
2
15 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Example 1 Question: does a high protein diet decrease calories consumed? Study design: participants randomized into one of two groups (high and low protein diet) Outcome: total kcalories consumed over 7 days Analysis: 2 sample t-test Want to detect: a change of at least 1500 kcal Known: from past studies, the standard deviation varies between 1400 and 2200 kcals Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
3
Example 2 Question: Do people living in agricultural areas have higher levels of pesticides in their houses ? Study design: observational study of 4 groups Outcome: detectable levels of pesticides swabbed from flooring. Expect values ~ 25 for controls Analysis: One way ANOVA Want to detect: a difference of at least 8 µg/sq ft Known: from past studies, the standard deviation ranges from 9 to 15 µg/sq ft Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
4
16 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
The Analysis Factor Workshop: Calculating Power and Sample Size
Example 3 Question:Does television exposure affect infants’ language development? Study design: Observational study, control for age, gender Outcome: the number of spoken words infants hear in a 24 hour period Analysis: Multiple regression analysis Want to detect: a 5% minimum decrease, or 500 words less than 11,000 for each additional hour of TV Known: from past studies, the standard deviation of Y varies between 900-1200 and of TV watching is .7 hours Example: http://www.newscientist.com/article/dn17227-tvwatching-totsmiss-out-on-vital-chat.html?DCMP=OTC-rss&nsref=online-news Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
5
Example 4 Question: Does toxoplasmosis infection affect accident rates and does Rh protein have a protective effect? Study design: Observational study Outcome: Odds of having a traffic accident Analysis: Logistic regression analysis Want to detect: And odds ratio of 2.0 or above Known: from past studies, the probability of accident at mean levels of taxo ranges from .05 to .15 Example: http://www.biomedcentral.com/1471-2334/9/72 Calculating Power and Sample Size Workshop Copyright 2011 http://TheAnalysisFactor.com
6
17 Copyright 2011 The Analysis Factor http://TheAnalysisFactor.com
