The Analysis Factor Tutorial Calculating Power and Sample Size
Calculating Power and Sample Size On Demand Tutorial Karen Grace-Martin
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
1
The Analysis Factor Tutorial Calculating Power and Sample Size
Tutorial Outline 1. 2. 3. 4. 5.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples
2
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
2
The Analysis Factor Tutorial Calculating Power and Sample Size
1. The Concepts • • • • •
Hypothesis Tests Rejection Regions Type I error Type II error Power
3
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
3
The Analysis Factor Tutorial Calculating Power and Sample Size
1. The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value
4
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
4
The Analysis Factor Tutorial Calculating Power and Sample Size
1. The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value •
One sample t test
H0: µ = 25 H1: µ ≠ 25
•
Two sample t test
H0: µ1 − µ2 = 0 H1: µ1 − µ2 ≠ 0
•
•
Regression Coefficient
H0: β = 0 H1: β ≠ 0
One-way ANOVA
H0: Var(µ) = 0 H1: Var(µ) ≠ 0
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
5
5
The Analysis Factor Tutorial Calculating Power and Sample Size
H0
20
25
30
35
40
X
6
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
6
The Analysis Factor Tutorial Calculating Power and Sample Size
H0
20 Reject H0
25
30
35
40
X
Reject H0
7
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
7
The Analysis Factor Tutorial Calculating Power and Sample Size
Truth H0 True Decision
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
8
8
The Analysis Factor Tutorial Calculating Power and Sample Size
Truth H0 True Decision
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
9
9
The Analysis Factor Tutorial Calculating Power and Sample Size
Truth H0 True Decision
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
10
10
The Analysis Factor Tutorial Calculating Power and Sample Size
Truth H0 True Decision
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
11
11
The Analysis Factor Tutorial Calculating Power and Sample Size
Truth H0 True Decision
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
H0 False
Reject H0
Type I error
Correct
Accept H0
Correct
Type II error
12
12
The Analysis Factor Tutorial Calculating Power and Sample Size
Tutorial Outline 1. 2. 3. 4. 5.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples
13
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
13
The Analysis Factor Tutorial Calculating Power and Sample Size
2. How Sample Size Relates to Power 1. One sample t test
2. Two sample t test
3. Regression Coefficient
t=
t=
X −µ s2
n
( X 1 − X 2 ) − ( µ1 − µ 2 ) s 2pooled ( 1 + 1 ) n1 n2
t=
b−β s2
n
SSB 4. One-way ANOVA
F=
SSE
k −1 n−k 14
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
14
The Analysis Factor Tutorial Calculating Power and Sample Size
t= H0
X −µ s2
n
H1
1−α 1−β
α/2
Power
α/2
β
X
25
37
Reject H0
Reject H0 −1.8
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
0
1.8
3.6
5.4
15
15
The Analysis Factor Tutorial Calculating Power and Sample Size
t=
X −µ s2
n
16
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
16
The Analysis Factor Tutorial Calculating Power and Sample Size
t=
X −µ s2
n
17
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
17
The Analysis Factor Tutorial Calculating Power and Sample Size
t=
X −µ s2
n
18
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
18
The Analysis Factor Tutorial Calculating Power and Sample Size
2. How Sample Size Relates to Power To increase power: 1. 2. 3. 4. 5.
Increase alpha Conduct a one-tailed test Increase the effect size Decrease random error Increase sample size
19
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
19
The Analysis Factor Tutorial Calculating Power and Sample Size
2. How Sample Size Relates to Power All else being equal: Lower alpha 2 tails Smaller effect size Higher Power Larger variance
larger n larger n larger n larger n larger n
20
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
20
The Analysis Factor Tutorial Calculating Power and Sample Size
Outline 1. A Review of the Concepts 2. How Power Relates to Sample Size 3. Estimating Sample Sizes Prospectively 4. Prospective vs. Post-hoc Power 5. Examples
21
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
21
The Analysis Factor Tutorial Calculating Power and Sample Size
3. Estimating Sample Sizes The Reasons: 1. Many published studies have low power. 2. To not waste your time and money pursuing something impossible. 3. To reduce risk and wasting resources 4. Funding agencies and many committees require it 22
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
22
The Analysis Factor Tutorial Calculating Power and Sample Size
3. Estimating Sample Sizes The Steps: 1. Specify a hypothesis test 2. Specify the significance level of the test 3. Specify the smallest effect size of scientific interest 4. Estimate the values of other parameters 5. Specify the intended power of the test 6. Plug all information into the appropriate equation 7. Solve for n 23
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
23
The Analysis Factor Tutorial Calculating Power and Sample Size
3. Estimating Sample Sizes z=
X −µ
σ2
n
E 1 − β = P( Z > zα / 2 + ) σ n φ = area under the normal curve to the right of a specified X value
z1− β = zα / 2 +
n = sample size E = effect size, difference between µ1 and µ0
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
σ n
za = value of the standard normal distribution with an area under the curve to the right = a
σ = standard deviation of X
E
n=
( z1−α / 2 + z1− β ) 2 σ 2 E2 24
24
The Analysis Factor Tutorial Calculating Power and Sample Size
Outline 1. 2. 3. 4.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis - Prospective vs. Post-hoc Power - Standardized Effect Sizes 5. Examples 25
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
25
The Analysis Factor Tutorial Calculating Power and Sample Size
4.1 Post-hoc Power Analyses Observed Power: Calculate power assuming the observed sample statistics equal the true parameter values Confidence Intervals: use the width to indicate likelihood of the null Equivalence Testing: The null states that the effect size is greater than a defined value 26
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
26
The Analysis Factor Tutorial Calculating Power and Sample Size
4.2 Standardized Effect Sizes x1 − x2 Cohen’s d = sp .2 = small .5 = medium .8 = large
27
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
27
The Analysis Factor Tutorial Calculating Power and Sample Size
Points to Remember 1. 2. 3. 4. 5. 6. 7. 8.
DesignAnalysisSample Size Sample size is an educated guess Identify the study’s primary outcomes Use the smallest meaningful effect size Increase the ‘real’ sample size to reflect loss to follow up, expected response rate, lack of compliance, etc. If you have a really tricky sampling situation, get some input from a statistician There is no point in doing power analysis after the study is done Use Equivalence Testing to show no effect 28
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
28
The Analysis Factor Tutorial Calculating Power and Sample Size
Power and Sample Size Software StudySize 2.0 http://www.studysize.com/ Power and Precision http://www.poweranalysis.com/software_overview.htm nQuery http://www.statsol.ie/index.php?pageID=2 29
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
29
The Analysis Factor Tutorial Calculating Power and Sample Size
References and Further Reading Colegrave, N. & Ruxton, G.D. (2003). Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology, 14,3: 446-447. http://beheco.oxfordjournals.org/cgi/content/full/14/3/446 Hoenig, J.M. and D.M. Heisey. 2001. The Abuse of Power: the Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1):19-24. http://www.fisheries.vims.edu/hoenig/pdfs/hoenig2.pdf Lenth, R. V. (2001), Some Practical Guidelines for Effective Sample Size Determination, The American Statistician, 55, 187-193. http://www.stat.uiowa.edu/techrep/tr303.pdf Kraemer, H.C. & Thiemann, S. (1987). How Many Subjects?: Statistical Power Analysis in Research. Keppel, G. (1991). Design and Analysis: A Researcher’s Handbook. Prentice Hall. Snijders, Tom A.B. (2005). Power and Sample Size in Multilevel Linear Models. In: B.S. Everitt and D.C. Howell (eds.), Encyclopedia of Statistics in Behavioral Science. Volume 3, 1570–1573. Wiley, Chicester (etc.). http://stat.gamma.rug.nl/PowerSampleSizeMultilevel.pdf 30
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
30
The Analysis Factor Tutorial Calculating Power and Sample Size
Outline 1. 2. 3. 4. 5.
A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples
31
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
31
The Analysis Factor Tutorial Calculating Power and Sample Size
5. Example 1 Question: does a high protein diet decrease calories consumed? Study design: participants randomized into one of two groups (high and low protein diet) Outcome: total kcalories consumed over 7 days Analysis: 2 sample t-test Want to detect: a change of at least 1500 kcal Known: from past studies, the standard deviation varies between 1400 and 2200 kcals 32
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
32
The Analysis Factor Tutorial Calculating Power and Sample Size
5. Example 2 Question: Do people living in agricultural areas have higher levels of pesticides in their houses ? Study design: observational study of 4 groups Outcome: detectable levels of pesticides swabbed from flooring. Expect values ~ 25 for controls Analysis: One way ANOVA Want to detect: a difference of at least 8 µg/sq ft Known: from past studies, the standard deviation ranges from 9 to 15 µg/sq ft 33
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
33
The Analysis Factor Tutorial Calculating Power and Sample Size
5. Example 3 Question:Does television exposure affect infants’ language development? Study design: Observational study, control for age, gender Outcome: the number of spoken words infants hear in a 24 hour period Analysis: Multiple regression analysis Want to detect: a 5% minimum decrease, or 500 words less than 11,000 for each additional hour of TV Known: from past studies, the standard deviation of Y varies between 900-1200 and of TV watching is .7 hours Example: http://www.newscientist.com/article/dn17227tvwatching-tots-miss-out-on-vital-chat.html?DCMP=OTCrss&nsref=online-news
34
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
34
The Analysis Factor Tutorial Calculating Power and Sample Size
5. Example 4 Question: Does toxoplasmosis infection affect accident rates and does Rh protein have a protective effect? Study design: Observational study Outcome: Odds of having a traffic accident Analysis: Logistic regression analysis Want to detect: And odds ratio of 2.0 or above Known: from past studies, the probability of accident at mean levels of taxo ranges from .05 to .15 http://www.biomedcentral.com/1471-2334/9/72 35
Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com
35