Sample Size Tutorial handout

Report 4 Downloads 74 Views
The Analysis Factor Tutorial Calculating Power and Sample Size

Calculating Power and Sample Size On Demand Tutorial Karen Grace-Martin

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

1

The Analysis Factor Tutorial Calculating Power and Sample Size

Tutorial Outline 1. 2. 3. 4. 5.

A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples

2

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

2

The Analysis Factor Tutorial Calculating Power and Sample Size

1. The Concepts • • • • •

Hypothesis Tests Rejection Regions Type I error Type II error Power

3

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

3

The Analysis Factor Tutorial Calculating Power and Sample Size

1. The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value

4

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

4

The Analysis Factor Tutorial Calculating Power and Sample Size

1. The Hypothesis Test H0: parameter = specified value H1: parameter ≠ that value •

One sample t test

H0: µ = 25 H1: µ ≠ 25



Two sample t test

H0: µ1 − µ2 = 0 H1: µ1 − µ2 ≠ 0





Regression Coefficient

H0: β = 0 H1: β ≠ 0

One-way ANOVA

H0: Var(µ) = 0 H1: Var(µ) ≠ 0

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

5

5

The Analysis Factor Tutorial Calculating Power and Sample Size

H0

20

25

30

35

40

X

6

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

6

The Analysis Factor Tutorial Calculating Power and Sample Size

H0

20 Reject H0

25

30

35

40

X

Reject H0

7

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

7

The Analysis Factor Tutorial Calculating Power and Sample Size

Truth H0 True Decision

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

H0 False

Reject H0

Type I error

Correct

Accept H0

Correct

Type II error

8

8

The Analysis Factor Tutorial Calculating Power and Sample Size

Truth H0 True Decision

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

H0 False

Reject H0

Type I error

Correct

Accept H0

Correct

Type II error

9

9

The Analysis Factor Tutorial Calculating Power and Sample Size

Truth H0 True Decision

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

H0 False

Reject H0

Type I error

Correct

Accept H0

Correct

Type II error

10

10

The Analysis Factor Tutorial Calculating Power and Sample Size

Truth H0 True Decision

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

H0 False

Reject H0

Type I error

Correct

Accept H0

Correct

Type II error

11

11

The Analysis Factor Tutorial Calculating Power and Sample Size

Truth H0 True Decision

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

H0 False

Reject H0

Type I error

Correct

Accept H0

Correct

Type II error

12

12

The Analysis Factor Tutorial Calculating Power and Sample Size

Tutorial Outline 1. 2. 3. 4. 5.

A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples

13

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

13

The Analysis Factor Tutorial Calculating Power and Sample Size

2. How Sample Size Relates to Power 1. One sample t test

2. Two sample t test

3. Regression Coefficient

t=

t=

X −µ s2

n

( X 1 − X 2 ) − ( µ1 − µ 2 ) s 2pooled ( 1 + 1 ) n1 n2

t=

b−β s2

n

SSB 4. One-way ANOVA

F=

SSE

k −1 n−k 14

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

14

The Analysis Factor Tutorial Calculating Power and Sample Size

t= H0

X −µ s2

n

H1

1−α 1−β

α/2

Power

α/2

β

X

25

37

Reject H0

Reject H0 −1.8

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

0

1.8

3.6

5.4

15

15

The Analysis Factor Tutorial Calculating Power and Sample Size

t=

X −µ s2

n

16

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

16

The Analysis Factor Tutorial Calculating Power and Sample Size

t=

X −µ s2

n

17

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

17

The Analysis Factor Tutorial Calculating Power and Sample Size

t=

X −µ s2

n

18

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

18

The Analysis Factor Tutorial Calculating Power and Sample Size

2. How Sample Size Relates to Power To increase power: 1. 2. 3. 4. 5.

Increase alpha Conduct a one-tailed test Increase the effect size Decrease random error Increase sample size

19

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

19

The Analysis Factor Tutorial Calculating Power and Sample Size

2. How Sample Size Relates to Power All else being equal: Lower alpha 2 tails Smaller effect size Higher Power Larger variance

    

larger n larger n larger n larger n larger n

20

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

20

The Analysis Factor Tutorial Calculating Power and Sample Size

Outline 1. A Review of the Concepts 2. How Power Relates to Sample Size 3. Estimating Sample Sizes Prospectively 4. Prospective vs. Post-hoc Power 5. Examples

21

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

21

The Analysis Factor Tutorial Calculating Power and Sample Size

3. Estimating Sample Sizes The Reasons: 1. Many published studies have low power. 2. To not waste your time and money pursuing something impossible. 3. To reduce risk and wasting resources 4. Funding agencies and many committees require it 22

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

22

The Analysis Factor Tutorial Calculating Power and Sample Size

3. Estimating Sample Sizes The Steps: 1. Specify a hypothesis test 2. Specify the significance level of the test 3. Specify the smallest effect size of scientific interest 4. Estimate the values of other parameters 5. Specify the intended power of the test 6. Plug all information into the appropriate equation 7. Solve for n 23

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

23

The Analysis Factor Tutorial Calculating Power and Sample Size

3. Estimating Sample Sizes z=

X −µ

σ2

n

   E  1 − β = P( Z >  zα / 2 + ) σ   n  φ = area under the normal curve to the right of a specified X value

z1− β = zα / 2 +

n = sample size E = effect size, difference between µ1 and µ0

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

σ n

za = value of the standard normal distribution with an area under the curve to the right = a

σ = standard deviation of X

E

n=

( z1−α / 2 + z1− β ) 2 σ 2 E2 24

24

The Analysis Factor Tutorial Calculating Power and Sample Size

Outline 1. 2. 3. 4.

A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis - Prospective vs. Post-hoc Power - Standardized Effect Sizes 5. Examples 25

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

25

The Analysis Factor Tutorial Calculating Power and Sample Size

4.1 Post-hoc Power Analyses Observed Power: Calculate power assuming the observed sample statistics equal the true parameter values Confidence Intervals: use the width to indicate likelihood of the null Equivalence Testing: The null states that the effect size is greater than a defined value 26

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

26

The Analysis Factor Tutorial Calculating Power and Sample Size

4.2 Standardized Effect Sizes x1 − x2 Cohen’s d = sp .2 = small .5 = medium .8 = large

27

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

27

The Analysis Factor Tutorial Calculating Power and Sample Size

Points to Remember 1. 2. 3. 4. 5. 6. 7. 8.

DesignAnalysisSample Size Sample size is an educated guess Identify the study’s primary outcomes Use the smallest meaningful effect size Increase the ‘real’ sample size to reflect loss to follow up, expected response rate, lack of compliance, etc. If you have a really tricky sampling situation, get some input from a statistician There is no point in doing power analysis after the study is done Use Equivalence Testing to show no effect 28

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

28

The Analysis Factor Tutorial Calculating Power and Sample Size

Power and Sample Size Software StudySize 2.0 http://www.studysize.com/ Power and Precision http://www.poweranalysis.com/software_overview.htm nQuery http://www.statsol.ie/index.php?pageID=2 29

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

29

The Analysis Factor Tutorial Calculating Power and Sample Size

References and Further Reading Colegrave, N. & Ruxton, G.D. (2003). Confidence intervals are a more useful complement to nonsignificant tests than are power calculations. Behavioral Ecology, 14,3: 446-447. http://beheco.oxfordjournals.org/cgi/content/full/14/3/446 Hoenig, J.M. and D.M. Heisey. 2001. The Abuse of Power: the Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1):19-24. http://www.fisheries.vims.edu/hoenig/pdfs/hoenig2.pdf Lenth, R. V. (2001), Some Practical Guidelines for Effective Sample Size Determination, The American Statistician, 55, 187-193. http://www.stat.uiowa.edu/techrep/tr303.pdf Kraemer, H.C. & Thiemann, S. (1987). How Many Subjects?: Statistical Power Analysis in Research. Keppel, G. (1991). Design and Analysis: A Researcher’s Handbook. Prentice Hall. Snijders, Tom A.B. (2005). Power and Sample Size in Multilevel Linear Models. In: B.S. Everitt and D.C. Howell (eds.), Encyclopedia of Statistics in Behavioral Science. Volume 3, 1570–1573. Wiley, Chicester (etc.). http://stat.gamma.rug.nl/PowerSampleSizeMultilevel.pdf 30

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

30

The Analysis Factor Tutorial Calculating Power and Sample Size

Outline 1. 2. 3. 4. 5.

A Review of the Concepts How Power Relates to Sample Size Estimating Sample Sizes Prospectively Issues in Power Analysis Examples

31

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

31

The Analysis Factor Tutorial Calculating Power and Sample Size

5. Example 1 Question: does a high protein diet decrease calories consumed? Study design: participants randomized into one of two groups (high and low protein diet) Outcome: total kcalories consumed over 7 days Analysis: 2 sample t-test Want to detect: a change of at least 1500 kcal Known: from past studies, the standard deviation varies between 1400 and 2200 kcals 32

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

32

The Analysis Factor Tutorial Calculating Power and Sample Size

5. Example 2 Question: Do people living in agricultural areas have higher levels of pesticides in their houses ? Study design: observational study of 4 groups Outcome: detectable levels of pesticides swabbed from flooring. Expect values ~ 25 for controls Analysis: One way ANOVA Want to detect: a difference of at least 8 µg/sq ft Known: from past studies, the standard deviation ranges from 9 to 15 µg/sq ft 33

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

33

The Analysis Factor Tutorial Calculating Power and Sample Size

5. Example 3 Question:Does television exposure affect infants’ language development? Study design: Observational study, control for age, gender Outcome: the number of spoken words infants hear in a 24 hour period Analysis: Multiple regression analysis Want to detect: a 5% minimum decrease, or 500 words less than 11,000 for each additional hour of TV Known: from past studies, the standard deviation of Y varies between 900-1200 and of TV watching is .7 hours Example: http://www.newscientist.com/article/dn17227tvwatching-tots-miss-out-on-vital-chat.html?DCMP=OTCrss&nsref=online-news

34

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

34

The Analysis Factor Tutorial Calculating Power and Sample Size

5. Example 4 Question: Does toxoplasmosis infection affect accident rates and does Rh protein have a protective effect? Study design: Observational study Outcome: Odds of having a traffic accident Analysis: Logistic regression analysis Want to detect: And odds ratio of 2.0 or above Known: from past studies, the probability of accident at mean levels of taxo ranges from .05 to .15 http://www.biomedcentral.com/1471-2334/9/72 35

Copyright 2010-17 The Analysis Factor http://theanalysisfactor.com

35