Repeated Measures ANOVA
Repeated Measures ANOVA: Part 1
Repeated Measures ANOVA Effects of Adventure Programming on Self-Determination – Part I
Repeated Measures ANOVA: Part 1
In this Module In this module, you will learn: How to conduct a Repeated Measures ANOVA and obtain the following types of sums-of-squares (SS): SS Between Groups SS Within Groups SS Block SS Residual MS Between Groups MS Residual F-test P-value
Repeated Measures ANOVA: Part 1
Introduction Repeated measures in the ANOVA design mean that each subject has more than 2 “measures” (scores) on the same (or equivalent) survey/test in the study. Recall that in the dependent t test each subject has “two” measures responding to the same test. These two measures are often called the “pretest” score and the “posttest” score. For the Type I error rate, repeated measures ANOVA designs should be used when we have more than 2 repeated measures in our study.
Repeated Measures ANOVA: Part 1
Introduction In the repeated measures ANOVA design, we usually use the “TIME” variable as the independent variable. And there are at least three levels of the TIME variable: TIME1, TIME2, and TIME3. Therefore, the independent variable is a qualitative variable, also referred to as a repeated measures variable or a within subject variable.
The dependent variable will be the measured scores, such as mathematics scores, which is a continuous (quantitative) variable. Let’s study a real research case for the repeated measures ANOVA design.
Repeated Measures ANOVA: Part 1
Case 1: Purpose
This case was adapted from a College of Education dissertation.
The purpose of the study was to examine whether adventure based programming could lead to changes in self-determination of adolescents who had been identified as at-risk for school failure.
Repeated Measures ANOVA: Part 1
Case 1: Participants
The participants were 216 middle school students who were identified as at-risk for school failure.
The students came from 6 public middle schools in a moderate sized school district in Florida. Each of the students had returned a signed parent permission form.
Repeated Measures ANOVA: Part 1
Case 1: Research Design
Within each of the six schools, the students were randomly assigned to one of four small cooperative groups, each consisting of 8 to 12 students. Groups of this size are considered optimal for the adventure programming curriculum. Two of the small groups were randomly assigned to the treatment condition and the other two to a wait-list control condition. The self-determination of the students was measured at the beginning of the study (Time1), after the treatment group went through the adventure programming (Time2), and at a follow-up time that was six weeks after the treatment group completed the adventure programming (Time3).
Repeated Measures ANOVA: Part 1
Case 1: Measures
Self-determination was assessed using the American Institute for Research (AIR) Self-Determination Scale.
The scores on this measure are recorded as the average response to a set of items using a 5-point Likert scale (1 to 5), where higher numbers indicate more selfdetermination.
Repeated Measures ANOVA: Part 1
Analysis for this Case In this module we are going to restrict ourselves to looking only at the treatment group. This will allow us to ease into repeated measures analysis by focusing on methods to analyze change in a single group. In the next module, I will add in the data for the control group, and do a more thorough analysis of the data from the study.
Repeated Measures ANOVA: Part 1
Unit of Analysis The students within each treatment group interacted heavily during the adventure programming, thus the individual students were not independent of each other. Repeated measures ANOVA assumes independent units, therefore, a decision was made to use group means as the unit of analysis. To recall unit of analysis from Statistics I, refer to the next two slides
Repeated Measures ANOVA: Part 1
Unit of Analysis = Individuals If each participant receives the treatment independently of the other participants, the degrees of freedom and the estimate of the dependent variable’s variance are unbiased.
The assumption of independence is met.
Repeated Measures ANOVA: Part 1
Unit of Analysis = Group Individuals Nested Within Contexts When participants receive the treatments in groups (such as students nested in classrooms), participants within each group are more similar to each other than participants in different groups. This phenomenon (referred to as intraclass correlation) produces an estimated variance that is too small and degrees of freedom for the ttest that are too large. The assumption of independence of observations is violated and the ttest does not maintain control of Type I error.
Repeated Measures ANOVA: Part 1
Data Set For each adventure programming group, the mean self-determination score is given at: Time 1 (prior to the treatment; pretest) Time 2 (immediately after the treatment; posttest) Time 3 (at the six week follow-up; follow-up)
Repeated Measures ANOVA: Part 1
Analysis Plan 1. Compute appropriate descriptive statistics to describe the distribution of self-determination at each time and to describe the relationship between the selfdetermination measures. 2. Construct difference variables (e.g., the change between Time1 and Time2), and look at them descriptively. 3. Conduct a Repeated Measures ANOVA: F-test, an omnibus or overall test Sphericity adjustment
4. If overall test is significant, run follow-up tests.
Repeated Measures ANOVA: Part 1
Descriptive Statistics: Pre, Post, Follow-up
We assume you are comfortable computing and interpreting descriptive statistics. We see that the average self-determination score is higher after adventure-programming than it was before, which is consistent with what we were expecting to see.
Repeated Measures ANOVA: Part 1
Graphical Display of the Change in Means Next we created a line graph showing the selfdetermination means as a function of time. Again note that the means follow a pattern that is consistent with the hypothesis that self-determination would increase by participation in the adventure programming.
Repeated Measures ANOVA: Part 1
Correlations Among Self-Determination Scores Next we looked at the relationships between the scores at the three points in time.
We see from the matrix that pre, post, and follow-up measure of self-determination are all positively correlated, which indicates that groups that were higher than average early in the study tended to be higher than average later.
Repeated Measures ANOVA: Part 1
Difference Variables After looking at the self-determination scores at each point in time, we turn our attention to how much selfdetermination changed between adjacent time points. For the first cooperative group, we see the selfdetermination was 2.33 at Time 1 and was 3.06 at Time 2, and thus the change in self-determination was 0.73. At Time 3 the self-determination was 3.04, which is a 0.02 decrease from the Time 2 self-determination. More formally we define two difference variables: Diff12 Time2 Time1 Diff 23 Time3 Time2
Repeated Measures ANOVA: Part 1
Describing the Distribution of Difference Scores We now see that mean of the first difference variable is larger than the second, which again indicates that in this sample the change in selfdetermination between Time 1 and Time 2 was greater than the change between Time 2 and Time 3. We note there is considerable variation in the amount of change, but the distributions do not depart markedly from normality besides the extreme leptokurtic values in Diff23.
Repeated Measures ANOVA: Part 1
Describing the Distribution of Difference Scores We also looked at the correlation between the two difference variables. The Pearson Product-Moment Correlation (PPMC) was -.373 which showed that greater gains in self-determination from Time 1 to Time 2 were associated with smaller gains between Time 2 to Time 3. At this point we have a pretty good feel for the sample data, and are ready to move into making inferences about the population. We would like to determine if the gain in self-determination exceeded that which could be attributed reasonably to sampling error. To do this, we turn to repeated measures ANOVA.
Repeated Measures ANOVA: Part 1
Repeated Measures ANOVA There are many parallels between Repeated Measures ANOVA and the ANOVAs you have been studying. We will follow the same general strategy of computing appropriate sums-of-squares (SS), dividing the sums-of-squares by the degrees of freedom to obtain mean squares (MS), and taking the ratio of mean squares to compute an F statistic.
The following table shows the general symbols used for the scores and means in a repeated measures context. This table can be used for reference when unpacking the details of how each of the sums-of-squares are computed. Show Notation
Repeated Measures ANOVA: Part 1
Notations for Repeated Measures ANOVA
Mean of Block 1 X11 represents the 1st observation in Time 1
The grand mean represents the mean of all time means.
X32 represents the 3rd observation in Time 2
Mean of Time 1 nth observation in Time k
If you put all observations in only one time, the grand mean will be the mean of all observations.
Repeated Measures ANOVA: Part 1
Sum-of-Squares Between Groups The sums-of-squares between groups is obtained by finding deviations from the means at the particular time points and the grand mean. As with ANOVA these deviations are squared, then summed, and then multiplied by the sample size, n. 𝑘
(𝑋.𝑗 −𝑋.. )2
𝑆𝑆𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = 𝑛 𝑗=1
For this case: SSbetween = 4.21
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Within Groups The sums-of-squares within groups is obtained by finding deviations of observations from the means at particular time points. These squared deviations are summed across all the observations at a particular time point, and across all the time points. k
n
SSwithin ( X ij X j )2 j 1 i 1
For this case: SSwithin = 10.32
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Blocks The sums-of-squares between groups and the sums-ofsquares within groups were the same computations we had used for a one-way ANOVA, and thus they were probably familiar. The sums-of-squares blocks, however, is unique to repeated measures analysis.
Repeated Measures ANOVA: Part 1
Sum-of-Squares Blocks The sums-of-squares blocks is obtained by looking at deviations between the block means and the grand mean. n
SSblocks k ( X i X ) 2 i 1
For this case: SSblocks = 7.20
In this study, the block means are the self-determination scores for each cooperative group (individual) when we average across time. That is, we average the Time 1, Time 2, and Time 3 assessments. As with other sums-of-squares, these deviations need to be squared and summed. the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Error The sums-of-squares error is conceptually the part of the within group variation that can not be explained by systematic differences between the blocks. To obtain the sums-of-squared errors we simply subtract the sums-ofsquares blocks from the sums-of-squares within. SSerror SS within SSblocks
For this case: SSerror = 3.11
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Mean Squares Between Groups Now that we have partitioned the variation in the selfdetermination scores into the various sums-of-squares (SS) we are ready to compute mean squares (MS). Like in the previous ANOVA cases, we compute the MS by dividing the SS by the appropriate degrees of freedom. We start by computing the MS between groups. MSbetween
SSbetween dfbetween
dfbetween k 1
For this case: dfbetween=2, because there are three time points (k=3). MSbetween = 2.10
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Mean Square Error The mean square error (MSerror) is computed by dividing the SSerror by the error degrees of freedom (dferror), where the dferror is defined as the number of blocks minus 1 multiplied by the number of time points minus 1. MSerror
SSerror df error
dferror (n 1)(k 1)
For this case: dferror = 22, because there are twelve blocks (n=12) and three time points (k=3). MSerror = .14
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
F-test The F-test used to determine if there is a statistically significant difference between the means at the different measurement occasions is conducted by taking the ratio of the MS between groups to the MS error. MSbetween F MSerror
For this case: F = 14.86
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
p-value for the F-test To determine if the F is large enough to reject the null hypothesis of no difference in the population means over time, we compute a p-value. As in our other ANOVA cases, this p-value can be thought of by imagining repeatedly conducting the study when the null hypothesis is true. If we did this, we would get a variety of F-values, each of which would depend on the particular sample selected.
Repeated Measures ANOVA: Part 1
p-value for the F-test As we did the study over and over we would accumulate a large number of F-values, which would make up a distribution of F-values. The proportion of the F-values in this distribution that were as large as or larger than the one obtained would equal the p-value. Put another way, the p-value is the probability of obtaining an F-value as large or larger than the one obtained, if the null hypothesis is true.
Repeated Measures ANOVA: Part 1
Assumptions The computational algorithm for the p-value is fairly complex and beyond the scope of this course, but it is important to recognize that in deriving this algorithm certain simplifying assumptions were made. If the data are inconsistent with these assumptions the accuracy of the pvalue and the validity of the statistical test become questionable. For a repeated measures ANOVA we assume: Independence Normality
Sphericity
Repeated Measures ANOVA: Part 1
Independence Assumption We assume that the blocks are independent of each other, but do not have to assume the repeated observations are independent. In this case we used the cooperative group as the unit of analysis because we could assume that each cooperative group operated independently of the other cooperative groups. We would not have been comfortable assuming the individuals within a cooperative group were independent because we would expect that within a cooperative group individuals would affect the selfdetermination of each other.
Repeated Measures ANOVA: Part 1
Independence Assumption We previously noted that the pre-assessment was related to the post-assessment, which implies the repeated observations are not independent, but that is fine. The assumption is of independence across blocks not across times.
Repeated Measures ANOVA: Part 1
Normality Assumption We assume that if we had the difference scores for the whole population we would see a normal distribution.
As with other ANOVAs we have some latitude in violating this assumption, because ANOVAs are relatively robust to violations of the normality assumption. The Type I error rate stays near the declared level (say .05) even if the sampled populations are not normal. As noted previously there were not gross departures from normality in our sample data, and thus we feel comfortable proceeding with the repeated measures ANOVA.
Repeated Measures ANOVA: Part 1
Sphericity Assumption We assume that the difference variables have equal variance and that they do not covary. Unfortunately, this assumption is often violated with educational data. In this data set we saw a correlation of -.373 between the difference variables, suggesting that they do covary. In addition, repeated measures ANOVA are not robust to violations of this assumption. Consequently, we routinely make an adjustment when we compute the p-values for a repeated measures ANOVA. To make the needed adjustment we estimate something called the sphericity parameter.
Repeated Measures ANOVA: Part 1
Sphericity Parameter The sphericity parameterε (epsilon) indexes the degree to which the population data are consistent with the sphericity assumption. When the data are consistent with the assumption, the sphericity parameter is 1.0. When there is a violation of the sphericity assumption the parameter is less than 1.0, with smaller values representing more substantial violations.
Repeated Measures ANOVA: Part 1
Sphericity Parameter If we knew the value of the sphericity parameter, we could adjust our probability calculations by redefining the degrees of freedom as the original degrees of freedom multiplied by the sphericity parameter. This would then lead to the correct probabilities with nonspherical data.
Repeated Measures ANOVA: Part 1
Sphericity Parameter Estimates We don’t know , but we can estimate it based on our sample data. There are two estimates that are commonly used: Greenhouse-Geisser Huynh-Feldt
ˆ
ˆ .8956 1.0585
In this case
~
Note the Greenhouse-Geisser value is smaller and thus it will lead to a more substantial adjustment, and is considered the more conservative of the two methods.
Repeated Measures ANOVA: Part 1
Sphericity Adjusted p-values We suggest you regularly adjust your p-value for potential violations of the sphericity assumption using either the Greenhouse-Geisseror Huynh-Feldt estimate. Let us suppose that we were in a conservative mood today, so we chose to go with the GreenhouseGeisser adjustment. We would note the p-value was less than .05, and conclude that the population mean had not remained the same over time. We have evidence that the self-determination changed by a greater amount than can be easily attributed to sampling error.
In this case Greenhouse-Geisser adjusted p-value is .0002 The Huynh-Feldt adjusted pvalue is
Repeated Measures ANOVA Effects of Adventure Programming on Self-Determination – Part I
Repeated Measures ANOVA: Part 1
In this Module In this module, you will learn: How to conduct a Repeated Measures ANOVA and obtain the following types of sums-of-squares (SS): SS Between Groups SS Within Groups SS Block SS Residual MS Between Groups MS Residual F-test P-value
Repeated Measures ANOVA: Part 1
Introduction Repeated measures in the ANOVA design mean that each subject has more than 2 “measures” (scores) on the same (or equivalent) survey/test in the study. Recall that in the dependent t test each subject has “two” measures responding to the same test. These two measures are often called the “pretest” score and the “posttest” score. For the Type I error rate, repeated measures ANOVA designs should be used when we have more than 2 repeated measures in our study.
Repeated Measures ANOVA: Part 1
Introduction In the repeated measures ANOVA design, we usually use the “TIME” variable as the independent variable. And there are at least three levels of the TIME variable: TIME1, TIME2, and TIME3. Therefore, the independent variable is a qualitative variable, also referred to as a repeated measures variable or a within subject variable.
The dependent variable will be the measured scores, such as mathematics scores, which is a continuous (quantitative) variable. Let’s study a real research case for the repeated measures ANOVA design.
Repeated Measures ANOVA: Part 1
Case 1: Purpose
This case was adapted from a College of Education dissertation.
The purpose of the study was to examine whether adventure based programming could lead to changes in self-determination of adolescents who had been identified as at-risk for school failure.
Repeated Measures ANOVA: Part 1
Case 1: Participants
The participants were 216 middle school students who were identified as at-risk for school failure.
The students came from 6 public middle schools in a moderate sized school district in Florida. Each of the students had returned a signed parent permission form.
Repeated Measures ANOVA: Part 1
Case 1: Research Design
Within each of the six schools, the students were randomly assigned to one of four small cooperative groups, each consisting of 8 to 12 students. Groups of this size are considered optimal for the adventure programming curriculum. Two of the small groups were randomly assigned to the treatment condition and the other two to a wait-list control condition. The self-determination of the students was measured at the beginning of the study (Time1), after the treatment group went through the adventure programming (Time2), and at a follow-up time that was six weeks after the treatment group completed the adventure programming (Time3).
Repeated Measures ANOVA: Part 1
Case 1: Measures
Self-determination was assessed using the American Institute for Research (AIR) Self-Determination Scale.
The scores on this measure are recorded as the average response to a set of items using a 5-point Likert scale (1 to 5), where higher numbers indicate more selfdetermination.
Repeated Measures ANOVA: Part 1
Analysis for this Case In this module we are going to restrict ourselves to looking only at the treatment group. This will allow us to ease into repeated measures analysis by focusing on methods to analyze change in a single group. In the next module, I will add in the data for the control group, and do a more thorough analysis of the data from the study.
Repeated Measures ANOVA: Part 1
Unit of Analysis The students within each treatment group interacted heavily during the adventure programming, thus the individual students were not independent of each other. Repeated measures ANOVA assumes independent units, therefore, a decision was made to use group means as the unit of analysis. To recall unit of analysis from Statistics I, refer to the next two slides
Repeated Measures ANOVA: Part 1
Unit of Analysis = Individuals If each participant receives the treatment independently of the other participants, the degrees of freedom and the estimate of the dependent variable’s variance are unbiased.
The assumption of independence is met.
Repeated Measures ANOVA: Part 1
Unit of Analysis = Group Individuals Nested Within Contexts When participants receive the treatments in groups (such as students nested in classrooms), participants within each group are more similar to each other than participants in different groups. This phenomenon (referred to as intraclass correlation) produces an estimated variance that is too small and degrees of freedom for the ttest that are too large. The assumption of independence of observations is violated and the ttest does not maintain control of Type I error.
Repeated Measures ANOVA: Part 1
Data Set For each adventure programming group, the mean self-determination score is given at: Time 1 (prior to the treatment; pretest) Time 2 (immediately after the treatment; posttest) Time 3 (at the six week follow-up; follow-up)
Repeated Measures ANOVA: Part 1
Analysis Plan 1. Compute appropriate descriptive statistics to describe the distribution of self-determination at each time and to describe the relationship between the selfdetermination measures. 2. Construct difference variables (e.g., the change between Time1 and Time2), and look at them descriptively. 3. Conduct a Repeated Measures ANOVA: F-test, an omnibus or overall test Sphericity adjustment
4. If overall test is significant, run follow-up tests.
Repeated Measures ANOVA: Part 1
Descriptive Statistics: Pre, Post, Follow-up
We assume you are comfortable computing and interpreting descriptive statistics. We see that the average self-determination score is higher after adventure-programming than it was before, which is consistent with what we were expecting to see.
Repeated Measures ANOVA: Part 1
Graphical Display of the Change in Means Next we created a line graph showing the selfdetermination means as a function of time. Again note that the means follow a pattern that is consistent with the hypothesis that self-determination would increase by participation in the adventure programming.
Repeated Measures ANOVA: Part 1
Correlations Among Self-Determination Scores Next we looked at the relationships between the scores at the three points in time.
We see from the matrix that pre, post, and follow-up measure of self-determination are all positively correlated, which indicates that groups that were higher than average early in the study tended to be higher than average later.
Repeated Measures ANOVA: Part 1
Difference Variables After looking at the self-determination scores at each point in time, we turn our attention to how much selfdetermination changed between adjacent time points. For the first cooperative group, we see the selfdetermination was 2.33 at Time 1 and was 3.06 at Time 2, and thus the change in self-determination was 0.73. At Time 3 the self-determination was 3.04, which is a 0.02 decrease from the Time 2 self-determination. More formally we define two difference variables: Diff12 Time2 Time1 Diff 23 Time3 Time2
Repeated Measures ANOVA: Part 1
Describing the Distribution of Difference Scores We now see that mean of the first difference variable is larger than the second, which again indicates that in this sample the change in selfdetermination between Time 1 and Time 2 was greater than the change between Time 2 and Time 3. We note there is considerable variation in the amount of change, but the distributions do not depart markedly from normality besides the extreme leptokurtic values in Diff23.
Repeated Measures ANOVA: Part 1
Describing the Distribution of Difference Scores We also looked at the correlation between the two difference variables. The Pearson Product-Moment Correlation (PPMC) was -.373 which showed that greater gains in self-determination from Time 1 to Time 2 were associated with smaller gains between Time 2 to Time 3. At this point we have a pretty good feel for the sample data, and are ready to move into making inferences about the population. We would like to determine if the gain in self-determination exceeded that which could be attributed reasonably to sampling error. To do this, we turn to repeated measures ANOVA.
Repeated Measures ANOVA: Part 1
Repeated Measures ANOVA There are many parallels between Repeated Measures ANOVA and the ANOVAs you have been studying. We will follow the same general strategy of computing appropriate sums-of-squares (SS), dividing the sums-of-squares by the degrees of freedom to obtain mean squares (MS), and taking the ratio of mean squares to compute an F statistic.
The following table shows the general symbols used for the scores and means in a repeated measures context. This table can be used for reference when unpacking the details of how each of the sums-of-squares are computed. Show Notation
Repeated Measures ANOVA: Part 1
Notations for Repeated Measures ANOVA
Mean of Block 1 X11 represents the 1st observation in Time 1
The grand mean represents the mean of all time means.
X32 represents the 3rd observation in Time 2
Mean of Time 1 nth observation in Time k
If you put all observations in only one time, the grand mean will be the mean of all observations.
Repeated Measures ANOVA: Part 1
Sum-of-Squares Between Groups The sums-of-squares between groups is obtained by finding deviations from the means at the particular time points and the grand mean. As with ANOVA these deviations are squared, then summed, and then multiplied by the sample size, n. 𝑘
(𝑋.𝑗 −𝑋.. )2
𝑆𝑆𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = 𝑛 𝑗=1
For this case: SSbetween = 4.21
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Within Groups The sums-of-squares within groups is obtained by finding deviations of observations from the means at particular time points. These squared deviations are summed across all the observations at a particular time point, and across all the time points. k
n
SSwithin ( X ij X j )2 j 1 i 1
For this case: SSwithin = 10.32
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Blocks The sums-of-squares between groups and the sums-ofsquares within groups were the same computations we had used for a one-way ANOVA, and thus they were probably familiar. The sums-of-squares blocks, however, is unique to repeated measures analysis.
Repeated Measures ANOVA: Part 1
Sum-of-Squares Blocks The sums-of-squares blocks is obtained by looking at deviations between the block means and the grand mean. n
SSblocks k ( X i X ) 2 i 1
For this case: SSblocks = 7.20
In this study, the block means are the self-determination scores for each cooperative group (individual) when we average across time. That is, we average the Time 1, Time 2, and Time 3 assessments. As with other sums-of-squares, these deviations need to be squared and summed. the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Sum-of-Squares Error The sums-of-squares error is conceptually the part of the within group variation that can not be explained by systematic differences between the blocks. To obtain the sums-of-squared errors we simply subtract the sums-ofsquares blocks from the sums-of-squares within. SSerror SS within SSblocks
For this case: SSerror = 3.11
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Mean Squares Between Groups Now that we have partitioned the variation in the selfdetermination scores into the various sums-of-squares (SS) we are ready to compute mean squares (MS). Like in the previous ANOVA cases, we compute the MS by dividing the SS by the appropriate degrees of freedom. We start by computing the MS between groups. MSbetween
SSbetween dfbetween
dfbetween k 1
For this case: dfbetween=2, because there are three time points (k=3). MSbetween = 2.10
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
Mean Square Error The mean square error (MSerror) is computed by dividing the SSerror by the error degrees of freedom (dferror), where the dferror is defined as the number of blocks minus 1 multiplied by the number of time points minus 1. MSerror
SSerror df error
dferror (n 1)(k 1)
For this case: dferror = 22, because there are twelve blocks (n=12) and three time points (k=3). MSerror = .14
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
F-test The F-test used to determine if there is a statistically significant difference between the means at the different measurement occasions is conducted by taking the ratio of the MS between groups to the MS error. MSbetween F MSerror
For this case: F = 14.86
the computations are available for download from the Attachments tab
Repeated Measures ANOVA: Part 1
p-value for the F-test To determine if the F is large enough to reject the null hypothesis of no difference in the population means over time, we compute a p-value. As in our other ANOVA cases, this p-value can be thought of by imagining repeatedly conducting the study when the null hypothesis is true. If we did this, we would get a variety of F-values, each of which would depend on the particular sample selected.
Repeated Measures ANOVA: Part 1
p-value for the F-test As we did the study over and over we would accumulate a large number of F-values, which would make up a distribution of F-values. The proportion of the F-values in this distribution that were as large as or larger than the one obtained would equal the p-value. Put another way, the p-value is the probability of obtaining an F-value as large or larger than the one obtained, if the null hypothesis is true.
Repeated Measures ANOVA: Part 1
Assumptions The computational algorithm for the p-value is fairly complex and beyond the scope of this course, but it is important to recognize that in deriving this algorithm certain simplifying assumptions were made. If the data are inconsistent with these assumptions the accuracy of the pvalue and the validity of the statistical test become questionable. For a repeated measures ANOVA we assume: Independence Normality
Sphericity
Repeated Measures ANOVA: Part 1
Independence Assumption We assume that the blocks are independent of each other, but do not have to assume the repeated observations are independent. In this case we used the cooperative group as the unit of analysis because we could assume that each cooperative group operated independently of the other cooperative groups. We would not have been comfortable assuming the individuals within a cooperative group were independent because we would expect that within a cooperative group individuals would affect the selfdetermination of each other.
Repeated Measures ANOVA: Part 1
Independence Assumption We previously noted that the pre-assessment was related to the post-assessment, which implies the repeated observations are not independent, but that is fine. The assumption is of independence across blocks not across times.
Repeated Measures ANOVA: Part 1
Normality Assumption We assume that if we had the difference scores for the whole population we would see a normal distribution.
As with other ANOVAs we have some latitude in violating this assumption, because ANOVAs are relatively robust to violations of the normality assumption. The Type I error rate stays near the declared level (say .05) even if the sampled populations are not normal. As noted previously there were not gross departures from normality in our sample data, and thus we feel comfortable proceeding with the repeated measures ANOVA.
Repeated Measures ANOVA: Part 1
Sphericity Assumption We assume that the difference variables have equal variance and that they do not covary. Unfortunately, this assumption is often violated with educational data. In this data set we saw a correlation of -.373 between the difference variables, suggesting that they do covary. In addition, repeated measures ANOVA are not robust to violations of this assumption. Consequently, we routinely make an adjustment when we compute the p-values for a repeated measures ANOVA. To make the needed adjustment we estimate something called the sphericity parameter.
Repeated Measures ANOVA: Part 1
Sphericity Parameter The sphericity parameterε (epsilon) indexes the degree to which the population data are consistent with the sphericity assumption. When the data are consistent with the assumption, the sphericity parameter is 1.0. When there is a violation of the sphericity assumption the parameter is less than 1.0, with smaller values representing more substantial violations.
Repeated Measures ANOVA: Part 1
Sphericity Parameter If we knew the value of the sphericity parameter, we could adjust our probability calculations by redefining the degrees of freedom as the original degrees of freedom multiplied by the sphericity parameter. This would then lead to the correct probabilities with nonspherical data.
Repeated Measures ANOVA: Part 1
Sphericity Parameter Estimates We don’t know , but we can estimate it based on our sample data. There are two estimates that are commonly used: Greenhouse-Geisser Huynh-Feldt
ˆ
ˆ .8956 1.0585
In this case
~
Note the Greenhouse-Geisser value is smaller and thus it will lead to a more substantial adjustment, and is considered the more conservative of the two methods.
Repeated Measures ANOVA: Part 1
Sphericity Adjusted p-values We suggest you regularly adjust your p-value for potential violations of the sphericity assumption using either the Greenhouse-Geisseror Huynh-Feldt estimate. Let us suppose that we were in a conservative mood today, so we chose to go with the GreenhouseGeisser adjustment. We would note the p-value was less than .05, and conclude that the population mean had not remained the same over time. We have evidence that the self-determination changed by a greater amount than can be easily attributed to sampling error.
In this case Greenhouse-Geisser adjusted p-value is .0002 The Huynh-Feldt adjusted pvalue is
