Repeated Measures ANOVA Part 2
Repeated Measures ANOVA: Part 2
Repeated Measures ANOVA Effects of Adventure Programming on Self-Determination – Part II
Repeated Measures ANOVA: Part 2
In this Module In this module, you will learn:
Within-subjects factors and between-subjects factors Notation for data Sources of variations Plethora of F Statistics Contrasts Graphs New pieces of SAS code
Repeated Measures ANOVA: Part 2
Case 1 Recap: Purpose
Case 2 is simply an extension of the last case. The purpose of the study is still to examine whether adventure based programming could lead to changes in self-determination of adolescents who had been identified as at-risk for school failure.
Repeated Measures ANOVA: Part 2
Case 1 Recap: Participants
The participants were 216 middle school students who were identified as at-risk for school failure.
The students came from 6 public middle schools in a moderate sized school district in Florida. Each of the students had returned a signed parent permission form.
Repeated Measures ANOVA: Part 2
Case 1 Recap: Research Design
Within each of the six schools, the students were randomly assigned to one of four small cooperative groups, each consisting of 8 to 12 students. Groups of this size are considered optimal for the adventure programming curriculum. Two of the small groups were randomly assigned to the treatment condition and the other two to a wait-list control condition. The self-determination of the students was measured at the beginning of the study (Time1), after the treatment group went through the adventure programming (Time2), and at a follow-up time that was six weeks after the treatment group completed the adventure programming (Time3).
Repeated Measures ANOVA: Part 2
Case 1 Recap: Measures
Self-determination was assessed using the American Institute for Research (AIR) Self-Determination Scale.
The scores on this measure are recorded as the average response to a set of items using a 5-point Likert scale (1 to 5), where higher numbers indicate more selfdetermination.
Repeated Measures ANOVA: Part 2
Analysis for Case 2 For Case 2, we are going to look at both the treatment and control group. This will mean that we have to add another factor to our analysis. Now we will not only group scores by time (pretest, posttest, or follow-up test), but also by treatment condition (treatment group or control group). Thus, we have two factors like we did when we talked about factorial ANOVAs, but one of these factors is a repeated measures (within-subjects) factor.
Repeated Measures ANOVA: Part 2
Analysis Terminology Some would refer to the repeated measures factor (time) as a within-subjects factor, because you move within the same participants as you move between levels of this factor (e.g. the pretest scores and posttest scores come from the same individuals). Treatment condition, would be referred to as a between-subjects factor because you move between participants as you move to different levels of this factor (e.g., the treatment group scores and the control group scores come from different people). Consequently, some would call this an ANOVA with one betweensubjects factor and one within-subjects factor. Others would refer to it as a mixed model ANOVA because it mixes between- and within-subjects factors.
Repeated Measures ANOVA: Part 2
Unit of Analysis As noted in the last case, the students within each treatment group interacted heavily during the adventure programming (a violation of the independence assumption if we use the individual scores as the unit of analysis), thus a decision was made to use group means as the unit of analysis.
Repeated Measures ANOVA: Part 2
Data Set: Adventure Group
For each adventure programming group, the mean selfdetermination score is given at Time 1 (prior to the treatment), at Time 2 (immediately after the treatment), and at Time 3 (at the six week follow-up).
Repeated Measures ANOVA: Part 2
Data Set: Control Group
For each control group, the mean self-determination score is given at Time 1 (prior to the treatment), at Time 2 (immediately after the treatment), and at Time 3 (at the six week follow-up).
Repeated Measures ANOVA: Part 2
Analysis Plan 1. Compute appropriate descriptive statistics to describe the distribution of self-determination at each time for each group and to describe the relationship between the selfdetermination measures. 2. Construct difference variables (e.g., the change between Time1 and Time2), and look at them descriptively.
3. Conduct a Mixed Model ANOVA: - F-test for group main effect - F-test for time main effect - F-test for interaction of group and time 4. If the omnibus (overall) test is significant, run follow-up tests.
Repeated Measures ANOVA: Part 2
Descriptive Statistics: Pre, Post, Follow-up
We see that the average self-determination score in the adventure-programming group goes up during treatment more than that of the control group.
Repeated Measures ANOVA: Part 2
Graphical Display of the Change in Means Next we created an interaction graph to show the selfdetermination means as a function of time for each of the two treatment conditions.
3.1 3 2.9 2.8 2.7 2.6 2.5 2.4 2.3 2.2
Control Adventure
Time 1
Time 2
Time 3
Repeated Measures ANOVA: Part 2
Correlations Among Self-Determination Scores Next we looked at the relationships between the scores at the three points in time.
We see from the matrix that pre, post, and follow-up measure of self-determination are all positively correlated, which indicates that cooperative groups that were higher than average early in the study tended to be higher than average later.
Repeated Measures ANOVA: Part 2
Difference Variables After looking at the self-determination scores at each point in time, we turn our attention to how much selfdetermination changed between adjacent time points. As in the last case, we define two difference variables. Diff12 Time2 Time1 Diff 23 Time3 Time2
Repeated Measures ANOVA: Part 2
Describing the Distribution of Difference Scores: Adventure Group As when we looked at the data from the adventure-programming group in the last case, we see that mean of the first difference variable is larger than the second, which again indicates that in this sample the change in self-determination between Time 1 and Time 2 was greater than the change between Time 2 and Time 3. We note there is considerable variation in the amount of change, but the distributions don’t depart markedly from normality besides the leptokurtic values in Diff23.
Repeated Measures ANOVA: Part 2
Describing the Distribution of Difference Scores: Control Group
When we look at the control group, we see that mean of the first difference variable is relatively small given the variation in the difference variable. We also note that the distributions don’t depart markedly from normality besides the platykurtic values in Diff12.
Repeated Measures ANOVA: Part 2
Describing the Correlation Between Difference Scores We also looked at the correlation between the two difference variables.
The PPMC was -.495 which showed that greater gains in self-determination from Time 1 to Time 2 were associated with smaller gains between Time 2 to Time 3.
Repeated Measures ANOVA: Part 2
Inferential Statistics At this point we have a pretty good feel for the sample data, and are ready to move into making inferences about the population. We saw a larger gain in self determination for the adventure-programming group than the control group for this sample. We would like to infer that such a discrepancy in gains exists for the population. To do this, we turn to a mixed model ANOVA.
Repeated Measures ANOVA: Part 2
Inferential Statistics At this point the computations get a bit tedious. You are welcome to leave the computations to SAS (however it is important to understand the concepts that are presented). We will show you the formulas, however, so you can see which deviations go into each sum-of-squares, and so you can appreciate what SAS is doing behind the scenes. For those of you who are mathematically inclined and feel you will understand the process better by computing things yourself, we encourage you to do so.
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA There are many parallels between mixed model ANOVAs and the factorial ANOVAs you studied previously. We will follow the same general strategy of computing appropriate sums-of-squares for each main effect and the interaction, dividing the sums-of-squares by the degrees of freedom to obtain mean squares, and taking the ratio of mean squares to compute F statistics. The following table shows the general symbols used for the scores and means in a mixed model ANOVA context. This table can be used for reference when unpacking the details of how each of the sums-of-squares are computed. Show Notation
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA Symbols
First observation in Time 1 in Treatment 1 Mean of Time 1 in Treatment 1 First observation in Time 1 in Treatment 2 Mean of Time 1 in Treatment 2
Xijk is the individual observation. The first subscript i represents the “Block” group (1-12 in this case). The second subscript represents the treatment group (1 and 2 in this case). The third is subscript is the time point (1, 2, and 3 in this case).
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA Symbols Mean of Block 1 in Treatment 1
Mean of Treatment 1 Mean of Block 1 in Treatment 2
Mean of Treatment 2 The grand mean represents the mean of all the time across two treatment groups. If you put all observations in only on group, the grand mean will be the mean of all observations.
Repeated Measures ANOVA: Part 2
Sum-of-Squares Computations We will now need to find the various Sums-of-Squares (SS): 1. Treatment group main effect: SStreatment 2. Within treatment groups: SSwithin
3. Time main effect: SStime 4. Time*treatment interaction effect: SStime*treatment 5. Residual: SSresidual
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Treatment Group Main Effect
The sums-of-squares for the treatment group main effect is obtained by finding deviations of the treatment means from the grand mean. These deviations are squared, then summed, and then multiplied by the sample size of each treatment group, n, and the number of time points, t. m
SStreatment tn ( X j 1
j
X )2
For this case: SStreatment = 5.0933
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Within Treatment Groups
The sums-of-squares within treatment groups is obtained by finding deviations of the block means from the treatment group means. These deviations are squared, then summed over blocks and treatment groups, and then multiplied by the number of time points, t. m
n
SSwithin t ( X ij X j ) j 1 i 1
2
For this case: SSwithin = 18.3114
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Time Main Effect
The sums-of-squares for the time main effect is obtained by finding deviations of the means at each point in time and the grand mean. These deviations are squared, then summed, and then multiplied by the sample size of each treatment group, n, and the number of treatment groups, m. t
SStime mn ( X k 1
k
X )2
For this case: SStime = 2.2139
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Time*Treatment
The sums-of-squares for the time by treatment interaction effect is obtained by using the cell means (the mean for a particular treatment group at a particular time). As when we computed the interaction effect in our other factorial ANOVA cases, the cell means are looked at relative to the marginal means and the grand mean.
m
t
SStime*treatment n ( X j 1 k 1
jk
X
j
X
k
X )2
For this case: SStime*treatment = 2.0627
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Residual
The sums-of-squares for the residual, which will be used in determining significance of the within-subjects factors, is obtained by looking at how individual cooperative group scores deviate from their block mean, their treatment group mean at a particular time, and the overall treatment group mean.
t
m
n
SSresidual ( X ijk X ij X k 1 j 1 i 1
jk
X j )
2
For this case: SSresidual = 8.775
Repeated Measures ANOVA: Part 2
Degrees of Freedom Computations We will also need to compute the degrees of freedom (df) associated with each sum-of-squares.
dftreatment m 1 df within N m dftime t 1
dftime*treatment m 1 t 1
For this case: df treatment = 2 – 1 = 1 df within = 24 – 2 = 22 dftime = 3 – 1 = 2 df time*treatment = (2-1) (3-1) = 2 df residual = (24-2) (3-1) = 44
df residual N m t 1
Where m is the number of treatment groups, t is the number of time points, and N is the total number of blocks.
Repeated Measures ANOVA: Part 2
Mean Squares Computations The mean squares can then be computed by dividing each of the sum-of-squares by the corresponding degrees of freedom.
MStreatment MS within MStime
SStreatment dftreatment
SS within df within
SStime dftime
MStime*treatment MS residual
SStime*treatment dftime*treatment
SS residual df residual
Repeated Measures ANOVA: Part 2
F-statistic Computations Using the mean-squares we can then make an F-ratio to test each main effect and the interaction. Note the between-subjects main effect uses the within group mean square for the error term, while the within-subjects main effect and interaction use the residual mean square for the error term.
Ftreatment Ftime
MStreatment MS within
MStime MSresidual
Ftime*treatment
MStime*treatment MSresidual
Repeated Measures ANOVA: Part 2
Summary Table Like usual, we can summarize the results of our ANOVA computations in a Table. The results for this case are:
Repeated Measures ANOVA: Part 2
p-Values Like usual, we can determine if each effect is statistically significant by either comparing the obtained F to a critical F, or we can look to see if the corresponding p-value is less than our alpha, say .05. We will show the p-value approach here. Recall that for our repeated measure factor we will want to adjust the p-value to compensate for possible violations of sphericity using either the Greenhouse-Geiser (GG) or Huynh-Feldt (HF) adjustment.
Repeated Measures ANOVA: Part 2
Statistical Decisions Let us say we decided to use the HF adjusted p-values. We would look in the last column and the last row to see if the interaction was significant, which it is (p
Repeated Measures ANOVA Effects of Adventure Programming on Self-Determination – Part II
Repeated Measures ANOVA: Part 2
In this Module In this module, you will learn:
Within-subjects factors and between-subjects factors Notation for data Sources of variations Plethora of F Statistics Contrasts Graphs New pieces of SAS code
Repeated Measures ANOVA: Part 2
Case 1 Recap: Purpose
Case 2 is simply an extension of the last case. The purpose of the study is still to examine whether adventure based programming could lead to changes in self-determination of adolescents who had been identified as at-risk for school failure.
Repeated Measures ANOVA: Part 2
Case 1 Recap: Participants
The participants were 216 middle school students who were identified as at-risk for school failure.
The students came from 6 public middle schools in a moderate sized school district in Florida. Each of the students had returned a signed parent permission form.
Repeated Measures ANOVA: Part 2
Case 1 Recap: Research Design
Within each of the six schools, the students were randomly assigned to one of four small cooperative groups, each consisting of 8 to 12 students. Groups of this size are considered optimal for the adventure programming curriculum. Two of the small groups were randomly assigned to the treatment condition and the other two to a wait-list control condition. The self-determination of the students was measured at the beginning of the study (Time1), after the treatment group went through the adventure programming (Time2), and at a follow-up time that was six weeks after the treatment group completed the adventure programming (Time3).
Repeated Measures ANOVA: Part 2
Case 1 Recap: Measures
Self-determination was assessed using the American Institute for Research (AIR) Self-Determination Scale.
The scores on this measure are recorded as the average response to a set of items using a 5-point Likert scale (1 to 5), where higher numbers indicate more selfdetermination.
Repeated Measures ANOVA: Part 2
Analysis for Case 2 For Case 2, we are going to look at both the treatment and control group. This will mean that we have to add another factor to our analysis. Now we will not only group scores by time (pretest, posttest, or follow-up test), but also by treatment condition (treatment group or control group). Thus, we have two factors like we did when we talked about factorial ANOVAs, but one of these factors is a repeated measures (within-subjects) factor.
Repeated Measures ANOVA: Part 2
Analysis Terminology Some would refer to the repeated measures factor (time) as a within-subjects factor, because you move within the same participants as you move between levels of this factor (e.g. the pretest scores and posttest scores come from the same individuals). Treatment condition, would be referred to as a between-subjects factor because you move between participants as you move to different levels of this factor (e.g., the treatment group scores and the control group scores come from different people). Consequently, some would call this an ANOVA with one betweensubjects factor and one within-subjects factor. Others would refer to it as a mixed model ANOVA because it mixes between- and within-subjects factors.
Repeated Measures ANOVA: Part 2
Unit of Analysis As noted in the last case, the students within each treatment group interacted heavily during the adventure programming (a violation of the independence assumption if we use the individual scores as the unit of analysis), thus a decision was made to use group means as the unit of analysis.
Repeated Measures ANOVA: Part 2
Data Set: Adventure Group
For each adventure programming group, the mean selfdetermination score is given at Time 1 (prior to the treatment), at Time 2 (immediately after the treatment), and at Time 3 (at the six week follow-up).
Repeated Measures ANOVA: Part 2
Data Set: Control Group
For each control group, the mean self-determination score is given at Time 1 (prior to the treatment), at Time 2 (immediately after the treatment), and at Time 3 (at the six week follow-up).
Repeated Measures ANOVA: Part 2
Analysis Plan 1. Compute appropriate descriptive statistics to describe the distribution of self-determination at each time for each group and to describe the relationship between the selfdetermination measures. 2. Construct difference variables (e.g., the change between Time1 and Time2), and look at them descriptively.
3. Conduct a Mixed Model ANOVA: - F-test for group main effect - F-test for time main effect - F-test for interaction of group and time 4. If the omnibus (overall) test is significant, run follow-up tests.
Repeated Measures ANOVA: Part 2
Descriptive Statistics: Pre, Post, Follow-up
We see that the average self-determination score in the adventure-programming group goes up during treatment more than that of the control group.
Repeated Measures ANOVA: Part 2
Graphical Display of the Change in Means Next we created an interaction graph to show the selfdetermination means as a function of time for each of the two treatment conditions.
3.1 3 2.9 2.8 2.7 2.6 2.5 2.4 2.3 2.2
Control Adventure
Time 1
Time 2
Time 3
Repeated Measures ANOVA: Part 2
Correlations Among Self-Determination Scores Next we looked at the relationships between the scores at the three points in time.
We see from the matrix that pre, post, and follow-up measure of self-determination are all positively correlated, which indicates that cooperative groups that were higher than average early in the study tended to be higher than average later.
Repeated Measures ANOVA: Part 2
Difference Variables After looking at the self-determination scores at each point in time, we turn our attention to how much selfdetermination changed between adjacent time points. As in the last case, we define two difference variables. Diff12 Time2 Time1 Diff 23 Time3 Time2
Repeated Measures ANOVA: Part 2
Describing the Distribution of Difference Scores: Adventure Group As when we looked at the data from the adventure-programming group in the last case, we see that mean of the first difference variable is larger than the second, which again indicates that in this sample the change in self-determination between Time 1 and Time 2 was greater than the change between Time 2 and Time 3. We note there is considerable variation in the amount of change, but the distributions don’t depart markedly from normality besides the leptokurtic values in Diff23.
Repeated Measures ANOVA: Part 2
Describing the Distribution of Difference Scores: Control Group
When we look at the control group, we see that mean of the first difference variable is relatively small given the variation in the difference variable. We also note that the distributions don’t depart markedly from normality besides the platykurtic values in Diff12.
Repeated Measures ANOVA: Part 2
Describing the Correlation Between Difference Scores We also looked at the correlation between the two difference variables.
The PPMC was -.495 which showed that greater gains in self-determination from Time 1 to Time 2 were associated with smaller gains between Time 2 to Time 3.
Repeated Measures ANOVA: Part 2
Inferential Statistics At this point we have a pretty good feel for the sample data, and are ready to move into making inferences about the population. We saw a larger gain in self determination for the adventure-programming group than the control group for this sample. We would like to infer that such a discrepancy in gains exists for the population. To do this, we turn to a mixed model ANOVA.
Repeated Measures ANOVA: Part 2
Inferential Statistics At this point the computations get a bit tedious. You are welcome to leave the computations to SAS (however it is important to understand the concepts that are presented). We will show you the formulas, however, so you can see which deviations go into each sum-of-squares, and so you can appreciate what SAS is doing behind the scenes. For those of you who are mathematically inclined and feel you will understand the process better by computing things yourself, we encourage you to do so.
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA There are many parallels between mixed model ANOVAs and the factorial ANOVAs you studied previously. We will follow the same general strategy of computing appropriate sums-of-squares for each main effect and the interaction, dividing the sums-of-squares by the degrees of freedom to obtain mean squares, and taking the ratio of mean squares to compute F statistics. The following table shows the general symbols used for the scores and means in a mixed model ANOVA context. This table can be used for reference when unpacking the details of how each of the sums-of-squares are computed. Show Notation
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA Symbols
First observation in Time 1 in Treatment 1 Mean of Time 1 in Treatment 1 First observation in Time 1 in Treatment 2 Mean of Time 1 in Treatment 2
Xijk is the individual observation. The first subscript i represents the “Block” group (1-12 in this case). The second subscript represents the treatment group (1 and 2 in this case). The third is subscript is the time point (1, 2, and 3 in this case).
Repeated Measures ANOVA: Part 2
Mixed Model ANOVA Symbols Mean of Block 1 in Treatment 1
Mean of Treatment 1 Mean of Block 1 in Treatment 2
Mean of Treatment 2 The grand mean represents the mean of all the time across two treatment groups. If you put all observations in only on group, the grand mean will be the mean of all observations.
Repeated Measures ANOVA: Part 2
Sum-of-Squares Computations We will now need to find the various Sums-of-Squares (SS): 1. Treatment group main effect: SStreatment 2. Within treatment groups: SSwithin
3. Time main effect: SStime 4. Time*treatment interaction effect: SStime*treatment 5. Residual: SSresidual
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Treatment Group Main Effect
The sums-of-squares for the treatment group main effect is obtained by finding deviations of the treatment means from the grand mean. These deviations are squared, then summed, and then multiplied by the sample size of each treatment group, n, and the number of time points, t. m
SStreatment tn ( X j 1
j
X )2
For this case: SStreatment = 5.0933
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Within Treatment Groups
The sums-of-squares within treatment groups is obtained by finding deviations of the block means from the treatment group means. These deviations are squared, then summed over blocks and treatment groups, and then multiplied by the number of time points, t. m
n
SSwithin t ( X ij X j ) j 1 i 1
2
For this case: SSwithin = 18.3114
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Time Main Effect
The sums-of-squares for the time main effect is obtained by finding deviations of the means at each point in time and the grand mean. These deviations are squared, then summed, and then multiplied by the sample size of each treatment group, n, and the number of treatment groups, m. t
SStime mn ( X k 1
k
X )2
For this case: SStime = 2.2139
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Time*Treatment
The sums-of-squares for the time by treatment interaction effect is obtained by using the cell means (the mean for a particular treatment group at a particular time). As when we computed the interaction effect in our other factorial ANOVA cases, the cell means are looked at relative to the marginal means and the grand mean.
m
t
SStime*treatment n ( X j 1 k 1
jk
X
j
X
k
X )2
For this case: SStime*treatment = 2.0627
Repeated Measures ANOVA: Part 2
Sum-of-Squares: Residual
The sums-of-squares for the residual, which will be used in determining significance of the within-subjects factors, is obtained by looking at how individual cooperative group scores deviate from their block mean, their treatment group mean at a particular time, and the overall treatment group mean.
t
m
n
SSresidual ( X ijk X ij X k 1 j 1 i 1
jk
X j )
2
For this case: SSresidual = 8.775
Repeated Measures ANOVA: Part 2
Degrees of Freedom Computations We will also need to compute the degrees of freedom (df) associated with each sum-of-squares.
dftreatment m 1 df within N m dftime t 1
dftime*treatment m 1 t 1
For this case: df treatment = 2 – 1 = 1 df within = 24 – 2 = 22 dftime = 3 – 1 = 2 df time*treatment = (2-1) (3-1) = 2 df residual = (24-2) (3-1) = 44
df residual N m t 1
Where m is the number of treatment groups, t is the number of time points, and N is the total number of blocks.
Repeated Measures ANOVA: Part 2
Mean Squares Computations The mean squares can then be computed by dividing each of the sum-of-squares by the corresponding degrees of freedom.
MStreatment MS within MStime
SStreatment dftreatment
SS within df within
SStime dftime
MStime*treatment MS residual
SStime*treatment dftime*treatment
SS residual df residual
Repeated Measures ANOVA: Part 2
F-statistic Computations Using the mean-squares we can then make an F-ratio to test each main effect and the interaction. Note the between-subjects main effect uses the within group mean square for the error term, while the within-subjects main effect and interaction use the residual mean square for the error term.
Ftreatment Ftime
MStreatment MS within
MStime MSresidual
Ftime*treatment
MStime*treatment MSresidual
Repeated Measures ANOVA: Part 2
Summary Table Like usual, we can summarize the results of our ANOVA computations in a Table. The results for this case are:
Repeated Measures ANOVA: Part 2
p-Values Like usual, we can determine if each effect is statistically significant by either comparing the obtained F to a critical F, or we can look to see if the corresponding p-value is less than our alpha, say .05. We will show the p-value approach here. Recall that for our repeated measure factor we will want to adjust the p-value to compensate for possible violations of sphericity using either the Greenhouse-Geiser (GG) or Huynh-Feldt (HF) adjustment.
Repeated Measures ANOVA: Part 2
Statistical Decisions Let us say we decided to use the HF adjusted p-values. We would look in the last column and the last row to see if the interaction was significant, which it is (p
