Chapter 14: Generalizing Results
Chapter 14: Generalizing Results
Issues of generalizing results to other population; problem with college students as participants. Issues of generalizing result to other cultures and ethnic groups. Problem of generalizing to other experimenters and possible solutions. Importance of replications, distinguishing between exact replications and conceptual replications. Narrative literature reviews and meta-analyses.
Recall: Internal validity—ability to infer a causal relationship exist between the variables. External validity—extent to which findings may be generalized.
Generalizing to other populations of research participants Rarely are participants randomly selected from the general population, usually they are selected because they are available: college students! So can we really generalize the results beyond that group? College students (and rats) High % (70%) of studies published used college student as participant. Potential problem is that such studies use a highly restricted population. College students: young, late adolescence—sense of self-identity still developing, social and political attitudes in flux, high need for peer approval, unstable peer relationships. Intelligent. Research shows those students groups are more homogeneous than non-student samples. Rats are hardy, cheap, easy to rear, well adapted to lab, just like first year students and sophomores, easy to obtain on a campus and readily available. Volunteers External validity of findings may also be limited, volunteers tend to be: more highly educated, higher socioeconomic status, and more in need of approval, more social. Different kinds of people volunteer for different kinds of experiments (based on the Title). Internet research Asking people on internet to volunteer for a survey: but the internet use is still more common in a particular demographic: urban/suburban area, college educated, younger, higher income. Gender considerations Sometimes one sex is used because convenient, or the procedures seem better suited to a gender. Gender bias: confounding gender with age or job status, selecting response measures that are gender-stereotyped. *Include both & recognize ways that they might differentially interpret variable manipulations. Locale Location that participants are recruited from: found even personality traits like extraversion and openness can vary across geographic areas.
Generalization as a statistical interaction Think of the problem of generalization as an interaction in a factorial design: -An interaction occurs when a relationship between variables exists under 1 condition but not another; when nature of relationship is different in one condition than in another. Ex: study used only males, perhaps an interaction between gender and independent variable. Perhaps can’t generalize the result to females. Ex: study on aggression and crowding in both sexes. -Graph C+D shows interaction between gender and crowding.
In defense of college students (and rats) Criticism of study using only college students should be backed with good reasons that a relationship would not be found with other types of subject. 1. Student bodies are increasingly diverse and representative of society as a whole. 2. Replication provides safeguard against limited external validity of a single study. (applying to other populations such as children, aging adults, other countries). *Internet samples are often used as a complement based on college student samples. Rats: we can applies findings to humans: bio bases of memory, food preferences, sexual behavior, choice behavior, drug addictions.
Cultural Considerations Cross culture generalization, critic—Psychology is built on study of WEIRD (western, educated, industrialized, rich, democratic) people. -Ex: theories of self concept: in western society – self means people are independent, self-enhancement comes from individual achievements. In other cultures: self is a collective concept, self-esteem derived from relationships with others. Japanese engage in self-criticism which is seen as relationship-maintaining. -Many are different and low generalization, ex: self. Same is: waist-hip ratio! Generalized across culture.
Generalizing to other experimenters
Experimenters are another source of external validity problems: often just one experimenter. Main goal is to ensure that any influence they has on subjects is constant throughout experiment. Important charac of experimenters: personality, gender, amount of practice as experimenter. Ex: found participants do better when tested by experimenter of other sex. (disclosure experiment, participants revealed more with experimenters of opp.sex). Solution: use two/more experimenters.
Pretests and generalization Use pretest to ensure that groups are equivalent at beginning and see the change. May limit ability to generalize to populations that did not receive a pretest. Using pretest let you assess mortality (attrition): see if people who withdraw are different from those who completed the study. Ex: Solomon four-group design half receive pretest, other half post test only. Mortality effects can be assessed in pretest conditions. Can also examine if there’s an interaction between the independent variable and pretest: are post test score different if had pretest or no. Generalizing from laboratory settings
Highly controlled conditions allow for internal validity. But the artificiality may limits ability to generalize. But it also depends on the purpose of experiment (it is fine if you want to determine causal relationships). Can also examine the results of field experiments: manipulates IV in natural setting. -Found, results of lab and field experiments complements each other. Similar effect size. When findings are replicated using multiple methods, confidence in external validity increases.
Importance of replications a) Exact replications: attempt to replicate precisely the procedures of a study to see if obtain same results. Ex: make sure the finding is reliable, or build on findings of prior study (expand). -Failure to replicate share the same problems as non-significant results: original results are invalid, but also may be that replication attempts was flawed, important procedure omitted. Better to write to the researcher to obtain detailed info on all the materials used. -Replication do not occur in isolation, if multiple repeated failures to replicate, perhaps original study was a type I error. Or the results can only be obtained under certain limited circumstances. b) Conceptual replication: use of different procedures to replicate a finding. -Using new, or different operational definitions of those variables. * More important in furthering our understanding of behavior, especially in social sciences, have multiple operational definition. -Can use different operational definition of IV, and use different measuring of DV. Even using alternative stimulus (a different composer to analyze the Mozart effect), or alternative dependent measure. -Or, same variable studied in both lab and field settings. -Operational definitions can change over time, but the underlying conceptual variable often remains consistent. (Conceptual replication of two variables can stand the test of time).
Evaluating generalizations with literature reviews and meta-analyses Literature review is often used to draw external validity of research findings. Literature review: a reviewer reads may studies addressing a topic and then writes a paper to summarizes and evaluates the literature. It functions to organize, integrate, evaluate previously published material, and considers progress of research toward a problem. a) Summarizes what has been found. b) Tells what findings are strongly supported, what weakly supported. c) Inconsistent findings and areas which research is lacking. d) Future directions. Narrative literature review: based on subjective impressions of reviewer. -Identifies trends, often qualitative conclusions. Meta-analysis: combines the actual results of many studies. Using statistical procedures employing effect sizes to compare findings across studies. -Draw quantitative statistical conclusions. Can lead to clear general conclusions. Using research to improve lives Research findings can be generalized to many fields in our daily lives. a) Health: programs to promote health related behaviors related to stress, STD, heart. b) Law & Criminal justice: data on effects of 6 vs 12 jurors. c) Education: methods to improve academic performance reduce conflict between ethnic groups. d) Work environment: more control and improve interactions with computers. Also, published information on parenting, education, mental health, on the internet.
Issues of generalizing results to other population; problem with college students as participants. Issues of generalizing result to other cultures and ethnic groups. Problem of generalizing to other experimenters and possible solutions. Importance of replications, distinguishing between exact replications and conceptual replications. Narrative literature reviews and meta-analyses.
Recall: Internal validity—ability to infer a causal relationship exist between the variables. External validity—extent to which findings may be generalized.
Generalizing to other populations of research participants Rarely are participants randomly selected from the general population, usually they are selected because they are available: college students! So can we really generalize the results beyond that group? College students (and rats) High % (70%) of studies published used college student as participant. Potential problem is that such studies use a highly restricted population. College students: young, late adolescence—sense of self-identity still developing, social and political attitudes in flux, high need for peer approval, unstable peer relationships. Intelligent. Research shows those students groups are more homogeneous than non-student samples. Rats are hardy, cheap, easy to rear, well adapted to lab, just like first year students and sophomores, easy to obtain on a campus and readily available. Volunteers External validity of findings may also be limited, volunteers tend to be: more highly educated, higher socioeconomic status, and more in need of approval, more social. Different kinds of people volunteer for different kinds of experiments (based on the Title). Internet research Asking people on internet to volunteer for a survey: but the internet use is still more common in a particular demographic: urban/suburban area, college educated, younger, higher income. Gender considerations Sometimes one sex is used because convenient, or the procedures seem better suited to a gender. Gender bias: confounding gender with age or job status, selecting response measures that are gender-stereotyped. *Include both & recognize ways that they might differentially interpret variable manipulations. Locale Location that participants are recruited from: found even personality traits like extraversion and openness can vary across geographic areas.
Generalization as a statistical interaction Think of the problem of generalization as an interaction in a factorial design: -An interaction occurs when a relationship between variables exists under 1 condition but not another; when nature of relationship is different in one condition than in another. Ex: study used only males, perhaps an interaction between gender and independent variable. Perhaps can’t generalize the result to females. Ex: study on aggression and crowding in both sexes. -Graph C+D shows interaction between gender and crowding.
In defense of college students (and rats) Criticism of study using only college students should be backed with good reasons that a relationship would not be found with other types of subject. 1. Student bodies are increasingly diverse and representative of society as a whole. 2. Replication provides safeguard against limited external validity of a single study. (applying to other populations such as children, aging adults, other countries). *Internet samples are often used as a complement based on college student samples. Rats: we can applies findings to humans: bio bases of memory, food preferences, sexual behavior, choice behavior, drug addictions.
Cultural Considerations Cross culture generalization, critic—Psychology is built on study of WEIRD (western, educated, industrialized, rich, democratic) people. -Ex: theories of self concept: in western society – self means people are independent, self-enhancement comes from individual achievements. In other cultures: self is a collective concept, self-esteem derived from relationships with others. Japanese engage in self-criticism which is seen as relationship-maintaining. -Many are different and low generalization, ex: self. Same is: waist-hip ratio! Generalized across culture.
Generalizing to other experimenters
Experimenters are another source of external validity problems: often just one experimenter. Main goal is to ensure that any influence they has on subjects is constant throughout experiment. Important charac of experimenters: personality, gender, amount of practice as experimenter. Ex: found participants do better when tested by experimenter of other sex. (disclosure experiment, participants revealed more with experimenters of opp.sex). Solution: use two/more experimenters.
Pretests and generalization Use pretest to ensure that groups are equivalent at beginning and see the change. May limit ability to generalize to populations that did not receive a pretest. Using pretest let you assess mortality (attrition): see if people who withdraw are different from those who completed the study. Ex: Solomon four-group design half receive pretest, other half post test only. Mortality effects can be assessed in pretest conditions. Can also examine if there’s an interaction between the independent variable and pretest: are post test score different if had pretest or no. Generalizing from laboratory settings
Highly controlled conditions allow for internal validity. But the artificiality may limits ability to generalize. But it also depends on the purpose of experiment (it is fine if you want to determine causal relationships). Can also examine the results of field experiments: manipulates IV in natural setting. -Found, results of lab and field experiments complements each other. Similar effect size. When findings are replicated using multiple methods, confidence in external validity increases.
Importance of replications a) Exact replications: attempt to replicate precisely the procedures of a study to see if obtain same results. Ex: make sure the finding is reliable, or build on findings of prior study (expand). -Failure to replicate share the same problems as non-significant results: original results are invalid, but also may be that replication attempts was flawed, important procedure omitted. Better to write to the researcher to obtain detailed info on all the materials used. -Replication do not occur in isolation, if multiple repeated failures to replicate, perhaps original study was a type I error. Or the results can only be obtained under certain limited circumstances. b) Conceptual replication: use of different procedures to replicate a finding. -Using new, or different operational definitions of those variables. * More important in furthering our understanding of behavior, especially in social sciences, have multiple operational definition. -Can use different operational definition of IV, and use different measuring of DV. Even using alternative stimulus (a different composer to analyze the Mozart effect), or alternative dependent measure. -Or, same variable studied in both lab and field settings. -Operational definitions can change over time, but the underlying conceptual variable often remains consistent. (Conceptual replication of two variables can stand the test of time).
Evaluating generalizations with literature reviews and meta-analyses Literature review is often used to draw external validity of research findings. Literature review: a reviewer reads may studies addressing a topic and then writes a paper to summarizes and evaluates the literature. It functions to organize, integrate, evaluate previously published material, and considers progress of research toward a problem. a) Summarizes what has been found. b) Tells what findings are strongly supported, what weakly supported. c) Inconsistent findings and areas which research is lacking. d) Future directions. Narrative literature review: based on subjective impressions of reviewer. -Identifies trends, often qualitative conclusions. Meta-analysis: combines the actual results of many studies. Using statistical procedures employing effect sizes to compare findings across studies. -Draw quantitative statistical conclusions. Can lead to clear general conclusions. Using research to improve lives Research findings can be generalized to many fields in our daily lives. a) Health: programs to promote health related behaviors related to stress, STD, heart. b) Law & Criminal justice: data on effects of 6 vs 12 jurors. c) Education: methods to improve academic performance reduce conflict between ethnic groups. d) Work environment: more control and improve interactions with computers. Also, published information on parenting, education, mental health, on the internet.
