Experiments and Observational Studies Observational Studies
Stats: Data and Models – Canadian Edition Chapter 13 – Experiments and Observational Studies Observational Studies - Studies where researchers don’t assign choices, they simply observe them - Retrospective study – subjects are identified first and then data is collected on an aspect of their past, can have errors because they are based on historical data - Used widely in public health and marketing, used for discovering trends and possible relationships - Prospective study – identifying subjects in advance and collecting data as events unfold - Observational studies cannot demonstrate causal relationships Randomized, Comparative Experiments - An experiment requires a random assignment of subjects to treatments - An experiments studies the relationship between two or more variables – at least one explanatory variable (a factor) that is manipulated, and at least one response variable to measure - The experimenter actively and deliberately manipulates the factors to control the details of the possible treatments and assigns subjects to those treatments at random - Experimental units – individuals on whom we experiment, human units are called subjects or participants - Levels of a factor – specific values of a factor that the experimenter chooses - Treatment – combination of specific levels from all factors that a unit receives The Four Principles of Experimental Design - Control: control sources of variation other than the factors we are testing by making conditions as similar as possible for all treatment groups; controlling extraneous sources of variation reduces the variability of responses, making it easier to detect differences among the treatment groups o Control a factor by assigning subjects to different factor levels to see how to response will change at different levels o Control other sources of variation to prevent them from changing and affecting the response variable - Randomize: randomization allows us to equalize the effects of unknown or uncontrollable sources of variation, assigning subjects to treatments at random reduces bias due to uncontrolled sources of variation - Replicate: each treatment should be applied to a number of subjects; the outcome of an experiment on a single subject is an anecdote, not data, replication of an entire experiment with the controlled sources of variation at different levels is essential - Blocking: compromise between randomization and control, uncontrollable attributes of experimental units (i.e. age) may affect the outcomes of an experiment so we group similar individuals together and randomize within each block to remove variability that was due to the difference amongst blocks Diagrams - A diagram of an experiment’s procedure emphasizes the random allocation of subjects to treatment groups, the separate treatments applied to these groups, and the ultimate comparison of results Does the Difference Make a Difference? - Even if the treatment made no difference, there would still be some variation between groups of an experiment
-
Differences are statistically significant if it can be decided that the difference is bigger than that that could be attributed to just the randomization alone; it can be attributed to the treatment
Experiments and Samples - Sample surveys try to estimate population parameters - Experiments try to assess the effects of treatments - In an experiment, the randomization is not actually in the selection of its experimental units, but in the assignment to their therapies - Unless experimental units are chosen from the population at random, you should be cautious about generalizing experiment results to larger populations until the experiment has been repeated under different circumstances - Results become more persuasive if they remain the same in completely different circumstances - Experiments can draw stronger conclusions than surveys - Experiments cancel out many sources of bias by only looking at the differences across treatment groups; i.e. if the groups involved in an experiment are biased in the same way, that bias cancels out and the differences due to the treatment are seen more clearly Control Treatments - Circumstance where no treatment is applied to those in the group - Baseline measurement - Experimental units included are called the control group Blinding - To avoid bias, participants are blinded, made unaware, of the treatment condition they are in - Experimenters often subconsciously behave in ways that favour what they believe, and therefore should be blinded from who is in which treatment as well - Two classes of individuals can affect the outcome of the experiment: o Those who could influence the results o Those who evaluate the results - When the individuals in one of the above classes are blinded, the experiment is single-blinded, when both classes are blinded, the experiment is double-blinded Placebos - Some of the improvement seen with a treatment can be due simply to the act of treating - Placebo: “fake” treatment that looks like the treatments being tested; mimics the treatment itself - Placebos are the best way to blind subjects from knowing whether they are receiving the treatment or not - When psychological attitude can affect the results, control group subjects treated with a placebo may show an improvement - The placebo effect highlights the importance of blinding and the importance of comparing treatments with a control - The best experiments are usually: randomized, double-blinded, comparative, placebo-controlled Blocking - when groups of experimental units are similar, they can be gathered together into blocks - Blocking isolates the variability attributable to the differences between the blocks, so the difference caused by the treatments can be seen more clearly - Completely randomized design: each experimental unit has an equal chance of landing in each of the treatment groups - Randomized block design: randomization only occurs within blocks
-
Matching subjects because they are similar in ways not under study can reduce variation in the same way as blocking
Confounding - When the levels of one factor are associated with the levels of another, the two are confounded - Confounding interferes with our ability to interpret our analyses simply - A confounding variable is a variable associated with the experimental factor that may also have some effect on the response
-
Differences are statistically significant if it can be decided that the difference is bigger than that that could be attributed to just the randomization alone; it can be attributed to the treatment
Experiments and Samples - Sample surveys try to estimate population parameters - Experiments try to assess the effects of treatments - In an experiment, the randomization is not actually in the selection of its experimental units, but in the assignment to their therapies - Unless experimental units are chosen from the population at random, you should be cautious about generalizing experiment results to larger populations until the experiment has been repeated under different circumstances - Results become more persuasive if they remain the same in completely different circumstances - Experiments can draw stronger conclusions than surveys - Experiments cancel out many sources of bias by only looking at the differences across treatment groups; i.e. if the groups involved in an experiment are biased in the same way, that bias cancels out and the differences due to the treatment are seen more clearly Control Treatments - Circumstance where no treatment is applied to those in the group - Baseline measurement - Experimental units included are called the control group Blinding - To avoid bias, participants are blinded, made unaware, of the treatment condition they are in - Experimenters often subconsciously behave in ways that favour what they believe, and therefore should be blinded from who is in which treatment as well - Two classes of individuals can affect the outcome of the experiment: o Those who could influence the results o Those who evaluate the results - When the individuals in one of the above classes are blinded, the experiment is single-blinded, when both classes are blinded, the experiment is double-blinded Placebos - Some of the improvement seen with a treatment can be due simply to the act of treating - Placebo: “fake” treatment that looks like the treatments being tested; mimics the treatment itself - Placebos are the best way to blind subjects from knowing whether they are receiving the treatment or not - When psychological attitude can affect the results, control group subjects treated with a placebo may show an improvement - The placebo effect highlights the importance of blinding and the importance of comparing treatments with a control - The best experiments are usually: randomized, double-blinded, comparative, placebo-controlled Blocking - when groups of experimental units are similar, they can be gathered together into blocks - Blocking isolates the variability attributable to the differences between the blocks, so the difference caused by the treatments can be seen more clearly - Completely randomized design: each experimental unit has an equal chance of landing in each of the treatment groups - Randomized block design: randomization only occurs within blocks
-
Matching subjects because they are similar in ways not under study can reduce variation in the same way as blocking
Confounding - When the levels of one factor are associated with the levels of another, the two are confounded - Confounding interferes with our ability to interpret our analyses simply - A confounding variable is a variable associated with the experimental factor that may also have some effect on the response
