Observational studies vs. experiments in statistics

Asking people what they already do gives us a different kind of evidence from assigning them something to try. We’ll use a school survey and a treatment study to see what each design can tell us.

Start with a population and a sample

The population is the whole group we want to understand. A sample is the part of that group we actually measure. A number calculated from the sample is a statistic. The corresponding number for the whole population is a parameter. For example, we can use a sample mean to estimate a population mean.

Collect observations without assigning a treatment

Suppose we ask a class whether they prefer peanut butter or jelly. We record one variable for each student, so we have one-way data. If 70% of the class prefers peanut butter, that is the sample proportion. A single class is a convenience sample, so how well that proportion describes the entire school depends on how representative the class is.

If we also record each student’s height, we have two variables for each student. For a two-way table, we could group the heights into categories and count preferences within each group. If the share preferring peanut butter increases with height, that shows an association in the data.

Age could be a confounding variable if older students tend to be taller and have different preferences. The observed association leaves open whether height affects preference, age helps explain both, or another factor is involved.

Assign a treatment in an experiment

Suppose we want to study whether a treatment changes blood pressure. Researchers could randomly assign participants to receive the treatment or a placebo and compare the groups’ changes in blood pressure. The assigned condition is the explanatory variable, and the measured change is the response variable.

Random assignment helps make the groups comparable before the treatment. Random sampling is a separate step that helps us generalize to the population we sampled from. A well-designed randomized experiment can provide evidence that the treatment caused a difference.

What changes between the two designs
Study designWhat the researcher does
Observational studyRecords variables without assigning a treatment.
ExperimentAssigns treatments, then compares the measured outcomes.

Use controls, blinding, and matched pairs

A placebo gives the control group a similar experience without the active treatment. In a single-blind design, participants can be kept unaware of their assignment. In a double-blind design, participants and the people giving the treatment or assessing the response can both be kept unaware.

To block, group similar participants before randomly assigning treatments within each group. In a matched-pairs design, the blocks have two similar participants. For example, match participants on age and sex, then randomly assign one person in each pair to the treatment and the other to the placebo. Compare the changes within each pair.

Repeat the study to see whether the result holds up

Using enough independent participants in each group gives us repeated observations within the experiment. A separate team can also repeat the study with new participants. Similar results from a new study give us more evidence that the finding holds up.