Comparing two groups – randomization test and two-sample t-test
MathematicsStatisticsAges 15–16
How do you mainly use this simulation?
Use it with your class: students open it from a link or a QR code, with no account needed.
Three real randomized experiments (caffeine and finger tapping, a nap or caffeine before a memory test, intrinsic or extrinsic motivation and creativity), generated data or your own values. Shuffle the group labels again and again to build the randomization distribution of the difference in means, read the share of shuffles at least as extreme as the observed difference, and compare this p-value with the Welch or pooled two-sample t-test.
Lesson: Inference for two groups: randomization tests and the two-sample t-test
What it shows
In a randomized experiment, chance decided who received which treatment. If the treatment made no difference (the null hypothesis), every subject's value would be the same in either group, so the observed difference in means is only one of the many ways the labels could have been dealt. Shuffling the labels again and again shows that randomization distribution, and the p-value is the share of shuffles at least as extreme as the observed difference. The two-sample t-test reaches a similar p-value from a formula, assuming roughly normal populations; Welch's version does not assume equal variances.
How to use
Choose the Data set and the alternative μ₁ > μ₂, μ₁ ≠ μ₂ or μ₁ < μ₂. Press Shuffle labels once to watch the values dealt into new groups, then +100 shuffles or +1000 shuffles to build the distribution. Tick Compare with the t-test to see t, df and its p-value, and switch between Welch and Pooled. Edit the values to use your own data, change the Random seed, and press Record.
Parameters you can change
- Data set Caffeine and finger taps, Nap or caffeine: words recalled, Motivation and creativity, Generated data, Your own data
- Alternative H₁ μ₁ > μ₂, μ₁ ≠ μ₂, μ₁ < μ₂
- t-test Welch (unequal variances), Pooled (equal variances)
- Compare with the t-test
- Random seed 1–999
- Size of each group n (generated data) 4–30
- True difference μ₁ − μ₂ (generated data) -6–6
- Population SD σ (generated data) 1–8
- Population shape (generated data) Normal, Right-skewed
- Shuffles already done when the page opens 0–10000
Questions to explore
- Why does the randomization p-value change a little when you change the seed, while the t-test p-value does not?
- For the nap or caffeine data, how close are the randomization p-value and the t-test p-value?
- What happens to both p-values when you add one very large value to group 1?