Simpson's paradox
In a 1986 study of 700 kidney-stone patients, treatment A had the higher success rate for small stones and for large stones — but treatment B had the higher success rate overall. Change who got which treatment and see why.
From the slides: Probability and what happens when you learn something.
| Treatment A | Treatment B | |
|---|---|---|
| small stones | 81/87 = 93.1% | 234/270 = 86.7% |
| large stones | 192/263 = 73.0% | 55/80 = 68.8% |
| all patients | 273/350 = 78.0% | 289/350 = 82.6% |
Data: C. R. Charig et al., British Medical Journal 292 (1986). The success rate inside each group is held fixed below; only the mix of patients changes.
A overall
B overall
Which looks better overall?
What to notice
- The overall rate is a weighted average: P(success | A) = P(success | A, small) P(small | A) + P(success | A, large) P(large | A). That is the law of total probability.
- In the study, A was given mostly to the hard cases (75% large stones) and B mostly to the easy ones (77% small). B's overall number is flattered by its easy caseload.
- Press Give both the same mix: with equal weights A wins overall, as it does in each group. Before trusting an aggregate, ask what it is averaging over.