Simpson's Paradox

What’s going on

Ann and Ben practise basketball shots. Look at the numbers in the scene above:

• Close shots: Ann hits 9 of 10 (90%), Ben hits 80 of 100 (80%). Ann is better.
• Long shots: Ann hits 30 of 100 (30%), Ben hits 2 of 10 (20%). Ann is better again.
• All shots together: Ann hits 39 of 110 (35%), Ben hits 82 of 110 (75%). Ben is far better.

Nobody made a mistake. Every number is correct, and an option can win in every group and still lose when the groups are put together. This is called Simpson’s paradox.

Where the trick hides

Look at the mix. Ann took 100 long shots and only 10 close ones. Ben did the opposite. Long shots are hard for everyone, so Ann’s total is dragged down by all her difficult attempts, while Ben’s total is lifted by his easy ones.

The overall rate is not a plain average of the two group rates. It is a weighted average, and the weights are the mix. The bars in the scene show this as area. Each part of a bar is as wide as the number of shots and filled up to the hit rate, so the filled area is the number of hits. The dashed line shows the level you would get if you spread that filling evenly across the whole bar. Ann’s tall column is a thin sliver; her wide column is mostly empty.

The chart says the same thing with lines. Each player’s line runs from their close-shot rate (left edge) to their long-shot rate (right edge). The overall rate must sit somewhere on that line, at the position given by the share of long shots. Ann’s line is above Ben’s everywhere, but Ann’s dot sits far to the right and Ben’s far to the left, so Ben’s dot ends up higher.

Drag the dots, or move the sliders:

• Push Ann’s dot to the left. With only 10 long shots, like Ben, she hits 93 of 110 (85%) and wins.
• Press Same mix for both. With the same share of long shots, the dots sit one above the other, and the player who is better in each group is better overall. The paradox needs the two options to have different mixes.

A real case

Press Kidney stones study. In a 1986 study of 700 patients, open surgery (treatment A) succeeded more often than a keyhole method (treatment B) for small stones (93% against 87%) and for large stones (73% against 69%). Yet over all patients B looked better: 83% against 78%.

The reason is the same as with the shots. Large stones are harder to treat, and treatment A was given mostly to patients with large stones. Here the fair comparison is the one inside each group. (Whether to trust the groups or the total always depends on why the mixes differ: that is a question about the real world, not about arithmetic.)

Try it yourself

1. Make your own paradox

Two footballers take penalties over two seasons. Find numbers so that Xia scores more often than Yuri in each season but less often over both seasons together.

Answer

For example: season 1, Xia 4 of 10 (40%), Yuri 35 of 100 (35%); season 2, Xia 25 of 100 (25%), Yuri 2 of 10 (20%). Xia wins both seasons. Together, Xia scores 29 of 110 (about 26%) and Yuri 37 of 110 (about 34%). The recipe: give the winner most of his attempts in the harder season.

2. Equal groups

Suppose both options have exactly the same number of cases in group 1 and the same number in group 2. Can Simpson’s paradox still happen?

Answer

No. With the same mix, both overall rates use the same weights: overall = rate₁ × w + rate₂ × (1 − w) with the same w. If one option has the bigger rate₁ and the bigger rate₂, its weighted average is bigger too. In fact the same share is enough; the totals can differ. This is why good experiments assign treatments at random: then each treatment gets roughly the same mix of easy and hard cases.

3. Which treatment for you?

You have a small kidney stone. Looking at the study above, which treatment does the data favour? And if you do not know the size of your stone?

Answer

For a small stone, A: 81 of 87 (93%) against 234 of 270 (87%). For a large stone, A again: 192 of 263 (73%) against 55 of 80 (69%). If you do not know the size, your stone is still either small or large, and in both cases A did better, so A is still the better bet. B's higher total only reflects that B was given mostly to easier cases.

Go deeper

• The paradox is named after Edward H. Simpson, who wrote about it in 1951, but Karl Pearson and Udny Yule had noticed the same effect around 1900.
• A famous example is the 1973 graduate admissions at the University of California, Berkeley: overall, men were admitted more often than women, yet department by department women did about as well or better. Women had more often applied to the departments that were hardest to get into.