Reading Numbers in the News: Risk, Causation and Sample Size

Imagine a headline that reads "Popular Snack Linked to Doubled Risk of Rare Condition." It sounds alarming, and it might even be accurate. But the headline alone cannot tell you whether you should change anything about your life, because it leaves out the numbers that matter most.

Numbers in the news are rarely made up. More often, real numbers are presented in a way that makes them seem bigger, smaller or more certain than they are. A few simple questions can help you see past the framing to what a study actually found.

This guide covers three of the most common traps: relative versus absolute risk, correlation versus causation, and small samples. The snack, towns and products below are hypothetical examples; the studies and guidance are real and named.

Relative Risk vs. Absolute Risk

Relative risk tells you how much a risk changed compared with where it started. Absolute risk tells you how many people in a group are actually affected.

The gap between the two can be dramatic. Suppose, hypothetically, that a rare condition affects 2 out of every 10,000 people, and a study finds that it affects 4 out of every 10,000 people who eat a certain snack daily. A headline can truthfully say the risk "doubled," which is a 100% increase. In absolute terms, the risk went up by 2 people in 10,000.

The same framing works for benefits. In the US Food and Drug Administration's 2011 guide Communicating Risks and Benefits, a chapter on quantitative information by Angie Fagerlin and Ellen Peters gives the example of a preventive treatment whose effect could be described as a 50% reduction in breast cancer risk (relative) or as a reduction from a 6% risk to 3% (absolute). The authors report that when information is presented in a relative format, the risk reduction seems larger and treatments are viewed more favorably than when the same information is given in absolute terms.

A 2007 review by Gigerenzer and colleagues in Psychological Science in the Public Interest makes the same point with a real case: the statement that mammography screening reduces the risk of dying from breast cancer by 25% in fact means that 1 less woman out of 1,000 will die of the disease. Both numbers describe the same data. Only one shows you the size of the effect.

So when you see a percentage change, ask: from what, to what? If the article doesn't say, the claim is incomplete.

Keep the Denominator and Time Frame Steady

Comparisons get slippery when numbers use different bases. Hypothetically, one article says a side effect occurs in "1 in 200" users of Product A, and another says "3 in 1,000" users of Product B. Which is more common? Put them on the same denominator: 1 in 200 is the same as 5 in 1,000, so Product A's rate is higher, even though "3" looks bigger than "1" at a glance.

The FDA chapter's recommendations for presenting numbers include giving absolute risks rather than only relative risks, keeping denominators constant when comparing options, and keeping time frames constant. You can apply the same rules as a reader. A risk "per year" and a risk "over a lifetime" are not comparable, and neither is a rate among people who already have a condition versus a rate in the general population.

Correlation Is Not Causation, and Headlines Blur the Line

Many health and lifestyle stories are based on observational studies, which look at what people already do and what later happens to them. Such studies can reveal that two things tend to go together, but on their own they usually cannot show that one causes the other. People who eat the hypothetical snack every day might also differ in sleep, age, work or other habits. Our guide Thinking Like a Scientist walks through the main alternatives to a causal explanation, such as chance, reverse causation and a common cause.

The trouble is that causal language tends to creep in on the way to the headline. A 2014 study in The BMJ by Sumner and colleagues examined 462 health-related press releases issued by 20 leading UK universities in 2011, along with the underlying journal articles and 668 related news stories. They found that 33% of the press releases contained causal claims stronger than the research supported, 40% contained exaggerated advice, and 36% contained exaggerated inference to humans from animal research. When a press release exaggerated a causal claim, 81% of the related news stories did too, compared with 18% when the press release did not.

The researchers sorted wording into a ladder from weakest to strongest claim. Here it is with a hypothetical example:

• Correlational: "Eating the snack is associated with higher rates of the condition."
• Ambiguous: "Snack linked to condition."
• Conditional causal: "The snack might raise your risk."
• Can cause: "The snack can raise your risk."
• Unconditionally causal: "The snack raises your risk."

If the study was observational, the first rung is usually the honest one. When you notice the language climbing, look for the study design. Also check whether the research was done in people at all; results in animals or cells are early steps, not conclusions about humans.

Sample Size: Small Groups Swing Wildly

Small samples produce extreme results more easily than large ones, simply by chance. Flip a fair coin 4 times, and getting all heads happens about 1 time in 16. Flip it 100 times, and all heads is so unlikely that you should not expect to see it in your lifetime. The same logic applies to groups of people. Hypothetically, a town of 500 people with three cases of a rare illness can show a rate many times the national figure, while a city of 500,000 rarely will, because its numbers average out.

Researchers worry about this too. In a 2013 analysis in Nature Reviews Neuroscience, Button and colleagues explained that a study with low statistical power, often because it is small, has a reduced chance of detecting a true effect, and that low power also reduces the likelihood that a statistically significant result reflects a true effect. They list overestimates of effect size and low reproducibility among the consequences. Their analysis focused on neuroscience, but the statistical point applies widely.

So when a story reports a striking result, look for how many people were studied and whether the finding has been repeated. A small study can be a valuable first step. It is rarely the last word.

A Quick Checklist for Any Number

1. From what, to what? Find the absolute numbers behind any percentage change.
2. Out of how many, and over what time? Put comparisons on the same denominator and time frame.
3. Who was studied? People or animals, and how similar are they to you?
4. What kind of study was it? Observational studies show associations; randomized experiments are better at testing causes.
5. How big was the sample, and has anyone repeated the result?
6. What does the original say? Press releases and headlines can be stronger than the paper behind them.

None of this means distrusting every statistic. It means asking the questions the headline skipped. And for decisions about your own health, a clinician can help put the numbers in the context of your situation.

Key takeaways

• A relative change such as "doubles" or "cuts in half" means little without the absolute numbers behind it.
• Compare risks using the same denominator and the same time frame.
• Observational studies show associations; watch for headlines that climb from "linked to" to "causes."
• Small samples produce extreme results more easily and can overestimate effects.
• When a claim matters to you, look for the original study or a careful summary of it.

Sources

• US Food and Drug Administration, Communicating Risks and Benefits: An Evidence-Based User's Guide (2011), including Chapter 7, Quantitative Information, by Angie Fagerlin and Ellen Peters
• Gigerenzer and colleagues, Helping Doctors and Patients Make Sense of Health Statistics, Psychological Science in the Public Interest (2007), PubMed record
• Sumner and colleagues, The Association Between Exaggeration in Health Related Science News and Academic Press Releases, The BMJ (2014)
• Button and colleagues, Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience, Nature Reviews Neuroscience (2013), PubMed record