Harvard on telling evidence from hype
Harvard T.H. Chan School of Public Health · From its longevity science course, which devotes a module to evaluating claims.
Harvard's public course on longevity science describes its own purpose as teaching what is known, what is not, and how to tell a credible intervention from hype. It gives a module to that last part alone: the hierarchy of evidence, the difference between a statistically real result and one large enough to matter, and how to read a conflict of interest. We think that is the most useful thing anyone can hand a reader in this field, so the five questions below are our version of it, together with the words you need to use them.
Five questions to ask of any claim
- 01
What kind of study was it?
A trial that assigned people at random can show cause. A study that followed people who chose their own habits can show association, and people who do one healthy thing tend to do several. Work in mice or in cells says something may be worth testing in people; it does not say it works in people.
- 02
How big was the effect, in people out of a hundred?
A 40 percent reduction can mean going from five people in a hundred to three. Ask for the absolute numbers. If they are not given, that is usually why.
- 03
Did it measure the thing that matters, or something standing in for it?
Lowering a blood marker, shrinking a lesion on a scan or improving a score is not the same as fewer heart attacks, better memory or more years independent. Several large trials have moved the marker and not the outcome.
- 04
Who paid, and who benefits?
Not a reason to dismiss a study, but a reason to read the methods more carefully and to look for someone independent finding the same thing.
- 05
Has anyone found it twice?
One study is a data point. A finding that has been repeated by a different group, in different people, is a result. Most of what makes the news is still at the first stage.
The five phrases we use
Every claim discussed on this site carries one of these, with the study type alongside. They are the answer to the first question above, given in advance.
- Strong and consistent
- Several good studies, in people, pointing the same way, with an effect large enough to matter to a person rather than a spreadsheet.
- Promising but early
- Real signal, but from small studies, short follow-up, animal work or a single trial that has not yet been repeated.
- Mixed
- Good studies disagree, or the effect appears in some groups and not others. Honest answer: it depends, and we say on what.
- Insufficient
- Too little research in people to say either way. Not the same as evidence of no effect; it means nobody has properly looked yet.
- Not something we offer
- We have looked, and either the evidence is against it or the risk is not justified by what it might do. We say which.
What this changes here
Our reading, not theirs.
You should be able to check our work, and these questions are how. Every claim discussed on this site also carries one of five plain phrases for how strong the evidence behind it is, which is the answer to the first question given in advance. The glossary below covers the words we use, the words research uses, and the words the market uses.
The topics this bears on
The words we use
- Healthspan
How long you remain healthy, capable and independent, as distinct from how long you live.
Why it matters here. It is the thing this practice exists to extend, and the reason we measure function and not only disease.
- Lifespan
How long you live.
Why it matters here. Lifespan has lengthened for decades. Healthspan has not kept pace, and the gap between them is where most of the difficulty of later life sits.
- Functional reserve
The capacity you hold above what daily life demands: strength, balance, stamina, and the ability to recover.
Why it matters here. An illness, a fall or an operation draws reserve down. How much you had beforehand largely decides what happens afterwards.
- Frailty
A state of reduced reserve across several systems at once, so that a small setback has a large effect.
Why it matters here. It is measurable years before it changes what someone can do, and much of it is preventable.
- Cognitive reserve
The brain's ability to keep working well despite age or damage, built up over a lifetime of education, work, activity and social contact.
Why it matters here. It is why two people with similar changes on a brain scan can function very differently.
- VO₂ max
The most oxygen your body can use during hard effort; a measure of cardiorespiratory fitness.
Why it matters here. It is one of the better predictors of how long and how well people live, and it can be trained at any age.
- Polypharmacy
Taking several medicines at once, usually five or more.
Why it matters here. Common past sixty-five, rarely reviewed as a whole, and a frequent cause of falls, confusion and fatigue that gets blamed on age.
The words research uses
- Randomized controlled trial
People are assigned by chance to receive a treatment or a comparison, so that the groups differ only in the treatment.
Why it matters here. The most reliable way to find out whether something works. Most claims in this field have never been through one.
- Cohort study
A group of people is followed over time and their habits or exposures compared with what happens to them.
Why it matters here. Good for spotting associations over decades; weak at proving cause, because people who do one healthy thing tend to do several.
- Systematic review and meta-analysis
A structured search for every study on a question, with the results combined where that is sensible.
Why it matters here. This is where our evidence briefs start. One study is a data point; a review is a body of evidence.
- Absolute and relative risk
Relative risk is the percentage change; absolute risk is how many people out of a hundred are actually affected.
Why it matters here. A 50 percent reduction sounds large and may mean two people in a hundred instead of four. We always give the absolute figure.
- Effect size
How big a difference the treatment made, not only whether it made one.
Why it matters here. A result can be statistically real and too small to matter to anyone.
- Statistical versus clinical significance
Statistical significance means the result is unlikely to be chance. Clinical significance means it is big enough to matter to a person.
Why it matters here. Headlines report the first. Decisions should rest on the second.
- Surrogate endpoint
A measurement that stands in for the outcome you care about: a blood marker instead of a heart attack, a scan instead of memory.
Why it matters here. Useful and often misleading. Several large trials have moved the marker and not the outcome.
- Preprint
A study published before other scientists have reviewed it.
Why it matters here. Fast, and unfiltered. We read them; we do not change practice on them.
- Conflict of interest
When the people running or reporting a study stand to gain from a particular result.
Why it matters here. Not a reason to dismiss a study, but a reason to read the methods more carefully and to look for independent confirmation.
The words the market uses
- Biomarker
Anything measurable in the body that indicates a state or a risk: blood pressure, cholesterol, a gene, a scan.
Why it matters here. Everything we track is a biomarker. The question is always whether changing it changes what happens to you.
- Biological age
An estimate of how old the body is, usually from patterns of DNA methylation, as against age in years.
Why it matters here. A real research tool; not yet reliable enough in one person to guide a decision. See Where we stand.
- Hallmarks of ageing
A list of the cellular processes that change with age, twelve in the current version, from DNA damage to chronic inflammation.
Why it matters here. A useful map of the biology, and the source of most of the mechanisms marketed as longevity treatments. A mechanism is not a treatment.
- Longevity medicine
Care aimed at extending healthy life, with a wide range of evidence behind the things offered under the name.
Why it matters here. Some of it is prevention with strong evidence. Some of it is not. The strength label is how we tell you which.
Where to go next
The other work we read