3.D.5
Evaluating prevention outcomes
ASPPB's competency statement 3.D.5: Evaluate prevention strategy outcomes
Prevention succeeds when fewer people develop the problem than would have otherwise, which is harder to show than a change in scores. A sound evaluation compares participants with a credible comparison group, preferably randomized, measures new cases as well as risk factors, follows people long enough for onset to occur, and reports effects in absolute terms. Reach and implementation decide how much of a trial's effect a community actually gains.
Key ideas
- Incidence versus risk factors
- Reducing risk factors is not the same as preventing disorders: in one review, 51% of eating disorder prevention programs reduced risk factors but only 29% reduced current or future eating pathology (Stice, Shaw and Marti, 2007).
- Credible comparison
- Without a comparison group, a drop may reflect maturation or wider trends (Shadish and colleagues, 2002). Prevention science's standards of evidence set requirements for efficacy, effectiveness and scale-up claims (Gottfredson and colleagues, 2015).
- Absolute and relative effects
- A relative risk reduction can sound large when the outcome is uncommon. The number needed to treat, the inverse of the absolute risk reduction, conveys clinical meaning (Cook and Sackett, 1995).
- Length of follow-up
- New cases take time to appear, and trials with shorter follow-up periods produced larger effects in eating disorder prevention (Stice, Shaw and Marti, 2007), so short follow-ups can overstate lasting benefit.
- Public health impact
- Impact depends on reach, efficacy, adoption, implementation and maintenance together, at both individual and organizational levels (Glasgow, Vogt and Boles, 1999).
Worked example
A college program to prevent first episodes of depression among students with elevated symptoms says it 'cut the risk in half.' In its randomized evaluation, 6 percent of participants and 12 percent of controls had a new depressive episode, diagnosed by interview, over one year.
- Check the design: random assignment and interview diagnosis of new episodes support a causal claim about incidence, not just about symptom scores.
- Convert the figures: the relative risk reduction is 50 percent, but the absolute risk reduction is 6 percentage points, so the number needed to treat is about 17 (1 divided by 0.06).
- Ask about time and reach: one year shows early effects, longer follow-up would show whether they last, and the share of eligible students who enrolled shows how much of the effect reaches the campus.
- Report both the relative and absolute figures to decision makers, with the cost per case prevented when costs are known.
Translate prevention results into absolute terms and judge them by design, follow-up and reach.
Check yourself
In a randomized prevention trial, 4 percent of participants and 8 percent of controls develop the disorder. What is the number needed to treat to prevent one case?
Sources
- Stice, Shaw and Marti (2007), A meta-analytic review of eating disorder prevention programs: encouraging findings, Annual Review of Clinical Psychology (PMID 17716054), Abstract; risk factors versus pathology, and follow-up length
- Gottfredson, Cook, Gardner and colleagues (2015), Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation, Prevention Science (PMID 25846268), Abstract; the revised standards
- Cook and Sackett (1995), The number needed to treat: a clinically useful measure of treatment effect, BMJ (PMID 7873954), Abstract
- Glasgow, Vogt and Boles (1999), Evaluating the public health impact of health promotion interventions: the RE-AIM framework, American Journal of Public Health (PMID 10474547), Abstract
- Shadish, Cook and Campbell (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Chapter 2, threats to internal validity