3.D.3
Monitoring and improving a prevention program
ASPPB's competency statement 3.D.3: Continually assess effectiveness and, as needed, modify or change prevention strategies, considering all relevant factors (e.g., client and/or community preferences)
A prevention program is not finished when it launches. Its team keeps checking whom it reaches, whether it is delivered as designed, whether it changes what it targets, and what participants and communities think, and then adjusts. Weak results often reflect weak implementation rather than a weak program, so the first question is what was actually delivered and to whom. Some programs can also cause harm, which monitoring should be able to detect.
Key ideas
- Implementation matters
- Across more than 500 studies, the level of implementation affected the outcomes of promotion and prevention programs, and at least 23 contextual factors influenced implementation (Durlak and DuPre, 2008).
- RE-AIM
- Reach, efficacy, adoption, implementation and maintenance, assessed at individual and organizational levels; failing to evaluate all five can waste resources and limit public health impact (Glasgow, Vogt and Boles, 1999).
- Fidelity and adaptation
- Adapting a program to local language and culture can improve fit, but the components that carry its effect should be kept; Durlak and DuPre (2008) treat both fidelity and adaptation as aspects of implementation to measure.
- Unintended harm
- Grouping high-risk adolescents together can increase problem behavior through peer 'deviancy training' (Dishion, McCord and Poulin, 1999), so monitoring should look for worsening as well as benefit.
- Community preferences
- Participants' and partners' views shape uptake and fit. Community-based approaches involve community members in all aspects of the work, including judging what should change (Israel and colleagues, 1998).
Worked example
A county offers a ten-session school group for children aged 8 to 11 who screen high for anxiety. After a year, children who attended most sessions improved more than expected, but few families who speak languages other than English enrolled, and several facilitators skipped the two sessions of graded practice facing feared situations.
- Separate the questions with RE-AIM: effects among attenders look promising, but reach is uneven and implementation is incomplete.
- Ask why: talk with facilitators about what made the practice sessions hard to run, and with families and community organizations about barriers to enrolling.
- Adjust delivery while keeping the core: translate materials, offer interpreters or bilingual facilitators, and coach facilitators to run the practice sessions rather than letting them drop.
- Keep measuring reach by language group, session-by-session fidelity and child outcomes, so the next review shows whether the changes worked.
Before judging a prevention program, find out whom it reached and what was delivered, and fix those first.
Check yourself
A parenting program works well for the families who attend, but only a small share of eligible families enroll, and those who do are mostly from one neighborhood. Which RE-AIM dimension is weakest?
Sources
- Durlak and DuPre (2008), Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation, American Journal of Community Psychology (PMID 18322790), Abstract; aspects of implementation
- Glasgow, Vogt and Boles (1999), Evaluating the public health impact of health promotion interventions: the RE-AIM framework, American Journal of Public Health (PMID 10474547), Abstract; the five dimensions
- Dishion, McCord and Poulin (1999), When interventions harm: Peer groups and problem behavior, American Psychologist (PMID 10510665), Abstract
- Israel, Schulz, Parker and Becker (1998), Review of community-based research: assessing partnership approaches to improve public health, Annual Review of Public Health (PMID 9611617), Abstract; key principles