4.C.1
Running an evaluation: plan, monitor and adapt
Énoncé de compétence 4.C.1 d’ASPPB : Develop, implement, continually monitor, and modify evaluation activities, considering all relevant factors (e.g., stakeholder preferences, logistic limitations, societal context)
An evaluation is planned with the people who will use it, then watched as closely as the program itself. Designs must fit what stakeholders will accept and what is logistically possible, and the societal context shapes who is reached and what counts as success. When problems surface midway, the evaluator revises the plan openly, documents the change and checks whether the problem has biased the results.
Notions clés
- Framework steps
- CDC's 2024 framework sets six steps: assess context, describe the program, focus the evaluation questions and design, gather credible evidence, generate and support conclusions, and act on findings. It treats them as iterative rather than strictly sequential.
- Logic model
- A picture of how a program's inputs and activities are expected to lead to short-term, intermediate and long-term outcomes. It shows what to measure and where an expected link may fail.
- Implementation monitoring
- A review of over 500 studies found that the level of implementation affects the outcomes of prevention and promotion programs, so an evaluation that measures only outcomes cannot tell a weak program from a weakly delivered one.
- Designs that fit constraints
- When withholding a program is not acceptable, a stepped wedge rollout (sites start in turn, so later sites serve as comparisons) or a regression discontinuity around an eligibility cutoff can still supply a credible comparison.
- Equity and context
- CDC's framework adds three actions for every step: engage collaboratively, advance equity, and learn from and use insights. Context includes values, power, history and the economic and political circumstances of the setting.
Exemple commenté
A psychologist evaluates a classroom social skills program in eight elementary schools using monthly classroom observations. By midyear, observation data are missing for most visits at three schools, all in lower-income neighborhoods, where schedule changes and substitute shortages keep cancelling visits.
- Treat the monitoring data as a warning: missing data clustered in particular schools could bias the results toward the better-resourced ones.
- Meet with principals and teachers to learn what is feasible, rather than pressing the original schedule.
- Add a measure every school can provide, such as a brief teacher rating each month, while keeping observations where they still work, and record the change and its date in the evaluation plan.
- In the analysis, compare schools with and without full observation data and report the change and its possible effects openly.
Monitor the evaluation itself, adapt it to real constraints and document every change so readers can judge the results.
Vérifiez vos acquis
Midway through an evaluation, outcome data are missing far more often at the sites that serve low-income families. What is the main concern, and the right response?
Sources
- Kidder and colleagues, CDC Evaluation Framework Work Group (2024), CDC Program Evaluation Framework, 2024, MMWR Recommendations and Reports, 73(6), 1-37 (PMID 39316770), The six steps, iterative and revisited when contextual constraints appear later; the logic model; the three cross-cutting actions; Step 1, Assess Context
- Durlak and DuPre (2008), Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation, American Journal of Community Psychology, 41(3-4), 327-350 (PMID 18322790), Abstract: over 500 studies show the level of implementation affects outcomes; implementation data essential in evaluations
- Moscoe, Bor and Barnighausen (2015), Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: A review of current and best practice, Journal of Clinical Epidemiology, 68(2), 122-133 (PMID 25579639), Abstract: regression discontinuity allows rigorous causal inference when treatment is assigned by scoring above or below a cutoff on a continuous measure
- Hemming, Haines, Chilton, Girling and Lilford (2015), The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting, BMJ, 350, h391 (PMID 25662947), The stepped wedge design, in which clusters cross from control to intervention in turn until all receive it (PubMed holds no abstract; content cited from the article)
- Shadish, Cook and Campbell (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Attrition and missing data as threats to validity; quasi-experimental designs
- Yarbrough, Shulha, Hopson and Caruthers (2011), The Program Evaluation Standards, Third Edition, Feasibility and accuracy standards