A completed programme is not an effective programme

#Question

What evidence separates programme delivery from programme effectiveness?

#Answer in brief

A programme can complete every planned activity and still fail to improve health. Process or implementation evaluation asks how the programme was delivered relative to its intended theory of change. Outcome evaluation asks whether intended outcomes occurred but, as the 2024 CDC framework states explicitly, does not by itself attribute causality. Impact evaluation goes further by comparing outcomes with and without the programme and asks whether a causal relationship can be established.^[1]

The distinction is not terminology. It determines what conclusions the evidence can support.

#Four statements that look similar but are not

Consider a synthetic prevention programme.

An activity statement says twelve community sessions were delivered. An output statement says eight hundred eligible people attended at least one session. An outcome statement says the proportion completing screening increased from 45% to 60%. An impact statement says the programme caused that increase.

The first three do not logically establish the fourth.

#Why completion is seductive

Activities are usually the easiest part of a programme to count. Budgets, contracts and work plans are often organized around sessions, visits, reports, doses, inspections, trainings, messages and referrals.

A programme can therefore appear successful because it completed its implementation plan.

But public-health value lies in what those activities change, not in their existence. CDC distinguishes outputs—the products of activities—from outcomes—the intended programme effects.^[1]

#Process evaluation answers a necessary question

If outcomes fail to improve, the first question is often whether the intervention was actually delivered as intended.

Process evaluation can examine implementation, mechanisms of impact, and context; within that structure, questions about reach, fidelity, adaptations and participant exposure help explain how delivery differed from the intervention as intended.^[2]

This prevents an important interpretive error. If a programme was never implemented adequately, the evaluation has not tested the intended intervention in the form described. Conversely, if implementation was strong and outcomes did not change, the programme theory itself may need revision.

#Fidelity is not obedience

Implementation fidelity is sometimes misunderstood as never adapting the intervention.

Complex programmes often require adaptation.

The more useful question is whether changes preserved or altered the functions believed to produce the outcome.

MRC guidance emphasizes the interaction of intervention, context and systems and encourages attention to adaptation and implementation rather than assuming a fixed intervention delivered identically everywhere.^[3]

A programme can therefore be locally adapted and still be faithful to its core mechanism—or delivered exactly according to a manual while failing because the manual does not fit the context.

#Outcome evaluation does not identify the cause

Suppose vaccination coverage rises after a catch-up programme.

Possible contributors include the programme, a concurrent national campaign, a denominator revision, seasonal service expansion, improved registry completeness, changes in vaccine availability, or public concern following an outbreak.

An outcome evaluation can document the rise.

It cannot automatically attribute the rise to the programme.

CDC's 2024 definition is explicit: outcome evaluation measures achievement of intended outcomes but cannot attribute causality.^[1]

#Impact evaluation needs a counterfactual

A causal effect is defined relative to what would have happened without the programme.

That alternative cannot be observed for the same population at the same time.

Evaluation therefore uses designs that approximate a counterfactual, including randomized assignment, controlled before-after designs, interrupted time series, difference-in-differences, regression discontinuity and other quasi-experimental approaches.

The appropriate design depends on the intervention, implementation, timing and assumptions.

A before-after comparison is not automatically an impact evaluation simply because it includes an after period.

#Reach can improve while equity worsens

Imagine an intervention that increases participation from 50% to 70%.

Overall performance looks good.

Stratified analysis shows affluent areas moving from 60% to 90%, while disadvantaged areas move from 40% to 45%.

The programme increased overall reach while widening the gap.

CDC's updated framework places equity among cross-cutting evaluation actions and encourages questions about for whom programmes are more or less effective.^[1]

An average can conceal distributional failure.

#Unintended effects count

A programme can achieve its target while producing harm elsewhere.

Examples include appointment reminders increasing penalties for people with unstable transport, intensified reporting increasing administrative burden, screening generating follow-up demand the system cannot meet, or a communication campaign increasing anxiety or stigma.

A serious evaluation asks about unintended positive and negative effects rather than treating the intended outcome as the only relevant endpoint.

#A synthetic vaccination recovery evaluation

A region runs a six-month recovery programme for overdue adolescent vaccination.

Reported results include 50,000 reminder messages, 12,000 appointments booked, 9,500 vaccinations administered and a six-point increase in final administrative coverage.

A superficial conclusion is that the programme increased coverage by six points.

A stronger evaluation separates process, outcome, alternative explanations, impact, equity and sustainability.

Were overdue adolescents correctly identified? Did vaccination timeliness improve? Was the denominator revised? Did another campaign overlap? What change would likely have occurred without the programme? Did recovery occur across all groups? Did the intervention reduce the creation of new overdue cases or only clear an old backlog?

The programme can be operationally valuable even if the causal impact estimate remains uncertain. That uncertainty should be stated rather than replaced with a stronger claim.

#Stage of development should shape the evaluation

A new programme may need to ask whether it is feasible, acceptable, implementable and mechanistically plausible.

A mature programme may be ready to ask what outcomes occur, for whom, at what cost and compared with what alternative.

CDC's framework explicitly links evaluation questions to programme stage.^[1]

Demanding a definitive impact estimate from a pilot can be as unhelpful as evaluating a mature national programme only by counting activities.

#A practical evidence ladder

A useful claim discipline separates activity, output, implementation, outcome and causal impact.

Activity proves that something was done. Output shows that something was delivered. Implementation describes reach, fidelity, quality and adaptation. Outcome shows that an intended state changed. Causal impact requires evidence that the programme contributed to the change relative to a credible alternative.

The ladder is not a universal scoring system. It is a way to keep conclusions aligned with evidence.

#How to write conclusions that match the evidence

Prefer statements such as: the programme delivered 80% of planned sessions and reached 65% of the eligible population; screening completion increased during implementation; or an interrupted time-series model detected a level change under stated assumptions.

Avoid simply saying the programme was effective unless the evaluation design and evidence support that conclusion.

#The relationship to intervention design

Evaluation cannot rescue an incoherent programme theory.

The companion guide Design the intervention before you design the indicator explains why programme mechanism and implementation conditions should be explicit before indicators are selected.

Evaluation then tests that model rather than merely describing activity.

#Implementation failure and theory failure require different responses

When a programme does not improve outcomes, the next action depends on what failed.

If implementation was weak, the intervention may still be plausible but needs changes in delivery, staffing, reach or adaptation.

If implementation was strong but the expected mechanism did not produce change, repeating the same programme more faithfully may not help.

This distinction is why process data are essential to outcome interpretation: process evaluation can help distinguish delivery problems from failures in the assumed mechanism, without itself turning the outcome analysis into a causal design.^[2]

Without them, a null result cannot easily distinguish "the programme was not delivered" from "the programme theory was wrong."

#Contribution is not the same as sole causation

Public-health outcomes often arise from multiple programmes and contextual changes.

An evaluation may support a conclusion that a programme contributed to change without claiming it was the only cause.

The wording should match the design.

A strong quasi-experimental estimate may support a causal effect under explicit assumptions. A mixed-method contribution analysis may support a more bounded conclusion about how the programme plausibly contributed within a system.

Both are more informative than treating "effective" as an undifferentiated label.

#Sustainability is an outcome domain

A short implementation burst can produce temporary improvement.

A mature evaluation should ask whether the programme can maintain its mechanism under routine staffing, budgets and demand.

For a catch-up programme, that can mean distinguishing a one-time reduction in backlog from a durable improvement in the processes that prevent new delays.

Sustainability therefore belongs in the programme theory rather than being left as an afterthought after a positive outcome.

#Limitations and uncertainty

Evaluation terminology varies across organizations and disciplines. This article follows the distinctions in the 2024 CDC framework.^[1] Complex programmes can produce heterogeneous effects and interact with context, making a single causal estimate an incomplete description even when valid. MRC guidance emphasizes these system interactions and the iterative nature of development and evaluation.^[3] The examples are synthetic and not assessments of actual public-health programmes.