When surveillance fails the decision

#Question

How can a surveillance system fail even when its records look accurate?

#Answer in brief

A surveillance system is not a database with a public-health label. It is an information system built to support decisions. A record can be complete, valid and correctly coded yet still be useless if it arrives after the decision window, systematically misses the population in which risk is changing, produces too many false signals for the available investigation capacity, or never reaches the people who can act. CDC guidance has long defined usefulness in relation to prevention, control and decisions, while recent WHO guidance makes the same logic explicit by starting surveillance design from the decisions that need to be made.^[1][2]

The practical question is therefore not only "Are the data good?" It is: good enough for which decision, at what time, and under what operating constraints?

#A synthetic case

Consider a synthetic regional system monitoring a severe respiratory syndrome. It receives daily records from emergency departments, laboratory feeds and mandatory notifications. The technical dashboard is stable. Most required fields are complete. Duplicate records are uncommon.

During a winter surge, however, local public-health teams discover four different failure modes.

#Failure 1: the signal is accurate but late

Laboratory-confirmed cases are coded correctly, but a batch interface delays some results by five days. For retrospective weekly burden estimates this delay may be tolerable. For deciding whether to activate additional testing capacity, it may make the signal operationally obsolete.

Timeliness is therefore not one number. It is a chain of intervals: onset to care, care to sampling, sampling to result, result to notification, notification to analysis, analysis to decision, and decision to action. CDC treats timeliness as a core system attribute because improved speed can allow earlier control measures.^[1]

#Failure 2: the signal is timely but selective

The emergency-department feed arrives within hours. Yet people with mild illness increasingly use primary care, telemedicine, or no care at all. The ED trend is real among ED users but no longer maps cleanly to incidence in the underlying population.

This is a representativeness problem, not a data-entry problem. ECDC notes that representativeness can shift as health-care access, clinical practice, reporting infrastructure and participating settings change.^[3]

#Failure 3: the signal is sensitive but overwhelms investigation

An automated rule is deliberately broad so that few clusters are missed. The result is a stream of alerts, most of which do not represent actionable outbreaks. The algorithm is doing what it was configured to do, but the investigation team cannot review every signal quickly.

CDC explicitly describes the interdependence of sensitivity, predictive value, complexity, timeliness and cost. Raising sensitivity can reduce positive predictive value and increase investigation burden.^[1] A technically impressive detector can therefore make the surveillance-to-action system worse if it saturates the human response layer.

#Failure 4: the information is good but has no decision owner

The weekly report accurately documents a rise in activity. It is emailed to several units, but no one is explicitly responsible for deciding whether the rise should trigger additional sampling, communication, or operational changes.

The defect sits downstream of analysis. WHO's multisource collaborative surveillance approach starts by identifying decision-making questions and then works backward to the information required for those decisions.^[2] If no decision, threshold or accountable user is defined, producing a better chart does not repair the system.

#Four failure modes, four different remedies

The synthetic system should not receive one overall label such as "high quality" or "poor quality." Each failure implies a different intervention.

Failure mode	What is wrong	Plausible remedy
Late but accurate	Information misses the decision window	Separate provisional from validated feeds; reduce bottleneck intervals
Timely but selective	Observed users no longer represent the target population	Combine sources; quantify coverage and changing care-seeking
Sensitive but noisy	False signals exceed response capacity	Recalibrate thresholds; tier alerts; measure investigation yield
Informative but ownerless	Analysis is disconnected from action	Define decision rights, triggers and feedback loops

This is why surveillance evaluation is better understood as fitness for decision than as a checklist score.

#What should be measured

A decision-centred evaluation can still use conventional attributes, but it should connect each one to a failure consequence.

• Sensitivity: what material events could the system miss?
• Positive predictive value: what does acting on false signals cost?
• Timeliness: which interval is actually limiting action?
• Representativeness: who or what is systematically under-observed?
• Data quality: which missing or invalid fields change interpretation?
• Acceptability: where does reporting burden reduce participation?
• Stability: does the system remain functional during the period when it is most needed?
• Flexibility: can definitions and flows change without losing continuity?
• Usefulness: what decisions have actually been informed?

CDC and ECDC frameworks remain useful precisely because they describe distinct dimensions rather than one synthetic quality score.^[1][3]

#The decision clock

One practical way to make the evaluation concrete is to draw a decision clock.

Start with the action deadline: when would an intervention still change the outcome? Then work backward.

last useful moment for action
← decision
← interpretation
← analysis
← validation
← notification
← clinical/laboratory event

Every stage consumes time. Improving a segment that is not on the critical path may leave decision usefulness unchanged.

The same logic applies to spatial and population coverage. Begin with the population about which the decision is made, then work backward to the data-generating process. If the dataset includes only people who crossed a particular health-care threshold, the system needs either an explicit interpretation boundary or additional sources.

WHO's 2025 multisource framework formalizes this backwards logic by asking first what decisions are required, then what information is necessary, and finally which surveillance approaches can produce it.^[2]

#Why this complements conventional system evaluation

A conventional surveillance evaluation asks whether the system is sensitive, timely, representative, acceptable and stable. That remains necessary.

This worked case asks a different question: how do those attributes fail in relation to a concrete decision?

The distinction matters because two systems with the same measured timeliness can have very different utility. A 72-hour delay may be irrelevant for annual planning and unacceptable for an event requiring same-day action. Likewise, lower sensitivity may be tolerable for estimating stable trends but dangerous for a rare high-consequence event.

The decision is the context that gives an attribute meaning.

#Practical implications

Before redesigning a surveillance platform, document three things:

1. the decisions it must support;
2. the failure that currently prevents those decisions;
3. the observable evidence that would show the failure has improved.

That discipline prevents expensive substitutions in which a new interface replaces an old interface while the real bottleneck—coverage, delay, incentives, case definition, investigation capacity, or decision ownership—remains unchanged.

It also makes trade-offs explicit. If a broader case definition improves sensitivity but doubles investigation workload, that is not automatically a reason to reject it. It is a requirement to decide whether the additional detection is worth the operational cost for the public-health objective.

#Limitations and uncertainty

This is a synthetic case designed to expose failure modes, not an evaluation of a real surveillance authority. The relative importance of timeliness, sensitivity, predictive value, representativeness and other attributes depends on the event, jurisdiction, legal framework, available resources and intended action. The CDC and ECDC frameworks cited here are established rather than recent; current disease-specific guidance and local requirements may add constraints. The WHO 2025 framework is oriented to multisource collaborative surveillance for health threats and should not be treated as a universal operational specification for every routine surveillance programme.^[2]