How to evaluate whether a surveillance system is useful

#Question

How do you judge whether a public-health surveillance system is actually doing its job?

#Answer in brief

Begin with the system's purpose and intended decisions, not with a generic quality score. Then assess a named set of attributes — sensitivity, timeliness, representativeness, data quality, acceptability, simplicity, flexibility, stability, and positive predictive value among them — and examine whether information is actually used for action. The CDC, ECDC, and WHO frameworks all treat surveillance evaluation as contextual: a system is useful when it produces information of adequate quality and speed for its stated public-health objectives, within realistic operational constraints.^[1][2][3]

A system can perform strongly on one attribute and poorly on another. Those trade-offs are often structural rather than accidental, so the evaluation must make them visible instead of collapsing them into one reassuring number.

#Key points

• Purpose comes first. The same architecture may be adequate for annual burden estimation and inadequate for early outbreak detection.
• Usefulness is multidimensional. Sensitivity, timeliness, representativeness, data quality, acceptability, simplicity, flexibility, stability, and predictive value describe different failure modes.^[1]
• Data quality and system quality are not synonyms. A perfectly completed record can still arrive too late, cover the wrong population, or never reach the people expected to act on it. ECDC therefore treats data-quality monitoring and system evaluation as connected but distinct activities.^[2]
• Surveillance and response should be linked. WHO's framework emphasizes that information should be usable at the levels where detection, investigation, and response occur, not only reported upward for accountability.^[3]
• An evaluation should end with priorities. The useful output is not a ceremonial rating; it is a defensible account of which limitation matters most for the next decision and what evidence would show improvement.

#Scope and exclusions

This guide covers evaluation of an existing or proposed public-health surveillance system at a conceptual and programme level. It does not provide a jurisdiction-specific audit instrument, legal interpretation of reporting obligations, statistical specifications for automated outbreak-detection algorithms, or disease-specific case definitions. Current disease-specific WHO, ECDC, national, or regional guidance should be used where it exists.

#Start with a decision map

Before calculating an indicator, write down:

1. Who is expected to use the information? Local investigators, regional programme managers, national authorities, laboratories, clinicians, or the public may need different outputs.
2. What decision should the information support? Detect a rare event, estimate burden, allocate resources, monitor equity, evaluate an intervention, or demonstrate compliance.
3. How quickly must the decision be made? Hours, days, months, and annual planning cycles imply very different timeliness requirements.
4. What error is more harmful? Missing a true event, investigating a false signal, misrepresenting a subgroup, or acting on unstable preliminary data.
5. What operating conditions must the system survive? Routine workload, an outbreak surge, staff turnover, laboratory disruption, or changes in reporting requirements.

This decision map prevents the common mistake of evaluating an abstract "surveillance system" without defining what success means.

#Core attributes

Adapted from the CDC and ECDC guidance:^[1][2]

• Sensitivity — the system's ability to identify the cases, events, or signals it is intended to detect.
• Positive predictive value — among records or alerts classified as cases or signals, the proportion that meet the intended definition after verification.
• Timeliness — the intervals from event onset through detection, notification, verification, analysis, communication, and action.
• Representativeness — whether the data adequately describe the relevant distribution across people, place, time, and other dimensions of interest.
• Data quality — completeness, validity, consistency, and interpretability of the information actually recorded.
• Acceptability — willingness and ability of reporters, laboratories, institutions, and users to participate as intended.
• Simplicity — operational and cognitive burden across reporting, processing, analysis, and use.
• Flexibility — ability to accommodate changes in definitions, threats, technologies, or information needs without disproportionate disruption.
• Stability — reliability, availability, and resilience of the people, processes, infrastructure, and funding on which the system depends.
• Use and usefulness — whether information reaches the relevant users and contributes to prevention, control, planning, evaluation, or accountability.

The precise definitions and measurements should be adapted to the system's objectives rather than copied mechanically.

#Trade-offs that an evaluation must expose

#Sensitivity versus investigation burden

Lowering an alert threshold can increase sensitivity while decreasing positive predictive value. That may be appropriate for a rare event with severe consequences, but harmful in a routine programme where false signals consume limited investigation capacity. The evaluation should estimate both sides of the trade-off and identify who absorbs the additional work.

#Timeliness versus verification

Rapid provisional reporting may support early action but include records that are later reclassified. Extensive verification can improve record accuracy while making the information operationally irrelevant. A mature system may need explicitly labelled preliminary and validated outputs rather than one undifferentiated dataset.

#Completeness versus representativeness

High completeness among participating facilities does not show that the participating facilities represent the jurisdiction. Coverage gaps can remain invisible if the evaluation looks only at fields within received records.

#Simplicity versus analytical ambition

Adding variables can answer more questions but increase reporting burden, missingness, and delay. Every field should have a defined use; a variable that is never analysed or acted upon imposes cost without demonstrated value.

#Worked example: regional foodborne-disease surveillance

A regional authority wants to know whether its surveillance system is useful for detecting and investigating clusters of foodborne illness.

#Step 1 — Define the purpose

The immediate purpose is not to estimate every sporadic infection. It is to identify possible clusters early enough for epidemiological, laboratory, and environmental teams to investigate and control a shared source.

#Step 2 — Map the pathway

symptom onset
→ care or laboratory contact
→ test and result
→ notification
→ case-definition assessment
→ cluster detection
→ investigation decision
→ control action

A single headline measure such as "time to notification" cannot locate delays across this chain.

#Step 3 — Select measurements

• Sensitivity: compare known outbreaks found through complaints, laboratory typing, or environmental investigation with clusters detected by the routine system.
• Timeliness: measure separate intervals from onset to specimen, specimen to result, result to notification, notification to cluster assessment, and assessment to investigation.
• Representativeness: compare reporting coverage and case profiles across geography, age, access to testing, facility type, and population groups.
• Data quality: audit exposure histories, onset dates, laboratory details, and linkage identifiers needed for cluster assessment.
• Acceptability: examine reporting burden, failed submissions, laboratory participation, and qualitative feedback from users.
• Stability: test whether staffing, data exchange, and analytical capacity remain functional during a surge.
• Usefulness: document decisions influenced by the system, including cases where information correctly prevented an unnecessary investigation.

#Step 4 — Interpret jointly

Suppose notification is fast but exposure fields are often missing, laboratory results cannot be linked reliably, and rural facilities are underrepresented. The system should not be labelled simply "timely" or "poor quality." It is fast at receiving incomplete, unevenly distributed records and may therefore be useful for some alerts but unreliable for characterising source and burden. That diagnosis leads to different improvements than replacing the entire platform.

#Evidence and method

The attribute framework and the purpose-led interpretation follow the CDC's updated evaluation guidance.^[1] The ECDC handbook adds detailed European practice for monitoring data quality and conducting surveillance-system evaluation, developed to improve comparability and daily use across EU/EEA communicable-disease systems.^[2] The WHO framework emphasizes the connection between surveillance, response, local use of information, and accountability across system levels.^[3]

These sources are complementary rather than interchangeable. None supplies one universal score or threshold for every system. The worked example is synthetic and demonstrates how to apply the frameworks; it is not an evaluation of a real authority or jurisdiction.

#Practical implications

A useful evaluation report should state:

• the system's purpose and users;
• the pathway from event to action;
• evidence for each material attribute;
• uncertainties and data gaps;
• trade-offs between attributes;
• priority improvements, owners, and measurable follow-up indicators;
• functions that should be retained because they already work.

Before commissioning a replacement, identify the failure precisely. A system that is slow but complete may be adequate for annual trend analysis and inadequate for outbreak response. A technically modern system may still be unacceptable to reporters or unrepresentative of underserved populations. The intervention should follow the diagnosed limitation, not the prestige of a new technology.

#Limitations and uncertainty

The principal general frameworks cited here are established but not new. Their continued value lies in the durability of the evaluation questions, while current disease-specific, jurisdictional, legal, interoperability, and equity requirements may add attributes or change acceptable thresholds. Evaluation also depends on the quality of the evidence collected: a neatly completed checklist does not compensate for missing users, untested assumptions, or indicators selected only because they are easy to measure.

#Connection to Weavidence

Weavidence Lab's intended separation of data validation, computation, interpretation, and review reflects one lesson of surveillance evaluation: record completeness, analytical output, and decision usefulness are different claims. Keeping their evidence and limitations visible can support evaluation; it does not itself prove that a surveillance system or analytical conclusion is fit for purpose.