The observer changes the epidemic

#Question

How should epidemiology evaluate an AI system when its own predictions change the population it is supposed to observe?

#Answer in brief

By treating prediction, intervention and subsequent measurement as one causal system.

If an AI system predicts where risk is high and that prediction causes more testing, more outreach, more vaccination or different allocation of resources, the next round of data partly reflects the action triggered by the model.

The model has not only described the distribution. It has helped change it.

The machine-learning literature calls this performative prediction: predictions influence the outcomes or distributions later observed.^[1] Clinical research has documented the same class of feedback problem in deployed prognostic models.^[2][3]

This article uses performative epidemiology as a public-health framing of that problem.

#The usual surveillance diagram is incomplete

A familiar sequence is:

population
→ surveillance data
→ analysis
→ decision
→ intervention

But surveillance repeats.

The next cycle is:

population_t
→ data_t
→ model_t
→ action_t
→ population_t+1
→ data_t+1

If action_t depends on model_t, then data_t+1 is partly downstream of an earlier prediction.

That matters because the future training and evaluation data are no longer independent of model deployment.

#A synthetic example

Suppose an AI system identifies neighbourhoods at elevated risk of respiratory infection.

Teams send additional testing capacity to those neighbourhoods.

Testing rises.

More infections are detected.

The next dataset contains more confirmed cases from the areas the model highlighted.

At least three explanations now coexist:

1. underlying transmission was higher;
2. targeted testing increased ascertainment;
3. both occurred.

If the system records cases but not the model-triggered testing intensity, retraining can treat intervention-generated ascertainment as if it were unchanged disease risk.

The observation process has become part of the intervention.

#Positive feedback can look like confirmation

A model can create a positive feedback loop when a high-risk prediction increases observation in a way that strengthens the signal used to justify the original classification.

Possible public-health examples include:

• more testing in predicted hotspots;
• more case finding in predicted high-risk populations;
• more inspection of predicted high-risk facilities;
• more diagnostic follow-up after algorithmic flagging.

This does not prove the initial prediction was wrong.

It means later data cannot be interpreted without knowing that observation intensity changed.

#Negative feedback can make success look like error

The reverse also occurs.

A model predicts high hospitalization risk.

A programme intervenes successfully.

Hospitalizations fall.

A naive audit compares the original risk estimate with the observed outcome and concludes that the model overpredicted.

But the observed outcome is lower partly because the prediction triggered action.

Clinical monitoring research has shown that such feedback can distort post-deployment performance estimates and retraining decisions.^[3]

In a successful intervention system, prediction error and intervention benefit can become statistically entangled.

#The missing variable is often the action

Many machine-learning evaluation datasets preserve:

• features;
• model output;
• final outcome.

For performative systems that is incomplete.

A minimal epidemiological record should preserve:

model version
prediction
human or automated decision
action taken
action intensity
observation intensity
outcome
policy version

The action is not administrative metadata.

It is part of the causal pathway connecting prediction to future data.

#Retraining can learn policy effects

Repeated retraining on post-deployment data creates a second problem.

Suppose a variable strongly predicts an adverse outcome before deployment.

The system then targets people with that variable for preventive intervention.

The association weakens.

A retrained model may conclude that the predictor is less important.

But the association weakened partly because the system acted on it.

Performative-prediction theory formalizes the broader principle that deployment can change the distribution on which future models are trained.^[1]

Public-health interventions make that distribution shift causally meaningful.

#Drift needs a causal diagnosis

After deployment, analysts often monitor:

• calibration;
• discrimination;
• feature distributions;
• prevalence;
• subgroup performance.

A change may reflect:

• a real change in the population;
• a change in measurement;
• a change in practice;
• a change in model version;
• a change caused by the model-mediated intervention itself.

Those mechanisms require different responses.

Automatic retraining can erase evidence about which mechanism occurred.

#A proposed performative epidemiology record

For AI systems that influence action, six elements should be reviewable:

#Prediction

What did the model output?

#Decision

Was the output accepted, overridden, ignored or escalated?

#Intervention

What action actually occurred?

#Intensity

How much testing, outreach, treatment or resource allocation followed?

#Observation

Did the intervention change who became visible to surveillance?

#Outcome

What happened after action?

This is a proposed causal bookkeeping device, not an established reporting standard.

Its purpose is to keep the intervention pathway visible during evaluation and retraining.

#Evaluation needs more than model accuracy

A post-deployment evaluation should separate at least three questions.

#Predictive validity

Did the model identify risk before the intervention?

#Intervention effect

What changed because action followed the prediction?

#System effect

How did deployment change future observation, behaviour and model performance?

Those are different estimands.

One accuracy score cannot answer all three.

#Some surveillance must remain independent of the model

A system that always sends observation resources to places it already believes are high risk can learn progressively more about those places and less about everywhere else.

That creates an exploration problem.

Where ethical and feasible, evaluation may need model-independent information from:

• sentinel sites;
• staggered rollout;
• random or quasi-random verification samples;
• independent surveillance streams;
• natural experiments;
• sensitivity analyses.

The objective is not to withhold useful interventions.

It is to retain some route for distinguishing population change from model-induced observation change.

#Equity can become performative

If one population receives more algorithm-triggered surveillance, more events may be detected there.

The data can then justify continued intensive surveillance.

Another population with less observation generates fewer recorded events.

The apparent difference may partly reflect unequal visibility rather than only unequal underlying risk.

This is not merely prediction bias.

It is policy-mediated data generation.

Equity review should therefore ask whether AI deployment changes the probability that disease, exposure or need becomes observable.

#Causal memory should survive retraining

Historical labels should ideally retain whether they were observed:

• before model deployment;
• after deployment but before intervention;
• after model-triggered intervention;
• under a different policy or threshold.

Without that memory, identical labels can represent different data-generating processes.

Retraining should not turn intervention effects into apparently neutral training data.

#The deeper shift

Epidemiology studies how exposures and interventions affect outcomes.

AI-mediated public health adds a new object:

a prediction that becomes part of the exposure environment.

A model output can influence who is tested, contacted, treated, vaccinated or monitored.

The model is therefore no longer external to the causal system being studied.

It becomes one of its components.

That is why performative epidemiology is not only an AI-governance problem.

It is a causal-inference problem.

#Practical questions

Before deploying an AI system that can influence public-health action, ask:

• Can the prediction change who gets observed?
• Can it change who receives an intervention?
• Will that action affect the outcome later used to judge the model?
• Are prediction, decision and action logged separately?
• Can observation intensity be quantified?
• Is there any model-independent surveillance signal?
• What will retraining treat as ground truth?
• Could successful intervention look like prediction failure?
• Could intensified ascertainment look like worsening disease?

If these questions are unanswered, conventional monitoring may be inadequate.

#Limitations and uncertainty

Performative epidemiology is proposed here as a public-health framing of an established performative-prediction problem, not as a claim that feedback effects are newly discovered.^[1] The clinical literature demonstrates that model-mediated interventions can complicate monitoring, but public-health systems differ from clinical prediction systems in scale, authority, intervention pathways and observation processes.^[2][3] Model-independent comparison streams or experimental designs may be infeasible or unethical in some settings. The proposed record requires empirical testing before it should be treated as a standard.