When the model becomes the measurement instrument

#Question

What changes when artificial intelligence is used to construct the epidemiological variables that public health later analyses?

#Answer in brief

The model stops being only an analytical tool and becomes part of the measurement system.

If an AI system extracts vaccination status from free text, classifies a death certificate, adjudicates whether a notification meets a case definition, derives a phenotype from an electronic health record, or infers an exposure from images or sensor data, its output becomes an epidemiological variable.

At that point, false positives and false negatives are not merely model-performance errors. They are misclassification. If error differs by subgroup, setting, time or outcome, it can become differential measurement error. If the model changes version during surveillance, the measurement process itself changes over time.

Classical epidemiology already has a theory for the consequences of measurement error and misclassification.^[1] The new problem is that AI can make the measurement instrument adaptive, context-sensitive and versioned.

This article proposes treating AI-derived variables as measurements whose validation must follow the epidemiological role they will play.

#Prediction and measurement are not the same job

A model can be excellent at prediction while still being unsuitable as a measurement instrument for a particular epidemiological question.

Suppose an LLM classifies clinical notes into disease phenotypes.

For one task, the objective may be to identify patients who probably have the condition so that a clinician can review them. A high-recall triage model may be useful even if some cases are false positives.

For another task, the same model output may define the outcome in a cohort study.

Now every false positive and false negative contributes directly to the estimated disease frequency and potentially to an exposure–outcome association.

The acceptable error structure changes with the use.

Recent studies show that LLMs can perform complex phenotyping tasks from clinical text and digital data, making this no longer a hypothetical measurement pathway.^[2][3]

The epidemiological question is therefore not only:

How accurate is the model?

It is:

What variable is the model creating, for whom, under which conditions, and what happens to the target estimate when that variable is wrong?

#AI can become part of the case definition

Public-health data systems often transform observations into operational categories.

A suspected event becomes a probable case. A narrative becomes a cause-of-death code. A clinical note becomes a phenotype. A free-text reason for attendance becomes a syndromic category.

Historically, these transformations have been performed through explicit rules, trained coders, laboratory criteria, clinical review or combinations of them.

AI can now sit inside the same chain.

That does not make the category invalid. It changes what must be documented.

A model-mediated case definition contains at least four layers:

1. the substantive definition of the event;
2. the source data available to the model;
3. the model and inference procedure translating source data into a label;
4. the threshold or decision rule turning model output into the analytical variable.

If any layer changes, the measured variable can change even when the underlying population does not.

#Model error maps onto epidemiological misclassification

Consider a binary AI-derived outcome.

Let the underlying condition be present or absent, and let the AI classify it positive or negative.

The familiar quantities sensitivity and specificity now describe part of the measurement process.

But epidemiology needs more than one global sensitivity and specificity.

Error may vary by:

• age;
• language;
• sex or gender;
• socioeconomic position;
• disease severity;
• care setting;
• documentation style;
• hospital;
• device;
• calendar period;
• model version.

A single aggregate F1 score can hide those differences.

Measurement-error theory distinguishes differential from non-differential error because the consequences for inference can differ materially.^[1]

An AI system that performs worse in one population therefore does not merely create a fairness concern. It can alter comparative epidemiological estimates.

#Differential error can distort disparities

Suppose an AI system identifies a condition from clinical notes with 95% sensitivity in one language group and 75% in another.

If those classifications are later used to estimate prevalence, the apparent difference between the groups combines:

• the real difference in disease;
• the difference in documentation;
• the difference in model measurement performance.

The model has entered the disparity estimate.

The same problem can affect exposures and confounders.

If an AI-derived measure of housing instability is more complete for people with rich narrative documentation than for people with sparse records, adjustment may not remove confounding as intended.

The variable exists in the dataset, but its meaning is conditional on the model and the data-generating process.

#Version drift can create artificial epidemiological trends

Traditional measurement instruments can drift.

AI makes drift unusually easy because the instrument can change through:

• model replacement;
• prompt changes;
• retrieval changes;
• threshold changes;
• fine-tuning;
• vendor updates;
• changes in source documentation;
• changes in preprocessing.

Imagine a surveillance programme using an LLM to classify emergency-department narratives.

In March, model version A has moderate sensitivity for a respiratory syndrome. In April, the provider deploys version B, which recognizes more linguistic variants.

Observed syndrome counts rise.

That rise could reflect a real epidemiological increase.

It could also reflect a more sensitive measurement instrument.

Without versioned measurement provenance, those two explanations become difficult to separate.

A time series should therefore record not only the case count but the state of the measurement system that produced it.

#Validation should follow the variable's epidemiological role

A useful validation design begins with the role of the AI-derived variable.

#If AI defines an outcome

Validate outcome misclassification and examine whether error varies by exposure, subgroup or setting.

#If AI defines an exposure

Examine whether exposure error depends on outcome status or factors related to selection.

#If AI defines eligibility

False inclusion and exclusion can change the study population before analysis begins.

#If AI defines a confounder

Measurement error can leave residual confounding even when the variable is included in the model.

#If AI defines a surveillance case

Assess sensitivity, predictive value, timeliness, representativeness and stability in relation to the decision the surveillance system must support.

The same model may therefore require different evidence for different epidemiological uses.

#A synthetic example

A regional programme uses an LLM to classify infectious-disease notifications from free text into:

• confirmed;
• probable;
• not a case.

The model is validated on 1,000 historical records and shows 92% overall agreement with expert review.

That sounds strong.

But further analysis shows:

• 96% sensitivity in records from tertiary hospitals;
• 78% sensitivity in primary-care notifications;
• lower performance for short notes;
• a substantial performance change after a model update.

If the objective is workload triage, the system may still be operationally useful.

If the objective is to compare incidence between hospital catchment areas, the same error structure could be unacceptable.

The validation question must match the epidemiological claim.

#A proposed validation matrix

For AI-derived epidemiological variables, a minimum review can be organized across four dimensions:

Dimension	Question
Construct	Does the AI output correspond to the epidemiological concept being claimed?
Error	What are sensitivity, specificity, calibration or measurement error, and how do they vary across relevant strata?
Stability	Does performance change across settings, time, source-data quality or model versions?
Inferential consequence	How would plausible error change the target estimate, comparison or decision?

This matrix is a proposed methodological device, not an established reporting standard.

Its purpose is to prevent validation from ending at a global benchmark score.

#Benchmark accuracy is not inferential validity

An AI system can outperform a baseline and still create biased epidemiological estimates.

Benchmark evaluation typically asks whether predictions agree with a reference set.

Epidemiological inference asks what happens when those predictions are used inside a population estimate, causal contrast, surveillance indicator or allocation rule.

Those questions overlap, but they are not identical.

A model with modest error may be acceptable if error is stable and non-differential for the target question.

A model with excellent average accuracy may be dangerous if its errors concentrate exactly where comparisons are being made.

#The reference standard also needs scrutiny

AI validation often treats human labels as ground truth.

But human classification can itself be inconsistent, incomplete or dependent on information unavailable to the model.

The phenotyping literature illustrates this difficulty: expert-labelled clinical tasks may have imperfect inter-rater agreement, so model–human agreement is not the same as agreement with an unobservable biological truth.^[2]

Validation should therefore document:

• who created the reference;
• which information they could access;
• how disagreements were resolved;
• whether the reference is appropriate for the intended epidemiological construct.

A weak gold standard cannot create a strong measurement claim.

#What should be preserved in the data lineage

If an AI-derived variable is retained for analysis, the lineage should identify at least:

• source record or source-data version;
• substantive variable definition;
• model provider and model/version where available;
• prompt or extraction specification where relevant;
• preprocessing;
• threshold or classification rule;
• validation evidence;
• uncertainty or confidence representation where meaningful;
• human review or override;
• date of derivation.

This is not a demand to archive unrestricted sensitive prompts or proprietary model internals.

It is a demand to preserve enough measurement provenance to know what the variable means.

WHO guidance on multimodal AI emphasizes transparency, accountability and appropriate human oversight in health and research contexts.^[4] Epidemiological use adds a specific requirement: the model-mediated measurement must remain interpretable as part of the data-generating process.

#The deeper methodological shift

AI does not eliminate measurement error.

It can industrialize measurement.

That can be enormously useful: thousands of unstructured records can become analysable variables in a way that manual review could never scale.

But scale changes the consequence of error.

A small classification bias repeated across millions of records can create a very precise estimate of the wrong quantity.

The relevant standard is therefore not:

Was the AI good?

It is:

Was the AI-derived measurement adequate for this epidemiological estimand and this public-health decision?

#Limitations and uncertainty

The proposed framing of AI as a measurement instrument is a methodological interpretation, not a new formal error model. Classical measurement-error theory remains the stronger foundation for analysing misclassification and measurement bias.^[1] The AI phenotyping studies cited here demonstrate that model-mediated variable construction is feasible; they do not establish that all such systems produce important epidemiological bias.^[2][3] Error structures will depend on task, population, model, source data and validation design. The proposed validation matrix requires empirical testing before it should be treated as a standard.