An analytical audit trail must preserve decisions, not just files

#Question

What information should an analytical audit trail preserve so another reviewer can reconstruct both the result and the reasoning behind it?

#Answer in brief

Saving the final script is not enough. Provenance should connect the entities used and produced, the activities that transformed them, and the agents responsible for those activities. W3C PROV formalizes those concepts because origin, transformation and responsibility are central to understanding and reproducing how an object was generated.^[1]

For analytical work, this article proposes extending that provenance skeleton with decision provenance: why records were excluded, why a model was chosen, which alternatives were considered, what diagnostics changed the plan, and how interpretation evolved. W3C PROV supplies the general entity–activity–agent vocabulary; it does not prescribe this epidemiological decision-audit architecture.^[1]

#Reproducibility needs identity

An analyst receives three files named data-final, analysis-final and figure-final.

Can the analysis be reconstructed?

Not necessarily.

Questions immediately arise: which source produced the data file, whether it is before or after exclusions, which code version created the figure, which parameters were used, whether the script was edited after output generation, whether manual changes were made, and which software environment was used.

File existence is not provenance.

#The W3C provenance skeleton

W3C PROV describes three foundational concepts: entities such as data and outputs, activities that use or generate entities, and agents associated with activities and responsibility.^[1]

For an epidemiological analysis, a raw extract can be an entity, cleaning an activity, the analysis dataset a new entity, model fitting another activity, and the model object or figure further entities.

The provenance graph can also record who or what performed each activity and which version was used.

#Lineage is necessary but not sufficient

Lineage can tell us that Dataset B was derived from Dataset A by Script X.

It may not tell us why 2,300 observations were removed.

The difference is crucial.

Some transformations are purely technical. Others encode scientific judgement: eligibility, episode definition, censoring, outlier treatment, missing-data strategy, model family, exposure window or reference category.

A serious audit trail should distinguish them.

#Preserve exclusions as first-class decisions

An exclusion should record its rule, rationale, stage applied, number affected, whether it was prespecified, and the source supporting it where relevant.

A weak note says invalid records were removed.

A stronger record says 418 records with event date preceding birth date were excluded because the sequence is impossible under the data model, and the rule was applied before outcome modelling.

A mature system can additionally bind the exclusion rule to a stable decision ID, code version, affected-record artifact and reviewer disposition.

The goal is not bureaucratic verbosity. It is reconstructability.

#Parameters need names and provenance

Many analyses depend on thresholds: washout period, allowable gap, minimum event count, date window, matching caliper, convergence tolerance or missingness threshold.

If the number appears only inside code, its scientific meaning can disappear.

An audit trail should link a parameter to definition, value, units, rationale, version and sensitivity alternatives.

Then a reviewer can distinguish a software constant from a scientific assumption.

#Alternatives reveal the decision boundary

Suppose Poisson, negative-binomial and quasi-Poisson models are plausible.

The final report shows only the chosen negative-binomial model.

A good audit trail records why alternatives were plausible, which diagnostics were inspected, what criterion favored the final choice, and whether conclusions changed materially.

This prevents the final output from falsely implying that the analytical route was inevitable.

#Diagnostics should be preserved with consequences

A diagnostic is useful only if it can influence the analysis.

For example, an overdispersion diagnostic may exceed a prespecified threshold and trigger a change from Poisson to negative binomial.

The audit trail should preserve the diagnostic result, interpretive rule and resulting decision.

Otherwise diagnostics can become decorative outputs generated after the decision.

#Reviewer interventions are part of provenance

A reviewer may identify a denominator mismatch, coding error, implausible exclusion or unsupported causal claim.

The correction should not erase the prior state.

A review-aware trail can preserve result version one, reviewer finding, analytical change, result version two and interpretation change.

This is particularly important when the numerical result changes little but the scientific claim becomes more cautious.

#Interpretation has versions too

Analytical provenance often stops at the table.

But the same table can support different descriptions.

A first draft might say treatment reduced mortality by 20%. After causal review, the statement might become: treatment exposure was associated with 20% lower mortality under the prespecified model; residual confounding remains possible.

The data and model can be identical.

The interpretation changed because the claim was recalibrated.

That is an editorial-scientific event worth preserving.

#A practical audit-trail schema

The schema below is a proposed operational pattern derived from the preceding reasoning; it is not part of W3C PROV. For each material analytical stage, preserve:

• input identity, source, version or hash and extraction date;
• transformation code, environment, parameters and generated entity;
• scientific decision, alternatives, rationale, evidence and accountable reviewer;
• checks, diagnostics, failures and remediation;
• output identity, upstream dependencies and interpretation version.

This does not require W3C PROV syntax. The standard provides a conceptual vocabulary; implementation can remain proportionate to the project.^[1]

#The audit trail should survive file renaming

If the workflow depends on remembering that final2-revised-real came after final2, it is fragile.

Stable identifiers, hashes, manifests and explicit parent-child relations are more reliable than filenames.

The National Academies' discussion of computational reproducibility emphasizes data, code, computational details and documentation sufficient to reproduce results.^[2]

#Human-readable and machine-readable views serve different users

A machine-readable provenance graph is useful for automation.

A human reviewer often needs a concise narrative: Data extract X was cleaned with pipeline Y. Eligibility rule Z removed N records. Model A was replaced by model B after diagnostic D. Reviewer R requested sensitivity analysis S. The final conclusion was narrowed because the result was unstable under S.

The two views can be generated from the same underlying records.

#What not to preserve indiscriminately

A complete log of every keystroke is not a useful audit trail.

Preserve information according to consequence.

High-value events include data identity changes, inclusion and exclusion, scientific parameters, model choices, overrides, reviewer decisions, interpretation changes and release state.

Noise can make important provenance harder to find.

#Why this is separate from reproducibility

The companion article Reproducible can still mean repeatably wrong explains why reproducibility does not establish validity.

An audit trail helps reviewers inspect validity by exposing how the analysis was constructed.

It is review infrastructure, not a certificate of correctness.

#Prespecification and deviation need separate records

A strong audit trail distinguishes what was planned from what was changed after seeing data.

For a prespecified analysis, preserve:

• protocol or analysis-plan version;
• date of approval;
• primary outcome and model;
• prespecified exclusions;
• planned sensitivity analyses.

If the analysis changes, preserve the deviation as a new decision rather than silently editing the original plan.

A deviation can be scientifically justified. The integrity requirement is that a reviewer can see when it occurred, why it occurred and whether the observed results influenced it.

#Data corrections should propagate through dependency graphs

Suppose a source dataset is corrected after publication of an internal report.

A useful dependency graph should answer:

• which cleaned datasets used the source version;
• which models depend on those datasets;
• which figures and tables depend on those models;
• which narrative claims cite those outputs.

That turns provenance into impact analysis.

Instead of asking the team to remember where a variable appeared, the system can identify affected artifacts mechanically.

#Release state is part of provenance

An artifact can exist without being approved.

Useful states can include draft, reviewed, approved, superseded and withdrawn.

The audit trail should therefore distinguish generation from release.

A model output generated during exploration is not equivalent to the output accepted for a final report. Preserving release state prevents later users from mistaking an abandoned intermediate artifact for an authoritative result.

#Limitations and uncertainty

W3C PROV is a general provenance model rather than a mandatory epidemiological audit format.^[1] Different regulated, clinical or research environments may impose stronger record-keeping requirements. The proposed schema is a methodological pattern, not a compliance standard. More provenance is not always better: retention, privacy, security and proportionality constraints can limit what should be stored. NIH guidance is used here for rigor and transparency principles rather than as a provenance specification.^[3]