Reproducible can still mean repeatably wrong

#Question

How can an analysis be perfectly reproducible and scientifically invalid?

#Answer in brief

Computational reproducibility means that the same input data, computational methods and conditions produce consistent results. The National Academies adopted that specific definition to distinguish reproducibility from replication with new data.^[1]

Validity is a different question: whether the design, measurements, analysis and interpretation support the inference being made.

A workflow can therefore be perfectly reproducible and reproduce the same selection bias, coding mistake or causal error every time.

#Two separate questions

Consider these statements:

I can rerun your code and obtain exactly the same estimate.

Your estimate answers the scientific question correctly.

The first is about reproducibility. The second is about validity.

They are related because reproducibility makes the analytical process inspectable and testable. But neither logically implies the other.

#A four-cell matrix

A useful conceptual matrix separates reproducible from non-reproducible analyses and valid from invalid inferences.

A reproducible and valid analysis is the desirable case. A reproducible but invalid analysis is repeatably wrong. A potentially valid analysis that cannot be reproduced remains difficult to verify. A non-reproducible and invalid analysis combines both failures.

The matrix is not a scoring system. It is a reminder that process reconstruction and scientific correctness are distinct dimensions.

#Example 1: reproducible selection bias

A study uses a complete scripted pipeline to estimate disease prevalence from people who attend a specialist clinic.

Every extraction, transformation and model is versioned.

The pipeline is perfectly reproducible.

But if the estimate is presented as population prevalence, the selected source population invalidates the inference.

Reproducibility has preserved the bias exactly.

#Example 2: reproducible denominator mismatch

An analyst computes vaccination coverage using vaccinations by service location divided by residents in the district.

The code runs identically every time.

But people can receive vaccination outside their district. The numerator and denominator describe different population rules.

Again, computational reliability does not repair conceptual mismatch.

#Example 3: valid idea, irreproducible execution

A well-designed study uses appropriate eligibility, measurement and analysis.

However, data-cleaning steps were performed manually and undocumented; some records were removed without a reproducible rule; the model specification in the manuscript differs from the final script; and software versions are unknown.

The scientific idea may be strong, but independent reconstruction is difficult.

This is why rigor and reproducibility are complementary rather than interchangeable.

#Rigor is broader than reproducibility

NIH defines scientific rigor as the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting.^[2]

That definition spans multiple stages.

Reproducibility focuses on whether the executed process and result can be reconstructed under the same inputs and conditions.

A reproducible analysis can still fail rigor because the study design is biased, an outcome is measured poorly, exclusions use future information, an inappropriate model is chosen, uncertainty is misinterpreted, or the conclusion exceeds the estimand.

#Why reproducibility still matters for validity

If reproducibility does not prove validity, why invest in it?

Because it improves the conditions under which validity can be assessed.

A reconstructable workflow allows another analyst to inspect transformations, exclusions, model parameters, intermediate datasets, sensitivity analyses and discrepancies between protocol and execution.

It converts hidden process into reviewable process.

That can reveal invalidity.

#Reproducibility is not replication either

The National Academies distinguishes computational reproducibility from replicability. Replicability concerns obtaining consistent results in a new study addressing the same scientific question with new data.^[1]

This distinction prevents another common overstatement.

Rerunning the same analysis successfully does not show that the finding will recur in a different population or dataset.

#A result can be reproducible and unstable

Suppose a model produces the same coefficient every time.

Changing one reasonable analytical choice—such as an exposure window—changes the estimate dramatically.

The original analysis is still reproducible.

But the scientific conclusion may be fragile.

A strong workflow therefore needs not only deterministic rerunning but sensitivity analysis around consequential choices.

#The audit question is not does it run?

A pipeline passing automated execution checks can establish that files exist, code executes, tests pass and expected outputs are produced.

It cannot establish by itself that the study question is important, the exposure is valid, confounding is controlled, interpretation is causal or the conclusion is epidemiologically meaningful.

Automation verifies defined properties.

Scientific judgment decides whether those properties are sufficient for the claim.

#The right relationship

A useful hierarchy is:

reconstructable process → inspectable decisions → methodological review → calibrated inference.

Reproducibility supports review. It does not replace it.

#Practical implications

For analytical work, preserve input data identity, code and versions, environment, transformation sequence, analysis specification and outputs.

Then separately review study design, measurement, causal structure, statistical assumptions, sensitivity and interpretation.

The companion article An analytical audit trail must preserve decisions, not just files develops what needs to be recorded beyond executable code.

#Reproducibility can expose analytical discretion

One benefit of a reproducible workflow is that discretionary choices become observable.

Suppose two analysts start from the same dataset and reach different estimates because they make different choices about:

• episode construction;
• missing-data handling;
• transformation of a continuous exposure;
• inclusion of a subgroup;
• model family;
• time window.

If both workflows are fully reconstructable, the disagreement becomes scientifically productive. Reviewers can ask which choice is better justified and whether conclusions are sensitive to it.

If only the final numbers remain, disagreement can be misread as software instability rather than a difference in scientific assumptions.

#Validity has several layers

Validity is not one switch that is either on or off.

An analysis can have strong measurement validity but weak causal identification. It can answer the intended estimand internally but fail to transport to another population. It can have appropriate design but poor data quality.

For practical review, separate at least:

• measurement validity — do variables represent the intended constructs?
• design validity — does the study structure support the comparison?
• statistical validity — do methods and uncertainty estimates fit the data and assumptions?
• causal validity — are the assumptions for a causal interpretation plausible?
• external validity — does the inference apply to the target population or context?

A reproducibility check intersects all of these only indirectly.

#Reproducibility can make corrections cheaper

When an error is found, a reproducible workflow narrows the repair problem.

If one diagnosis code was misclassified, the analyst can identify the transformation, change it, regenerate dependent outputs and compare versions.

In a manually assembled workflow, the correction may require rebuilding tables and figures one by one, increasing the chance of inconsistent revisions.

This is an operational reason to value reproducibility even when it does not establish validity: it makes scientific correction more tractable.

#Limitations and uncertainty

Reproducibility is used differently across disciplines. This article follows the National Academies' computational definition.^[1] Other communities may use reproducibility to include what the report calls replicability. The four-cell matrix is a conceptual teaching device, not a formal classification system. Scientific validity is multidimensional and cannot usually be reduced to one binary judgement.