What makes an analytical workflow reproducible?

#Question

What does it actually take for an analytical workflow to be reproducible, and how is that different from being replicable?

#Answer in brief

The National Academies define reproducibility as obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. Replicability concerns consistent results across studies that collect new data to answer the same scientific question.^[1] The distinction matters because a workflow can reproduce exactly and still encode a biased design, invalid measurement, or incorrect scientific claim. Conversely, a finding can replicate across studies even when one study's computational record is incomplete.

A reproducible workflow needs more than a notebook that happens to run on its author's computer. It needs identifiable input data, executable and versioned code, captured dependencies and parameters, controlled randomness, recorded manual decisions, machine-checkable expected outputs, and a lawful access path for an authorized person to rerun the work. Where data cannot be public, the goal becomes controlled reproducibility, not abandonment of the standard.

#Key points

• Reproducibility is not truth. It shows that a result can be recomputed from a recorded analytical state; it does not establish that the question, design, data, model, or interpretation is correct.
• A file name is not a version. Input data, code, configuration, and output need persistent identifiers, hashes, commits, immutable snapshots, or another mechanism that identifies exactly what was used.
• Manual work is part of the workflow. Spreadsheet edits, exclusions, recoding decisions, and interactive clicks must be captured or eliminated.
• The environment is an input. Language, package, operating-system, locale, and sometimes hardware versions can affect results.
• Randomness must be controlled and described. A recorded seed is necessary for exact reruns of stochastic steps, but one seed is not a robustness analysis.
• FAIR and reproducible are related but different. Findable, Accessible, Interoperable, and Reusable data and metadata make reuse more feasible; FAIR does not itself guarantee that an analysis can be rerun or that access is unrestricted.^[2]
• Reporting is part of reproducibility. RECORD extends STROBE for studies using routinely collected health data and requires transparent reporting of data provenance, linkage, selection, and coding processes needed to interpret and reconstruct the study.^[3]

#Scope and exclusions

This guide addresses computational and procedural reproducibility of analytical workflows, especially in health, public-health, and biomedical research. It does not define statistical validity, causal identification, research integrity, privacy law, information-security controls, software validation for regulated medical devices, or the probability that a scientific finding will replicate. Those questions interact with reproducibility but require additional standards.

#Reproducible, replicable, robust, and reusable are different claims

#Reproducible

An authorized analyst can use the recorded inputs and procedures to obtain the same material outputs, allowing for explicitly documented numerical tolerances where exact bitwise identity is not realistic.

#Replicable

A separate study with new data addressing the same question obtains a result that is consistent under an appropriate definition. Replicability concerns the finding, not merely the pipeline.^[1]

#Robust

The conclusion remains materially similar under reasonable alternative analytical choices, assumptions, specifications, or perturbations. A workflow can be exactly reproducible and highly fragile.

#Reusable

Data, code, metadata, and documentation can be understood and applied beyond the original execution. FAIR principles support this by making digital objects findable, accessible under stated conditions, interoperable, and reusable.^[2]

Goodman and colleagues documented that different fields use "reproducibility" for several distinct concepts, which is why every claim should state what was repeated, with what data, and by whom.^[4]

#The minimum reproducibility record

#1. Identifiable source data

Record:

• source system and extraction date;
• query or extraction logic;
• cohort or record-selection criteria;
• data version, snapshot, hash, or immutable identifier;
• linkage procedures;
• coding dictionaries and reference-data versions;
• transformations performed before the analytical repository received the data;
• access and disclosure constraints.

"The current production database" is not a reproducible input because it changes. A copy labelled final_data_v2.csv is not sufficient when its origin and transformations are unknown.

For routinely collected health data, provenance and selection are especially important because the dataset exists for an operational purpose and may not represent the population or variables assumed by the research question. RECORD therefore adds reporting requirements concerning database selection, codes, algorithms, linkage, and access to supplementary information.^[3]

#2. Versioned executable procedures

Retain the code that actually generated the result, including:

• cleaning and recoding;
• exclusions;
• linkage and deduplication;
• derived variables;
• model fitting;
• diagnostics;
• tables, figures, and reports.

A prose method is necessary for interpretation but usually insufficient for exact reconstruction. Code should be tied to a commit or immutable release, not merely stored in a folder that continues to change.

#3. Captured environment

Record or construct:

• language and runtime versions;
• package names and exact versions;
• operating-system or container image;
• locale, encoding, and time-zone assumptions;
• external software and system libraries;
• hardware or numerical backend where materially relevant.

Lockfiles, environment files, containers, and virtual machines solve different parts of this problem. A container improves portability but does not preserve external data, credentials, mutable services, or indefinitely available image registries.

#4. Parameters and configuration

Values that can change the output should be explicit:

• inclusion thresholds;
• date windows;
• missing-data rules;
• model formulae and reference categories;
• convergence settings;
• multiplicity decisions;
• hyperparameters;
• random seeds;
• report options.

Defaults are still parameters. An upgraded package can change them without the analysis code changing.

#5. Manual decisions and interventions

A workflow is not reproducible when decisive operations occur only in someone's memory or through undocumented interface actions.

Examples include:

• removing "obvious errors" in a spreadsheet;
• merging categories after viewing results;
• changing a model because a graph looked unusual;
• manually copying values into a report;
• choosing one of several outputs without recording the criterion.

Either automate the operation or retain an auditable decision record with the input, action, rationale, author, date, and effect.

#6. Expected outputs and tests

A rerun needs a criterion for success. Preserve:

• key result values or hashes;
• row counts and schema checks after major stages;
• invariant and plausibility tests;
• model convergence and diagnostic expectations;
• allowed numerical tolerances;
• generated report identifiers.

Tests catch silent drift. A workflow that runs to completion but produces a different cohort is not reproduced merely because no error appeared.

#7. Instructions and authorized access

A second person should know:

• how to obtain authorized access;
• where to start;
• which command or pipeline target to run;
• expected execution order;
• required secrets or services;
• approximate resource requirements;
• where outputs are written;
• how to compare them with the reference result.

Public release is not always lawful or ethical. Reproducibility can be tested in a secure environment, through controlled access, by an independent internal analyst, or with disclosure-controlled synthetic data that verifies pipeline behavior while the real-data result is checked in place.

#Exact reproducibility is not always bitwise identity

Floating-point arithmetic, parallel execution, nondeterministic algorithms, database query order, hardware accelerators, and external services can create small differences. A defensible workflow should classify outputs:

• exact: hashes or values must match;
• tolerance-based: values must agree within a predefined numerical bound;
• distributional: stochastic procedures must reproduce a defined distribution or summary, while a fixed seed supports debugging;
• semantic: a report must contain the same conclusions and evidence despite non-material formatting differences.

Tolerances should be chosen before comparison and connected to scientific meaning. "Close enough" after seeing a discrepancy is not a reproducibility standard.

#Worked example: a programme participation estimate

An analyst reports that participation increased by 12% year over year using routine service data.

#Incomplete record

The repository contains one notebook and a chart. The notebook reads from a live database, uses an unversioned lookup table, excludes records through a manual spreadsheet, and depends on the analyst's installed packages. The 12% figure may be correct, but the workflow is not reproducible.

#Reproducible record

1. A dated, access-controlled extract is assigned a hash and documented query.
2. Inclusion and deduplication code is committed.
3. The lookup table and coding dictionary are versioned.
4. Manual exclusions are replaced by a reviewed exclusions file containing record identifier, rule, rationale, and approver.
5. Package versions and runtime are locked.
6. The pipeline runs from one documented command.
7. Tests assert source row count, included cohort count, missingness ranges, and the final estimate within a declared tolerance.
8. A second authorized analyst runs the workflow in a clean environment and obtains the expected result.

This supports the claim that the calculation is reproducible. It does not show that participation was measured without bias, that the year-to-year comparison supports a causal claim, or that 12% is practically important.

#A reproducibility review checklist

#Inputs

• Can each source dataset and reference table be identified exactly?
• Are extraction, linkage, inclusion, and coding procedures recorded?
• Are access restrictions and lawful-use conditions stated?

#Execution

• Is the complete analytical path executable?
• Are dependencies, parameters, seeds, and services captured?
• Are undocumented manual steps absent?

#Outputs

• Are expected results and tolerances defined?
• Are diagnostics and quality checks retained?
• Can every published table and figure be traced to a pipeline target?

#Independent rerun

• Has someone other than the original analyst rerun the workflow?
• Did they start from the documented materials rather than informal coaching?
• Were discrepancies recorded and resolved without rewriting history?

#Interpretation

• Are reproducibility, robustness, replicability, and validity kept distinct?
• Are unavailable data or proprietary dependencies disclosed?
• Does the report follow relevant guidance such as RECORD for routinely collected health data?^[3]

#Evidence and method

The reproducibility–replicability distinction follows the National Academies' 2019 consensus report.^[1] Goodman and colleagues provide the broader taxonomy showing why the term must be qualified.^[4] FAIR supplies principles for managing digital research objects so that machines and people can find, access under stated conditions, interoperate with, and reuse them.^[2] RECORD supplies domain-specific reporting expectations for observational research based on routinely collected health data.^[3]

The implementation checklist above synthesizes these principles into an analytical-workflow record. It is not a certification standard and has not been validated as a scoring instrument.

#Practical implications

Replace the vague question "Is this analysis reproducible?" with:

Can a named, authorized person start from the identified data state and documented environment, execute the recorded procedures without undocumented assistance, and obtain outputs that satisfy predefined comparison criteria?

Then ask the separate questions:

• Is the design valid for the research question?
• Are the data fit for purpose?
• Are the analytical choices robust?
• Is the interpretation proportionate?
• Does an independent study replicate the finding?

This separation prevents a technically polished pipeline from borrowing scientific credibility it has not earned.

#Limitations and uncertainty

Complete reproducibility may be constrained by privacy, consent, intellectual property, proprietary software, disappearing external services, computational cost, or legacy systems. These constraints should produce a documented access and preservation strategy, not an unqualified reproducibility claim. Containers and lockfiles can decay; secure environments can become unavailable; data controllers may lawfully deny secondary access; and exact reruns may be impossible after hardware or software obsolescence. Reproducibility is therefore both an engineering property and an institutional maintenance commitment.

#Connection to Weavidence

Weavidence Lab is designed to keep data-quality findings, transformations, analytical procedures, parameters, review decisions, and registered artifacts connected through lineage. That architecture is intended to support an inspectable reproducibility record. Lab remains in active development and is not publicly available or externally validated; the design does not itself prove that any future workflow will be reproducible, robust, or scientifically valid.