Organisational moves that actually improve reproducibility

Why organisational moves matter

Reproducibility is a sociotechnical problem: it sits at the intersection of people, incentives, process and tools. A single developer can write cleaner code or add a README, but long‑term reproducibility requires durable organisational practices. Research into scaling reproducibility in institutions shows that frameworks and aligned stakeholders enable sustainable change — leaders, managers and researchers must talk the same language and share responsibilities for reproducible outputs (see the Open Science framework below).

1. Invest in engineering systems thinking

Treat experiments and model development as systems engineering projects, not one‑off scripts. Practically this means creating explicit interfaces, ownership, and lifecycle plans for experiments:

• Define an experiment spec: inputs, environment, hardware requirements, exact dataset versions and expected outputs.
• Use reproducible execution environments: containers, declared package manifests and machine images. Capture exact dependency versions and OS layers where possible.
• Instrument experiments with lightweight telemetry and provenance metadata so runs can be audited and compared.
• Make experiment runs first‑class artefacts: store logs, random seeds, config files and evaluation pipelines alongside code.
• Apply continuous integration ideas to experiments: automated checks that run training/evaluation on small samples to verify that pipelines still execute.

These steps frame research work as maintainable systems, reducing the ‘works on my laptop’ problem and making replication tractable for others within the organisation.

2. Make reproducibility part of code review and acceptance criteria

Organisational norms matter. If reproducibility is optional, it will be deprioritised. Practically, add reproducibility checkpoints into existing gates:

• Code review checklist items: does the PR include environment spec, a minimal runbook and tests that exercise core logic?
• Paper or report acceptance criteria: require a reproducible artefact (container, dataset pointers, minimal script) or a clear explanation why one isn’t possible.
• Use badges or internal metadata that tag outputs as 'reproducible' or 'requires special access', so reviewers and readers immediately see the status.

Journals and funders already affect behaviour by setting reporting standards, so internal acceptance criteria can mirror those external expectations and reduce friction when submitting work (see references below on journal and funder roles).

3. Negotiate contractual terms for archival access and snapshots

Cloud services and third‑party providers are essential, but vendor changes can break reproducibility. Organisations should negotiate contracts that preserve the ability to reproduce results long term:

• Request archival access or long‑term snapshots for data, models and execution environments where possible.
• Include clauses for escrow or export of critical artefacts if a provider changes terms or shutters a service.
• Plan for institutional archiving: deposit immutable snapshots in institutional repositories, object stores with versioning, or national archives when appropriate.
• Avoid single‑provider lock‑in for the canonical trace of an experiment; if you must use a managed service, replicate metadata and critical binaries to an internal archive.

These practices echo funders’ and public institutions’ concerns about long‑term access and preservation of research artefacts.

4. Build incentives for documentation and reproducible artefacts

Incentives steer behaviour more than exhortation. Make reproducibility part of reward structures:

• Recognise reproducible contributions in performance reviews and promotion criteria, not only novel results or prototype speed.
• Allocate small grants or 'reproducibility sprints' that fund researchers to turn promising prototypes into reproducible packages.
• Create internal metrics: number of reproducible artefacts, time to reproduce a result, or presence of required metadata, and use them in portfolio decisions.
• Publicly credit people who document and maintain reproducible workflows — social recognition matters.

Putting it into practice: a short implementation roadmap

Start small and iterate. A pragmatic rollout might look like:

• Quick wins (1–3 months): add reproducibility checklist items to code reviews and acceptance templates; pilot a reproducible artefact badge.
• Mid term (3–9 months): define an experiment spec template, run experiments through a CI pipeline for basic execution checks, negotiate minimal data/export clauses with key vendors.
• Longer term (9–18 months): embed reproducible outputs into promotion criteria, set up institutional archiving and automated snapshotting for critical artefacts, and run cross‑team reproducibility audits.

Assign clear owners for each step: team leads handle local checklists, platform or research engineering teams provide templates and automation, legal negotiates contractual terms, and leadership sets incentives.

Conclusion: the sociotechnical lever

Technical fixes matter, but without organisational design they are fragile. By investing in systems thinking, gating reproducibility in reviews, negotiating archival access, and rewarding documentation, organisations make reproducibility an enduring capability rather than a heroic exception. These moves align the social and technical parts of the system so reproducibility becomes a shared, supported practice.

Expanded from

This post expands on a point from Sociotechnical systems in software and AI: Reproducibility in the age of AI-assisted research.

Sources

• Ouvrir la Science — Approaches to scaling up reproducibility in research organisations
• ATCC — Improving accuracy and reproducibility in life science research
• NCBI Bookshelf — Improving Reproducibility and Replicability
• National Academies — Reproducibility and Replicability in Science