Sociotechnical systems in software and AI: Reproducibility in the age of AI-assisted research

Reproducibility in AI research is as much a social and organisational problem as a technical one. This post unpacks the sociotechnical dynamics that make reproducibility tricky today, and offers practical steps teams can take to keep research verifiable and resilient when AI becomes a collaborator.

Introduction: reproducibility beyond the code

Reproducibility used to feel like a tidy problem: check your code into version control, pin your libraries, and ship the data. Today, AI-assisted research has stretched that tidy picture into a knotted tapestry of models, cloud services, people, incentives and legal fences. The emerging literature on AI as a sociotechnical system reminds us that reproducibility is not only a technical reproducibility problem but an organisational and social one too. In plain terms: it matters who controls the model, who owns the data, and who documents the social choices embedded in the research pipeline.

Why reproducibility matters — and what’s changed

At its core, reproducibility means someone else (or you, six months later) can obtain the same or similar results using the same dataset, algorithm and environment. That definition, helpfully summarised in industry guides, highlights three parts you must control: data, algorithm and environment. If any of them shifts, reproducibility drifts away.

But in the age of large proprietary models and cloud-hosted pipelines, two extra complications appear. First, the epistemic capacity — our ability to create and verify knowledge — can be enclosed by proprietary services. A recent white paper points out that when a widely used model or API is discontinued, researchers lose a crucial part of the experiment’s foundation (the paper cites the discontinuation of Codex as a concrete example). Second, the tying of cloud and AI services to specific providers makes long-term stability and independence harder for smaller teams and organisations.

The sociotechnical twist: systems, not instruments

Thinking sociotechnically means shifting attention from isolated tools to interconnected systems made of people, processes, institutions and artifacts. One commentary on this transition argues that our cultural habits prefer 'tools' and mechanisms, but AI forces us to read interactions, layers and emergent behaviours instead. That perspective reframes reproducibility as a property of a system: do we have enough visibility and governance over the social and technical layers to reproduce results?

Practical examples include simulation and digital twin technologies, which bridge virtual and physical worlds in Industry 4.0; these remind us that reproducibility depends on how well we model real-world conditions and how faithfully we document them. The National Institute work on socio-technical approaches also urges holistic evaluation — aligning accuracy with societal values and legal requirements. In short, reproducibility is entangled with governance, documentation, and the socio-economic context.

Concrete sources of reproducibility failure (observed in practice)

Proprietary model disappearance: a hosted model or API you rely on can be deprecated, changed, or gated behind new terms, undermining reproducibility.
Opaque model behaviour: closed models make it hard to understand internal changes or nondeterministic updates.
Hidden environment drift: cloud images, dependency updates and hardware differences introduce variability if not captured.
Social/organisational churn: personnel changes, undocumented handoffs and shifting incentives reduce institutional memory needed to reproduce work.

A practical reproducibility checklist for AI-assisted research

Below is a pragmatic list you can adopt in a project retrospective or bake into your workflow. These suggestions are rooted in the literature that treats AI as a sociotechnical system and in practical reproducibility guidance.

Document decisions, not just code: capture why you chose a model, dataset cleaning steps, and any social constraints (licenses, access limits).
Record environment precisely: use container images (Docker/OCI), record hardware specs, and keep dependency manifests with hash pins.
Log experiments and seeds: record random seeds, hyperparameters, and run IDs in an experiment tracking system so results can be replayed.
Prefer open models or snapshot your artefacts: when possible use open weights or archive model checkpoints. If you must use a hosted API, record API version, query logs and the exact timestamps.
Archive datasets and licences: keep immutable copies of training and evaluation sets where legally permissible, and store associated licence metadata and provenance.
Model cards and data statements: publish short, standardised descriptions of model scope, training data composition and limitations so others understand the socio-technical context.
Simulate dependency failure: run a 'provider outage' test—can you rerun experiments if an API becomes unavailable? If not, what minimal local artefacts would you need?
Human workflows and roles: define who is responsible for archiving artefacts and for granting future access; avoid relying on single-person knowledge.
Governance checklist: align reproducibility practices with organisational policies for IP, privacy and export controls that might affect later verification.

Organisational moves that help

Reproducibility is rarely fixed by a single developer alone. Organisations can create durable practices that reflect the sociotechnical view:

Invest in engineering systems thinking: treat experiments as systems engineering projects, not one-off scripts.
Make reproducibility part of code review and acceptance criteria for papers or internal reports.
Negotiate contractual terms with providers that allow archival access or long-term snapshots where possible.
Build incentives for documentation: reward reproducible artifacts, not just fast prototype results.

Trade-offs and reality checks

There are trade-offs to be explicit about. Full reproducibility can be costly: archiving large datasets and models eats storage, and snapshotting proprietary APIs is sometimes legally or technically impossible. The sociotechnical lens helps here: make a risk-based decision about which experiments require deep archiving and which can be treated as ephemeral prototypes. The point is to make those decisions consciously and document them.

Conclusion: design for longevity, not just speed

AI-assisted research will keep accelerating. If we want the work to be trustworthy and useful beyond the next quarter, we must design reproducibility into our sociotechnical systems. That means better documentation, engineering practices tuned to AI artefacts, and organisational policies that acknowledge models and providers as part of the research apparatus. Treat reproducibility as a systemic property — not an optional appendix — and you'll be better placed when a favourite API goes read-only or a model is retired. And while we’re pragmatic about the limits, a little foresight and a few reproducible habits can spare future teams (and future-you) a lot of detective work.