When a Model Forges a Reference: Sociotechnical Consequences and Practical Fixes

Citation hallucinations are more than a technical quirk. Seen through a sociotechnical lens, forged references erode the social and procedural scaffolding that makes scholarly communication reliable. This post explains why that matters and gives concrete guidance for engineers, editors and research teams.

Why citations are a sociotechnical problem, not just a model bug

In scholarly systems, citations are a form of social infrastructure: they encode provenance, credit, and a pathway for verification. From a sociotechnical perspective, technology and social practice are mutually constituted — you cannot fix citation problems by tweaking models alone. The Minds and Machines paper on sociotechnical AI argues that AI must be understood as part of larger systems of interdependent elements: models, datasets, retrieval systems, people and the rules they follow. That means a model that invents citations does harm across the whole system because it interferes with people’s ability to verify, evaluate and trust research claims.

How a hallucinated citation ripples through the system

When a model supplies a plausible but false reference, the immediate harm is to the consumer who may accept the claim without checking. But the sociotechnical consequences go further.

Editors and reviewers who rely on automated checks or model summaries face higher verification costs and risk passing flawed work.
Legal and ethical norms that depend on traceable sources become harder to enforce if provenance is unreliable.
Research communities begin to distrust automated tools, slowing adoption and fragmenting workflows that could otherwise scale verification.

These outcomes match core ideas in sociotechnical literature: technical capacities interact with organisational procedures and social expectations, producing emergent behaviours that are context dependent. You can design a very capable model, but how it functions in practice depends on the social rules and infrastructures around it.

Practical guidance: design, governance and workflows

Treat trust as an engineering constraint. If citations are a currency of scholarly trust, the system must ensure the currency is verifiable. Below are pragmatic measures teams can adopt.

Design for provenance: For models that generate or surface citations, require explicit provenance metadata: where the citation came from (index, local dataset, web scrape), retrieval confidence and a unique identifier (DOI, arXiv ID). Make provenance part of the API contract so downstream tools and humans can always inspect origin.
Integrate retrieval-augmented techniques with verified corpora: Combine model generation with deterministic retrieval from curated bibliographic databases. Retrieval can be engineered to return only records that resolve to persistent identifiers, reducing the space of plausible but nonexistent references.
Automated verification checks: Before a citation is surfaced as authoritative, run lightweight automated checks: resolve the DOI, compare title and author strings, verify publication year. Fail-fast if the check cannot confirm the reference.
Human-in-the-loop gates for uncertain cases: Where automated checks are inconclusive, route the item to an editorial or librarian workflow rather than publishing the claim. Sociotechnical research emphasises that human practices and judgement remain essential when uncertainty is social and contextual.
Clear labelling and uncertainty communication: Don’t present generated citations as if they were verified facts. Use clear labels (e.g. "candidate reference") and show confidence scores, so consumers understand what requires further checking.
Logging, audit trails and reproducibility: Maintain immutable logs of model outputs, retrieval calls and verification steps. Logs support later audits and help communities learn about failure modes and how often they occur.

Testing, metrics and organisational practice

Adopt test suites that exercise citation behaviour as a first-class requirement. Examples include:

Benchmarks that measure the rate of fabricated references under representative prompts.
Integration tests that verify end-to-end resolution of cited identifiers.
Usability tests with editors, librarians and researchers to learn where automation helps and where manual checks are essential.

From an organisational viewpoint, embed citation behaviour into roles and norms: update editorial guidelines, train researchers in using AI-assisted tools critically, and create feedback loops so detected hallucinations refine both models and organisational rules.

Why this matters for adoption and the broader research ecosystem

Sociotechnical accounts emphasise that systems which ignore social practices and ethical imperatives struggle to gain traction. If tools systematically undermine the ability to verify scholarship, communities will either ban or heavily circumscribe their use — losing potential productivity gains. Conversely, designing tools that respect citation norms can speed adoption and strengthen epistemic practices by reducing routine verification cost and making provenance explicit.

Expanded from

This post expands on a point from When the Model Hallucinates a Citation: Trust, Verification and Scientific Norms in Sociotechnical AI.