When the Model Hallucinates a Citation: Trust, Verification and Scientific Norms in Sociotechnical AI

Citation hallucinations are not just technical glitches; they expose how AI sits inside social practices. Practical steps — verification, provenance, organisational rules and evaluation reform — can reduce harm and restore trust.

Introduction

When a language model invents a perfectly plausible-but-fictional citation, it doesnt merely make a small mistake. It calls into question the social contract that underpins scholarly work: that citations point to verifiable, shared facts. In a sociotechnical view, models, people, workflows and norms form a system. Fixing citation hallucinations therefore means more than adjusting loss functions; it requires changes to tools, rules and practices that surround those tools.

What is a citation hallucination?

Different communities use slightly different language, but the core idea is clear. The Stanford study on legal research tools adopts a useful working definition: a hallucination occurs when a response is incorrect or misgrounded — including when a model asserts a source supports a point when it does not. Practical examples include invented articles, wrong page references, or claims that a cited case nonetheless supports a proposition when it has been overturned.

Recent work on detecting citation hallucinations emphasises that these are not the same as the old kind of human citation slips. Rather than transcription errors driven by tiredness or inattentive typing, model hallucinations are systematic fabrications produced by statistical patterns in language generation. Models are very good at mimicking the surface features of citations: plausible author names, journals in the right field and titles that look researchy. That verisimilitude is precisely what makes them dangerous.

Why this matters for sociotechnical systems

From a sociotechnical perspective, technologies never exist in isolation. The "system" includes technical parts (models, datasets, retrieval systems), people (researchers, editors, counsel), and rules (publication norms, legal standards). The Minds and Machines paper on sociotechnical AI argues that we must see AI as part of larger systems with interdependent elements; citation norms are one of those elements.

NISTs analysis further warns that technology that fails to respect social and ethical imperatives will struggle to gain adoption. Trust is not a cosmetic feature; it is an engineering constraint. If citations are the currency of scholarly trust, then models that counterfeit that currency undermine the whole marketplace of ideas.

How hallucinations arise — beyond the myth of black-box magic

There are technical and social contributors. OpenAIs write-up on why models hallucinate points out that post-training (finetuning and alignment) can reduce some hallucinations, but that the problem persists because evaluation and incentives sometimes inadvertently reward fluent but ungrounded outputs. The arXiv paper on citation hallucinations emphasises that models exploit the conventions of academic writing, producing references that look authentic without being anchored in real sources.

Meanwhile, the Women in AI whitepaper reminds us not to fetishise the label "AI". Pseudo-objectivity the tendency to treat a formula, score or opaque output as inherently scientific is a broader sociotechnical problem. When decision criteria are hidden behind commercial secrets or opaque pipelines, it becomes harder for users to interrogate whether a citation is real or contrived.

Practical guidance for practitioners

Here are steps teams can take today. They are pragmatic and system-oriented: changes to tools, workflows and culture.

Assume scepticism by default. Train users and downstream readers to treat model-provided citations as hypotheses to verify, not proofs. Make citation-checking part of ordinary review workflows.
Require provenance and retrieval traces. When a model cites a paper, show the retrieval snapshot (document snippet, URL, DOI and retrieval timestamp). This makes it possible to confirm that the reference was actually seen by the system and to audit later.
Automate basic verification. Tools can check whether DOIs resolve, whether author names match, or whether a title is indexed in major databases. Automation wont catch everything, but it rapidly filters out many fabrications.
Use conservative RAG patterns. Retrieval-augmented generation can reduce hallucinations if the retrieval component is trustworthy. Prefer systems that surface exact text spans and make it easy to navigate back to the original source rather than just providing an in-model paraphrase.
Document model limitations in outputs. Make the systems confidence and grounding explicit: "Source verified" vs "Source unverified" flags give readers quick cues about reliability.
Embed human-in-the-loop review where consequences are high. For legal, clinical or policy outputs, mandate human verification of all citations before publication or filing. The Stanford legal-research study shows how misleading citations can materially misrepresent the law; dont leave that to chance.
Change evaluation incentives. Align your benchmarks and internal leaderboards with groundedness metrics, not just fluency. OpenAI notes that current leaderboards can reinforce particular hallucinations; changing what we measure changes what gets optimised.
Build organisational norms and archives. Keep a changelog of model versions, datasets and prompt templates. When a hallucination surfaces, the ability to trace it to a specific model run or dataset makes remediation and learning possible.

Policy and cultural measures

Technical fixes are necessary but insufficient. Scientific norms exist for a reason citations are a social contract. Encourage teams to adopt minimal disclosure standards for AI-aided drafting, and require checks for claims that influence public decisions. Reward careful scholarship: peer review and editorial processes should treat model-assisted claims the same way they would treat an unfamiliar human-cited source.

Closing thoughts

Citation hallucinations are an emblematic sociotechnical problem. They are produced by models statistical tendencies, amplified by opaque toolchains and by social incentives that prize speed and fluency. Addressing them requires an ecological response: technical measures (provenance, verification), organisational changes (workflows, audits) and cultural reinforcement of scientific norms. If we treat citations as merely decorative metadata, we will be surprised by the cost. If instead we treat them as part of the contractual fabric of science, we can design systems that restore trust without sacrificing practicality.