Sociotechnical systems in software and AI: Who bears the risk when AI-assisted medical decisions go wrong?

When AI nudges a clinician toward a harmful decision, responsibility is rarely clear-cut. A sociotechnical perspective helps practitioners, hospitals and developers share the load of risk — and reduce it.

Introduction: a sociotechnical problem, not just a technical one

When an AI system flags a diagnosis or recommends a treatment, it is doing so inside a sociotechnical system: software, data, clinical workflows, human judgement, organisational incentives and legal frameworks all interlock. If something goes wrong, asking "Who is at fault?" is important, but it’s only part of the picture. We must also ask how people, processes and technology combined to produce the error — and who can act to prevent the next one.

Why liability feels messy with healthcare AI

Traditional malpractice and product-liability frameworks rely on familiar concepts: duty of care, breach of a standard, and causation. Courts use those to allocate responsibility when a tool injures a patient. But AI blurs these lines. As the Stanford HAI policy brief explains, assessing duty and standard of care becomes complex when a clinician consults an algorithm whose internal reasoning they cannot fully inspect.

That opacity matters in two directions. If clinicians blindly follow an authoritative AI and harm results, they may be accused of over-reliance. Conversely, if a clinician disregards an AI alert that later proves correct, they may face criticism for ignoring useful information. The resulting "double-bind" affects professional behaviour and the pace of AI adoption.

Who are the usual actors — and what each one risks

Clinicians: Remain responsible for verifying diagnoses and treatment plans. Several legal analyses and practitioner commentaries emphasise that clinicians can be liable if they fail to exercise judgement rather than treating an AI output as gospel.
Hospitals and healthcare organisations: Responsible for procurement, deployment, training and monitoring. Stanford HAI advises that organisations must document which model versions are in use and tailor monitoring intensity to the risk posed by each tool.
Developers and vendors: May face product-liability claims if the system is defective, biased, or inadequately tested. Plaintiffs increasingly frame AI tools as consumer products that should be reasonably safe.
Insurers and payers: The International Journal of Communication piece highlights how automation in administrative decisions — such as prior authorisation and payout adjudication — has already shaped harmful outcomes when automated systems err.

Sociotechnical dynamics that decide who actually bears the risk

Responsibility is shaped by three intertwined dynamics:

Design and explainability: If a system is a black box, clinicians are less able to evaluate when to trust it. Legal authors note that this creates a tension: clinicians may rely on powerful but opaque tools, then be held accountable for consequences they could not foresee.
Organisational governance: Hospitals that fail to implement risk-based monitoring, version control and documentation invite liability. Stanford HAI’s recommendations include focusing oversight on highest-risk technologies and keeping precise records of deployed versions and configurations.
Regulatory and contractual coverage: Contracts and insurance shape ultimate economic responsibility. Developers may shift risk contractually, insurers may deny payouts when automated decisions look suspect (as illustrated in the IJOC analysis), and courts will evaluate duty and breach under familiar legal tests adapted to AI.

Practical guidance for practitioners and organisations

Reducing who bears the risk is about shifting from reactive blame to proactive prevention. The following steps synthesise recommendations from legal analyses, policy briefs and practitioner sources.

Adopt a risk-based monitoring posture: Prioritise intensive oversight for high-impact tools. Monitor more closely where errors are likely to be harmful or the patient group is vulnerable, as advised by Stanford HAI.
Document relentlessly: Record which model version, training data snapshot and software package are in use. If something goes wrong, good records are the first line of defence — legally and clinically.
Test before you deploy: Vendors and hospitals should verify performance across relevant populations and edge cases. Legal guidance repeatedly emphasises thorough pre-deployment testing to reduce foreseeable harm.
Make decision roles explicit: Define who has final authority in each workflow. A sociotechnical approach clarifies when the clinician must overrule the algorithm and when an automated alert requires escalation.
Train people, not just systems: Provide clinicians with practical training about system limitations and failure modes, including what types of mistakes the tool is prone to.
Embed human-in-the-loop controls appropriately: Design interfaces and workflows that encourage verification where it matters most, rather than creating a passive "accept recommendation" button that invites deskilling.
Agree contractual risk-sharing and insurance: Clarify liabilities in procurement contracts and obtain professional indemnity and cyber/tech insurance that reflect AI-specific risks.
Plan for incident response and learning: Have a clear process for near-miss reporting, audit trails and model recall or rollback. Regulators and courts view timely remediation favourably.

Lessons from cases and commentary

Practitioner accounts and case studies demonstrate predictable patterns: systems optimised without social context, incomplete testing, and misaligned incentives often precede harm. In some reported cases, plaintiffs have argued that companies should be treated like manufacturers of consumer products, while other analyses show that intense liability concerns can chill valuable innovation.

Legal scholarship also warns of perverse incentives. If clinicians expect to bear the full legal risk for following system recommendations, they may avoid using helpful tools or introduce expensive double-checks that slow care. Conversely, vendors offloading all risk can reduce the incentive to make robust, explainable systems.

Conclusion: shared responsibility, practical governance

No single actor should carry the entire burden when AI-assisted decisions go wrong. The sociotechnical perspective insists that responsibility maps to where control and influence lie: developers must build reliable, well-documented systems; organisations must govern deployment and monitoring; clinicians must exercise professional judgement; and regulators and insurers must craft rules and coverage that align incentives.

Put simply: design robust systems, operationalise them carefully, train the people who use them, and make sure contracts and insurance reflect the risks. Do that, and when an error does occur the system will be better prepared to learn rather than merely assign blame. That’s good for patients, practitioners and the technology alike — and a bit less finger-pointing is good for everyone’s blood pressure.