Who really bears the risk? Practical lessons when AI-assisted medical decisions go wrong

Why this matters

The observation in the original piece is familiar to anyone working at the intersection of clinical care and machine intelligence: harm rarely appears out of nowhere. Rather, it tends to follow predictable sociotechnical patterns — systems optimised without social context, incomplete testing, and misaligned incentives. Those patterns produce legal and operational dilemmas: should vendors be treated like manufacturers, or do liability rules chill innovation? How do we avoid perverse incentives that push clinicians to abandon useful tools or vendors to offload risk?

What the patterns look like in practice

From practitioner accounts and case studies, three recurring themes emerge.

• Optimisation without social context — algorithms are tuned to datasets and objectives that don’t reflect the messy realities of frontline care (workarounds, local protocols, data gaps). That mismatch means a perfectly performing model in development may behave poorly in situ.
• Incomplete testing — limited real-world validation, weak post-deployment monitoring and little user-centred testing. Edge cases and interactions with clinical workflow are often discovered only after harm has occurred.
• Misaligned incentives — commercial, regulatory and clinical incentives push actors toward extremes. Vendors may prioritise market speed; clinicians may fear legal exposure for following tool recommendations. Both sides can then make conservative choices that reduce benefit or safety.

How legal framing amplifies sociotechnical dynamics

Two competing legal logics shape behaviour. One treats companies as product manufacturers with attendant duties; the other emphasises clinician responsibility for final decisions. Both logics expose perverse incentives noted in scholarship and practice:

• If clinicians expect to bear most legal risk for following recommendations, they may stop using helpful tools or add costly double‑checks that slow care.
• If vendors sign away responsibility entirely, they lose incentives to design robust, explainable systems and to invest in monitoring and improvement.

Neither extreme is sociotechnically wise. Risk allocation affects not just legal outcomes but everyday clinical behaviour and system safety.

Practical guidance: what clinicians and health organisations can do

Clinicians and purchasers can act now to reduce harm without waiting for perfect law or regulation.

• Insist on site-specific validation. Require vendors to support or fund local trials and retrospective validation on representative data before clinical rollout.
• Integrate tools into workflow deliberately. Map existing decision paths, identify where an AI suggestion will sit, and design fail‑safes and escalation routes so the tool augments rather than interrupts practice.
• Document rationale and deviations. Make it straightforward for clinicians to record when and why they follow or reject a recommendation; these records are crucial for learning and for defensible care decisions.
• Maintain continuous monitoring. Track performance metrics, user acceptance and near‑misses. Use these signals to trigger retraining, interface changes or pausing deployments.

Practical guidance: what vendors should do

Vendors can reduce downstream risk and build trust by designing for the social environment in which their product will operate.

• Test beyond accuracy. Demonstrate usability, explainability and how recommendations behave in edge cases and under distribution shift.
• Design for meaningful human oversight. Explainability is not just a feature — it should enable clinicians to reason about when a model might be wrong.
• Commit to post‑market surveillance. Share monitoring data with purchasers and agree triggers for updates or rollbacks.
• Offer realistic warranties and shared‑risk contracts. Rather than blanket indemnities, consider limited liability tied to adherence to integration and monitoring requirements.

Procurement and contract levers that reduce perverse incentives

Procurement is the lever where sociotechnical design meets legal allocation of risk. Contracts can nudge safer behaviour without blocking innovation.

• Require evidence of local validation and user testing as conditions precedent to payment or full deployment.
• Use phased rollouts with predefined safety gates — pilots, audits and go/no‑go criteria mitigate the chance of wide‑scale harm.
• Define shared responsibilities for monitoring, incident response and communication to regulators and patients.
• Set incentive structures such as performance‑linked payments or joint commitments to fund corrective work post‑deployment.

Governance, not just liability

Focusing only on who pays after something goes wrong is short sighted. Robust governance — governance that blends clinical leadership, technical assurance, procurement standards and patient representation — reduces both incidents and adversarial cascades. Practically, this means standing up multidisciplinary review boards, formal incident learning processes and transparent reporting.

Takeaway

Legal framing matters because it changes behaviour across the sociotechnical stack. The safer path is not to try to pin blame on a single party but to design systems, contracts and clinical workflows so that incentives align with safety and continuous learning. Shared risk, realistic testing, explainability and active governance keep useful innovation moving while protecting patients and clinicians.

Expanded from

This post expands on a point from Sociotechnical systems in software and AI: Who bears the risk when AI-assisted medical decisions go wrong?.

Sources

Sociotechnical systems in software and AI: Who bears the risk when AI-assisted medical decisions go wrong?