Sociotechnical systems and accountability gaps: who takes the blame when software and AI go wrong?

When an AI system misbehaves, blame often ricochets between the human operator, the model and the organisation. Framing AI as a sociotechnical system helps practitioners close those accountability gaps with concrete design, governance and incident practices.

Introduction: the familiar blame game

You know the scene: an automated decision goes sideways, a customer complains, and the organisation produces a statement. Engineers point to model behaviour. Legal points to contracts. The model, of course, offers no comment — it is a statistical artefact, not a moral agent. What follows is often a messy round of finger-pointing that leaves the harmed person with little clarity and organisations with slow, expensive fixes.

AI is not just code — it is sociotechnical

One of the clearest lessons from recent scholarship and policy work is that AI systems are not isolated lines of code. They are sociotechnical systems: engines of code and data embedded in organisational practices, market relations, operational infrastructure and social expectations. Treating them as purely technical solves only part of the problem; the rest lives in teams, contracts, incentives and everyday work routines (see Women in AI White Paper and TechPolicy.Press discussion).

Why accountability gaps emerge

There are several recurring reasons that accountability becomes diffuse in practice:

Diffusion of agency. Models do not have legal or moral agency, so responsibility must be traced back to people and organisations — but those parties often share partial roles in design, deployment and oversight. This fragmentation is sometimes called the “problem of many hands” (Springer Nature: Rethinking responsible AI).
Opacity and unpredictability. Modern machine learning models can be opaque and produce behaviours that are hard to predict in novel contexts, making it difficult to point to a single causal error.
Siloed expertise. Sociotechnical knowledge — sociology, anthropology, legal, policy — is often available inside organisations but underused or confined to governance teams rather than integrated into early design and engineering (Center for Democracy and Technology).
Complex supply chains. Systems are assembled from components, APIs and foundation models sourced from multiple vendors. Contracts and SLAs complicate who is accountable for which failure modes.
Regulatory and policy gaps. Governance frameworks are evolving; policy design must treat AI as sociotechnical to create workable levers beyond narrow technical compliance (TechPolicy.Press).

Two practical concepts: tracking and tracing

A useful operational pair to guide accountability work is tracking and tracing, derived from multidisciplinary efforts on meaningful human control (Minds and Machines).

Tracking asks: does the system’s behaviour align with the intentions and norms of the humans who operate or commission it? Practically, this means designing signals and controls so operators can recognise and steer model behaviour.
Tracing asks: can responsibility be allocated based on who had influence over a decision, and who had authority to change the system? Tracing demands logs, decision records and documented responsibilities that survive personnel changes.

Concrete steps for practitioners

If you build or operate AI systems, the following actions turn sociotechnical thinking into practice. They are practical, incremental and oriented towards accountability, not just compliance.

Embed sociotechnical expertise early. Bring people with social science, policy, legal and operations backgrounds into product design and prototyping, not just ethics sign-off. CDT highlights that these experts are often available but sidelined; integrate them into design sprints and architecture reviews.
Create decision and intent logs. Record why model choices were made: data decisions, parameter settings, intended operational envelope and human-in-the-loop expectations. Logs enable tracing of responsibility when outcomes deviate from intent.
Define roles with rigour. Use clear RACI-style (Responsible, Accountable, Consulted, Informed) mappings that cover model development, deployment, monitoring and incident response. Avoid vague hand-offs like “the team” — name the role and its authority.
Design human verification loops. Practitioners increasingly observe that verification, not generation, is the bottleneck (see Noam Schwartz’s discussion). Build fast human-in-the-loop verification for high-risk outputs and automate confidence thresholds to flag content for review.
Instrument for observability and context. Capture model inputs, outputs, environment signals and operator decisions. Observability must include socio-context: which templates were used, which business rule overrode the model, and who approved that override.
Stress-test in realistic socio-contexts. Test in the messy environments where your system will run, not only in pristine datasets. Include non-technical stakeholders in scenario planning to spot sociocultural failure modes.
Lean on contracts and supplier governance. When using third-party models, ensure contracts specify responsibilities for correctness, provenance, update windows and incident cooperation. Don’t assume the vendor owns “all accountability”.
Run regular incident drills. Practice cross-disciplinary incident response so engineers, product managers, legal and comms know how to act and whom to call when things go wrong. Drills reduce the reflex to punt blame publicly.
Make remediation pathways visible. Document how affected users can seek redress and how the organisation will operationalise fixes. Clarity reduces harm and helps regulators and stakeholders understand your response logic.
Use governance levers beyond tech. Policy design benefits from a sociotechnical frame: procurement rules, incentive structures, and public reporting can change behaviour as much as code-level constraints (TechPolicy.Press).

An incident workflow that reduces finger-pointing

When an incident happens, follow a short, repeatable workflow that prioritises containment, evidence preservation and clear responsibility lines.

Contain and stabilise. Limit immediate harm with rollbacks, throttles or temporary manual gates.
Preserve context. Capture inputs, outputs, operator notes and timestamps. Preserve logs for tracing.
Assemble a cross-disciplinary team. Include engineering, ops, product, legal and the sociotechnical expert who understands the organisational context.
Conduct tracing and tracking analysis. Ask: did the system act outside intended parameters (tracking)? Which decisions and agents contributed to the outcome (tracing)?
Assign corrective actions with owners and deadlines. Make the remediation plan explicit: code fix, policy change, training, contract amendment.
Communicate clearly and promptly. Tell affected users what happened, how you are fixing it, and how they can seek redress. Avoid vague comms that fuel blame games.

Closing: design accountability as a feature

Accountability gaps are not mysterious failures of ethics; they are predictable consequences of how we build socio-technical systems. The solution is practical: treat accountability as an engineering requirement, embed sociotechnical expertise early, instrument systems so responsibilities can be traced, and design human verification into operational loops.

In short: make responsibility visible before you need to explain it. If you can turn the finger-pointing into a shared flowchart, you’ll likely fix the problem faster — and spare your comms team an awkward weekend.