Measuring AI and software productivity realistically: expect a phased curve

Organisational productivity gains from AI and software arrive unevenly. Track short-term leading indicators, design medium-term business metrics, and avoid conflating model scores with realised value. Practical steps for measuring through the phased adoption curve.

Why a tidy metric rarely maps to organisational value

When teams pilot a new AI system or software workflow, it’s tempting to celebrate a neat model metric or a lab-based time saving and call the job done. The sociotechnical reality is messier. Systems interact with users, existing processes, incentives and capital allocation decisions. That interaction produces a phased adoption curve: early disruption, a reconfiguration period, and only then the potential for durable productivity gains.

Expectation management: the phased curve

Practitioners should expect three broad phases.

Initial disruption. Introducing new tools often creates friction. Roles shift, tasks get redistributed, and people learn new ways of working. The original article highlights that some sectors — notably manufacturing in past studies — saw substantial early declines before recovery over several years. That matches other analyses that caution against simplistic summaries of success.
Reconfiguration. Teams adapt processes, managers redesign flows, and organisations begin to reinvest savings into higher-value activities. During this window, task-level automation gains can appear while aggregate throughput or revenue metrics lag.
Realised uplift. If redesign, training and reinvestment happen effectively, organisations see sustained gains: higher throughput, better quality and improved revenue per head. Early adopters may report strong task-level improvements, but these need time and systems change to translate to firm-level productivity.

What to measure, and when

Design a measurement portfolio that covers short, medium and long horizons.

Short-term leading indicators (weeks to months): time saved on specific tasks, error or rework rates, user satisfaction and adoption rates. These tell you if the tool is doing what it promises at the task level and whether people are willing to use it.
Medium-term business metrics (quarters): throughput, cycle time, revenue per head, customer response times and defect rates. These show whether task-level gains aggregate into operational improvements.
Long-term strategic outcomes (year+): market share, profit margins, cost structure changes and capability shifts. These depend on how organisations reinvest gains and adjust strategy.

Practical measurement design

Good measurement is not one-off reporting; it’s experiment design plus continuous learning.

Baseline first. Capture pre-adoption measures for every indicator you plan to report. Without a baseline you can’t tell whether a change is due to the new system or background noise.
Use control groups or phased roll-outs. Comparing similar teams that adopt at different times helps isolate the effect of the technology from other variables.
Mix quantitative and qualitative signals. Numbers show scale, qualitative feedback explains why. Regular interviews, support logs and ethnographic notes are inexpensive but revealing.
Measure adoption, not just capability. Model accuracy or latency means little if users bypass the tool. Track daily active users, task completion with vs without the tool, and time-to-first-value for new users.
Track reinvestment. Note where time or cost savings are redeployed — training, new features, capacity for innovation — because realised productivity depends on these decisions.

Avoid common traps

Don’t conflate a tidy model metric with organisational value. A 95% model F1 is valuable only if the organisation can absorb and act on its outputs.
Don’t cherry-pick short-term wins. Task-level gains of 14–55% reported by some early adopters are encouraging, but they may not imply instant firm-level productivity — the conversion takes time and systems change.
Don’t let incentives mislead measurement. If bonuses reward custodial metrics (e.g. number of cases closed), teams may optimise those instead of genuine customer or throughput improvements.

Operational advice for managers and practitioners

Plan measurement as part of rollout. Make measurement a deliverable of the project, with clear owners and reporting cadence.
Run short learning cycles. Use A/B or staggered deployments to learn fast and avoid wholesale disruption.
Invest in change management. Training, playbooks and role redesign accelerate the move from task gains to business outcomes.
Be explicit about time horizons. Communicate that early dips are possible, but that the organisation is prepared to manage and measure the reconfiguration phase.

Bringing it back to sociotechnical systems

Software and AI operate inside human organisations. Measuring success therefore needs to recognise socio-technical interactions: incentives, routines and capital allocation all shape whether an algorithmic improvement becomes a sustained productivity gain. As Forbes and other practitioners remind us, how you measure shapes what you manage — design your metrics to reflect the phased, sociotechnical nature of change.

Expanded from

This post expands on a point from Sociotechnical systems in software and AI: Who actually captures the productivity gains?.