Measuring AI risk and return

Executive summary

Every AI pilot comes with a business case. Very few come with a way to check it. Six months in, the sponsor says the system saves time, the finance team can't find the saving, and the risk team has no record of what went wrong. The board is asked to fund the next phase on belief.

That is a common way for AI work to end. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, and names three causes: rising costs, unclear business value and weak risk controls. All three are measurement problems. You can't defend a cost you never tracked, or a gain you never baselined.

Our view is that return and risk belong on the same page, measured from the same log. An AI system is worth running when each completed task costs less than it did before, quality holds, and every known risk has a named owner. This article sets out the measures we use, how to take a baseline before launch, and how to put the result in front of a board.

1. Why AI business cases are hard to check

Three habits make AI results hard to prove.

• No baseline. Teams measure the new process with care and guess at the old one. The saving is then the gap between a number and a guess.
• Counting the wrong unit. "Prompts answered" or "hours saved" say little. What matters is work finished to the right standard, at a known cost.
• Return and risk in separate rooms. Finance tracks spend. Risk tracks incidents. Nobody sees that the cheapest month was also the month approvals were waved through.

The firms that get value from AI tend to fix this early. In McKinsey's The state of AI in 2026, the small group of high performers (6% of respondents) were about twice as likely as others to report defined processes to measure the impact of AI. Measurement is not paperwork added at the end. It is part of how the value gets made.

HMR insight: If you can't show what a task cost before the AI arrived, you can't show what the AI saved.

2. Take a baseline before launch

The baseline is the one step you can't do later. Once the new system is live, the old process is gone.

Start with a single workflow and define its unit of work: one invoice matched, one claim settled, one support case closed. Then measure how that work runs today, on real cases, for several weeks:

• Cost: people time per case, from a time study or system timestamps, at a loaded hourly rate agreed with finance.
• Cycle time: from request in to outcome out, including waiting time.
• Quality: errors found by a reviewer in a random sample, plus cases reopened or redone.
• Risk: incidents, complaints and audit findings linked to the workflow over the past year.

Two rules keep the baseline honest. First, measure the same way you will measure the new system, from the same sources. Second, where you can, keep part of the work on the old path for the first weeks of live running. A side-by-side comparison beats a before-and-after one, because it removes seasonal swings and other changes happening at the same time.

3. Eight measures that answer two questions

The board has two questions about any AI system. Is it worth running? Is it safe to keep running? We track four measures for each.

Every measure needs a definition, a data source and an owner, agreed before launch. A shared table stops arguments later.

Most of these come from the audit trail described in Part 2. If every tool call and approval passes through one gateway, the measures are a query, not a project.

4. Cost per completed task: the number finance needs

Cost per task is the clearest test of return, but only if both halves are honest.

The top half is the full cost to run the system for a period: model usage, hosting, the gateway and monitoring, and the people time spent reviewing, approving and fixing its work. Leave out review time and a system can look cheap while it keeps a team busy checking it. The bottom half is tasks completed to standard, with no rework. Tasks the agent attempted and a person then redid don't count.

The FinOps Foundation's guidance on AI costs takes the same line. It suggests pairing token and inference costs with measures of the work done, such as cost per call or time to close. For the raw usage, the OpenTelemetry conventions for generative AI define standard fields for input and output tokens on each model call. Tag each call with the workflow and task it served, and model cost per task follows.

Watch the trend as well as the level. Model prices, prompt length and retries all move. A system that was cheap at pilot volume can drift upward as edge cases pile up.

5. The risk measures

Approval override rate

This is the share of tier 2 and tier 3 steps where the approver rejects or edits what the agent proposed. It is the best early signal you have, and it needs reading in both directions.

A rising rate means the agent is getting things wrong, often after a data or model change. A rate near zero for months, with approvals given in seconds, may mean people have stopped reading. Part 4 covers how to design approvals that stay meaningful. Here, track the override rate next to the time each approval took, and review a sample of approved steps each month.

Incidents and near misses

Agree severity levels before launch and count incidents by level. Count near misses too: a wrong refund caught at the approval gate is a free lesson. For each incident, record the time to detect and the time to stop. Those two numbers tell you whether your stop controls work. Part 6 covers the response itself.

Drift from what you tested

The NIST AI RMF playbook asks organizations to monitor how measures seen in production differ from those collected in testing (MEASURE 2.4). In practice, run the same test set after every model or prompt change, and compare live quality against the pre-launch result each month.

A risk register with owners

Measures show what is happening. The register says who acts when a limit is crossed. Keep it short, specific to the workflow and reviewed monthly.

Each owner is a named person, not a team. Data risks link to the boundaries in Part 3. If you run an AI management system under ISO/IEC 42001, this register can sit inside it.

6. Presenting to the board

Boards don't need a dashboard. They need one page each quarter, with a decision at the bottom.

We lay the page out in four parts:

1. Return. Cost per completed task and cycle time, each against the baseline, with the trend.
2. Quality. Error and rework rates from the sample, against the limits agreed at launch.
3. Risk. Override rate, incidents by severity, and any register items past their limit, each with its owner.
4. Decision. Widen, hold or stop, with the reason. If you want more scope or more autonomy, say which gate you want to loosen and what evidence supports it.

Regulators expect this kind of oversight to reach the top. The Reserve Bank's FREE-AI committee report calls for each regulated entity to have a board-approved AI policy, with standard formats for AI incident reporting (summary by Chambers and Partners). A quarterly page built from the log is the simplest way to show the board oversees the system as well as the policy.

7. Illustrative scenario: a manufacturer's payables team

This scenario is illustrative. The company is fictional.

Beas Auto Components wants an agent to match supplier invoices to purchase orders and goods receipts. Matched invoices go to a finance officer as payment proposals (tier 2). Payments above a set value need a second approver from risk (tier 3).

Before build, the payables team times a sample of invoices across a full month-end. They log how long each takes, how many are exceptions, and how many payments later need correcting. That is the baseline. For the first eight weeks of live running, one supplier group stays on the manual path as a comparison.

The first board page shows cost per matched invoice and cycle time against both the baseline and the comparison group. It shows the override rate, and a note that approvals on large payments take real time, which is the sign that people are reading them. One near miss is listed: a duplicate invoice caught at the gate. It has an owner and a fix. The decision asked is to add a second supplier group and keep every gate as it is. The board can agree because every line on the page traces back to the log.

8. Putting it in place

1. Assess. Our three-week Readiness Audit picks the workflow, defines its unit of work, agrees the eight measures and their owners, and starts the baseline.
2. Build. Wire the measures into the gateway and the finance data while the system is built, so the first live day produces the first data point.
3. Run and improve. Report monthly to the process owner and quarterly to the board. Loosen a gate only when the measures support it.

Moving forward

Pick the AI system you are most likely to be asked about at the next board meeting. Write down its unit of work and what one unit costs today. If nobody can answer, start there. To see how we govern AI in our own delivery, read our AI governance policy. To have the baseline and measures set up with you, book a readiness audit.

Sources

1. Gartner, Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027, press release, 25 June 2025
2. McKinsey, The state of AI in 2026: On the road to ROI, August 2026
3. FinOps Foundation, FinOps for AI overview
4. OpenTelemetry, Semantic conventions for generative AI
5. NIST, AI RMF Playbook: Measure
6. Reserve Bank of India, FREE-AI committee report, August 2025 (summary by Chambers and Partners)
7. ISO, ISO/IEC 42001:2023 AI management systems