AI project KPIs: what to measure

AI projects are measured poorly in a predictable way: one looks at how good the system is and ignores how much it is used. The first number is technical, the second decides the fate of the project.

Useful metrics are divided into three families, and they serve all three. By themselves, each can tell a false story.

The Three Families

The third family is the one missing in the reports and explaining the failures. A system with excellent process and quality metrics but low adoption is a system that works on a fraction of the cases, and that data remains invisible until someone searches for it.

The Five Numbers That Enough

1. The starting number, remeasured by the same method: hours, errors, times.
2. System-Managed Volume with respect to the total applicable volume.
3. Necessary Corrections: how many times the output needs to be changed before using it.
4. Exceptions: how many and of what kind, with the trend over time.
5. Overrides: how many times you have returned to the old method, and for what cases.

Point 2 requires knowing the denominator, i.e. how many occasions there have been. It is the number that few calculate and that reveals whether the system covers ten or ninety percent of the actual work.

Because accuracy alone deceives

It is the preferred metric in presentations and the least significant on its own.

A system with ninety-five percent accuracy can be excellent or disastrous depending on what happens with the other five percent. If the person checking catches the error before sending, it costs a few seconds. If it reaches the customer, the cost is quite different.

Metrics that are not needed

It is worth listing them, because they consume attention without producing decisions.

• Number of requests handled without denominator: it always grows, says nothing
• Generic Satisfaction collected with a questionnaire: no one responds poorly to hot
• Average time without distinguishing the types of request: mixes different things
• Comparisons with industry benchmarks: processes vary too much between similar companies

The last point deserves attention because it is what suppliers propose most often. An industry benchmark on processes you don't know in detail is not a comparison term: your comparison term is you first.

Practical examples

Document automation. The five numbers: documents handled on total received, percentage passed without corrections, exceptions per type, time per document, cases still inserted by hand. All collectable with a few minutes a month.

Customer Assistant. Total managed conversations, how many required an operator, how many generated a second request from the same customer, time to first response. The third is the most honest: it measures whether the problem has really been solved.

Project with low adoption. Excellent quality metrics, but only thirty per cent of the opportunities to use it went through the system. Asking why revealed that the system had not been configured for two common request types. A two-day fix increased adoption.

How they collect without becoming a job

The most common objection to measurement is that no one has time to do it. It is founded if one imagines a system of structured detection, and unfounded if one accepts that five approximate numbers are worth more than twenty precise ones that no one collects.

• Prefer an automatic count where it exists, even if imperfect
• Where manual detection is needed, reduce it to a tick
• Set a fixed time: the first Monday of the month, ten minutes
• Assign collection to those who look at numbers, not those who produce them

The last point avoids the conflict of interest: if those who collect the data are also those who pushed for the project, the collection tends to become optimistic without bad faith.

The metric that says when to stop

There is an indicator that almost no one defines and that is worth having: the threshold beyond which the project is to be reviewed.

It is not necessary to give up the first hitch, it is necessary not to drag something that does not work endlessly. It can be a minimum adoption level after three months, a rate of corrections that does not fall, a number of circumventions that grows.

Defining it first costs nothing and changes the conversation: when that number is exceeded, there is no discussion of whether the project is going well. it has already been decided previously what to do. It is the same principle as the closing criterion described in AI Pilot Project, applied to the exercise phase.

The five numbers in the table

The first four are derived from systems; the fifth is not, and must be asked. It is also the most useful.

Mistakes to Avoid

• Measuring only technical quality. It hides adoption issues.
• Not having the denominator. Volume without occasion does not inform.
• Not asking about workarounds. It is the most useful information.
• Building dashboards that no one opens. Five numbers is enough.
• Comparing with external benchmarks. The comparison is you first.
• Stopping measuring after release. Drift appears later.

How to apply it in the company

Prepare a sheet with five columns and one line per month. Fill it out yourself or have the project contact fill it out: ten minutes per month.

In the first two months look at it every week, because that's where you intercept problems while they're small. Then monthly. And every three months you ask the question about bypasses, which does not appear in any automatic monitoring system.

The return calculation is in ROI of an AI project, the initial setting in AI Pilot Project.

What to look at in the long run

After the first few months the metrics change function: they no longer serve to correct the system, but to notice that something around has changed.

The typical signal is a stable metric that slowly begins to get worse: corrections increase, exceptions grow, adoption declines. Almost never the cause is in the system: it's an upstream update, a modified process, a new category of cases that didn't exist before.

For this reason the monthly review must be maintained even when everything is fine. It costs ten minutes and intercepts the degradation described in how much it costs to maintain an AI system, which is the most common way a successful project silently stops working.

Measurement also counts in advertising within AI services. With the new ChatGPT Ads attribution formats and tools, the useful control remains that on qualified leads and sales, considering geographic availability and event configuration.

Metrics in the pilot phase

During a pilot metrics have a different function: they do not serve to manage the system but to make a decision within a date.

This changes what to look at. Above all, the comparison with the starting number and the adoption rate is needed; fine quality metrics count less, because at that stage the system is by definition to be calibrated.

• Direct comparison with initial measurement, same method
• How many times has it been used on how many occasions
• What categories of cases does it not cover
• If the trend improves week after week

The last point is the one that orients the final decision: a mediocre system that constantly improves deserves the extension more than a good but flat one, because the first is still learning from the context and the second has already given what it could. The complete setting is in AI Pilot Project.

Conclusion

Measuring an AI project requires five numbers and constancy, not sophisticated tools. Process metrics say if the work is better, quality metrics if the system is reliable, adoption metrics if all of this is really happening.

The question that is worth more than any dashboard remains the simplest: in what cases do people continue to do it by hand? The answers to that question are the list of things to fix, compiled by those who know the job.

If you want to understand what processes in your company can be automated with AI, Giallo Studio can help you turn the problem into a real workflow.

One last recommendation: share the numbers with those who use the system, not just those who decided to buy it. People who see the effect of their work tend to report more and adopt better, and often have explanations for the deviations that no one else can provide.