AI Pilot Project: How to Set It Up
Set up an AI pilot with scope, duration and predefined success criteria. Prevent it from becoming an endless prototype.
Reading time: 9 min
A pilot project is a realization Limited in perimeter but true in operation: it is used by those who work, on real work, for a defined period. It is not a prototype that demonstrates feasibility, and it is not a full shrunken project.
Its function is to answer a question that no analysis can solve: is it really necessary, in our specific context? And that's a question you answer only by putting the system in the hands of those who do that work.
In summary
- The pilot enters the Real Work: is what distinguishes it from a prototype.
- Restricted perimeter but full operation, not a halved version.
- The Success Criteria must be written first, in one line.
- We need one. End Date, or stay in the pilot forever.
- It has to be done with who actually does that process, not with a sample.
- A pilot who says "doesn't fit" worked, not failed.
How to set
- Perimeter: a process, a type of request, a department. Written in a line.
- Starting Measure: as it is now, with numbers collected before we start.
- Success criterion: what needs to happen for it to be worth extending.
- Closing criterion: what would make you decide to stop.
- Duration: a date, not "until it works".
- Who uses it: the people who do that job every day.
- What is measured during, and who looks at it.
Points 3 and 4 are the ones that almost nobody writes and that determine whether the pilot will produce a decision. Without it, we eventually discuss for weeks whether "it went well", with positions that depend on how much everyone has grown fond of the job.
Pilot, prototype and proof of concept
It's three different things that are used as synonyms.
| Question he answers | Get into real work | Duration | |
|---|---|---|---|
| Proof of concept | Can it be done technically? | No | Days |
| Prototype | What do you mean? | No | Days or weeks |
| Pilot | Does it serve, in our context? | Yes | Months |
The most expensive confusion is between proof of concept and pilot: you build something that demonstrates feasibility, you show it in a meeting, everyone is excited, and you go straight to the full project without ever verifying whether that process, in the real work, benefits. The theme is deepened in proof of concept AI.
What to measure during
- The number you had measured before: hours, errors, waiting times
- How many times the system was actually used, not how many was available
- How many exceptions he encountered and what kind
- How many times have people bypassed the system, and why
The last point is the most informative and nobody picks it up. If those who work go back to the old method for certain cases, those cases say exactly where the system doesn't hold up — and they say it better than any questionnaire.
The pilot who doesn't finish
It's the most common failure and the most difficult to recognize, because it doesn't seem like a failure: the system works, it's used, nobody complains. You just never decide whether to extend it.
Healthy Pilot
- It has a written end date
- It has criteria decided before
- Someone looks at the numbers during
- At the deadline, a decision is made
Pilot dragging
- "Let's see how it goes"
- Functions added during
- No one collects data
- After a year he is still a pilot
The second point of the right column is the most frequent cause: adding functions during the pilot shifts the target and makes it impossible to evaluate what was being evaluated.
Practical examples
Pilot on a type of document. A company that receives invoices from a hundred suppliers starts from twenty, the most recurring ones. Criterion: If eighty percent goes uncorrected in three months, it extends. Clear perimeter, simple measurement, automatic decision.
Pilot on a ward. An internal assistant activated only for the administration before for the whole company. It allows to correct the wrong answers with a restricted and tolerant audience.
Pilot closed with negative result. One company has tried to automate returns management. After two months the exceptions were more numerous than the standard cases and the system was bypassed regularly. Closed, with the decision documented. Low cost, valuable information: that process needs to be simplified first.
Who is to participate
The composition of the group that uses the pilot determines the quality of the information that comes out of it, and is almost always chosen with the wrong criterion: you take the most available or the most enthusiastic people.
Instead, you need the people who do that job. every day, including skeptical ones. Those who are enthusiastic tend to justify the limits of the system and not report problems; those who are skeptical report them all, and that is exactly what is needed at this stage.
- Include those who carry out the process daily, not occasionally
- Include at least one skeptical person: they will signal what others are silent about
- Avoid involving only those who proposed the project
- Make it clear that reporting problems is the task, not a complaint
The last point must be said explicitly. In many companies reporting that something is not working is perceived as opposition to the project, and people are silent about quiet living. A pilot in which no one reports anything is not a successful pilot: he is a pilot without information.
An incentive can fund testing, but the project must work even after its expiration. The program Claude for Startups with Team and API credits is a concrete case: the application, the suitability and the cost after the bonus are part of the pilot's plan.
What to do at the deadline
Once the date has arrived, there are four possible decisions and they must be named first in order not to find themselves improvising.
Extend. The criteria are met, the perimeter widens. It is the simplest decision but it must be accompanied by the question: extend to what, exactly?
One-time extension. The data is promising but insufficient. It should be granted once, with a new date and without adding functions, otherwise it becomes the infinite pilot.
Resize. The system works on part of the cases and not on others. You keep the part that works and you give up the rest: it is a frequent outcome and often the best.
Close. The criteria are not met. It is a decision that must be made without dramatizing: it produced information that was worth the cost of the pilot, and avoided a full wrong project.
The Pilot Compared to Alternatives
| Approach | What it proves | Risk |
|---|---|---|
| No proof, full project | None | High: it all turns out in the end |
| Proof of concept | Technical feasibility | Medium: doesn't say if it's needed |
| Pilot | Utilities in the Real Context | Low: contained perimeter |
| Infinite Pilot | Nothing, after a while | High: consumes without deciding |
The last line is the one where you end up unwittingly, and that's why the end date should be written along with the rest.
Mistakes to Avoid
- Not writing the success criterion beforehand. In the end, it is discussed endlessly.
- Not planning for closure. The pilot drags on forever.
- Adding functions during. Move the target.
- Not measuring the starting point. It makes evaluation impossible.
- Do it with those who don't do that job. He won't meet the real exceptions.
- Confusing it with a proof of concept. They answer different questions.
How to apply it in the company
Take a paper and write five lines: what the pilot covers, what I measure, what it is now, what I consider a success, when I decide. If you can't write all five of them down, the pilot is not ready to go.
Then choose the two or three people who do that work every day and involve them in the definition, not just in the use. The exceptions they know are the ones that will decide the outcome.
The complete picture is in AI project: how to get started, the metrics in KPI of an AI project.
The pilot and the rest of the company
An aspect that is overlooked: while the pilot is in progress, the rest of the company looks. And what he sees determines how the extension will be welcomed.
If the pilot is told how "we're trying something", people observe it with curiosity and without fear. If presented as "the beginning of digital transformation", it generates expectations and anxieties disproportionate to an experiment on a process.
It is also worth deciding what to communicate at the end, whatever the outcome. A closed driver with no explanation leaves the impression that it went wrong, even when the decision was simply to resize or postpone. Saying what you learned costs five minutes and keeps the door open for the next project.
The framework of metrics to follow during is in KPI of an AI project, and the typical causes of failure in Why AI Projects Fail.
Conclusion
A pilot serves to answer only one question —does he really serve us? eand he succeeds only if he has a tight perimeter, criteria written before and an end date.
The value is not in proving that AI works: we know that. It's in discovering, with real data, if it works in this process with these people. And it is an information that is also valid, and perhaps most importantly, when the answer is no.
If you want to understand what processes in your company can be automated with AI, Giallo Studio can help you turn the problem into a real workflow.
Finally, keep what you learned in written form, even a few lines: what you felt, what emerged, what you would decide differently. In SMEs this memory does not exist, and two years later we start from scratch on the same topic with the same questions.
Giallo Studio guides: methodology and sources
How we apply it in the company
Related resources
FAQ
What is an AI pilot project?
It is an implementation with limited scope but complete functionality, deployed to production for part of the real workload for a defined period. It differs from a prototype because the people doing the work actually use it, and from a full project because its scope is deliberately narrow.
How long should the training last?
Enough to collect significant data, typically a few months of real use. Less is not enough because you don't see the exceptions; much more is a project that never decided to finish, and in practice it stays in the pilot forever without anyone evaluating it.
How do you define if it went well?
By writing down in advance, in one line, what you would consider success. “Handling time falls below X”, “errors fall below Y”. Without a criterion written in advance, the final evaluation depends on the evaluator's mood and their attachment to the work done.
What distinguishes a pilot from a prototype?
A prototype demonstrates that something is possible without entering the real workflow. A pilot enters the real workflow within a reduced scope and produces data on what actually happens. The former answers “can it be done?”, the latter “is it useful?”.
What happens if the pilot fails?
It is a legitimate outcome and must be anticipated from the start, with a written criterion for closing it. A pilot that demonstrates a process is not worth automating has produced useful information at a contained cost: that is exactly what it was meant to do.
How many people should take part?
The few people who actually perform that process, rather than a representative sample. In an SME, this often means two or three people. What matters is that they are the people doing the work every day, because they are the only ones who will encounter the real exceptions.