How to Assess the Risks of an AI System
A six-step method for assessing business AI risks: classification, data, impact on people and controls.
Reading time: 10 min
Assessing the risks of an AI system does not require a complicated method. It requires the right questions in the right order, and to distinguish two types of risk that are always confused: the legal risk, the one that interests the AI Act, and the operational risk, the one that can harm your company.
They're independent. An assistant who writes business drafts is at minimal legal risk, but if his drafts start without review it can cost you a client. On the other hand, a high-risk classified system can be handled very well and not produce any damage.
In summary
- There are two distinct risks: Legal (standard) and operational (business). Both should be evaluated.
- The decisive question is not "how much is wrong" but "What Happens When You're Wrong".
- The factor that changes everything is the degree of automation: suggest or act.
- Evaluation should be done by those who know the process, not by those who know the technology.
- A documentation page is sufficient and demonstrates government.
- It must be remade when the use changes, even at unchanged software.
The six-step method
- Describe what it does in a sentence, without jargon: «read the incoming emails and propose a response that an operator approves».
- List Data What does it touch. Is there any personal data? Confidential customer information? Business know-how?
- Establish the degree of automation: suggests, performs with approval, or performs alone?
- Identify those who suffer the consequences of a mistake: no one, a colleague, a customer, a person who depends on that decision.
- Check who notices and in what time. It is the most neglected point.
- Decide the controls proportionate to what has emerged.
Steps 3 and 5 are the ones that make the difference between a useful assessment and a form exercise.
The degree of automation changes everything
The same system, with the same error rate, has completely different risk profiles depending on how autonomous it is.
| Grade | What it does | Risk | Typical control |
|---|---|---|---|
| Suggests | Propose, decide a person | Low | None particular |
| Run with approval | Prepare for action, need a go-ahead | Medium | Approval must be informed |
| Runs autonomously | It acts without human passages | High | Limits, thresholds, logs, possibility to stop it |
Who suffers the error
It is the question that determines whether you are in the territory of the AI Act or only in that of operational common sense.
Low risk
- Error produces a draft to be corrected
- He knows who's working, right away.
- Damage is lost time
Serious risk
- Error reaches a client or candidate
- No one checks it first
- The affected person does not know that there was a system
- Damage is a wrong decision about someone
When you are in the right column, you are in the perimeter described in the four levels of risk of the AI Act, and we need the guarantees provided.
The blind spot: who notices
In most of the evaluations I see it is missing completely. Yet it is the variable that turns a manageable error into a problem.
A system that fails 5% of the time, with an operator rereading everything, produces zero damage. The same system, with the same 5%, in a stream where no one rereads, produces a mistake in twenty that arrives at its destination — and no one finds out until someone complains.
- Is there a point where a person sees the output before it produces effects?
- Does that person have a way and time to realize it's wrong?
- Is there a log to reconstruct what happened?
- Is there a quick way to stop the system?
If the answers are no, the risk is not the model: it is the process around.
Practical examples
Assistant preparing estimates. Data: technical descriptions, sometimes customer names. Automation: suggests, the holder approves. Who suffers: no one, the mistake is seen first. Who notices: who signs. Low risk on both fronts, no additional control needed.
Agent responding on WhatsApp to customers. Personal data yes. Automation: Performs autonomously. Who suffers: the customer. Who notices: no one, if you don't watch the conversations. Here the controls serve: declare that it is automatic, predict the switch to operator, periodically review a sample of conversations, have a way to turn it off.
Tool that sorts CVs by score. Personal data of candidates. Automation: suggests an order, but in fact orients the decision. Who suffers: the person excluded. Who notices: no one, because the discarded does not call back. High risk: real human surveillance, information to candidates, criteria verification are needed. See also GDPR and AI in the company.
Automation that reconciles invoices. No relevant personal data, high automation, but the error is seen in accounting and is corrected. Average operational risk, low legal: just thresholds and an exception report.
Risk-proportioned controls
Having identified the level of risk, the next question is: which controls do I put? The wrong answer is "all those possible", because disproportionate controls are bypassed and produce the illusion of security.
| Level | Sufficient controls |
|---|---|
| Low | None particular. Basic training on use |
| Medium | Human review before sending, log available |
| High | Thresholds and Limits, Informed Human Surveillance, Sample Review, Switch, Full Traceability |
Three controls deserve a note, because they are the ones that companies forget.
The thresholds. A system that acts should have explicit limits: maximum amount, maximum number of shares per hour, casuistics on which it stops and asks. They are the difference between a contained error and a mistake repeated two hundred times before anyone notices.
The sample review. You don't need to reread everything: you need to watch twenty conversations a month chosen at random. It's the cheapest way to notice that the system has begun to behave differently.
The switch. It has to exist and someone has to know where it is, it seems trivial until it's served at seven o'clock in the evening with an agent who's sending the wrong messages to customers.
The Risk of Doing Nothing
Risk assessments always tell what can go wrong if you adopt a system. They almost never consider the other side: What happens if you don't adopt it.
It's not commercial rhetoric, it's part of an honest assessment. A manual process also has an error rate, often not measured precisely because no one controls it. A manually managed email box loses requests; a manual data entry is wrong; a response that comes after three days makes the customer lose.
AI System Risks
- model Errors
- Dependence on a supplier
- Data leaving the perimeter
- Behavior that changes in updates
Risks of the manual process
- Unmeasured human errors
- Dependence on individual persons
- Times That Lose Opportunities
- No traceability of what was decided
The correct comparison is not "AI versus perfection", but "AI versus how we work now". Many evaluations fail a system with 3% error to replace a manual process that makes 7% without anyone ever measuring it.
The way out of this is to measure the starting situation before introducing anything — it's the sense of AI for SMEs: a practical guide and of KPI of an AI project. For an overview of cases where the game is worth the candle: examples of AI automation in the company.
Mistakes to Avoid
- Evaluating technology instead of use. The model does not determine the risk, the process does.
- Stopping at the error rate. The consequence counts, not the frequency.
- Not reassessing when the automation changes. From suggester to performer is a new project.
- Delegating evaluation to those who do not know the process. Only those who work know what mistakes are tolerable.
- Not planning for the switch. Every system that acts must be able to be stopped quickly.
How to apply it in the company
Take a sheet and for each AI system write six lines: what it does, what data it touches, how autonomous it is, who suffers the error, who notices, what controls you have. Ten minutes per system.
The result is a page that serves three things: deciding where checks are needed, proving government in case of verification, and noticing the cases where you are about to automatically put something that shouldn't be. Many of the causes of failure described in Why AI Projects Fail They intercept right here.
Conclusion
The risk assessment of an AI system boils down to two honest questions: what happens when it fails, and who notices. Everything else is outline.
Companies that get in trouble are not the ones that use less good models: they are the ones that have let a system go from suggesting to deciding without anyone making that decision consciously.
If you want to go from testing with ChatGPT to an AI system integrated into business processes, Giallo Studio custom designed agents and automations, with real risk sized controls. You can see some systems working in The Lab.
Giallo Studio guides: methodology and sources
How we apply it in the company
Related resources
FAQ
Do you need a risk assessment even for a simple assistant?
Not a formal assessment, but you do need to think it through, which takes a few minutes. The questions are always the same: what data does it access, who suffers the consequences of mistakes and who notices? For an assistant summarising internal documents, the answers are reassuring and the assessment ends there.
What is the difference between AI Act risk and operational risk?
The AI Act considers risks to people's fundamental rights and safety. Operational risk concerns the harm a system can cause your business: mistakes in customer replies, poor decisions and interruptions. A system can be minimal-risk under the regulation and high-risk for you.
How do you assess the risk of the system being wrong?
It is not enough to ask how often it makes mistakes: ask what happens when it does. An error producing a draft to correct has negligible impact; the same error rate in messages sent automatically to customers is a problem. What matters is frequency multiplied by consequence.
Who should do the assessment in an SME?
Someone who knows the process, rather than the technology. The person performing the activity every day knows which errors are tolerable and which are not, and that is the decisive information. Technical support helps explain how the system works, rather than determine what is acceptable.
How often does it need to be redone?
Every time the use of the system changes, the data source or the degree of automation. The most typical case is the transition from suggestion to automatic execution: the same system that previously proposed and now acts has a completely different risk profile, even if the software is identical.
Do I need to document the assessment?
A page is sufficient in most cases and is more useful than it seems: it demonstrates governance during an inspection or dispute, and forces reasoning to be made explicit when it would otherwise remain implicit and unshared.