# Bias and prejudice in AI models: where businesses go wrong and how to avoid it

> How an SME can spot prejudice in AI models and limit it: where it comes from, five mistakes to avoid and a practical human check.

Source: Giallo Studio (giallostudio.it) · Page: https://giallostudio.it/en/blog/bias-pregiudizi-modelli-ai-pmi · Published: 2026-11-12 · Updated: 2026-11-12

A **bias** (or prejudice) in an AI model is a systematic distortion in its answers, arising from the data the model was built on and from the human choices made along the way. An SME can spot it by comparing the model's answers with the judgment of a competent person, and can limit it by always keeping a human check before text leaves the company.

The starting point is to dismantle a comfortable idea: that a model is "neutral" because it is a machine. According to [Google's machine learning course](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en), models are not objective by nature: those who train them feed them examples, and human intervention in collecting and curating that data can make predictions subject to distortions.

In this article we look at where prejudice comes from, how it shows up in the work of a small company or a professional firm, which mistakes are made most often and how to set up a simple check. Where the sources give no practical guidance, we say so, without filling the gaps.

<div class="key-points">
<p class="kp-title">In summary</p>
<ul>
<li>An AI model can reproduce prejudice present in the data it was built on, including sexist or racist content.</li>
<li>Even fine-tuning the model on your own data does not fully remove the starting bias.</li>
<li>Anthropic asks users not to use Claude as the only source of truth, and Google asks them not to treat Gemini's answers as professional advice.</li>
<li>The first useful check is comparison with a competent human assessment, not trust in the result.</li>
<li>The sources do not indicate thresholds or sampling methods: each company must define them for its own case.</li>
</ul>
</div>

## What does "bias" really mean in an AI model?

It is also called prejudice or distortion. It is not an occasional flaw that appears only in extreme cases: it is a characteristic that can enter the model from the very source material. The [Hugging Face guide](https://huggingface.co/learn/llm-course/it/chapter1/8) warns that anyone using a pre-trained model or a fine-tuned version in production must remember that models are powerful tools but have limits.

The main limit, according to the same guide, is that to pre-train a model on a large amount of data, researchers often include all the content they can access, taking the best and the worst of what the internet offers. For people working in a company the consequence is concrete: the model can generate sexist, racist or homophobic content, and fine-tuning on your own data does not erase this intrinsic bias.

It is useful to distinguish bias from "hallucinations", meaning invented information. They are different problems: if you want to explore the latter, you will find an analysis in [AI hallucinations and how to avoid them](https://giallostudio.it/en/blog/ai-hallucinations). Here we deal with the first family of errors, the one that stems from distortion.

## Where do a model's prejudices come from?

Google's documentation lists several types of bias that can enter the data or the way it is handled. You do not need all of them in an SME, but knowing them helps you ask the right questions of your supplier and your own team.

| Type of bias | What it means | Useful question in the company | Source |
|---|---|---|---|
| Historical bias | Historical data reflects inequalities that existed in the world at that time | Are we using data from the past to decide about the future? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |
| Reporting bias | The frequency of events or outcomes in the dataset does not reflect their real frequency | Are the cases the model has seen representative? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |
| Confirmation bias | Those building the model process the data in a way that confirms beliefs they already hold | Did whoever prepared the data already want a certain result? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |
| Implicit bias | Assumptions based on one's own personal experience are taken as valid | Do our assumptions also hold for other customers? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |
| Selection bias | Examples are chosen in a way that is not representative of the real distribution | Who is missing from the examples? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |
| Sampling bias | True randomness was not used during data collection | How was the data collected? | [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) |

An interesting detail from the Hugging Face guide: the problem appears [even with BERT](https://huggingface.co/learn/llm-course/it/chapter1/8), which is one of the rare Transformer models not built by gathering data of every kind from the internet, but using apparently neutral data such as English Wikipedia and BookCorpus. So even a starting base that looks clean does not guarantee the absence of distortions.

<div class="callout">
<p><strong>Remember:</strong> "neutral data" is an impression, not a guarantee. Always ask your supplier where the data comes from and how it was checked, and write down the answer.</p>
</div>

## Why do we trust automatic answers too much?

There is a second mechanism, which concerns people and not the model. Google's guide defines **automation bias** as the tendency to favor results produced by automated systems over those of non-automated systems, regardless of the error rates of each. In other words: if the answer comes from the screen, we tend to believe it.

The same page offers a useful example, that of a sprocket manufacturer. The machine learning engineers were excited to put into production the new "revolutionary" model trained to detect tooth defects, until the factory manager pointed out that the model's precision and recall were both 15% lower than those of human inspectors, as you can read in [Google's fairness course](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en). The figure refers to that specific case and should not be read as a general statistic: the lesson is that comparison with those who do the work by hand comes before enthusiasm.

Suppliers' guidance points in the same direction. The [Anthropic help center](https://support.claude.com/it/articles/8525154-claude-sta-fornendo-risposte-scorrette-o-fuorvianti-cosa-sta-succedendo) explains that users should not rely on Claude as the only source of truth and should carefully examine any high-risk advice. It adds that Claude may show citations that seem authoritative or convincing but are not grounded in fact. The [Gemini apps guide](https://support.google.com/gemini/answer/16279220?hl=it) says that answers about people can be inaccurate or inappropriate, that Gemini can hallucinate, presenting inaccurate information as factual, and that its answers should not be treated as equivalent to medical, legal, financial or other advice.

## How does the problem show up in an SME?

The three examples that follow are hypothetical: they serve to show where the risk described by the sources can touch day-to-day work. They are not client cases and do not report measured results.

**A recruitment agency in a provincial town.** It uses a model to summarize résumés and propose a first ranking. If historical data reflects unbalanced past hiring, the historical bias described by [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en) can push the model to repeat those choices. Here human review is not optional: the ranking is a draft, and every rejection must be justified by a person. For a broader picture of the topic, there is a guide on [AI for recruiting and staff selection](https://giallostudio.it/en/blog/ai-for-recruiting).

**An online clothing store.** It entrusts an AI assistant with the first reply to customers and the writing of product descriptions. The risk, according to the [Hugging Face guide](https://huggingface.co/learn/llm-course/it/chapter1/8), is that the model produces stereotyped or offensive expressions. A simple rule: no text that talks about people, or to people, goes out without being read by a staff member.

**An accounting firm.** It uses AI to prepare draft communications to clients. The answers are not advice: [Google for Gemini](https://support.google.com/gemini/answer/16279220?hl=it) asks users not to treat them as equivalent to medical, legal, financial or other advice, while [Anthropic for Claude](https://support.claude.com/it/articles/8525154-claude-sta-fornendo-risposte-scorrette-o-fuorvianti-cosa-sta-succedendo) asks users to examine high-risk advice with care. A competent person must reread and approve the content before it is sent.

## Which mistakes should you avoid to keep from being misled by bias?

There are five recurring mistakes. Each has a recognizable cause and a remedy that does not require sophisticated tools.

### Believing a model is objective because it is a machine

It happens because the result arrives in an orderly, confident way. But, as [Google](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en)reminds us, models are not objective by nature and human curation of the data can make them subject to distortions. How to avoid it: treat every answer as a proposal to be evaluated, not a verdict.

### Thinking that fine-tuning on your own data solves everything

It is a widespread belief: "we train it on our documents and it becomes reliable". The [Hugging Face guide](https://huggingface.co/learn/llm-course/it/chapter1/8) says instead that even fine-tuning on your own data cannot make the original model's intrinsic bias disappear. How to avoid it: after every customization, repeat the tests on sensitive cases.

### Using the model as the only source of truth

It happens when time is short and the answer seems convincing. Anthropic recommends the opposite, that is, to [carefully examine high-risk advice](https://support.claude.com/it/articles/8525154-claude-sta-fornendo-risposte-scorrette-o-fuorvianti-cosa-sta-succedendo). How to avoid it: identify in writing which decisions (staff, terms for customers, advice on sensitive topics) always require a second human check.

### Trusting citations just because they seem authoritative

A text with precise references gives an impression of solidity. The sources warn that citations can sound convincing without being grounded in fact, as [Anthropic](https://support.claude.com/it/articles/8525154-claude-sta-fornendo-risposte-scorrette-o-fuorvianti-cosa-sta-succedendo)explains. How to avoid it: always open the cited source and check that it really says what the model attributes to it; a useful checklist is in [verifying AI answers before using them](https://giallostudio.it/en/blog/verificare-risposte-ai-prima-di-inviarle).

### Not comparing the model with human work

This is the sprocket mistake: the tool is adopted because it is new and is not measured against the people who do the job today. How to avoid it: before adopting it, have a competent person assess a set of real cases and compare the two assessments, as the [Google example](https://developers.google.com/machine-learning/crash-course/fairness/types-of-bias?hl=en).

## How to apply it: a simple check in five steps

The sources do not indicate thresholds or sampling methods, so what follows is a working method derived from the principles we have seen, not an official procedure.

<ol class="steps">
<li><strong>Choose the process.</strong> Start with a single use, for example replies to customers, and decide who is responsible for it in the company.</li>
<li><strong>Collect real cases.</strong> Put together typical examples and a few sensitive cases, involving people, gender, origin or age.</li>
<li><strong>Have a person assess them.</strong> A competent colleague judges the same cases, without seeing the model's answer first.</li>
<li><strong>Compare and note.</strong> Where the model and the person diverge, write down why and what kind of distortion it is.</li>
<li><strong>Decide the rule.</strong> Establish which texts always require human reading and test again every time you change model, prompt or data.</li>
</ol>

To fit these steps into a broader framework you need two internal documents. The first is a [corporate policy on the use of AI](https://giallostudio.it/en/blog/company-ai-use-policy), which says what can and cannot be done. The second is a risk assessment: you will find it explained in [How to Assess the Risks of an AI System](https://giallostudio.it/en/blog/ai-risk-assessment). If the system touches personal data of candidates or customers, also look at [AI and GDPR](https://giallostudio.it/en/blog/ai-and-gdpr) and the [AI Act obligations for an SME](https://giallostudio.it/en/blog/ai-act-sme-obligations).

<ul class="checklist">
<li>I know where the data of the model I use comes from, or I have asked the supplier.</li>
<li>I have specified in writing which decisions cannot be left to the model.</li>
<li>A competent person rereads texts that talk about people or to people.</li>
<li>I periodically compare the model with human judgment.</li>
<li>I check cited sources before forwarding them.</li>
</ul>

## What do we still not know?

It is right to say so clearly. The materials consulted do not explain how to measure the frequency of prejudice in a specific company, nor which thresholds to use to decide that a model is acceptable. They report no statistics on how widespread the problem is in SMEs, and for this reason we cite none.

The question of who is legally answerable for the result produced by the model also remains open: the sources do not address it. On this point professional advice is needed, and this article is not legal advice.

## What can a company do right away?

An SME can start in a concrete way: choose a single process, collect a few real cases, have a person assess them and compare the results with those of the model. It can then write a simple rule on which texts do not go out without human reading. These are small steps, but consistent with what the sources say: the model is a powerful tool with limits, not a source of truth.

If you want to set up this check together with someone who knows your company's processes, you can start with an [AI Consulting](https://giallostudio.it/en/ai-consulting) engagement to understand where AI really helps and where more caution is needed.

## Frequently asked questions

### What is bias in an AI model?

It is a systematic distortion in the model's answers, arising from the data used to build it and from human intervention in collecting that data. According to Google's documentation, models are not objective by nature. For this reason, their predictions can be subject to prejudice.

### Does fine-tuning the model on my own data remove prejudice?

No. The Hugging Face guide states that even fine-tuning on your own data cannot make the original model's intrinsic bias disappear. After any customization, it is therefore a good idea to repeat the tests on the most sensitive cases.

### What is automation bias?

It is the tendency to favor results produced by automated systems over those of non-automated systems, regardless of the error rates of each. In practice, people tend to believe an answer simply because it comes from software. The remedy is to compare it with the judgment of a competent person.

### Can I use answers from Claude or Gemini as advice?

No. Google asks users not to treat Gemini's answers as equivalent to medical, legal, financial or other advice. Anthropic recommends not using Claude as the only source of truth and carefully examining any high-risk advice.

### How can I tell whether my model is biased?

The sources consulted do not indicate a measurement method or thresholds to apply. A reasonable approach is to have a competent person assess a set of real cases, including sensitive ones, then compare their judgment with the model's answers and note the differences.

### Who is responsible if the model produces a discriminatory result?

The materials consulted do not answer this question. To assess legal liability in your own case, you need professional advice. In the meantime, it is prudent to keep a person who reviews and approves texts before they are sent.
