Local LLMs in Italian: comparing without misleading tests

A benchmark of local LLMs in Italian is useful when it compares defined tasks, documented configurations and verifiable answers. A collection of impressions or scores taken from different sources does not demonstrate which model works best for your business.

Here we propose an evaluation protocol, not results obtained by Giallo Studio. It is intended for comparing candidates such as Qwen3.8-27B, Gemma 4 or other checkpoints compatible with your hardware, distinguishing language quality, correctness, source use and operational performance.

What must the comparison demonstrate?

The question 'which is best?' is too broad. Ask instead which configuration best completes a specific task on the available machine with the intended checks.

An assistant for consulting procedures needs different tests from a model preparing sales drafts. Document versions and citations may be decisive for the former; style must be separated from fidelity to supplied data for the latter.

The Guide to how to choose an AI model helps turn the objective into requirements. Also state exclusions: a model preparing a draft is not automatically authorized to send it, and a writing test does not demonstrate the ability to operate business software.

Why is 'supports Italian' not enough?

Declared language support does not equal correctness on your industry's vocabulary, documents and ambiguities. The Gemma 4 model card describes the family's multilingual capabilities and evaluations; testing the required Italian work remains necessary.

A natural-sounding answer may contain a transposed date, omitted exception or conclusion absent from the sources. The evaluation must therefore distinguish readability from fidelity.

Prepare realistic wording: internal abbreviations, incomplete requests, synonyms and questions written as people would ask them. Do not merely translate an English quiz. If people unfamiliar with technical terminology will also use the service, include those requests in the sample.

How do you identify candidate models precisely?

Record the producer, repository, checkpoint and revision. Add the file, precision or quantization, runtime version and generation settings. The family name is insufficient.

The Qwen3.8-27B model card distinguishes the checkpoint's capabilities and controls. If a test enables reasoning, tools or a different context, disclose it: you are comparing configurations, not an abstract entity called Qwen.

When you evaluate Qwen3.8-27B and Gemma 4 31B, specify whether you want to isolate one feature or choose the final system. In the first case, keep as many conditions identical as possible; in the second, different configurations are acceptable, but document their costs and limits.

How do you build the sample without inventing results?

Collect representative tasks and prepare an expected answer or rubric for each. Use synthetic or authorized material, avoiding turning the test into an indiscriminate copy of the business archive.

The sample should include ordinary cases, exceptions, missing information and documents with different versions. Decide in advance which errors are disqualifying. Do not increase a question's weight only after seeing that it makes your preferred model win.

Keep some requests for final validation rather than repeatedly using them to modify the prompt. Otherwise you risk building a configuration tailored to a few examples that looks valid in testing but is fragile on new work. There is no universal sample size here: size the evaluation according to variety and the consequences of error.

Have someone familiar with the documents and process write expected answers before generating outputs. Where several answers are acceptable, describe the constraints they must all meet rather than demanding identical wording. For a sales draft, for example, you can accept different styles while requiring that no version adds a discount absent from the data. This strengthens the evaluation and prevents rewarding a model merely for using words similar to the evaluator's.

Which scorecard should you use to record answers?

The following table is a blank template to complete, not a presentation of measured results.

Do not immediately merge everything into a single score. An invented citation or access violation can be disqualifying even when style and speed are excellent. If you use weights, disclose them and show how they affect conclusions.

Also report measurement denominators and keep error counts by category. An isolated percentage may hide a tiny sample or an unbalanced task distribution. If many questions involve simple summaries and few concern critical exceptions, the average may look reassuring without representing operational risk. Show task groups and disqualifying cases separately, stating when data is insufficient for a general conclusion.

How do you check citations and abstention?

Include a question with an answer in the material, one requiring several passages and a plausible one unsupported by the material. The model must distinguish these conditions.

Check that the source exists, is the intended one and supports the answer's point. A genuine but irrelevant link is not a correct citation. For document systems, the guide to local RAG distinguishes retrieval errors from generator errors.

Abstention does not mean refusing every difficult question. The criterion is recognizing when available information is insufficient and explaining what is missing. Keep both inventions and unnecessary refusals: an assistant that never makes a mistake because it never answers does not complete the required work.

How do you keep retrieval and generation separate?

If the system searches an archive, a wrong answer may result from failure to retrieve the source. Changing LLMs without seeing what entered the context makes diagnosis unreliable.

Prepare a retrieval check: which documents and passages should reach the model for each question? Then test the generator with the correct passages supplied directly. These evaluations answer different questions.

You can also compare the complete system, since that is what people will use. Keep the intermediate path, however, so you know whether to change extraction, indexing, retrieval or the answer. This avoids blaming the language model for a limitation introduced before its call.

How do you measure speed and memory correctly?

Distinguish loading, prompt processing, initial wait and generation. The Ollama usage metrics provide counts and durations useful for describing some of these stages.

For engine tests, llama-bench supports documented technical measurements. Do not confuse those results with the duration of a complete workflow that includes file loading, retrieval and review.

Record whether the model was already loaded, which other applications were active and how many requests were concurrent. Repeat important cases and retain the variation, not just the most favorable value. A number without a configuration cannot estimate what will happen on another PC, with another context or under load.

If models produce answers of very different lengths, interpret total time alongside useful content. A short but incomplete answer does not automatically win because it finishes first; a verbose answer does not by itself justify a longer wait. Define the required format and check task completion, keeping generated tokens separate from value delivered to the user. For shared services, add representative load: a single-person test does not demonstrate an office's experience during peaks.

How do you compare quantizations without changing the question?

To isolate compression's effect, use the same starting checkpoint and keep prompts, material and relevant settings identical. The guide to Q4 and Q8 in GGUF explains why the container name is not enough to describe the weights.

If you also change models, you can still choose a practical configuration, but do not attribute every difference to quantization. A smaller, less compressed model and a larger, heavily compressed one are different systems.

Measure what matters to the workflow: a configuration that fits in memory might allow more useful context, while a larger one might need a different hardware path. Report both effects alongside quality without claiming a universal percentage loss.

A purchasing department: how do you evaluate extraction and constraints?

Consider a hypothetical scenario: a purchasing department needs codes, quantities and terms from quotation documents. The sample may contain similar rows, excluded options and notes changing the table's meaning.

The expected answer must identify the correct fields and reference. Convincing style does not compensate for a quantity assigned to the wrong row. If the task requires a structured format, also verify that the next application can read the data.

Add a quotation with missing information. The system should flag it rather than complete it by analogy. None of these examples represents a result already obtained: they describe material to prepare for a process-specific evaluation.

Technical support: how do you avoid the wrong manual?

In a second hypothetical example, a support team compares models for consulting manuals. The question is not just whether the model knows the product, but whether it uses the correct revision and cites the relevant passage.

Prepare pairs of nearly identical documents with operational differences and questions requiring clarification. The system should distinguish certain information from information depending on an unspecified version.

Also evaluate how limitations are presented. A cautious but vague answer may not help the operator; a useful one identifies the necessary information or source to check. For PDFs and scans, separate multimodal document reading from the ability to reason over text already extracted.

How should you read benchmarks published by others?

Record who performed the evaluation, which dataset they used and which conditions they disclose. Producer results are information attributable to the producer, not independent tests conducted by your team.

Do not combine figures from different versions or tasks. Tools such as LM Evaluation Harness make many evaluation details explicit: tasks, configurations and execution methods are part of the result.

Even when benchmark names match, check the budget, prompts, tools and checkpoints. A measurement can guide initial selection but does not replace testing the Italian workflow. If a table does not reveal these conditions, state the limitation rather than turning it into a definitive ranking.

When should you use an automated judge and blind review?

An automatic evaluator can help organize many answers if the rubric is clear and checked. Do not treat it as infallible: it may favor long or polished text even when it contains errors.

To reduce brand-related preferences, present answers without model names and vary their order. This is a methodological choice to document, not a guarantee of impartiality.

Two evaluators disagree

Keep the disagreement and check the rubric against the source. If a criterion is ambiguous, clarify it before deciding. Do not automatically choose the judgment confirming your initial preference.

For objective data, prioritize direct checks against the document and format. Use editorial judgment for aspects genuinely requiring human assessment.

The comparison between DeepSeek R1 Distill 32B and Qwen3.8-27B provides an example of candidates with different inputs. If the sample contains images, an absent capability should not be treated as an Italian-language error: distinguish format suitability from textual-answer accuracy.

What should the final verdict contain?

State which configuration meets the task, which errors remain and under what conditions the conclusion holds. Keep requests, answers, sources and versions needed to repeat the test.

Connect the decision to the cost of local and cloud AI, because corrections, attempts and maintenance affect the service. Repeat critical cases when updating the model or runtime; do not change the comparison date without substantial new work.

To prepare an evaluation around your workflows, Giallo Studio can define the AI pilot: representative tasks, acceptance criteria and verifiable tests before choosing which model to introduce into the business.