Local RAG for business documents: a practical design

A local RAG for business documents retrieves relevant passages from an archive and supplies them to a model running on the infrastructure chosen by the business. To call it fully local, extraction, embeddings, retrieval and any services processing the files must also be local.

The useful outcome is a verifiable answer linked to sources the user can access. RAG does not guarantee correctness or replace document organization: an archive with confused versions can produce wrong answers even with an excellent model. For the general definition, see the guide to what RAG is; here we focus on the decisions needed for a local business system.

When do you need RAG instead of attaching a document?

If you occasionally work with a short file, attaching it to the conversation may be sufficient. The LM Studio documentation distinguishes placing a document in context from retrieving relevant portions when the material is longer.

RAG becomes useful when questions span an archive, sources change or different people need access to different collections. The problem is then not just 'how much text fits into the model', but how to find the right version and make it verifiable.

Do not build a knowledge base to compensate for files that were never organized. First identify the documents answering recurring questions. A reliable subset is a better foundation than indiscriminately importing an entire shared drive.

Also state which activities remain out of scope. Looking up a rule in a procedure differs from approving spending, changing an order or sending a communication. A document RAG system can be limited to displaying information and preparing drafts; actions require additional design with their own permissions and controls. Clarifying this boundary makes the first release easier to evaluate: success depends not on how many features appear in the demo, but on completing the defined work without opaque steps.

Which stages must stay within the local boundary?

The workflow includes acquisition, extraction, text splitting, embeddings, indexing, retrieval and generation. OCR, rerankers, transcription, logs and connected tools may add further stages.

The Guide to privacy and offline verification helps check external connections. 'The model runs on our server' does not answer the other rows in the table.

How should you prepare PDFs, tables and scanned documents?

For each format, check that the extracted text preserves what retrieval needs. Headings, columns and references can lose meaning when treated as an undifferentiated sequence of words.

A scanned document needs a suitable processing stage: OCR or a compatible vision model, depending on the configuration. The guide to multimodal AI for PDFs and images explains why visual reading and text extraction are different.

Prepare difficult examples before full import: a table spanning pages, one code similar to another, an exception in a footnote. Keep the link between extracted text and the source file. If you cannot trace back to the page or section, checking the answer later becomes much harder.

How can you split text without losing its meaning?

The chunking process divides documents into units for search and retrieval. There is no magic length that works for every archive. An operating procedure, price list and contract have different structures.

Avoid separating a condition from its exception or a table row from the heading explaining it. Add useful metadata: title, section, version, effective date and source reference. Evaluate these units against questions, not just ease of import.

A simple check is to read the retrieved passage without opening the rest of the file: does it contain enough information for a correct answer? If the subject or scope of the rule is missing, retrieve additional context or change how the document is split.

Attachments need the same care. A procedure may refer to a separate form or a table containing the decisive information. Recording relationships between documents helps reconstruct context without relying on incidental textual similarity. During tests, explicitly ask about information in attachments and check that the system distinguishes a reference from the actual answer. If an attachment is unavailable or unauthorized, the answer must state that limitation instead of hiding it behind generic wording.

Which embedding model should you use for Italian?

The model generating answers is not necessarily the one producing embeddings. Embeddings represent queries and documents for retrieval; the Sentence Transformers documentation describes this function and the differences between candidate models.

For an Italian archive, evaluate requests using synonyms, company abbreviations and the language people actually use. A multilingual label alone does not prove the system retrieves a procedure containing specialist terminology correctly.

Ollama provides a dedicated embedding endpoint. Using it locally is a technical option, not a guarantee that any embedding model will suit the task. Also check for truncation: if an important portion is not represented, the generator may never receive it.

When should you add lexical search and reranking?

Semantic search can help when wording differs from the documents. For codes, case numbers or exact abbreviations, also evaluate lexical search. Test the combination against the corpus rather than assuming one technique always replaces the other.

A reranker evaluates previously retrieved candidates more closely. The Sentence Transformers retrieve-and-rerank pipeline shows the distinction between initial selection and subsequent reordering.

Adding one means extra processing and another component to manage. Do it when tests show the right documents appear among candidates but do not reach the model in the final selection. If text was extracted incorrectly, a reranker cannot retrieve what is missing from the index.

How do you prevent users from reading unauthorized documents?

The system must enforce authorization before retrieval, or at least before content is exposed to the model and user. Do not rely on a prompt sentence such as 'do not show confidential information'.

The documentation on tenant separation in Qdrant describes filters and strategies for isolating datasets. The Open WebUI permissions instead concern resources and features accessible within the application. Both layers must be designed consistently.

Also check file names, previews and citations: they can reveal the existence of unauthorized documents. Prepare test users with different roles and identical requests. Check both the answer and what is retrieved along the way.

How do you make citations genuinely verifiable?

An answer with a link is not automatically a grounded answer. The reference must point to the version used, and the passage must support the claim rather than merely discuss the same subject.

Open WebUI documents citations and metadata in RAG. Whatever interface you choose, check that users can quickly find the relevant passage and have permission to open it.

The document exists, but the answer is not there

The system must be able to state that it found insufficient support in the available sources. A generic citation to the document does not fill that gap. Include plausible questions whose answers are deliberately absent from the test material.

For answers combining several sources, connect each conclusion to the relevant passages. This reveals where the model has added an inference requiring human review.

A manufacturer: how do you distinguish manuals and revisions?

Consider a hypothetical scenario: a machinery manufacturer wants to help support staff consult manuals. The same commercial product name may cover different revisions and instructions.

RAG should search within the specified model and revision, asking for clarification when they are missing. An answer correct for another version is an error even if its citation leads to a genuine manual.

The test sample must include nearly identical procedures with an important difference. If retrieval favors the longest or newest document without checking compatibility, changing only the LLM does not fix the problem. First correct the relationship between machine, manual and metadata.

Authorization for operational use belongs within the business process: the assistant can prepare information, while responsibilities and checks on instructions remain defined.

A consultancy: how do you search internal procedures?

In a second hypothetical example, a company wants to answer questions about reporting procedures. Some documents apply to everyone, others to a division or contractual period.

Before indexing, the owner identifies the current source and authorized audience. Test questions include requests in colleagues' everyday language, not just exact document titles.

A useful result contains the applicable rule, any exception and a link to the correct procedure. If the source is insufficient, the system should identify the appropriate contact rather than invent a company policy.

This project has verifiable value without promising percentages of time saved: fewer unnecessary retrieval steps and an easier way to check which rule is being applied.

How do you manage archive updates and removals?

The design must recognize when a file is replaced, its permissions change or it should no longer be used. Merely adding new documents can leave outdated passages in the index that continue to appear in answers.

Keep a stable document identifier and version. When updating content, check text, embeddings, metadata and references. If you change the embedding model, verify that the index and queries remain consistent: do not treat different representations as automatically comparable.

Include a question that previously retrieved the old document in the procedure. Repeating it after an update checks the effect on the end user. Also consider saved conversations and copies: removal from the index does not mean data disappears from every system archive.

Which tests distinguish retrieval errors from model errors?

Maintain two evaluations. In the first, check whether retrieval finds the expected sources and respects permissions. In the second, give the generator the correct passages and check how it uses them.

If the right document is not retrieved, investigate extraction, splitting, search and filters. If it reaches the context but the answer contradicts it, assess the prompt, model and request format. The Italian-language local benchmarking protocol helps document this distinction.

Add tests for missing answers, obsolete versions and attempts to obtain material outside the user's role. Average quality must not hide an access violation. Define acceptance thresholds according to the consequences of error.

For each error, keep the question, retrieved sources, configuration and expected correction. This makes maintenance practical: after a change, repeat previously failed cases and check that the improvement has not damaged anything else. Avoid indiscriminately archiving real conversations with confidential data merely for development convenience. Select and manage evaluation material using the same access and retention criteria as the service.

How do you move from a demo to a useful service?

Start with a collection that has an owner, reliable documents and recurring questions. Define who updates sources, who checks errors and how to report a wrong answer. Only then expand the scope.

Choose the generator from local models compatible with the work, considering hardware and maintenance too. The Open WebUI offline mode reminds us that models and dependencies must be prepared before isolation: a checkbox in settings does not replace a full test.

To connect permissions, documents and existing tools, Giallo Studio designs custom AI software. The first outcome to build is a verifiable path from question to source, with clear limits and defined operational responsibilities.