AI assistant for internal documents: RAG architecture, permissions and real limits

An assistant that answers from internal documents (contracts, procedures, manuals, minutes) is today one of the AI projects with the clearest cost-benefit ratio. It is also one of the easiest to get wrong: the demo works in an afternoon, the reliable version needs precise choices about how text is prepared, how it is searched, who can see what and how quality is measured.

The architecture: RAG, not training

The language model does not "learn" the documents. For each question the system finds the relevant passages and passes them to the model along with the question, with the instruction to answer only from them and cite them. This technique is called RAG (Retrieval-Augmented Generation). It has two decisive advantages: documents update without retraining anything, and every answer can be checked against its source.

The real work is not in the model but in the three phases around it: preparation, retrieval, controlled generation.

Preparation: where quality is decided

Extraction

Native PDFs, scanned PDFs, Word documents, spreadsheets, internal web pages. Each needs a different extractor. Tables are the critical point: extracted badly they become strings of numbers without headers, and the model answers confidently about data it has misread.

Chunking

Documents are split into chunks. Splitting every N characters is the quickest method and the worst. It works better to respect structure: sections, contract clauses, procedure steps, with some overlap and the section title repeated in the chunk text. Each chunk carries metadata: document, version, date, section, authorised roles.

Versions

If two versions of the same procedure are both indexed, the assistant may cite the outdated one. You need an explicit rule: only the latest version, or all of them with the date shown and a preference for the most recent.

Retrieval: hybrid, not only semantic

Semantic similarity search (embeddings) finds passages with different words but similar meaning. On its own, though, it fails on product codes, clause numbers, proper names and acronyms. The combination I usually use:

• hybrid search: semantic plus classic text search (BM25), merged into a single ranking;
• metadata filters before the search: language, document type and above all permissions;
• reranking of the top results with a dedicated model, which greatly improves the relevance of the passages passed to the model.

Controlled generation

The model's instructions require it to answer only from the passages provided, cite document and section, and say when the information is not there. Below a relevance threshold the assistant does not try to answer: it says it found nothing and points to who to ask. An "I don't know" is better than a plausible, wrong answer about a contract clause.

Permissions: the part discovered too late

An assistant that has read everything can reveal everything. Permissions must be applied at retrieval time, filtering passages by the user's roles, not after generation. If a restricted passage never reaches the model, the model cannot cite it. You also need a log of questions and sources consulted, log retention rules and a GDPR assessment when documents contain personal data.

Measuring quality

Before release you build a set of real questions, with the expected answer and the correct source, written by people who know the documents. It is re-run after every change to chunking, retrieval or instructions. Two separate things are measured: whether retrieval found the right passage, and whether the answer is faithful to that passage. Without this test set, every change is an impression.

Limits to state

• Quality drops on questions that require combining many documents.
• Calculations and numeric comparisons should be delegated to code, not the model.
• The assistant supports judgement, it does not replace it, especially on legal and contractual matters.
• Quality depends on the documents: if they contradict each other, so will the answers.

Costs and maintenance over time

A document assistant is not delivered once. Documents change and the index must be updated: ideally incrementally, reprocessing only modified files and deleting the chunks of removed ones. If the embedding model changes, the index must be rebuilt entirely, because vectors from different models are not comparable.

Running costs have three parts: indexing (once and on every update), retrieval (minimal) and generation (proportional to questions, passages sent and answer length). They are kept in check by passing the model only truly relevant passages, short answers by default and a model sized to the task.

The most valuable source of improvement is the logs: questions the assistant could not answer point to missing or badly written documents. Reviewing them monthly with whoever curates the content improves the assistant more than any model change.

My take

I believe in narrow assistants: few sources, well curated, and the courage to answer "I don't know". It is the same principle as the chat on this site, which answers only from a single hand-written knowledge file and refuses everything else. And it is the principle behind the Captain in Fix and Sail, the assistant that answers based on the manuals of the accessories installed on that specific boat, not generically.

I am wary of projects that start from the most powerful model and "let's upload everything". AI is the easy part here. What makes the difference is document care, retrieval and permissions: developer work, not prompt work.

Why work with me

I have been programming since 1999, delivered more than 300 projects and work as an external developer with about 30 agencies, from Rome and remotely. I design RAG assistants with hybrid search, citations, role-based permissions and a test set to measure their quality.

If you want to know whether your documents lend themselves to an assistant, get in touch: in a call of about 20 minutes we look at sources, users and risks. The service is in AI for businesses.