From document to evidenced answer

Embedding models only process a limited amount of text at once: one widely used model accepts at most 8,191 tokens, anything longer would be cut off. Documents are therefore split into sections. Microsoft recommends roughly 512 tokens per section with a quarter of overlap as a starting point; beyond that, only measuring against your own material helps. The glossary entry chunking explains the details.

The most common mistake is loss of context. An isolated section no longer knows which project or contract it refers to. The search then misses it or finds it wrongly. Anthropic therefore prepends an explanatory context sentence to each section before embedding and, by its own account, measures up to 49 percent fewer retrieval misses. Combined with reranking it reports up to 67 percent. The most useful rule of thumb remains: if an excerpt makes sense to a human without its surroundings, it usually does for the model too.

Vector search finds meaning, not wording: question and sections live as number sequences in a vector database, and the search looks for closeness in meaning. For exact values such as error codes, article numbers or names, classic keyword search (like BM25) remains superior.

Good systems therefore combine both. The two hit lists are merged, typically via reciprocal rank fusion: the ranks of both lists are combined instead of comparing raw scores. Meaning-based and wording-based hits complement each other without one side dominating.

The first search stage is fast but rough. A second model, the cross-encoder, reads question and candidate together and judges the match far more accurately. It is too slow for millions of entries, well suited for the leading hits. Microsoft's search service, for example, re-sorts only the top fifty this way. More in the glossary entry reranking.

The lever is direct: only what sits on top after re-sorting ends up in the request to the language model. A better second sorting step therefore directly improves the answers, at the price of extra computing time per query.

Whether a change helps is decided by a test run, not an opinion. Microsoft explicitly recommends a structured, scientific approach for RAG solutions: fixed test questions with known target answers, against which each stage is measured individually. Whoever changes the section size, the embedding model or the reranker compares against the frozen baselines afterwards, otherwise you are optimising blindly.

The finished answer needs checking too: does it really rest on the retrieved passages (grounding), or did the model add its own? Even the best models still invent around two to three percent of the statements when merely summarising supplied texts (Vectara leaderboard, data as of May 2026).

An assistant may only find what the person asking would be allowed to see. To achieve this, the access rights of the source documents are carried along in the search index and every search is filtered by them – Microsoft calls this security trimming. Without this step, a knowledge assistant bypasses departmental boundaries: confidential folders become a full-text search for everyone.

For personal data, the GDPR applies on top. In its October 2025 guidance, the German data protection conference requires a legal basis for such data in the knowledge base. It warns against purpose limitation violations and demands a role concept. It also names advantages: entries remain individually erasable, and a RAG system can be run entirely locally without transferring data to third parties. This page does not replace legal advice.

The base pattern stays, the edges are moving. Current systems let the model plan and refine search queries itself (agentic retrieval) instead of retrieving exactly once. For questions across entire collections (“What are the main topics of all reports?”), GraphRAG additionally links the content into a knowledge graph.

And the large context windows? They shift the threshold at which a search index pays off, but do not replace it. The fuller the context window, the less reliably the model accesses details within it. The discipline of feeding the right content at the right time now has its own name: context engineering.

For our software development we run a RAG system we built ourselves. The tool itself comprises around 240,000 lines of Python and runs entirely on our own hardware. Seven connected projects hang off one index, and their code never leaves the building. Our coding agent queries the system before every change and receives evidenced context instead of guesses. It learns which symbol is defined where, who calls it, and what breaks if you change it.

The pipeline follows exactly the steps on this page, with one special twist in chunking. Source code is not cut by token windows but split along the syntax into whole functions and classes. Search is hybrid: meaning-based and keyword search in one database, merged via reciprocal rank fusion and re-sorted by a cross-encoder on our own graphics card. Every hit brings its callers and references along. Uncommitted work in progress is layered in as an overlay, so the agent sees the current state rather than last night’s.