Arrange a call
DE EN

Workshop · basics As of: 29 Aug 2026

RAG: look it up instead of guessing

Retrieval-augmented generation sounds unwieldy but is a simple idea: the AI does not answer from memory, it first looks things up in your own documents. Here you can read, without jargon and with sources, what it delivers, what it costs and when you actually need it.

When is it worth it? On to the technical page

For search engines and AI assistants in one sentence:

RAG (retrieval-augmented generation) lets an AI language model retrieve matching passages from a defined knowledge base before answering and base the answer on them. This makes answers more current and more verifiable, but does not prevent errors entirely. For a small knowledge base of a few hundred pages, Anthropic says the context window (the text a model can process at once) is often enough.

Source: arXiv: Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)

Explained very simply

A language model without RAG answers like someone who read a book years ago: fluent, confident and sometimes wrong on the details. It knows nothing at all about your business. No price list, no contracts, no internal wiki.

RAG changes exactly that. Before the AI answers, the system puts the matching pages from your own documents on its desk. It is told to stick to those pages and to name where it found things. The abbreviation stands for retrieval-augmented generation, in plain terms: look it up first, then answer. The method was introduced in 2020 by a research team around Patrick Lewis.

Our glossary explains the key terms one by one: RAG, vector database, chunking, reranking and grounding.

How RAG works – four steps

Every knowledge assistant runs on the same sequence. Knowing it also tells you where quality is won and lost.

  1. Prepare

    Split the knowledge base

    Manuals, contracts or wiki pages are split into small sections (chunking) and stored as number sequences that capture meaning (embeddings).

  2. Understand

    Translate the question into numbers

    The question is translated into such a number sequence too. That makes it possible to compute which sections come closest in meaning, regardless of wording.

  3. Retrieve

    Select the best passages

    The search delivers candidates, a second sorting step (reranking) moves the truly fitting ones to the top. Only these few passages are shown to the model.

  4. Answer

    Write the answer with evidence

    The language model bases the answer on the supplied passages (grounding) and names the source. Anyone in doubt can look them up.

Decision aid

Good search, RAG or fine-tuning?

Three routes, three purposes. The columns are not a ranking. The simplest solution is often the right one.

Criterion Good search RAG Fine-tuning
Suited for Finding documents and reading them yourself Questions in everyday language, answers with evidence Changing a model's style and behaviour
Currency Immediate, as soon as the file exists High, the knowledge base is updated continuously Low, knowledge is frozen into the training
Verifiable answers Yes, you read the original Yes, via a source reference per answer No, provenance cannot be traced
Effort Low Medium: setup plus ongoing maintenance High: every change means new training
Running costs Hardly any Retrieval, embeddings, longer requests Training plus operation
Honestly: If your entire knowledge fits into a few hundred pages, you often need no RAG at all. Current models process whole manuals directly in one request. Anthropic gives a rule of thumb of around 200,000 tokens, roughly 500 pages.

Where RAG hits its limits

RAG makes errors rarer, not impossible. When merely summarising supplied texts, even the best models still invent around two to three percent of the statements (Vectara leaderboard, data as of May 2026). A Stanford study in 2024 found 17 to 33 percent faulty answers in RAG-based legal research services, depending on the provider. Important output should still be checked by a second pair of eyes, all the more when money or law is involved.

And the knowledge base sets the ceiling. Outdated documents deliver outdated answers, and a source reference only makes them more convincing. Then there are access rights: an assistant may only find what the person asking would be allowed to see. For personal data in the knowledge base, the German data protection conference requires a legal basis and a role concept in its October 2025 guidance. On the upside, entries there remain individually erasable. This page does not replace legal advice.

We don't just explain this. For our software development we run a RAG system of our own, on our own hardware, with seven connected projects. One principle from it applies everywhere: every answer carries its evidence. And when the system does not know something, it says so instead of guessing.

What that looks like in the engine room (chunking, vector search, reranking and our own system in detail) is on the follow-up page: RAG in practice.

Frequently asked questions about RAG

What does RAG stand for?
Retrieval-augmented generation, a method where a language model retrieves matching sources before answering. It was introduced in 2020 by a research team around Patrick Lewis. The idea: the model does not rely on trained knowledge alone but receives current passages from a knowledge base for every question. The English term is standard across languages; German authorities use it as well.

Source: arXiv: Lewis et al. (2020)

Does RAG prevent hallucinations?
No. It makes them much rarer and answers verifiable, but does not eliminate them. A Stanford study in 2024 found 17 to 33 percent faulty answers in commercial legal research services using RAG, depending on the provider. That is clearly better than general-purpose chatbots without source access, but far from error-free. Human review remains part of the process.

Source: Stanford HAI: AI on Trial (2024)

Do we always need a vector database for this?
No. Small knowledge bases often fit into the request to the model in full. Anthropic gives a rule of thumb of around 200,000 tokens, roughly 500 pages. Up to that point the entire knowledge base can be sent along directly. Only beyond it does a search index pay off. The figure is a vendor statement and shifts with new model generations.

Source: Anthropic: Introducing Contextual Retrieval

What is the difference between RAG and fine-tuning?
RAG brings current knowledge into the answer, fine-tuning changes the model's behaviour and style. Microsoft puts it in this formula: RAG when answers should rest on private or frequently changing data. Fine-tuning when behaviour, style or task performance should change, not the knowledge. Both can be combined.

Source: Microsoft Learn: Retrieval-augmented generation (Azure AI Foundry)

What does running a RAG system cost?
Beyond the setup, mainly recurring items: retrieval, embeddings, longer requests, and maintaining the knowledge base. Every question triggers extra work: the search itself, converting into embeddings, and more tokens per request because the retrieved passages are sent along. In our experience the biggest hidden item is maintenance: a knowledge base nobody keeps current produces convincing-sounding old answers.

Source: Microsoft Learn: Retrieval-augmented generation (Azure AI Foundry)

May personal data go into the knowledge base?
Only with a legal basis and a role concept. German supervisory authorities published dedicated guidance on this in 2025. The guidance of the German data protection authorities of 17 October 2025 requires a legal basis for personal data in the knowledge base. It warns against purpose limitation violations and demands access restrictions. One advantage of RAG: entries in the knowledge base are directly addressable and thus individually erasable. This does not replace legal advice.

Source: DSK: guidance on generative AI systems using RAG (PDF, German, October 2025)

Is RAG still current in 2026?
Yes. It remains the standard pattern and is combined with hybrid search, rerankers and agentic workflows. Large context windows only shift the threshold at which a search index pays off. They do not replace it, because as the window fills up, the model's recall accuracy drops. Current systems extend the base pattern with agentic retrieval, where the model plans and refines search queries itself.

Source: Anthropic: Effective context engineering for AI agents

Wondering whether a knowledge assistant fits your business?

Initial consultation, 30–45 minutes, free of charge. We will also tell you if a folder with good search is all you need.

Book a conversation

Terms related to this service

Technical terms that appear on this page – explained in one sentence in our glossary.