RAG explained for business: the four steps
RAG chat answers a question in four steps. Your documents are split into passages and stored as embeddings, numbers that place similar meanings close together. Each question is embedded the same way, and the nearest passages are pulled. The model is told to answer from them or say it cannot, and the reply cites the passages it used.
The name comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(opens in a new tab), which combined a model's trained memory with documents it looks up at answer time. For a business the point is simpler: each answer reads your current documents, so a changed policy needs a re-index, not a retrain.
Index
Documents are split into passages and stored as embeddings
Retrieve
The question is embedded the same way and the nearest passages are pulled
Constrain
The model is told to answer from those passages or say it cannot
Answer
The reply cites the passages it used
What is RAG chat with your documents? One policy, one question
Take a returns policy that allows unused items back within 30 days, and a shopper who asks whether a jacket bought three weeks ago can be sent back. Nothing in the question uses the word returned, yet its embedding sits close to the policy passage, so retrieval finds it. Both are invented to show the steps.
Keyword matching on send, back and jacket finds nothing in that policy sentence. Matching by meaning is what lets the bot answer a question phrased nothing like the document.
| Step | Example |
|---|---|
| Index | Stored passage: Unused items can be returned within 30 days of purchase. |
| Question | A shopper asks: Can I send this jacket back? I got it three weeks ago. |
| Retrieve | The returns passage ranks closest by meaning, though few words match |
| Constrain | Answer only from the passages below, or say you cannot |
| Answer | Yes, if it is unused. Returns are open for 30 days. Source: returns policy. |
How does a document chatbot work, and where it fails
Document chatbots fail in predictable ways, and each failure traces to the documents or the retrieval step. If the right passage is not retrieved, the model has nothing correct to quote, and a bot that was not told to decline will guess anyway.
In our experience the model matters less than these. Answer quality comes mostly from the content you index and the tests you run before launch.
| Failure | Fix |
|---|---|
| Stale document | Re-index whenever the source changes, triggered by publishing |
| Two documents disagree | Remove or merge the old one, since the bot may quote either |
| Question phrased unlike the documents | Expect weak passages. The right behavior is to decline |
| Model answers from general knowledge | Test with questions the documents cannot answer |
| Passage cut mid-thought | Split on headings so a passage keeps its context |
RAG loop methods: retrieve, grade, retry or decline
Plain RAG retrieves once and answers. Loop methods add a check in between. The CRAG paper(opens in a new tab) scores the retrieved passages with a lightweight evaluator and triggers different actions depending on its confidence. The Self-RAG paper(opens in a new tab) trains the model to decide when to retrieve and to reflect on what it found.
Anthropic's guidance on agents(opens in a new tab) describes the same idea as an evaluator-optimizer workflow: one model call generates, another evaluates in a loop, and for complex search tasks the evaluator decides whether further searches are warranted. Two safeguards come with it: ground truth from the environment at each step, and stopping conditions such as a maximum number of iterations.
A loop needs a stop rule, because each extra pass costs time and tokens. The loop below is one sensible default, not a method from those sources. Looping does not replace the wrong-answer test suite, but it adds one more behavior to test, which is whether the bot declines when the retry fails.
Retrieve passages
Grade them against the question
Rewrite the query and retrieve once more if they are weak
Decline and hand off to a person if they are still weak
What has changed in retrieval lately
Contextual retrieval(opens in a new tab) prepends context to each chunk before indexing, and Anthropic reports 49% fewer failed retrievals, or 67% with reranking, on its own benchmark. Hybrid search merges keyword and vector results into one ranking. Results on your content will differ, so measure with your own question set.
A different approach skips the per-question lookup. Karpathy's LLM Wiki gist(opens in a new tab) has the model build and maintain a wiki of Markdown pages, with a lint pass that looks for contradictions and stale claims. Written for personal knowledge bases, it teaches the same lesson: someone has to check for stale and contradictory content on a schedule.
| Technique | What it changes |
|---|---|
| Contextual retrieval | Each chunk gets a short description of its place in the document before it is indexed |
| Hybrid search | Keyword and vector search run side by side and merge into one ranking |
| Reranking | A second model scores the retrieved chunks and keeps the most relevant |
| Compiled wiki | The model maintains a wiki of notes instead of searching raw documents each time |
How we keep answers inside your documents
Three things keep a document chatbot inside your documents, and each has to be built and tested. Retrieval has to return passages with their source. The model needs an instruction to answer from them or decline. Tests have to include questions the documents cannot answer. Our build ships all three.
The AI knowledge base service is a fixed-price $4,500 build on Vercel AI SDK and Supabase pgvector. Monitoring after launch is its own job, covered in AI chatbot maintenance and monitoring.
- Retrieval returns each passage with its source, so every answer can cite the document it came from.
- Before launch, a wrong-answer test suite runs against the bot.
- Spend is capped by a token budget.
- When the documents do not answer, the assistant hands off to a person instead of guessing.
- Monitoring comes with every build. We do not ship unmonitored AI.

What to ask a vendor before you buy
Four questions separate a grounded assistant from a demo: what happens when your content changes, which answers it declined to give, where its citations appear, and what test suite ran before launch. Any vendor that cannot show each of them has not built them, however good the demo looks.
Ask what the assistant costs after launch as well as to build. AI chatbot maintenance cost lists the recurring parts.
- Content changes: a good answer names a trigger, such as a publish event, not a person who remembers.
- Declined answers: ask for the log of questions the bot refused, and read ten.
- Citations: each answer should link to its passage, so a person can check it in seconds.
- Test suite: ask to see the questions, including ones built to tempt the bot into guessing.

Questions about RAG chat with your documents
Does RAG train the model on my documents?
No. Documents sit in an index, and the passages that match a question are handed to the model for that question only. Nothing about the model changes, which is why updating a document means re-indexing it.
Can a RAG chatbot still give a wrong answer?
Yes. It can retrieve the wrong passage, quote a stale document, or ignore its instruction to stay within the passages. Known-answer tests, citations a person can check and timely re-indexing narrow the risk. None removes it.
What happens when the answer is not in my documents?
If the system is well built, it says so and offers a person. That behavior has to be set and tested. A loosely built bot may fall back on the model's general knowledge instead.
What is agentic RAG?
RAG where the model decides what to do next instead of following one fixed retrieve-then-answer path. Rewriting the query, searching again or stopping are all allowed. That can help on hard questions, but it adds cost and latency, so test it against your own question set before adopting it.
