Skip to content

What Is RAG Chat With Your Documents, and Where Does It Break?

What is RAG chat with your documents? Your files are split into passages, the passages closest to each question are found, and the model is told to answer only from them. It reads your current documents for every answer instead of relying on memory, so it can quote your real refund policy.

RAG explained for business: the four steps

RAG chat answers a question in four steps. Your documents are split into passages and stored as embeddings, numbers that place similar meanings close together. Each question is embedded the same way, and the nearest passages are pulled. The model is told to answer from them or say it cannot, and the reply cites the passages it used.

The name comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks(opens in a new tab), which combined a model's trained memory with documents it looks up at answer time. For a business the point is simpler: each answer reads your current documents, so a changed policy needs a re-index, not a retrain.

The four steps behind every answer
  1. Index

    Documents are split into passages and stored as embeddings

  2. Retrieve

    The question is embedded the same way and the nearest passages are pulled

  3. Constrain

    The model is told to answer from those passages or say it cannot

  4. Answer

    The reply cites the passages it used

What is RAG chat with your documents? One policy, one question

Take a returns policy that allows unused items back within 30 days, and a shopper who asks whether a jacket bought three weeks ago can be sent back. Nothing in the question uses the word returned, yet its embedding sits close to the policy passage, so retrieval finds it. Both are invented to show the steps.

Keyword matching on send, back and jacket finds nothing in that policy sentence. Matching by meaning is what lets the bot answer a question phrased nothing like the document.

One invented question, followed through the four steps
StepExample
IndexStored passage: Unused items can be returned within 30 days of purchase.
QuestionA shopper asks: Can I send this jacket back? I got it three weeks ago.
RetrieveThe returns passage ranks closest by meaning, though few words match
ConstrainAnswer only from the passages below, or say you cannot
AnswerYes, if it is unused. Returns are open for 30 days. Source: returns policy.

How does a document chatbot work, and where it fails

Document chatbots fail in predictable ways, and each failure traces to the documents or the retrieval step. If the right passage is not retrieved, the model has nothing correct to quote, and a bot that was not told to decline will guess anyway.

In our experience the model matters less than these. Answer quality comes mostly from the content you index and the tests you run before launch.

Common failures and the fix for each
FailureFix
Stale documentRe-index whenever the source changes, triggered by publishing
Two documents disagreeRemove or merge the old one, since the bot may quote either
Question phrased unlike the documentsExpect weak passages. The right behavior is to decline
Model answers from general knowledgeTest with questions the documents cannot answer
Passage cut mid-thoughtSplit on headings so a passage keeps its context

RAG loop methods: retrieve, grade, retry or decline

Plain RAG retrieves once and answers. Loop methods add a check in between. The CRAG paper(opens in a new tab) scores the retrieved passages with a lightweight evaluator and triggers different actions depending on its confidence. The Self-RAG paper(opens in a new tab) trains the model to decide when to retrieve and to reflect on what it found.

Anthropic's guidance on agents(opens in a new tab) describes the same idea as an evaluator-optimizer workflow: one model call generates, another evaluates in a loop, and for complex search tasks the evaluator decides whether further searches are warranted. Two safeguards come with it: ground truth from the environment at each step, and stopping conditions such as a maximum number of iterations.

A loop needs a stop rule, because each extra pass costs time and tokens. The loop below is one sensible default, not a method from those sources. Looping does not replace the wrong-answer test suite, but it adds one more behavior to test, which is whether the bot declines when the retry fails.

A retrieval loop with a stop rule
  1. Retrieve passages

  2. Grade them against the question

  3. Rewrite the query and retrieve once more if they are weak

  4. Decline and hand off to a person if they are still weak

What has changed in retrieval lately

Contextual retrieval(opens in a new tab) prepends context to each chunk before indexing, and Anthropic reports 49% fewer failed retrievals, or 67% with reranking, on its own benchmark. Hybrid search merges keyword and vector results into one ranking. Results on your content will differ, so measure with your own question set.

A different approach skips the per-question lookup. Karpathy's LLM Wiki gist(opens in a new tab) has the model build and maintain a wiki of Markdown pages, with a lint pass that looks for contradictions and stale claims. Written for personal knowledge bases, it teaches the same lesson: someone has to check for stale and contradictory content on a schedule.

Four retrieval techniques and what each changes
TechniqueWhat it changes
Contextual retrievalEach chunk gets a short description of its place in the document before it is indexed
Hybrid searchKeyword and vector search run side by side and merge into one ranking
RerankingA second model scores the retrieved chunks and keeps the most relevant
Compiled wikiThe model maintains a wiki of notes instead of searching raw documents each time

How we keep answers inside your documents

Three things keep a document chatbot inside your documents, and each has to be built and tested. Retrieval has to return passages with their source. The model needs an instruction to answer from them or decline. Tests have to include questions the documents cannot answer. Our build ships all three.

The AI knowledge base service is a fixed-price $4,500 build on Vercel AI SDK and Supabase pgvector. Monitoring after launch is its own job, covered in AI chatbot maintenance and monitoring.

  • Retrieval returns each passage with its source, so every answer can cite the document it came from.
  • Before launch, a wrong-answer test suite runs against the bot.
  • Spend is capped by a token budget.
  • When the documents do not answer, the assistant hands off to a person instead of guessing.
  • Monitoring comes with every build. We do not ship unmonitored AI.
Diagram titled How a document chatbot answers, and when it declines: a visitor question goes to retrieval of passages with their source, then to an instruction to answer from the passages or decline, which ends in either an answer with a citation when the passages cover it, or a hand-off to a person when they do not.

What to ask a vendor before you buy

Four questions separate a grounded assistant from a demo: what happens when your content changes, which answers it declined to give, where its citations appear, and what test suite ran before launch. Any vendor that cannot show each of them has not built them, however good the demo looks.

Ask what the assistant costs after launch as well as to build. AI chatbot maintenance cost lists the recurring parts.

  • Content changes: a good answer names a trigger, such as a publish event, not a person who remembers.
  • Declined answers: ask for the log of questions the bot refused, and read ten.
  • Citations: each answer should link to its passage, so a person can check it in seconds.
  • Test suite: ask to see the questions, including ones built to tempt the bot into guessing.
A long library aisle of shelves filled with books, standing in for the document collection a chatbot retrieves its passages from

Questions about RAG chat with your documents

Does RAG train the model on my documents?

No. Documents sit in an index, and the passages that match a question are handed to the model for that question only. Nothing about the model changes, which is why updating a document means re-indexing it.

Can a RAG chatbot still give a wrong answer?

Yes. It can retrieve the wrong passage, quote a stale document, or ignore its instruction to stay within the passages. Known-answer tests, citations a person can check and timely re-indexing narrow the risk. None removes it.

What happens when the answer is not in my documents?

If the system is well built, it says so and offers a person. That behavior has to be set and tested. A loosely built bot may fall back on the model's general knowledge instead.

What is agentic RAG?

RAG where the model decides what to do next instead of following one fixed retrieve-then-answer path. Rewriting the query, searching again or stopping are all allowed. That can help on hard questions, but it adds cost and latency, so test it against your own question set before adopting it.

What is RAG chat with your documents worth to your site? Start with an audit.

A senior engineer reviews your live site and returns a written, prioritized fix list, so you know which changes to make first and why.