Why your RAG chatbot gives wrong answers, and how to fix retrieval
When a RAG assistant makes things up, the model is usually not the problem. Retrieval is. The three levers that fix it.
When a RAG chatbot gives a wrong or made-up answer, the instinct is to blame the model. Usually the model is fine and retrieval is the problem: it handed the model the wrong documents, or no useful documents, and the model did its best with what it was given. Fix retrieval and most wrong answers disappear.
Retrieval quality comes down to three levers: how you split your documents, how you match questions to them, and how you rank what you found. Here is how each fails, and how to fix it.
The cause is usually retrieval, not the model
A RAG system answers from the documents it retrieves. If those documents do not contain the answer, the model has two options: say it does not know, or fill the gap with a plausible guess. Most wrong answers are the second case, and they trace back to retrieval handing over the wrong context, not to the model reasoning badly. Diagnose retrieval first by looking at exactly which chunks were retrieved for a failed question.
Lever one: chunking
Chunking is how you split documents into pieces to store and retrieve. Chunk too large and each piece mixes several topics, so a match on one part drags in irrelevant text and dilutes the answer. Chunk too small and you sever the context a sentence needed to make sense. The fix is chunks that hold one coherent idea, often split on document structure like headings, with a little overlap so nothing important falls on a boundary.
Lever two: matching
Retrieval usually matches questions to chunks by meaning, using embeddings, but meaning-matching alone misses exact terms like product codes, names, and numbers. A user asking for 'SKU 4471' wants that exact string, which a purely semantic search can miss. The fix is hybrid search: combine semantic matching with keyword matching so both the meaning and the exact terms count.
Lever three: reranking
The first retrieval pass is fast and approximate, so the truly best chunk is often in the top twenty but not the top three. A reranker is a second, more careful pass that reorders the candidates by how well they actually answer the question, then passes only the best few to the model. Adding a reranker is often the single biggest quality jump in a struggling RAG system.
Fixing it in order
Work the levers in sequence. First look at what was retrieved for real failures, so you are fixing the actual problem. Fix chunking so each piece is coherent. Add hybrid search so exact terms are not lost. Add reranking so the best chunk reaches the model. And underneath all of it, keep the source data clean and current, because retrieval can only be as good as the documents it draws from.
Questions
Why does my RAG chatbot make things up or give wrong answers?
Usually because retrieval handed the model the wrong documents or none at all, so it filled the gap with a guess. Fixing chunking, matching, and ranking removes most wrong answers before touching the model.
What is chunking in RAG?
Chunking is splitting your documents into pieces for storage and retrieval. Chunks that are too large mix topics and too small lose context, so each chunk should hold one coherent idea, often split on headings.
What is a reranker in a RAG system?
A reranker is a second pass that reorders retrieved candidates by how well they answer the question, then passes only the best few to the model. It is often the biggest single quality improvement for a weak RAG system.
Two ways to make a model know your world. When to retrieve, when to fine-tune, and when you need both.
The per-question price is the small part. Where the real cost of a RAG system lives, and how to control it.
SEO ranks a page for a person to click. GEO gets a passage quoted by a model. What actually differs, and what to do about it.