What a production RAG system actually costs to run
The per-question price is the small part. Where the real cost of a RAG system lives, and how to control it.
The cost of a production RAG system is not one number, and the part everyone quotes, the price per model call, is usually the smallest. The real cost lives in the retrieval infrastructure, the data work underneath, and the engineering that keeps it correct at volume.
Here is where the money actually goes, so you can budget for the whole system rather than the demo.
The per-question cost is the visible tip
Each question has a direct cost: retrieving the relevant chunks, then paying the model for the tokens in the prompt and the answer. This scales with usage and with how much context you stuff into each prompt, which is why disciplined retrieval, sending the few best chunks rather than many mediocre ones, is a cost control as well as a quality one. Even at volume this is often the smaller line item.
The infrastructure that runs continuously
Underneath each question sits infrastructure that costs money whether or not anyone asks anything. A vector store to hold and search your embeddings. Compute to generate embeddings when documents change. Hosting for the application and its APIs. Monitoring and logging so you can see failures. These are steady monthly costs that a per-question estimate ignores entirely.
The data work that never shows up in the model bill
The largest cost in most RAG projects is the pipeline that keeps the source data clean and current, and it does not appear on any model invoice. Extracting from your systems, standardizing formats, refreshing on a schedule, and re-embedding what changed is ongoing engineering. This is the same foundational data work every AI project needs, and skipping it does not save money, it just moves the cost to wrong answers later.
The levers that control it
Cost is controllable if you design for it. Retrieve fewer, better chunks to cut tokens per question. Cache answers to repeated questions instead of recomputing them. Match the model to the task, using a smaller, cheaper model where it is good enough and a larger one only where it earns its price. Re-embed only what changed rather than the whole corpus. And set alerts on spend, because an AI feature that is cheap at a hundred questions can surprise you at a hundred thousand.
Budgeting for the real system
Budget in three buckets, not one: per-question usage that scales with traffic, steady infrastructure that runs regardless, and the data engineering that keeps answers correct. Teams that only budget the first are the ones whose pilot looked cheap and whose production system did not. Price the whole thing up front and the numbers hold at volume.
Questions
How much does a RAG system cost to run?
It has three parts: per-question model and retrieval cost that scales with usage, steady infrastructure like a vector store and hosting, and the ongoing data engineering that keeps sources current. The per-question price is usually the smallest of the three.
Why is my AI feature cheap in testing but expensive in production?
Testing runs a handful of questions, so only the small per-question cost shows. Production adds steady infrastructure, continuous data refresh, and high volume, which is where most of the cost lives.
How do you reduce the cost of a RAG system?
Retrieve fewer and better chunks, cache repeated answers, use a smaller model where it suffices, re-embed only changed documents, and set spend alerts so volume does not surprise you.
Two ways to make a model know your world. When to retrieve, when to fine-tune, and when you need both.
When a RAG assistant makes things up, the model is usually not the problem. Retrieval is. The three levers that fix it.
Error handling, monitoring, and the 10,000th request: why the impressive prototype is the easy 20%.