What Is Retrieval Augmented Generation (RAG)? Plain Words
What is retrieval augmented generation (RAG)? A plain-words guide to how it works, when founders should use it, and how to build an AI product on it.
If you are asking what is retrieval augmented generation (RAG), in plain words, here is the short answer. RAG is a way to make an AI model answer from your documents instead of from memory. Before the model replies, your system looks up the most relevant pieces of text and hands them to the model along with the question. The model then writes its answer using that text.
This post explains the idea, walks through the steps, gives a worked example, and shows where it goes wrong. It also covers when a startup should use it and how to check that it works.
What is retrieval augmented generation (RAG) in one paragraph
The idea was named in a 2020 research paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Its authors joined a pre-trained language model to a searchable index of documents, and reported more specific, diverse and factual language than a model with no index. RAG has two parts. Retrieval means finding the right passages from a pile of your own content. Generation means the language model writes a reply. You put them together: search first, then write. The model does not need to have learned your content during training. It reads the passages you give it at the moment of the question, the way a person might open a handbook before answering a customer.
Why AI models need RAG: the problem it solves
A language model works by predicting the next token, one after another. It learned from a large pile of text, and then it stopped learning. That creates three problems for a builder:
- It does not know your data. Your help articles, contracts, price list and product notes were not in its training.
- It can be out of date. Anything that changed after training is missing.
- It can sound sure and be wrong. When it lacks the facts, it may still produce a fluent answer.
The Age of AI Entrepreneur lessons Next-Token Prediction and What AI Gets Wrong cover how models predict text and what they get wrong. RAG is one of the standard ways to design around it. You give the model the facts it needs, right when it needs them.
How RAG works step by step
A basic RAG system has a preparation stage and a question stage.
Preparation (done ahead of time)
- Collect your content. Help pages, PDFs, policies, notes.
- Split it into chunks. Each chunk is a short passage, often a few paragraphs, so you can fetch only the part that matters.
- Index the chunks. Many systems turn each chunk into a list of numbers that captures its meaning, so a search can match by meaning and not only by exact words. Others use plain keyword search. Some use both.
When a user asks a question
- Search. The system finds the few chunks closest to the question.
- Build the prompt. It puts those chunks and the question into one request to the model.
- Generate. The model writes an answer using the chunks.
- Show sources. Good products point to the chunk the answer came from, so the user can check it.
Each request costs tokens, and the chunks you add make the request longer. The lesson Model APIs, Tokens and What a Request Costs in the Intermediate course The AI Product Stack covers what a request costs. The next lesson in that course, Giving Models Your Data: Retrieval, is the one that covers this topic directly.
A simple everyday example of RAG
Say you run a small online shop that sells bike parts. The policy and prompt below are made up for this example. A customer asks: "Can I return a chain I already installed?"
Without RAG, the model guesses what a typical shop does. It may invent a 30 day policy you never wrote.
With RAG, this happens instead:
- Your return policy was split into chunks earlier.
- The system searches and finds the chunk about used and installed parts.
- It sends the model a prompt like this:
Answer the customer using only the policy text below. If the text does not answer the question, say you are not sure and offer to pass it to a person. Policy text: "Parts that have been installed or show wear cannot be returned. Unused parts in original packaging can be returned within 14 days." Question: Can I return a chain I already installed?
The model replies that installed parts cannot be returned, and it can quote the line. The key instruction is "using only the policy text" plus the fallback for when the text is silent. That small piece of prompt does a lot of work.
RAG vs fine-tuning vs a bigger prompt
You have three common ways to get a model to use your information.
- Put it all in the prompt. Simplest option. It works when your content is short, like a one page FAQ. It gets costly and slow as the content grows, and the model can miss details buried in a very long prompt.
- RAG. Best when you have a lot of content that changes. You update the documents and the answers update. No retraining.
- Fine-tuning. You train a model further on examples. It is better suited to teaching a style, a format or a narrow behavior than to storing facts that change often.
A practical order: try a good prompt first, add RAG when the content outgrows the prompt, and look at fine-tuning only when you have a clear behavior problem that the first two do not fix.
When your startup should use RAG (and when not to)
Use RAG when:
- Answers must come from your own content, such as docs, policies or a knowledge base.
- The content changes often.
- Users need to see where an answer came from.
Skip it, or wait, when:
- Your content fits comfortably in one prompt.
- The task is general writing or reasoning that needs no private facts.
- You have not yet confirmed anyone wants the product. Test demand first with a landing page and a waitlist.
One more warning. If the only thing you built is a chat box on top of a model plus some documents, anyone can copy it. Read Is my AI startup just a thin wrapper around ChatGPT? before you decide RAG alone is your edge. Your advantage is more likely to be the data you have, the workflow you fit into and how you reach customers. See how to find the right distribution channel.
Also watch the cost. Longer prompts mean more tokens per answer. Check your margins with how to price an AI product without losing money per user.
Common RAG mistakes that make answers wrong
- Bad chunks. If you cut a passage in the middle of a rule, the retrieved piece can mislead the model. Split on natural boundaries like headings.
- Retrieving the wrong text. If the search returns an unrelated passage, the model will still try to answer from it. A wrong answer can look like the model making things up when the real fault is in the search step.
- Too many chunks. Stuffing in ten passages when two matter adds noise and cost.
- No permission to say "I don't know." Without a fallback instruction, the model fills gaps with a guess.
- Stale documents. RAG is only as current as the files you indexed. Old policy in, old answer out.
- No sources shown. Users cannot catch errors if they cannot see where the answer came from.
How to test a RAG product with evals
Do not judge by reading a few answers and feeling good about them. Build a small set of real questions with known right answers, run them through your system, and score the results. The Advanced course Evals: Shipping AI That Works teaches this, including Building an Eval Set and Judging Outputs. For the background, read what LLM evals are and why your AI product needs them.
For RAG, check two things separately:
- Did retrieval find the right chunk? For each test question, note which passage should come back and see if it did.
- Did the model use it correctly? Even with the right chunk, the answer can be wrong or can add claims the text does not support.
Include trick questions too: ones your documents do not answer. The right behavior is a clear "I am not sure," not a confident guess. Re-run the set every time you change chunk size, the search method or the prompt, so you can see whether the change helped.
Frequently asked questions
Does RAG stop an AI from making things up?
It lowers the risk, but it does not remove it. The model can still misread the text, or retrieval can return the wrong passage. Show sources and test with evals.
Do I need to know how to code to use RAG?
To understand it, no. To build a custom version you will need some code or a tool that handles it for you. In the Intermediate level, the course Your First Builds moves from no-code to light-code projects, so you can build up gradually.
Is RAG the same as fine-tuning?
No. RAG fetches your content at question time and gives it to the model. Fine-tuning changes the model itself through extra training. RAG suits facts that change, and fine-tuning suits style or format.
How much content do I need before RAG makes sense?
If your content fits in one short prompt, start there. Move to RAG when it grows too large or changes often enough that pasting it in becomes slow, costly or hard to keep current.
Get started
The lesson Giving Models Your Data: Retrieval sits in the course The AI Product Stack, alongside lessons on request costs and choosing models. Like every lesson, it is short, hands-on and free with an account. It does not build a product for you, but it gives you the working picture you need to build one.
Every lesson is free. Start learning with Age of AI Entrepreneur: sign in on the home page and take your first short lesson today.