Skip to content

Expert: deep learning and AI · Building with generative AI, and MLOps · topic 53 of 62 · 35 min

Retrieval-augmented generation (RAG)

Explanation

RAG lets an LLM answer from your own documents. You split documents into chunks, turn each chunk into an embedding, and store them. For each question you retrieve the most similar chunks and put them in the prompt with an instruction to answer only from them and cite them. This gives fresher, private and checkable answers with fewer hallucinations.

Why RAG?

An LLM only knows what was in its training data, which is old and does not include your company's HR policy, your product manual or yesterday's circular. Retraining it for every document change is slow and costly. RAG (retrieval-augmented generation) instead looks up the relevant text at question time and hands it to the model. The model then works like a student in an open-book exam.

The pipeline

  1. Load documents: PDFs, web pages, FAQs, tickets.
  2. Chunk: split each document into pieces of a few hundred tokens, often with some overlap so a sentence cut at the edge still appears whole in one chunk.
  3. Embed: turn each chunk into a vector with an embedding model (module 10).
  4. Store: keep the vectors, chunk text and metadata (source, page, date, who may see it) in a vector index or database.
  5. Retrieve: embed the user's question and find the top-k most similar chunks.
  6. Generate: build a prompt with the question and the retrieved chunks, numbered, and ask the model to answer only from them and cite the numbers.
output
Answer the question using only the sources below.
If the answer is not in the sources, say "I don't know".
Cite sources like [1].

[1] Employees get 12 days of casual leave per calendar year.
[2] Unused casual leave lapses on 31 December.

Question: Can I carry forward my casual leave?
A RAG prompt. A good answer: "No. Unused casual leave lapses on 31 December [2]."

Getting retrieval right

  • Chunk size: too small loses context; too big dilutes the match and wastes tokens. Try a few sizes on real questions.
  • Hybrid search: combine vector similarity with keyword search (good for codes, names and numbers like "Form 16" or "Section 80C").
  • Re-ranking: retrieve 20 candidates cheaply, then re-order them with a more accurate model and keep the best 3–5.
  • Metadata filters: only search the current policy version, or the user's own state or branch.
  • Access control: filter by what the user is allowed to see BEFORE the text reaches the model, or the model may reveal it.
Tip: Keep the source name, page or section and the document date with every chunk. Showing "HR Policy 2025, section 4.2" next to an answer lets people check it in seconds, and lets you hide outdated versions with a simple filter.

What RAG does not fix

If retrieval brings the wrong chunk, the model answers from the wrong text. If documents conflict or are outdated, so are the answers. The model can still ignore or misread its sources. That is why you evaluate retrieval and answers separately (topic 3) and show citations so users can check.

Common mistakes

  • Skipping evaluation of retrieval. If the right chunk is not retrieved, no prompt can save the answer.
  • Embedding whole documents as single vectors. Chunk them.
  • Leaking confidential documents by searching everything for everyone.
  • No "I don't know" instruction, so the model fills gaps from memory.

Need help?

Stuck on a word or an idea? Search this course, or every course, and jump straight to the explanation or a worked question.

Try:

Watch the lesson

Retrieval-augmented generation (RAG) · 1/7

RAG

RAG lets a language model answer from your own documents, like an open-book exam.

Prefer a video by a teacher? Find free videos on “retrieval augmented generation RAG explained chunking embeddings vector database” on YouTube ↗

In real life: A GST help bot for small businesses

An accounting software company builds a help bot over its own GST guides and the official notifications it tracks. Questions such as "What is the due date for GSTR-3B for quarterly filers?" retrieve the relevant chunks, and the bot answers with citations and the date of the source document. Because rules change, the team re-indexes documents whenever a new notification is added and shows "check the official portal" for anything time-sensitive.

Practice (0/3 done)

Task 1 · Code · Python

Write chunk_words(text, size, overlap) that splits text into words (text.split()) and returns chunks of `size` words joined by single spaces. Chunks start at word 0 and each next chunk starts `size − overlap` words later. Stop after the chunk that reaches the last word. Return [] for empty text. Raise ValueError if overlap >= size.

Task 2 · Code · Python

Write top_k(query, vectors, k) that returns the indices of the k vectors most similar to query by cosine similarity, best first. Ties keep the original order. A zero vector has similarity 0.0.

Task 3 · Code · Python

Write rag_prompt(question, chunks) that returns exactly these lines joined with "\n": "Answer the question using only the sources below.", "If the answer is not in the sources, say \"I don't know\".", an empty line, one line "[n] chunk" for each chunk numbered from 1, an empty line, and "Question: " followed by the question.

Quick quiz

Question 1

1.What problem does RAG mainly solve?

Question 2

2.Why add overlap between chunks?

Question 3

3.Users search for "Section 80C limit". Pure vector search sometimes misses exact codes. What helps?

Question 4

4.Where should access control be applied in a RAG system?

Question 5

5.The right chunk is not among the retrieved chunks. What happens?

Read more (free)

Open a question to see the answer and why. The first ones are this topic's own quiz, so try the quiz above before you peek.

What problem does RAG mainly solve?

Answer: Letting an LLM use up-to-date or private documents at question time

It retrieves relevant text and passes it to the model.

Why add overlap between chunks?

Answer: So text cut at a chunk boundary still appears whole in some chunk

Overlap reduces the chance of splitting an answer in two.

Users search for "Section 80C limit". Pure vector search sometimes misses exact codes. What helps?

Answer: Hybrid search combining keywords with vectors

Keyword search is good at exact terms and numbers.

Where should access control be applied in a RAG system?

Answer: Before retrieved text reaches the model

Text the user may not see must never enter their prompt.

The right chunk is not among the retrieved chunks. What happens?

Answer: The model cannot ground its answer and may say it does not know or answer wrongly

Retrieval quality limits answer quality.

A RAG bot gives wrong answers. recall@5 is 0.4. What should you fix first?Related

Answer: Retrieval: the right source is often not even retrieved

Without the right source, the answer cannot be grounded.

From Evaluating LLM apps

Your 2,000 training examples contain many inconsistent labels. What will fine-tuning likely produce?Related

Answer: A model that copies the inconsistency

The model learns whatever patterns are in the examples, including mistakes.

From Fine-tuning, and when not to

You embed documents with model A and queries with model B. What happens?Related

Answer: Meaningless comparisons, because the vector spaces are different

Use one model for both sides.

From Embeddings and cosine similarity

Which instruction best reduces made-up answers?Related

Answer: "If the answer is not in the text, reply UNKNOWN"

A clear way out lowers the pressure to invent an answer.

From Prompting well

Pass the quiz (60% or more) or mark this topic as done when you are ready.

Your progress is saved on this device only.

Next: Fine-tuning, and when not to →

Next steps

  1. 4 more topics in “Building with generative AI, and MLOps” before its exam: Fine-tuning, and when not to, Evaluating LLM apps, AI safety and responsible deployment, MLOps: from notebook to production.
  2. Module 11 exam: Building with generative AI, and MLOps · 12 questions · 25 min, without help. 5 topics of this module still to finish first.
  3. Then the next module, Ethics, privacy and careers. 62 topics left in the course before the final exam.

← Previous: Prompting well