How Retrieval-Augmented Generation Works
Large language models can generate fluent answers, but they do not automatically know your private documents, and their internal knowledge is not guaranteed to be current or perfectly accurate.
Retrieval-augmented generation, usually shortened to RAG, solves a different problem:
Retrieve relevant information from an external knowledge source, place that information into the model's context, then ask the model to answer using it.
A good RAG system is therefore not just an LLM with a vector database attached.
It is a retrieval pipeline, a context-construction pipeline and a generation pipeline working together.
SENA visual explainer: How Retrieval-Augmented Generation Works.
Why RAG exists
Imagine an employee asks:
What is our current travel reimbursement limit for Singapore?
A general-purpose model may not know the company's internal policy.
Even if a similar policy appeared in training data, relying on the model's memory would be inappropriate.
A RAG system can:
- search the company's policy documents,
- retrieve the relevant reimbursement section,
- send that text together with the question to the model,
- produce an answer grounded in the retrieved policy,
- optionally cite the source.
The external source becomes part of the model's temporary context.
The RAG pipeline at a glance
A typical pipeline has two broad phases.
Offline or ingestion phase
Documents
↓
Parse and clean
↓
Split into chunks
↓
Create embeddings
↓
Store chunks + vectors + metadata
Query-time phase
User question
↓
Query transformation / embedding
↓
Retrieve candidates
↓
Rerank / filter
↓
Construct context
↓
LLM generates answer
↓
Citations / validation
Most RAG quality problems can be traced to one of these stages.
Step 1: ingest the source material
RAG begins with documents.
They might come from:
- PDFs,
- websites,
- knowledge bases,
- support tickets,
- databases,
- internal policies,
- product documentation,
- contracts,
- source code.
The first challenge is often not AI at all. It is reliable data ingestion.
A PDF parser that loses headings, merges columns incorrectly or drops tables can poison retrieval before the embedding model ever sees the content.
Good ingestion preserves:
- text,
- document structure,
- source URLs,
- titles,
- dates,
- permissions,
- page numbers,
- sections,
- relevant metadata.
Step 2: split documents into chunks
Passing every document into the LLM for every question is expensive and often impossible.
Documents are usually divided into smaller chunks.
A naive method is fixed-length chunking, for example every few hundred tokens.
Better systems often take structure into account:
- headings,
- paragraphs,
- clauses,
- list boundaries,
- code functions,
- table sections.
The goal is to create chunks that are both retrievable and self-contained enough to be useful once retrieved.
Why chunking is harder than it looks
Suppose a policy contains:
Section 4: Reimbursement limits
The limit is RM500 per trip.
Exceptions
Senior management may approve expenses above this amount.
If chunking separates the exception from the rule, retrieval may return only "RM500 per trip" and omit the approval condition.
If the chunk is too large, it may contain several unrelated policies and become harder to rank.
Chunk design affects the evidence the model can see.
Step 3: create embeddings
Each chunk can be converted into a vector embedding.
The embedding represents semantic properties of the chunk in a numerical space.
At query time, the question is embedded using the same embedding model.
The retrieval system then searches for nearby vectors.
A question such as:
How much can I claim for a hotel?
may retrieve a chunk titled:
Accommodation reimbursement limits
even without exact keyword overlap.
Step 4: store vectors and metadata
The system stores each vector together with its chunk text and metadata.
A typical record might contain:
{
"chunk_id": "policy-2026-04-section-3",
"text": "...",
"document_title": "Travel Policy",
"country": "Singapore",
"published_at": "2026-04-01",
"access_group": "employees",
"page": 7
}
Metadata is essential for filtering and citations.
Vector similarity alone cannot enforce rules such as:
- only retrieve documents the user is authorised to see,
- only use the newest policy,
- limit results to Malaysia,
- exclude archived material.
Step 5: retrieve candidate passages
When a question arrives, the retrieval system finds a set of candidate chunks.
This can use:
- dense vector similarity,
- keyword/BM25 retrieval,
- structured filters,
- combinations of these approaches.
Vector retrieval is good at semantic similarity.
Keyword retrieval is good at exact identifiers, names and rare terms.
A robust system often uses hybrid retrieval so it can benefit from both.
Step 6: improve the query before retrieval
Users do not always ask questions in a form that is easy to search.
The system may transform the query before retrieval.
Examples include:
- expanding abbreviations,
- generating alternate queries,
- adding context from conversation state,
- decomposing a broad question into subqueries,
- extracting entities,
- rewriting follow-up questions into standalone form.
For example:
User:
What about Singapore?
The retrieval layer may rewrite it as:
What is the travel reimbursement limit for Singapore?
using conversation context.
Step 7: rerank the candidates
Initial retrieval is designed to find plausible candidates efficiently.
The top vector matches are not always the best evidence.
A reranker scores the candidate passages again using a model that can examine the query and candidate more directly.
Pipeline:
Retrieve 30 candidates
↓
Rerank candidates
↓
Keep best 5
This two-stage approach balances speed and precision.
The retriever casts a wide net. The reranker decides which passages are most useful.
Step 8: construct the prompt context
The selected passages are assembled into context for the LLM.
A simplified prompt might look like:
Answer the question using only the sources below.
If the sources do not contain the answer, say that you do not know.
Cite the source IDs used.
SOURCE 1:
...
SOURCE 2:
...
QUESTION:
What is the reimbursement limit?
Prompt construction matters because context can contain:
- duplicated passages,
- contradictions,
- irrelevant material,
- stale documents,
- instructions embedded in source text.
A RAG system needs policies for ordering, labelling and delimiting retrieved content.
Step 9: generate the answer
The language model reads the user question and the retrieved context, then generates an answer.
The LLM is still performing ordinary inference.
RAG has not inserted new permanent knowledge into the model's parameters.
The retrieved text is temporary context available for the current request.
This distinction is useful:
- fine-tuning changes model behaviour or parameters,
- RAG supplies external information at inference time.
They solve different problems and can also be used together.
Step 10: attach citations and provenance
For factual applications, an answer is far more useful when the reader can verify it.
Because each retrieved chunk should retain source metadata, the system can return:
- document title,
- URL,
- section,
- page,
- publication date,
- source ID.
Citations are not merely a user-interface feature. They also help debugging.
If the answer is wrong, you can inspect whether:
- retrieval found the wrong document,
- the right document was found but reranked poorly,
- the prompt omitted important context,
- the model ignored or misinterpreted the evidence.
The three major RAG failure modes
Failure 1: the right evidence was never retrieved
This is a retrieval failure.
Possible causes:
- poor chunking,
- weak embeddings,
- query mismatch,
- missing documents,
- overly strict filters.
No prompt can recover evidence that never reaches the model.
Failure 2: the right evidence was retrieved but buried
This is a ranking or context-construction failure.
Too many irrelevant chunks can distract the model or consume the context budget.
Reranking, deduplication and better selection can help.
Failure 3: the evidence was present but the answer was still wrong
This is primarily a generation or instruction-following failure.
Useful mitigations include:
- stronger grounding instructions,
- structured extraction before synthesis,
- citation requirements,
- answer verification,
- choosing a more capable model.
Why "top-k" is not a quality strategy
A common implementation retrieves the top five vectors and sends them directly to the model.
That works as a prototype.
In production, the number of candidates should be justified by evaluation.
Retrieving too few can miss evidence.
Retrieving too many can:
- add noise,
- increase cost,
- increase latency,
- consume context,
- surface conflicting content.
RAG is an information-retrieval problem before it is a prompt-engineering problem.
Freshness and document lifecycle
Knowledge changes.
A policy from 2024 may conflict with a policy from 2026.
Production systems need document lifecycle rules:
- version documents,
- track effective dates,
- remove or archive stale records,
- prefer authoritative sources,
- re-index changed content.
Without freshness controls, RAG can confidently ground an answer in outdated evidence.
Permissions must be enforced before generation
If the corpus contains private data, access control must be applied during retrieval.
Do not retrieve confidential passages and rely on the LLM to decide not to reveal them.
The retrieval system should filter by the user's permissions before sensitive text enters model context.
This is a fundamental security boundary.
How to evaluate a RAG system
Evaluate retrieval and generation separately.
Retrieval metrics
Ask:
- Did the correct source appear in the candidates?
- How highly was it ranked?
- Were irrelevant passages included?
- Did the system retrieve the current version?
Useful metrics include recall at k and ranking measures.
Answer metrics
Ask:
- Is the answer correct?
- Is every factual claim supported by retrieved evidence?
- Are citations accurate?
- Does the system abstain when evidence is missing?
- Does it follow the required format?
A single overall "looks good" score makes debugging much harder.
When RAG is the right approach
RAG is useful when information is:
- private,
- frequently changing,
- too large to place in every prompt,
- source-sensitive,
- expected to be cited,
- distributed across many documents.
Examples include customer support knowledge, internal policies, financial documents, research archives, legal materials and technical documentation.
When RAG is not necessary
You may not need a retrieval system when:
- the task does not depend on external knowledge,
- the required reference text is small enough to include directly,
- deterministic database lookup is more appropriate,
- the problem is primarily transformation rather than information retrieval.
Do not add a vector database simply because an application uses an LLM.
Key takeaways
- RAG retrieves external evidence and supplies it to an LLM at inference time.
- Data ingestion and chunking directly affect retrieval quality.
- Embeddings enable semantic retrieval but should often be combined with keyword search.
- Metadata supports filtering, permissions, freshness and citations.
- Reranking improves candidate ordering before context is sent to the model.
- The right evidence must reach the LLM before prompt engineering can help.
- Retrieval and answer quality should be evaluated separately.
- RAG does not permanently teach the model new knowledge.
- Security controls belong in the retrieval layer, not only in the prompt.
Frequently asked questions
Is RAG the same as fine-tuning?
No. Fine-tuning changes model parameters or behaviour. RAG retrieves information at request time and inserts it into context.
Does RAG eliminate hallucinations?
No. It can reduce unsupported answers when retrieval and prompting are good, but the model can still misinterpret, ignore or invent information.
Do I need embeddings for RAG?
Not always. Retrieval can use keywords, databases or other search methods. Many modern systems combine lexical and embedding retrieval.
What is reranking?
Reranking takes an initial set of retrieved candidates and scores them again with a more precise relevance model before selecting context.
What is the most important part of RAG?
There is no single part, but retrieval quality is a hard prerequisite. If the correct evidence is absent from the context, the generator cannot reliably use it.