RAG Cheatsheet

Retrieval-Augmented Generation — from retrieval methods to production deployment

1. RAG Overview

RAG augments LLM responses by retrieving relevant documents from a knowledge base before generation.

Benefits

Better responses (grounded in facts) • Up-to-date info (not limited by training cutoff) • Domain knowledge injection • Reduced hallucinations

Keyword

Exact matches, high sensitivity

Semantic

Similar meaning, flexible

Metadata

Date, author, access control (SQL-like)

2. Search Methods

TF-IDF

Inverted index (word → doc mapping)
Score = count / doc_length
IDF = log(total_docs / docs_with_word)
Rare words get higher weight

BM25

Term freq saturation (diminishing returns)
Document length normalization
k — saturation param
b — length norm param

Semantic Search

Embeddings measure closeness in vector space. Cosine similarity: measures direction (−1 to 1), not magnitude.

Hybrid Search + Reciprocal Rank Fusion (RRF)

Score = Σ 1/(k + rank in list) — rewards docs ranked high across multiple lists
Single high rank doesn't dominate overall ranking
Beta: weight semantic vs keyword (0.7 = 70% semantic)

3. Evaluating Retrieval

Recall@K

Relevant Retrieved / Total Relevant

Precision@K / MAP

Relevant Retrieved / Total Retrieved

MRR

1 / rank of first relevant doc

⚠️ All metrics require ground truth labeled data

3b. Vector DB & Advanced Retrieval

KNN

Exact K nearest neighbors — high quality but doesn't scale.

ANN

Approximate nearest neighbor — significantly faster, slight quality tradeoff.

HNSW

Hierarchical proximity graph (1000→100→10 vectors). O(log n) query time but expensive to build.

Chunking Strategies

Fixed + overlap — simple, fast
Recursive character — respects natural boundaries
Semantic — smart but expensive
Language-based / Context-aware — domain-specific

Re-ranking

Cross-Encoders: Concat prompt+doc → relevance score. Better quality but can't preprocess — scales terribly.
ColBERT: MaxSim between word embeddings in prompt & doc. Higher quality but needs more storage.

Query Rewriting

Use LLM to rewrite the prompt for better retrieval. Apply named entity recognition (places, dates, orgs) to expand the query.

4. LLMs & Text Generation

Sampling Strategies

Greedy (top token) • Top-K (sample from K most likely) • Top-P / Nucleus (sample from smallest set covering probability P) • Temperature (lower = more deterministic)

LLM Characteristics

Model size (1–500B params) • Cost ($/M tokens) • Context window • Latency • Training cutoff

Benchmarks

Automated (scripted evals) • Human-evaluated (preference ratings) • LLM-as-a-judge (model scores model)

Prompting Best Practices

Add examples (few-shot)
Chain of thought ("think step-by-step")
Clear goals & strict output formats
Provide full context; manage context windows dynamically

Reduce Hallucinations

Self-consistency: repeat prompt, verify agreement across responses
Citation generation: ask model to cite sources (note: model may hallucinate citations)
ContextCite: tag which sentences in the retrieved context support the answer

RAGAS Evaluation Library

Response Relevancy: Can you reconstruct the original question from the answer?
Faithfulness: Is the response consistent with the retrieved context?

4b. Agentic Workflows

Patterns

Iterative: loop until task is done (ReAct, tool-calling loops)
Parallel: multiple agents running simultaneously
Router LLM: decide at runtime whether to use RAG, direct generation, or a tool

💡 Fine-tuning = domain adaptation (baking in style/format) | RAG = knowledge injection (dynamic external facts)

5. Production

Challenges

Scaling (latency, memory) • Unpredictable prompts • Messy real-world data • Security & privacy • Costly mistakes

Software Metrics

Latency, throughput, memory, tokens/sec

Quality Metrics

User satisfaction, response quality, faithfulness

Observability Tools

Phoenix (Arize), Datadog, Grafana — trace prompt paths, run A/B tests, log prompts & responses for analysis.

Cost Optimization

Vector DB: Smaller embedding models • Quantization (Matryoshka) • Tiered storage (RAM→Disk→Cloud) • Multi-tenancy
LLM: Smaller models • Router LLM • Caching (exact/semantic) • Dedicated endpoints

Security

Risks: KB leakage to LLM provider • Metadata filtering failures • Vector reconstruction attacks • DB hacking

Mitigations: Authenticate users • User-scoped databases • Self-hosted infrastructure • Encrypt text chunks

Multimodal RAG

PDFs contain text, charts, and images. Split into patches → vectorize. Vision-Language Models tokenize images directly. Re-ranking uses ColBERT-style scoring.