GyaanStack. Blog

Articles / ragcheat-sheet

The RAG cheat sheet I actually use

Every stage of a RAG pipeline on one page, with a symptom-to-fix table, the metrics that matter and a runnable hybrid search example.

By Raghunath · · 9 min read

Most RAG failures are retrieval failures. The model gets the wrong chunks, or the right chunk cut in half, and then writes a confident answer from them. So this sheet spends most of its space before the LLM call. Keep it open while you build, and come back to the table in the middle when answers go wrong.

The pipeline in one line#

ingest -> clean -> chunk -> embed -> index -> (rewrite query) -> retrieve -> rerank -> build prompt -> generate with citations -> evaluate

Each stage below has three parts: what it does, the default I start with, and what to change when it fails.

1. Ingest and clean#

Get text out of PDFs, HTML, docs and tickets, and keep the structure you will need later.

  • Keep metadata from the start: source URL or file path, page number, section heading, last-modified date, and who is allowed to read it. You cannot add permissions back after indexing.
  • PDFs with tables and two columns break simple text extraction. Check ten pages by eye before you index ten thousand.
  • Remove headers, footers and cookie banners. Repeated boilerplate matches every query and pushes real content down.
  • Store a content hash per document so re-ingesting only re-embeds what changed.

2. Chunking#

Method When I use it
Fixed size (300 to 800 tokens, 10 to 20% overlap) First version, plain prose
Split on headings and paragraphs Docs, wikis, manuals with clear sections
Parent-child (embed small chunks, return the larger parent) Answers need surrounding context
One chunk per record FAQs, tickets, product rows

Rules I follow:

  • Put the document title and section heading at the top of every chunk. A chunk that says "Click Reset" with no heading is useless on its own.
  • Never split a table row or a code block across two chunks.
  • Tune chunk size against your eval set. Don't pick it from a blog post, this one included.

3. Embeddings#

  • Pick a model that has been benchmarked on retrieval (the MTEB retrieval scores are a reasonable first filter), then test it on your own queries.
  • Embed queries and documents with the instructions the model expects. Many models need a "query:" or "passage:" prefix, and skipping it costs recall without any error.
  • Text in Hindi, Hinglish or another mixed language needs a multilingual model. Test with real user queries, not translated ones.
  • Changing the embedding model means re-embedding everything. Record the model name and version next to each vector.

4. Indexing#

  • Under about a million chunks, a flat or HNSW index in pgvector, Qdrant, Chroma or FAISS is fine. Choose the store that is easiest to run next to your app.
  • Store metadata fields you will filter on (tenant, language, date, permission group) as real filter fields, not inside the text.
  • Keep a keyword index (BM25) next to the vector index. You will want it for hybrid search.

5. Query rewriting#

Users type short, vague or chatty queries. Fix the query before you search with it.

  • Condense follow-ups. In a chat, rewrite "what about for premium members?" into a standalone question using the history.
  • Multi-query. Ask the LLM for three paraphrases, retrieve for each, merge the results.
  • HyDE. Generate a fake answer and embed that. It helps when queries and documents use very different words.
  • Decompose. Split "compare plan A and plan B" into two searches.

Each rewrite adds an LLM call. Add one only when your eval shows a gain.

6. Retrieval: keyword, vector and hybrid#

  • BM25 wins on exact tokens: error codes, product names, IDs, rare words.
  • Vector search wins when the query and the document say the same thing in different words ("money back" and "refund").
  • Hybrid search runs both and merges the two ranked lists. Reciprocal rank fusion (RRF) is the simplest merge and needs no score tuning: each list gives a document 1 / (k + rank), with k around 60, and you sort by the sum.
  • Retrieve wide (top 20 to 50), then let the reranker cut it down.

A runnable example#

This needs no API key. It compares BM25, a character n-gram TF-IDF ranker (standing in for embeddings so the script runs anywhere), and their RRF fusion, and it scores each one with recall@k and MRR.

pip install rank_bm25 scikit-learn
from rank_bm25 import BM25Okapi
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

docs = [
    "Refunds are issued within 7 business days after the returned item reaches our warehouse.",
    "To reset your password, open Settings, choose Security and click Reset password.",
    "Error code E-1042 means the payment gateway timed out. Retry after two minutes.",
    "Our warehouse in Pune ships orders Monday to Saturday.",
    "Premium members get free shipping on orders above Rs 499.",
]
queries = {  # query -> index of the document that answers it
    "what does E-1042 mean": 2,
    "how long until I get my money back": 0,
    "change my login password": 1,
}

def tok(s):
    return s.lower().replace(",", " ").replace(".", " ").split()

bm25 = BM25Okapi([tok(d) for d in docs])
# char n-grams stand in for embeddings here: they match "password" to "passwords" etc.
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5)).fit(docs)
dmat = vec.transform(docs)

def rank_bm25(q):
    s = bm25.get_scores(tok(q))
    return sorted(range(len(docs)), key=lambda i: -s[i])

def rank_dense(q):
    s = cosine_similarity(vec.transform([q]), dmat)[0]
    return sorted(range(len(docs)), key=lambda i: -s[i])

def rrf(*rankings, k=60):
    # reciprocal rank fusion: each list votes 1/(k + rank)
    score = {}
    for r in rankings:
        for rank, i in enumerate(r):
            score[i] = score.get(i, 0) + 1 / (k + rank + 1)
    return sorted(score, key=lambda i: -score[i])

def recall_at_k(ranker, k):
    return sum(ranker(q)[:k].count(gold) for q, gold in queries.items()) / len(queries)

def mrr(ranker):
    return sum(1 / (ranker(q).index(gold) + 1) for q, gold in queries.items()) / len(queries)

hybrid = lambda q: rrf(rank_bm25(q), rank_dense(q))
for name, r in [("bm25", rank_bm25), ("char-tfidf", rank_dense), ("hybrid", hybrid)]:
    print(f"{name:11s} recall@1={recall_at_k(r, 1):.2f}  recall@3={recall_at_k(r, 3):.2f}  MRR={mrr(r):.2f}")

for q, gold in queries.items():
    print(f"{q!r:40s} gold doc at rank  bm25={rank_bm25(q).index(gold)+1}  "
          f"char={rank_dense(q).index(gold)+1}  hybrid={hybrid(q).index(gold)+1}")

Output:

bm25        recall@1=0.67  recall@3=1.00  MRR=0.83
char-tfidf  recall@1=0.67  recall@3=0.67  MRR=0.75
hybrid      recall@1=0.67  recall@3=1.00  MRR=0.83
'what does E-1042 mean'                  gold doc at rank  bm25=1  char=1  hybrid=1
'how long until I get my money back'     gold doc at rank  bm25=2  char=4  hybrid=2
'change my login password'               gold doc at rank  bm25=1  char=1  hybrid=1

The second query is the useful one. "Money back" shares no words with "Refunds are issued", so both lexical rankers miss it at rank 1. That is the gap a real embedding model closes. Swap rank_dense for an embedding model (sentence-transformers runs locally) and rerun. With three queries the numbers mean little; with fifty labelled queries from real users, this same script becomes your retrieval test.

7. Reranking#

  • A cross-encoder reranker reads the query and each candidate together and scores them. It is slower than vector search and much better at ordering.
  • Retrieve 30, rerank, keep the top 3 to 8.
  • If the reranker keeps putting the right chunk at the top, you can shrink the prompt and cut cost.

8. Building the prompt and generating with citations#

  • Give each chunk an ID in the prompt ([doc-3]) and tell the model to cite IDs after each claim.
  • Tell it what to do when the context does not contain the answer: say so and stop. Then test that it does.
  • Put the most relevant chunk first. Long contexts lose detail in the middle.
  • Check citations in code: every cited ID must exist in the retrieved set. Reject or retry answers that cite nothing.

When answers go wrong#

Symptom Likely cause Fix
Right document exists but never comes back Query and document use different words Add vector or hybrid search; try HyDE or multi-query
Exact codes or names not found Vector-only search Add BM25 and fuse with RRF
Retrieved chunk has half the answer Chunks too small or split mid-section Split on headings; parent-child chunks; more overlap
Answer mixes two products or versions No metadata filters Filter on product, version or date before ranking
Correct chunk retrieved, wrong answer Too many chunks, or right chunk buried Rerank, keep fewer, put the best first
Model answers from its own knowledge Prompt doesn't forbid it Require citations; refuse when context lacks the answer; test it
Old policy quoted Stale index Re-ingest on change using content hashes; add dates; boost recent
User sees a document they shouldn't Permissions not applied at retrieval Filter by user's groups inside the search query, not after
Fine in English, poor in Hindi English-only embedding model Multilingual model; test with real mixed-language queries
Slow responses Too many LLM calls in rewriting, big prompts Cache rewrites, rerank to fewer chunks, stream output

Metrics#

Measure retrieval and generation separately. If retrieval is broken, prompt changes won't fix it.

Retrieval (needs a labelled set of queries and their correct chunks):

  • Recall@k: share of queries where a correct chunk is in the top k. This is the first number I look at.
  • MRR: average of 1 / rank of the first correct chunk. Rewards putting it near the top.
  • nDCG@k: use it when you have graded relevance (very relevant, somewhat relevant).

Generation (needs questions, retrieved context and answers):

  • Faithfulness: is every claim in the answer supported by the retrieved context? Usually scored by an LLM judge that splits the answer into claims and checks each one.
  • Answer relevance: does the answer address the question that was asked?
  • Citation accuracy: do the cited chunks contain the claim they're attached to?
  • Refusal rate on unanswerable questions: put 10% questions with no answer in your set. The right output is "I don't know."

If you use an LLM judge, label 30 to 50 answers yourself and check how often the judge agrees with you before you trust its scores.

Production checklist#

  • [ ] Eval set of at least 50 real queries with correct chunks labelled, run on every change
  • [ ] Permission filters applied inside retrieval, with a test that two users get different results
  • [ ] Embedding model and chunking settings versioned next to the index
  • [ ] Incremental re-ingestion with content hashes and a deletion path
  • [ ] Hybrid search with a reranker, or a written reason why not
  • [ ] Citations checked in code before the answer is shown
  • [ ] "Not in the documents" path tested
  • [ ] Prompt-injection test: a document containing "ignore previous instructions" must not change behaviour
  • [ ] Tracing of query, rewrites, retrieved IDs, scores, prompt and answer for every request
  • [ ] Cost and latency per request logged, with a budget alert
  • [ ] User feedback (thumbs up or down) stored with the trace so bad answers become new test cases

Start with the eval set. Without it, every other change on this page is a guess.