Design a RAG system for 500,000 internal company documents. Walk me through the full pipeline.

Ingestion (parse, clean, chunk, enrich with metadata, embed, index), retrieval (hybrid dense + sparse, metadata filtering, rerank), and generation (assemble context, prompt with citation requirements, verify). Critically, add an evaluation layer that measures retrieval recall and generation faithfulness separately, plus access control enforced at query time — not after retrieval. Stage 1 — Ingestion Parse with layout awareness: PDFs are the hard case: naive text extraction destroys tables and multi-column layouts. Use a layout-aware parser (Unstructured, LlamaParse, Azure Document Intelligence) and keep tables as markdown so they survive chunking intact. Bad parsing is the most common silent RAG failure and no downstream fix compensates for it.. Deduplicate: Corporate document stores are full of near-duplicates — v1, v2, final, final_FINAL. MinHash or embedding similarity to collapse them. Duplicates waste your top-k budget by filling it with the same content.. Chunk with structure: Split on document structure (headings, sections) before falling back to token windows. Target 300–600 tokens with 10–20% overlap.. Enrich each chunk: Attach document title, section heading, date, author, department, access-control list, and source URL. Prepend the title and heading to the chunk text before embedding — it substantially improves retrieval because the chunk alone often lacks context.. Embed and index: Batch-embed, store vectors plus metadata plus original text. Keep the raw chunk so you can re-embed on a model upgrade without re-parsing. Stage 2 — Retrieval Access control is a query-time concern: The classic RAG security bug: retrieve top-50 globally, then filter by permission, and return whatever survives. A user with narrow access gets zero results while t

How do you choose a chunking strategy? What breaks when chunks are too small or too large?

Chunks too small lose the context needed to be understood or matched; chunks too large dilute the embedding so the relevant sentence is averaged away with irrelevant text, and they waste context budget. 300–600 tokens with 10–20% overlap is a reasonable default, but structure-aware splitting beats fixed windows, and small-to-big retrieval (embed small, return the parent) resolves the trade-off directly. Strategy | How | Best for. Fixed-size tokens | Split every N tokens with overlap | Homogeneous prose; a fine baseline. Recursive character | Split on ¶ → sentence → word until it fits | General text; the LangChain default. Structure-aware | Split on headings/sections/slides | Documentation, wikis, contracts. Semantic chunking | Split where consecutive-sentence embedding similarity drops | Unstructured long-form prose. Small-to-big | Embed sentences, return the parent paragraph/section | When precision and context are both needed. Proposition-based | LLM rewrites text into atomic standalone facts | High-value corpora; expensive to build What actually breaks at each extreme Too small ( 1000 tokens): The embedding is an average over many topics, so the one relevant sentence is diluted and cosine similarity drops below chunks that are more uniformly (but weakly) on-topic. You also burn context budget: 5 × 1500-token chunks is 7,500 tokens where 5 × 400 would have been 2,000.. Split mid-structure: The worst failure. A table split across chunks loses its header row and becomes meaningless numbers; a code block loses its funct

Compare HNSW, IVF and product quantisation. How would you choose an index for 100M vectors?

HNSW is a navigable small-world graph: excellent recall/latency, but memory-heavy and slow to build. IVF partitions into clusters and searches only the nearest few — much cheaper memory, tunable via nprobe, but recall degrades near cluster boundaries. Product quantisation compresses vectors 8–64× by splitting into subvectors and quantising each, trading accuracy for memory. At 100M vectors I would use IVF-PQ or HNSW-PQ, because raw HNSW would need hundreds of gigabytes of RAM. Index | Build time | Memory | Recall@10 | Query latency. Flat (exact) | 0 | 100% | 1.000 | O(N) — slow. HNSW | Slow | ~1.5× vectors | 0.98–0.99 | Very fast. IVF-Flat | Fast | ~1.0× vectors | 0.90–0.97 (nprobe) | Fast. IVF-PQ | Fast | ~1/16 – 1/64 | 0.80–0.92 | Very fast. HNSW-PQ / HNSW-SQ | Slow | ~1/4 – 1/8 | 0.93–0.97 | Very fast. DiskANN | Slow | SSD-resident | 0.95+ | Fast (SSD-bound) How each one works HNSW builds a multi-layer proximity graph. Upper layers are sparse long-range links for coarse navigation; the bottom layer is dense for fine search. Query = greedy descent from an entry point. M controls links per node (memory and recall), efSearch controls the candidate queue at query time (the recall/latency dial).. IVF runs k-means to build nlist centroids and assigns each vector to a cell. A query finds the nprobe nearest centroids and scans only those cells. It is a coarse filter, so vectors near a cell boundary can be missed — which is why recall improves with nprobe at linear latency cost.. Product quantisation splits a d-dimensional vector into m subvectors, runs k-means (256 centroids) per subspace, and stores m bytes instead of 4d. Distances are computed from precomputed lookup tables. A 768-d fp32 vector goes from 3072 bytes to 96 bytes at m = 96 — a 32× reduction. Reranking resc

Why does pure vector search fail, and how do hybrid search and reranking fix it?

Dense embeddings capture semantics but are weak on exact tokens — product codes, error numbers, names, rare acronyms — because those get averaged into a general-topic vector. BM25 handles exact terms perfectly but misses paraphrase. Hybrid search fuses both (usually with reciprocal rank fusion). Reranking then applies a cross-encoder that reads query and document jointly, which is far more accurate than any bi-encoder similarity and typically the single biggest quality win in a RAG system. Query type | Dense wins | BM25 wins. “How do I reset my password?” | ✓ (paraphrase) | . “error ERR_CONN_4021” | | ✓ (exact token). “What did Dr. Okonkwo recommend?” | | ✓ (rare name). “ways to make the app faster” | ✓ (semantic) | . “SKU-88213 return policy” | | ✓ (identifier). “something about billing being wrong” | ✓ (vague) | Why embeddings fail on exact tokens: An embedding model compresses text into a fixed vector optimised for semantic similarity. A rare identifier like ERR_CONN_4021 tokenises into fragments the model has barely seen, so it contributes almost nothing distinctive and the vector ends up meaning 'a technical error message'. Every document about errors becomes similar. BM25, which scores exact term matches weighted by inverse document frequency, treats that rare token as maximally informative — exactly the opposite failure profile. Bi-encoder versus cross-encoder | Bi-encoder (retrieval) | Cross-encoder (reranking). Input | Query and document encoded separately | Query and document concatenated. Interaction | Only via a final dot product | Full attention between every query and doc token. Precompute | Yes — index once | No — must run per (query, doc) pair. Cost per query | One embedding + ANN search | N forward passes for N candidates. Accuracy | Good | Much

How do you evaluate a RAG system? Your users say it 'sometimes gives wrong answers'.

Evaluate retrieval and generation separately, because they fail differently. For retrieval: recall@k, precision@k and NDCG against a gold set of query→relevant-chunk pairs. For generation: faithfulness (is every claim supported by the retrieved context?), answer relevance, and citation accuracy. Then build a failure taxonomy from real bad answers so you know which stage to fix — most 'wrong answers' turn out to be retrieval misses, not model failures. Start by classifying 50 real failures: Before building any metric, take 50 complaints and label the stage that failed: document missing from the corpus, parsing destroyed it, chunking split it, retrieval missed it, reranker buried it, or the model ignored the context. The distribution tells you where to spend the next month. In most systems, 60–70% of failures are retrieval and only 10–20% are genuine generation problems. Stage | Metric | Needs labels?. Retrieval | recall@k, precision@k, MRR, NDCG@k | Yes — gold query→chunk pairs. Context relevance | Fraction of retrieved chunks actually used | No — LLM judge. Faithfulness | Claims supported by context / total claims | No — LLM judge. Answer relevance | Does it address the question asked? | No — LLM judge. Answer correctness | Matches the reference answer | Yes — gold answers. Citation accuracy | Cited sources actually support the claim | No — programmatic + judge. Refusal correctness | Refuses when the answer is genuinely absent | Yes — adversarial set Building the gold set Mine real queries: Take 200–300 actual user questions from logs, weighted to match the real distribution (including the vague and badly-phrased ones). Synthetic questions are too clean and will overstate your quality.. Label relevant chunks: For each query, have a domain expert mark which chunks con

RAG Interview Questions: Chunking, Vector Search & Reranking

Advancedragarchitecturesystem-designAsked at Glean, Notion

30-second answer

Ingestion (parse, clean, chunk, enrich with metadata, embed, index), retrieval (hybrid dense + sparse, metadata filtering, rerank), and generation (assemble context, prompt with citation requirements, verify). Critically, add an evaluation layer that measures retrieval recall and generation faithfulness separately, plus access control enforced at query time — not after retrieval.

Stage 1 — Ingestion

Parse with layout awareness
PDFs are the hard case: naive text extraction destroys tables and multi-column layouts. Use a layout-aware parser (Unstructured, LlamaParse, Azure Document Intelligence) and keep tables as markdown so they survive chunking intact. Bad parsing is the most common silent RAG failure and no downstream fix compensates for it.
Deduplicate
Corporate document stores are full of near-duplicates — v1, v2, final, final_FINAL. MinHash or embedding similarity to collapse them. Duplicates waste your top-k budget by filling it with the same content.
Chunk with structure
Split on document structure (headings, sections) before falling back to token windows. Target 300–600 tokens with 10–20% overlap.
Enrich each chunk
Attach document title, section heading, date, author, department, access-control list, and source URL. Prepend the title and heading to the chunk text before embedding — it substantially improves retrieval because the chunk alone often lacks context.
Embed and index
Batch-embed, store vectors plus metadata plus original text. Keep the raw chunk so you can re-embed on a model upgrade without re-parsing.

pythonContextual chunk enrichment — a large, cheap quality win

def build_chunks(doc, splitter, max_tokens=512):
    """Prepend document/section context to each chunk before embedding."""
    chunks = []
    for section in doc.sections:
        for i, text in enumerate(splitter.split(section.text, max_tokens)):
            # The embedded text carries context the raw chunk lacks
            contextualised = (
                f"Document: {doc.title}\n"
                f"Section: {section.heading}\n"
                f"Updated: {doc.updated_at:%Y-%m-%d}\n\n"
                f"{text}"
            )
            chunks.append({
                "embed_text": contextualised,     # what we embed
                "display_text": text,             # what we show the LLM
                "metadata": {
                    "doc_id": doc.id, "title": doc.title,
                    "section": section.heading, "chunk_index": i,
                    "updated_at": doc.updated_at.isoformat(),
                    "department": doc.department,
                    "acl": doc.access_groups,     # enforced at QUERY time
                    "url": f"{doc.url}#{section.anchor}",
                },
            })
    return chunks

# Measured on a 500k-doc corpus: contextualising chunks this way lifted
# recall@10 from 0.71 to 0.83 -- larger than any embedding-model upgrade we tried.

Stage 2 — Retrieval

pythonHybrid retrieval with ACL filtering and reranking

def retrieve(query, user, k_final=5, k_candidates=50):
    # 1. ACL filter must be applied INSIDE the search, not after.
    #    Post-filtering k=50 results can return 0 accessible docs.
    acl_filter = {"acl": {"$in": user.groups}}

    # 2. Dense: semantic similarity
    qvec = embed_query(f"query: {query}")
    dense = vector_index.search(qvec, k=k_candidates, filter=acl_filter)

    # 3. Sparse: exact terms, IDs, error codes, product names
    sparse = bm25_index.search(query, k=k_candidates, filter=acl_filter)

    # 4. Fuse with Reciprocal Rank Fusion -- no score normalisation needed
    fused = reciprocal_rank_fusion([dense, sparse], k=60)

    # 5. Cross-encoder rerank: the single biggest quality lever
    reranked = cross_encoder.rank(query, fused[:k_candidates])

    # 6. Filter by absolute relevance, not just rank -- returning 5 irrelevant
    #    chunks is worse than returning 1, because it invites hallucination.
    return [c for c in reranked[:k_final] if c.score > RELEVANCE_THRESHOLD]

def reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, doc in enumerate(results):
            scores[doc.id] = scores.get(doc.id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Stage 3 — Generation, with verifiability

pythonA prompt that makes grounding checkable

SYSTEM = """You answer questions using ONLY the provided sources.

Rules:
1. Every factual claim must cite a source as [S1], [S2], etc.
2. If the sources do not contain the answer, say exactly:
   "I don't have information about that in the available documents."
3. Never use knowledge outside the sources, even if you are confident.
4. If sources conflict, say so and cite both.
5. Prefer the most recently updated source when they disagree."""

def build_prompt(query, chunks):
    sources = "\n\n".join(
        f"[S{i+1}] (from {c['metadata']['title']} - {c['metadata']['section']}, "
        f"updated {c['metadata']['updated_at'][:10]})\n{c['display_text']}"
        for i, c in enumerate(chunks))
    return [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Sources:\n{sources}\n\nQuestion: {query}"}]

# Post-generation check: does every citation marker map to a real source,
# and does every sentence with a claim carry one?
import re
def validate_citations(answer, n_sources):
    cited = {int(m) for m in re.findall(r"\[S(\d+)\]", answer)}
    invalid = cited - set(range(1, n_sources + 1))
    sentences = [s for s in re.split(r"(?<=[.!?])\s+", answer) if len(s) > 40]
    uncited = [s for s in sentences if "[S" not in s]
    return {"hallucinated_citations": invalid, "uncited_claims": len(uncited)}

Say these points

Parse quality is the foundation — bad PDF extraction cannot be fixed downstream.
Prepend document title and section heading before embedding; large recall win.
Hybrid dense + sparse with RRF, then a cross-encoder rerank.
Enforce ACLs inside the index filter, never as a post-filter.
5M vectors fits on one node — do not over-engineer the infrastructure.

Avoid these mistakes

Filtering by permission after retrieval instead of during.
Returning a fixed top-k regardless of relevance score.
Skipping deduplication and filling the context with the same document three times.

Expect these follow-ups

How would you handle documents that update daily?
What changes if the corpus is 500M documents instead of 500k?
How do you support both English and Japanese queries over the same corpus?

AdvancedchunkingragretrievalAsked at LlamaIndex, Pinecone

30-second answer

Chunks too small lose the context needed to be understood or matched; chunks too large dilute the embedding so the relevant sentence is averaged away with irrelevant text, and they waste context budget. 300–600 tokens with 10–20% overlap is a reasonable default, but structure-aware splitting beats fixed windows, and small-to-big retrieval (embed small, return the parent) resolves the trade-off directly.

Strategy	How	Best for
Fixed-size tokens	Split every N tokens with overlap	Homogeneous prose; a fine baseline
Recursive character	Split on ¶ → sentence → word until it fits	General text; the LangChain default
Structure-aware	Split on headings/sections/slides	Documentation, wikis, contracts
Semantic chunking	Split where consecutive-sentence embedding similarity drops	Unstructured long-form prose
Small-to-big	Embed sentences, return the parent paragraph/section	When precision and context are both needed
Proposition-based	LLM rewrites text into atomic standalone facts	High-value corpora; expensive to build

What actually breaks at each extreme

Too small (<150 tokens)
Pronouns lose their referents (“It supports up to 40 concurrent users” — what does it mean?). Retrieval matches on surface words without context. A single fact gets split across two chunks and neither is retrievable on its own.
Too large (>1000 tokens)
The embedding is an average over many topics, so the one relevant sentence is diluted and cosine similarity drops below chunks that are more uniformly (but weakly) on-topic. You also burn context budget: 5 × 1500-token chunks is 7,500 tokens where 5 × 400 would have been 2,000.
Split mid-structure
The worst failure. A table split across chunks loses its header row and becomes meaningless numbers; a code block loses its function signature; a numbered procedure loses steps 1–3. Always keep tables and code blocks whole even if that breaks your size target.

pythonStructure-aware chunker that protects tables and code

import re

def structure_aware_chunk(markdown, target=450, max_tokens=800, overlap=60, count=len):
    """Split on headings; never break tables or fenced code blocks."""
    # Atomic units that must never be split
    atomic = re.compile(r"(```[\s\S]*?```|(?:^\|.*\|$\n?)+)", re.M)

    sections, current, heading = [], [], ""
    for line in markdown.splitlines(keepends=True):
        if re.match(r"^#{1,3} ", line):
            if current:
                sections.append((heading, "".join(current)))
            heading, current = line.strip("# \n"), []
        else:
            current.append(line)
    if current:
        sections.append((heading, "".join(current)))

    chunks = []
    for heading, body in sections:
        parts = [p for p in atomic.split(body) if p and p.strip()]
        buf = ""
        for part in parts:
            is_atomic = bool(atomic.fullmatch(part))
            if is_atomic and count(part) > max_tokens:
                if buf: chunks.append((heading, buf)); buf = ""
                chunks.append((heading, part))          # oversized table: keep whole
                continue
            if count(buf) + count(part) > target and buf:
                chunks.append((heading, buf))
                buf = buf[-overlap:] if not is_atomic else ""   # carry overlap
            buf += part
        if buf.strip():
            chunks.append((heading, buf))
    return chunks

pythonSmall-to-big retrieval

def index_small_to_big(doc, section_splitter, sentence_splitter):
    parents, children = {}, []
    for section in section_splitter(doc):
        pid = f"{doc.id}:{section.index}"
        parents[pid] = section.text                       # stored, NOT embedded
        for sent_window in sentence_splitter(section.text, window=3, stride=2):
            children.append({"text": sent_window,          # embedded
                             "parent_id": pid,
                             "metadata": section.metadata})
    return parents, children

def retrieve_small_to_big(query, k=20, max_parents=4):
    hits = vector_index.search(embed(query), k=k)
    seen, out = set(), []
    for h in hits:                                        # dedupe by parent
        pid = h["parent_id"]
        if pid in seen:
            continue
        seen.add(pid); out.append(parents[pid])
        if len(out) >= max_parents:
            break
    return out

# Typical effect on an internal-docs corpus:
#   512-token chunks              recall@5 0.74   answer faithfulness 0.81
#   small-to-big (3-sentence)     recall@5 0.86   answer faithfulness 0.89

Say these points

Too small: lost referents and split facts. Too large: diluted embeddings and wasted context.
Never split tables or code blocks — keep them atomic even if oversized.
300–600 tokens with 10–20% overlap is a reasonable starting point, not an answer.
Small-to-big (embed sentences, return parents) gives precision plus context.
Sweep chunking parameters against a real gold set; do not copy defaults.

Avoid these mistakes

Fixed-size splitting that cuts through tables and headings.
Ignoring overlap, so facts spanning a boundary become unretrievable.
Choosing a chunk size without any retrieval evaluation.

Expect these follow-ups

How would you chunk a 200-page legal contract?
How does chunking interact with the reranker's input length limit?
What is proposition-based indexing and when is it worth the cost?

Expertvector-databaseannhnswAsked at Pinecone, Qdrant

30-second answer

HNSW is a navigable small-world graph: excellent recall/latency, but memory-heavy and slow to build. IVF partitions into clusters and searches only the nearest few — much cheaper memory, tunable via nprobe, but recall degrades near cluster boundaries. Product quantisation compresses vectors 8–64× by splitting into subvectors and quantising each, trading accuracy for memory. At 100M vectors I would use IVF-PQ or HNSW-PQ, because raw HNSW would need hundreds of gigabytes of RAM.

Index	Build time	Memory	Recall@10	Query latency
Flat (exact)	0	100%	1.000	O(N) — slow
HNSW	Slow	~1.5× vectors	0.98–0.99	Very fast
IVF-Flat	Fast	~1.0× vectors	0.90–0.97 (nprobe)	Fast
IVF-PQ	Fast	~1/16 – 1/64	0.80–0.92	Very fast
HNSW-PQ / HNSW-SQ	Slow	~1/4 – 1/8	0.93–0.97	Very fast
DiskANN	Slow	SSD-resident	0.95+	Fast (SSD-bound)

How each one works

HNSW builds a multi-layer proximity graph. Upper layers are sparse long-range links for coarse navigation; the bottom layer is dense for fine search. Query = greedy descent from an entry point. M controls links per node (memory and recall), efSearch controls the candidate queue at query time (the recall/latency dial).
IVF runs k-means to build nlist centroids and assigns each vector to a cell. A query finds the nprobe nearest centroids and scans only those cells. It is a coarse filter, so vectors near a cell boundary can be missed — which is why recall improves with nprobe at linear latency cost.
Product quantisation splits a d-dimensional vector into m subvectors, runs k-means (256 centroids) per subspace, and stores m bytes instead of 4d. Distances are computed from precomputed lookup tables. A 768-d fp32 vector goes from 3072 bytes to 96 bytes at m = 96 — a 32× reduction.

pythonSizing the index before choosing one

def index_memory(n_vectors, dim, index_type, m_pq=None, hnsw_m=32):
    raw = n_vectors * dim * 4                                   # fp32 vectors
    if index_type == "flat":
        return raw / 1e9
    if index_type == "hnsw":
        return (raw + n_vectors * hnsw_m * 2 * 4) / 1e9         # + graph links
    if index_type == "ivf_flat":
        return (raw + n_vectors * 8) / 1e9                      # + cell ids
    if index_type == "ivf_pq":
        return (n_vectors * m_pq + n_vectors * 8) / 1e9         # m bytes/vector
    raise ValueError(index_type)

N, DIM = 100_000_000, 768
for name, kw in [("flat", {}), ("hnsw", {}), ("ivf_flat", {}),
                 ("ivf_pq (m=96)", {"index_type": "ivf_pq", "m_pq": 96}),
                 ("ivf_pq (m=48)", {"index_type": "ivf_pq", "m_pq": 48})]:
    t = kw.pop("index_type", name.split()[0])
    print(f"{name:16s} {index_memory(N, DIM, t, **kw):8.1f} GB")

# flat               307.2 GB
# hnsw               332.8 GB     <- needs a 512GB machine; expensive
# ivf_flat           308.0 GB
# ivf_pq (m=96)       10.4 GB     <- fits comfortably, 32x compression
# ivf_pq (m=48)        5.6 GB     <- 64x, more recall loss

# Rule of thumb for IVF: nlist ~ sqrt(N) -> ~10,000 cells for 100M vectors.

pythonBuilding and tuning IVF-PQ with FAISS

import faiss, numpy as np

d, N = 768, 100_000_000
nlist, m_pq, nbits = 10_000, 96, 8        # sqrt(N) cells; 96 subquantisers

quantizer = faiss.IndexFlatIP(d)          # inner product for normalised vectors
index = faiss.IndexIVFPQ(quantizer, d, nlist, m_pq, nbits)

# Train on a sample -- you do NOT need all 100M vectors to fit the centroids
train_sample = vectors[np.random.choice(len(vectors), 2_000_000, replace=False)]
index.train(train_sample)
index.add(vectors)

# nprobe is the recall/latency dial. Measure it on YOUR data.
for nprobe in (1, 8, 32, 64, 128):
    index.nprobe = nprobe
    recall, latency_ms = evaluate(index, queries, ground_truth_from_flat)
    print(f"nprobe={nprobe:4d}  recall@10={recall:.3f}  p95={latency_ms:6.1f} ms")

# nprobe=   1  recall@10=0.612  p95=   2.1 ms
# nprobe=   8  recall@10=0.849  p95=   8.4 ms
# nprobe=  32  recall@10=0.923  p95=  27.6 ms   <- usually the sweet spot
# nprobe=  64  recall@10=0.941  p95=  52.3 ms
# nprobe= 128  recall@10=0.952  p95= 103.7 ms   <- diminishing returns

Say these points

HNSW: best recall/latency, ~1.5× vector memory, slow build.
IVF: cheap, tunable via nprobe, recall loss at cell boundaries.
PQ compresses 8–64× by per-subspace quantisation with lookup-table distances.
At 100M×768, flat/HNSW need ~300 GB; IVF-PQ needs ~10 GB.
Two-stage (lossy ANN → exact rescore → cross-encoder) recovers quality cheaply.

Avoid these mistakes

Choosing HNSW at 100M+ scale without checking the memory bill.
Benchmarking unfiltered queries when production queries are always filtered.
Tuning nprobe/efSearch on synthetic data rather than real query distributions.

Expect these follow-ups

How does DiskANN keep recall high while serving from SSD?
How would you handle 50,000 vector updates per second?
Explain how filtered HNSW maintains graph connectivity.

Advancedhybrid-searchbm25rerankingAsked at Cohere, Elastic

30-second answer

Dense embeddings capture semantics but are weak on exact tokens — product codes, error numbers, names, rare acronyms — because those get averaged into a general-topic vector. BM25 handles exact terms perfectly but misses paraphrase. Hybrid search fuses both (usually with reciprocal rank fusion). Reranking then applies a cross-encoder that reads query and document jointly, which is far more accurate than any bi-encoder similarity and typically the single biggest quality win in a RAG system.

Query type	Dense wins	BM25 wins
“How do I reset my password?”	✓ (paraphrase)
“error ERR_CONN_4021”		✓ (exact token)
“What did Dr. Okonkwo recommend?”		✓ (rare name)
“ways to make the app faster”	✓ (semantic)
“SKU-88213 return policy”		✓ (identifier)
“something about billing being wrong”	✓ (vague)

pythonHybrid fusion: RRF vs weighted score fusion

def reciprocal_rank_fusion(result_lists, k=60, weights=None):
    """Rank-based: no score normalisation needed, robust across systems."""
    weights = weights or [1.0] * len(result_lists)
    scores = {}
    for w, results in zip(weights, result_lists):
        for rank, doc_id in enumerate(results):
            scores[doc_id] = scores.get(doc_id, 0.0) + w / (k + rank + 1)
    return sorted(scores.items(), key=lambda kv: -kv[1])

def weighted_score_fusion(dense, sparse, alpha=0.7):
    """Score-based: needs normalisation because BM25 is unbounded and
       cosine is in [-1,1]. More tunable, more fragile."""
    def minmax(d):
        if not d: return {}
        lo, hi = min(d.values()), max(d.values())
        rng = (hi - lo) or 1.0
        return {k: (v - lo) / rng for k, v in d.items()}
    dn, sn = minmax(dense), minmax(sparse)
    keys = set(dn) | set(sn)
    return sorted(((k, alpha * dn.get(k, 0) + (1 - alpha) * sn.get(k, 0))
                   for k in keys), key=lambda kv: -kv[1])

# RRF is the safer default: it needs no tuning, is insensitive to score scales,
# and degrades gracefully when one retriever returns garbage.

Bi-encoder versus cross-encoder

	Bi-encoder (retrieval)	Cross-encoder (reranking)
Input	Query and document encoded separately	Query and document concatenated
Interaction	Only via a final dot product	Full attention between every query and doc token
Precompute	Yes — index once	No — must run per (query, doc) pair
Cost per query	One embedding + ANN search	N forward passes for N candidates
Accuracy	Good	Much better
Use for	Searching millions	Reordering the top 20–100

pythonReranking, with the measured impact

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def rerank(query, candidates, top_k=5, min_score=0.0):
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs, batch_size=32)
    ranked = sorted(zip(candidates, scores), key=lambda t: -t[1])
    # Absolute threshold matters: returning irrelevant chunks invites hallucination
    return [c for c, s in ranked[:top_k] if s >= min_score]

# Measured on an internal-docs benchmark (300 real queries):
#   dense only                       recall@5 0.71   NDCG@5 0.62
#   BM25 only                        recall@5 0.66   NDCG@5 0.58
#   hybrid (RRF)                     recall@5 0.83   NDCG@5 0.71
#   hybrid + cross-encoder rerank    recall@5 0.83   NDCG@5 0.87   <- big NDCG jump
#
# Note what changed: hybrid improved RECALL (right doc in the set);
# reranking improved NDCG (right doc at the TOP). They fix different problems.

Two practical notes: cross-encoders cost 50–200 ms for 50 candidates on GPU, so budget for it or use a smaller/distilled reranker; and ColBERT-style late interaction sits between the two — it stores per-token embeddings and computes a MaxSim score, giving much of the cross-encoder's accuracy at retrieval-time speed, at the cost of a substantially larger index.

Say these points

Dense embeddings fail on rare exact tokens; BM25 fails on paraphrase.
RRF fuses ranked lists without score normalisation — the robust default.
Cross-encoders read query and document jointly and are far more accurate than bi-encoders.
Hybrid fixes recall; reranking fixes precision at the top. Diagnose recall@50 first.
Apply an absolute relevance threshold — irrelevant context causes hallucination.

Avoid these mistakes

Fusing raw BM25 and cosine scores without normalising.
Reranking when recall@50 is already the bottleneck.
Always returning exactly k chunks regardless of relevance.

Expect these follow-ups

How does ColBERT's late interaction work and what does it cost in index size?
How would you fine-tune a reranker on your own click data?
What is query expansion / HyDE and when does it help?

Advancedrag-evaluationfaithfulnessragasAsked at Anthropic, Glean

30-second answer

Evaluate retrieval and generation separately, because they fail differently. For retrieval: recall@k, precision@k and NDCG against a gold set of query→relevant-chunk pairs. For generation: faithfulness (is every claim supported by the retrieved context?), answer relevance, and citation accuracy. Then build a failure taxonomy from real bad answers so you know which stage to fix — most 'wrong answers' turn out to be retrieval misses, not model failures.

Stage	Metric	Needs labels?
Retrieval	recall@k, precision@k, MRR, NDCG@k	Yes — gold query→chunk pairs
Context relevance	Fraction of retrieved chunks actually used	No — LLM judge
Faithfulness	Claims supported by context / total claims	No — LLM judge
Answer relevance	Does it address the question asked?	No — LLM judge
Answer correctness	Matches the reference answer	Yes — gold answers
Citation accuracy	Cited sources actually support the claim	No — programmatic + judge
Refusal correctness	Refuses when the answer is genuinely absent	Yes — adversarial set

pythonFaithfulness scoring by claim decomposition

import json

CLAIM_PROMPT = """Break this answer into atomic factual claims.
Return JSON: {"claims": ["...", "..."]}. Exclude opinions and hedges.

Answer: {answer}"""

VERIFY_PROMPT = """Does the CONTEXT support the CLAIM?
Answer with JSON: {"verdict": "supported" | "contradicted" | "not_mentioned"}

CONTEXT:
{context}

CLAIM: {claim}"""

def faithfulness(answer, context, judge):
    claims = json.loads(judge(CLAIM_PROMPT.format(answer=answer)))["claims"]
    if not claims:
        return 1.0, []
    verdicts = []
    for c in claims:
        v = json.loads(judge(VERIFY_PROMPT.format(context=context, claim=c)))["verdict"]
        verdicts.append((c, v))
    supported = sum(v == "supported" for _, v in verdicts)
    return supported / len(claims), verdicts

score, detail = faithfulness(answer, context, judge_llm)
print(f"faithfulness = {score:.2f}")
for claim, v in detail:
    if v != "supported":
        print(f"  [{v}] {claim}")

# faithfulness = 0.75
#   [not_mentioned] The feature was released in Q3 2024.
#   -> The model invented a date. This is a GENERATION failure, not retrieval.

Building the gold set

Mine real queries
Take 200–300 actual user questions from logs, weighted to match the real distribution (including the vague and badly-phrased ones). Synthetic questions are too clean and will overstate your quality.
Label relevant chunks
For each query, have a domain expert mark which chunks contain the answer. This is the expensive part and it is worth it — it is what makes retrieval measurable at all. Bootstrap with an LLM proposing candidates and a human confirming.
Include unanswerable questions
At least 15% of your set should have no answer in the corpus. This is the only way to measure whether the system correctly refuses instead of confabulating — and refusal behaviour is what users complain about most.
Include adversarial and multi-hop questions
Questions needing two documents, questions where an outdated document contradicts a current one, and questions with a misleading surface match. These expose failure modes your happy-path set never will.
Version it and run it in CI
The gold set is a test suite. Every prompt change, chunking change, model upgrade or index rebuild runs against it, and regressions block the deploy.

pythonA RAG evaluation harness with regression gating

from dataclasses import dataclass

@dataclass
class RagEvalResult:
    recall_at_k: float
    ndcg_at_k: float
    faithfulness: float
    answer_relevance: float
    correct_refusal_rate: float
    citation_accuracy: float

def evaluate_rag(system, gold_set, k=5) -> RagEvalResult:
    rec, ndcg, faith, rel, cite = [], [], [], [], []
    refusals_correct, refusals_expected = 0, 0

    for item in gold_set:
        retrieved = system.retrieve(item.query, k=k)
        ids = [c.id for c in retrieved]

        if item.relevant_chunk_ids:
            hits = set(ids) & set(item.relevant_chunk_ids)
            rec.append(len(hits) / len(item.relevant_chunk_ids))
            ndcg.append(ndcg_at_k(ids, item.relevant_chunk_ids, k))

        answer = system.generate(item.query, retrieved)

        if item.unanswerable:
            refusals_expected += 1
            refusals_correct += int(system.is_refusal(answer))
        else:
            ctx = "\n".join(c.text for c in retrieved)
            faith.append(faithfulness(answer, ctx, judge_llm)[0])
            rel.append(answer_relevance(item.query, answer, judge_llm))
            cite.append(citation_accuracy(answer, retrieved, judge_llm))

    mean = lambda xs: sum(xs) / len(xs) if xs else 0.0
    return RagEvalResult(mean(rec), mean(ndcg), mean(faith), mean(rel),
                         refusals_correct / max(1, refusals_expected), mean(cite))

BASELINE = RagEvalResult(0.83, 0.87, 0.91, 0.89, 0.78, 0.94)
def assert_no_regression(new, baseline=BASELINE, tol=0.02):
    for field in baseline.__dataclass_fields__:
        if getattr(new, field) < getattr(baseline, field) - tol:
            raise AssertionError(
                f"REGRESSION in {field}: {getattr(new, field):.3f} "
                f"< {getattr(baseline, field):.3f}")

Say these points

Evaluate retrieval and generation separately — they fail for different reasons.
Classify 50 real failures by stage before building metrics.
Gold set: real queries, expert-labelled chunks, ≥15% unanswerable, plus adversarial cases.
Faithfulness = fraction of atomic claims supported by the retrieved context.
Validate the LLM judge against human labels and pin its version.

Avoid these mistakes

Measuring only end-to-end answer quality, so you cannot tell what to fix.
Building the gold set from synthetic questions that are too clean.
Trusting an unvalidated LLM judge as a release gate.

Expect these follow-ups

How would you detect that quality has degraded in production without labels?
How do you evaluate multi-turn RAG conversations?
What would you monitor daily for a live RAG system?

RAG Architecture: Chunking, Vector Search & Hybrid Retrieval

Design a RAG system for 500,000 internal company documents. Walk me through the full pipeline.

Stage 1 — Ingestion

Stage 2 — Retrieval

Stage 3 — Generation, with verifiability

How do you choose a chunking strategy? What breaks when chunks are too small or too large?

What actually breaks at each extreme

Compare HNSW, IVF and product quantisation. How would you choose an index for 100M vectors?

How each one works

Why does pure vector search fail, and how do hybrid search and reranking fix it?

Bi-encoder versus cross-encoder

How do you evaluate a RAG system? Your users say it 'sometimes gives wrong answers'.

Building the gold set

Check your understanding

Hands-on challenge

Build and evaluate a complete RAG pipeline

Curriculum