RAG beyond the tutorial: chunking, hybrid search, and metadata filtering
Production RAG needs more than embed-and-retrieve — semantic chunking, BM25 + vector fusion for exact tokens, tenant-scoped retrieval, and the discipline to know when NOT to use RAG at all.
The tutorial version of RAG — embed your docs, retrieve the top chunks, stuff them in the prompt — is a fine starting point (see the RAG-fundamentals and RAG-improvement posts). Production adds a handful of requirements the tutorial quietly skips, and getting them wrong is why so many RAG systems feel almost-but-not-quite right.
Chunking is a product decision
Fixed-size splits cut sentences in half and strand ideas across chunks. Structure-aware, semantic chunking — respecting headings, keeping a coherent idea together, tuning size and overlap — is one of the biggest quality levers, and it's specific to your documents, not a default you accept.
Hybrid search for exact tokens
Pure vector search is great at meaning and bad at exact matches — product codes, error strings, names, acronyms. Fuse lexical search (BM25) with dense vectors so you get keyword precision and semantic recall together. This one change fixes a whole category of 'why didn't it find the obvious document' complaints.
Metadata filtering and tenant scoping
Real corpora need filtered retrieval — by date, source, or permissions — so the model only sees documents it should. Most importantly, scope retrieval to the tenant so users can never pull each other's documents, and enforce that filter in code, not in the prompt (the same rule as query safety).
When NOT to use RAG
RAG is for retrieving relevant text from a large corpus — not a universal answer. If the knowledge fits in the prompt, just put it there. If the question is really about structured data, an NLQ query beats semantic search. If your 'knowledge base' is a database, don't wrap it in embeddings. RAG is a tool, not a reflex.
Tutorial RAG retrieves chunks. Production RAG retrieves the right chunks, for the right tenant, including the exact-match ones — and knows when not to retrieve at all.