RAG & Vector-DB Infrastructure
Stop stuffing docs into the prompt: chunk them, embed them, index them for approximate search, rerank the best hits, and keep the index fresh — with the latency budget to prove it.
An interactive AI Infrastructure lesson: 21 steps, about 32 minutes, on a live simulation in your browser.
The shop's support bot answers from 12,400 documentation pages: returns policy, pricing tiers, pro-plan limits. Today it stuffs the whole manual into the prompt on every question, and at 4,000 tokens a question it is slow and expensive.
RAG, retrieval-augmented generation, is the alternative: fetch the five chunks that answer the question and prompt with those. The generator stays the same Llama 3.1 8B you served in "The KV Cache"; everything around it changes.
What you will learn
Stuffing stops working
- The support bot and its docs: RAG trades one giant prompt for a pipeline: retrieve the evidence, then generate from it.
- Stuff 4,000 tokens in: Prompt tokens set the time to the first token and the price; output tokens set the rest.
- What when docs grow 10×: Stuffing scales with the corpus; retrieval scales with the question.
Cut the docs into chunks
- Cut the docs into chunks: Chunks are the unit of retrieval: small enough to be precise, overlapping enough to survive their edges.
- Overlap zero to save space?: A boundary through one sentence makes two chunks that each look irrelevant.
- Drill: chunks per page: Chunks per page = 1 + ceil((page − size) / (size − overlap)).
Embeddings as a service
- Embed everything once: Embeddings are encoded once and read on every query; generation is paid on every answer.
- Batch the encoder: Backfills batch wide for throughput; live queries run alone and pay the full pass.
- Drill: size the vector RAM: Vector RAM = chunks × dimensions × bytes per value; the index adds its graph on top.
Indexes: exact vs approximate
- Compare against everything: Exact search never misses and never ships; every production index rents recall to buy latency.
- HNSW or IVF for the bot?: HNSW rents RAM for speed; IVF rents probes for recall. Spend whichever you have.
- Grade with goldens: Hit rate says the answer came back; MRR says how far down you had to look.
Rerank and the budget
- Cheap net, careful hands: Retrieve wide with cheap scoring, then rerank narrow with the expensive scorer.
- Hybrid search: keywords plus vectors: Dense vectors catch paraphrase, BM25 catches exact tokens; RRF fuses both lists into one.
- The whole budget on one line: In a RAG answer, decode dominates; retrieval is single-digit percent.
- Drill: add the budget: A latency budget is a sum: quote each part, then name the largest.
Freshness and failure
- The price changed yesterday: A stale index fails silently: version the index with the docs it was built from.
- The answer straddles a cut: A split sentence is two irrelevant chunks; audit boundaries, do not assume them.
- New model, old vectors: Vectors belong to the model that made them; mixing versions scrambles every distance.
Recap & playground
- Cheat sheet
- Playground