Deep Dive: RAG Parameters Under the Hood

The lab wired ChromaDB and a Streamlit app together, ingested Acme's runbooks, and watched Learning Mode show a question retrieve the right chunk and generate a grounded answer. It never asked why chunk_size=500, k=3, or nomic-embed-text were the right choices — they were just the values already in app/main.py. This page opens those knobs: what moving each one actually does to retrieval quality, what distance metric ChromaDB is silently using under add_documents, how to size a context-window budget instead of guessing at top-k, and how to tell a retrieval miss from a generation miss when the app answers wrong. It closes by re-ingesting the same corpus under two different chunking strategies and comparing what each one actually retrieves.

▶ Module 5 Deep Dive — RAG ParametersOpen fullscreen ↗

Where this picks up

You need the m5 stack up — ChromaDB and genai-app from the lab. This works whether the stack is currently running or was torn down after the lab; up.sh is idempotent, so re-running it is safe.

cd labs/m5
bash up.sh

Expected output

m5 ready: chromadb + genai-app healthy.

---

1 — Chunk size and overlap: cutting the document without losing the sentence

The lab's splitter runs with chunk_size=500, chunk_overlap=50 — 500 characters per chunk, the last 50 characters of each chunk repeated at the start of the next. Nobody explained why those two numbers, or what changing them buys or costs you.

Analogy: think of chunking as cutting a long runbook into index cards for a study deck. Cut the cards too small and a card can end mid-sentence — the fact "restart the payments service" lands on one card and the actual command lands on the next, so anyone reading a single card gets half an idea. Cut the cards too large and each card tries to hold four unrelated procedures at once; when you ask the librarian for "the payments card," she hands you a card whose average meaning is a blur of four topics, and the embedding for that card is a mediocre match for anything in particular. Overlap is photocopying the last couple of lines from each card onto the front of the next one, so a fact that happens to fall right on a cut boundary is never orphaned on one card with no context.

Concretely, RecursiveCharacterTextSplitter tries to cut on paragraph and line boundaries first (its separator list is ["\n\n", "\n", " ", ""] — paragraphs, then lines, then words, then a hard character cut as the last resort), so it avoids slicing through a word or a mid-sentence run wherever a paragraph break gives it a cleaner place to cut. At 500 characters, the whole ~800-character Acme corpus is small enough that the splitter packs roughly two of the four runbook sections into each chunk — which is exactly why the lab's ingest reported 2 chunks for the whole document. That's small enough that overlap barely matters here; overlap earns its keep on longer documents where a procedure genuinely spans a chunk boundary.

Overlap means the tail of chunk 1 and the head of chunk 2 share text — a fact sitting on that boundary appears whole in at least one chunk instead of being split with no context on either side.

Too small (say chunk_size=100): chunks approach single-sentence granularity. Embeddings get more precise about what each chunk means, but each retrieved chunk carries less surrounding context — the model gets the command but not the caveat two sentences later ("the payments service depends on the Postgres primary"). You also multiply the chunk count, which raises storage and search cost for no retrieval-quality gain once chunks are smaller than a single coherent idea.

Too large (say chunk_size=2000 on this corpus): a chunk this big would swallow two or three of Acme's runbook sections into one embedding. The embedding model has to compress "payments restart," "database backup," and "checkout 503" into a single 768-number average — noise dilution. A query about payments now has to compete with unrelated content baked into the same vector, and worse, retrieving that chunk burns your context-window budget (§4) on paragraphs the question never asked about.

There is no universal right answer — it depends on how self-contained a "fact" is in your corpus. A FAQ where every Q&A pair is already a natural unit wants small chunks close to that unit's size. A long runbook where a single procedure spans several paragraphs wants larger chunks or overlap generous enough to keep a straddling step intact.

---

2 — top-k: how many cards you hand the model

The lab retrieves with search_kwargs={"k": 3} — the three most similar chunks, every query, regardless of how many chunks actually exist or how relevant the third-nearest one is.

More is not automatically better. Every chunk you retrieve is text the model has to read before it can answer, and it competes for two scarce things: your context-window budget (§4) and the model's attention. A k too small can miss the answer entirely if the right chunk lands 4th on the similarity ranking and never gets retrieved. A k too large hands the model near-duplicate or off-topic chunks alongside the real answer — on a small corpus like Acme's four runbook sections, k=3 out of only 2 total chunks (the lab's ingest count) means you're already retrieving everything indexed, so k isn't filtering at all here. On a larger corpus, a k that's too generous makes the model do the filtering work retrieval was supposed to do for it, and a plausible-sounding but wrong nearby chunk sitting in the context is exactly what produces a confidently wrong answer (§5).

The right k is corpus-shaped: a narrow FAQ where one chunk fully answers most questions wants k=1–2. A runbook where the answer commonly spans two related sections (the command and the prerequisite two sentences later) wants k=3–5. Above that, you're usually diluting more than you're helping — this is the point at which naive top-k retrieval gives way to re-ranking (§5), which this module's naive RAG doesn't have.

---

3 — Similarity metric and embedding model: what "similar" even means

The lab's Chroma(...) constructor in init_vectorstore() passes no collection_metadata argument — no hnsw:space override anywhere in app/main.py, rag_roundtrip.py, or the checks.json collection-creation call. That means the documents collection runs on ChromaDB's default index space: squared L2 (Euclidean) distance, not cosine similarity. Every "Found N relevant chunks" line Learning Mode prints in the lab is ChromaDB returning the N chunks with the smallest L2 distance to the query vector — a distance, not a similarity score, which is why smaller numbers mean "more relevant" if you ever query the metric directly (§6).

This matters in one specific way: for nomic-embed-text, whose embeddings are close to unit-norm by construction, L2 distance and cosine similarity produce the same nearest-neighbour ranking — L2² between two unit vectors is a monotonic function of their cosine similarity (‖a-b‖² = 2 - 2·cos(a,b) when ‖a‖=‖b‖=1). So the metric choice is largely academic for this specific embedding model. It stops being academic the moment you swap in an embedding model that does not normalize its output vectors — L2 distance then starts rewarding vector magnitude alongside direction, which cosine similarity ignores entirely. If you ever change embedding models, check whether the new model's vectors are normalized before assuming the ranking behaves the same way.

Why the embedder — not the LLM — defines retrieval quality. The LLM (qwen3.5:2b) never sees your documents until after retrieval hands it a slice of text. Every retrieval decision — which chunk counts as "similar" — is made entirely by nomic-embed-text's 768-dimensional representation of meaning. If the embedding model doesn't distinguish "restart" from "backup" in its vector space, no amount of LLM quality downstream fixes a bad retrieval. This is also why you cannot swap embedding models without re-embedding the whole corpus: two different models place the same sentence at different, incomparable coordinates — nomic-embed-text's vector for "restart the payments service" and a hypothetical second model's vector for the same sentence are not points in the same space, so comparing across them (or storing embeddings from one and querying with the other) produces nonsense distances. nomic-embed-text was chosen for this course because it is small enough to run natively on Ollama alongside qwen3.5:2b within the 6 GB budget and produces reasonable general-purpose retrieval quality without a GPU. A larger domain-tuned embedder (e.g. a code- or legal-specific embedding model) would out-retrieve it on that domain — at a proportionally larger memory and latency cost.

---

4 — Context-window budget: what actually has to fit

init_llm() sets num_ctx=4096 on the Ollama LLM — that is the real ceiling every prompt this app builds has to fit inside, independent of qwen3.5:2b's much larger native training context. The augmented prompt the app sends to the model (main.py, the augmented_prompt f-string) has four parts, and every one of them eats into that 4096-token budget:

Based on the following context, answer the question.

Context:
{context}                    <- retrieved chunks, joined with "\n\n"

Question: {prompt}           <- the user's question

Answer:

Work the arithmetic for this lab's actual numbers. As a rough rule of thumb, 500 characters of English prose is roughly 100–130 tokens (about 4 characters per token). With chunk_size=500 and k=3:

Budget line	Rough size
Template scaffolding ("Based on the following context...", "Question:", "Answer:")	~20 tokens
Retrieved context: 3 chunks × ~500 chars	~300–390 tokens
User question	~15–30 tokens
Prompt total	~350–450 tokens
Headroom for the answer inside num_ctx=4096	~3,650–3,700 tokens

At this lab's scale the prompt uses under 15% of the 4096-token ceiling — there's no pressure on this corpus. The arithmetic starts to matter the moment you scale either axis: raise k to 10 on a corpus with chunk_size=1500, and 10 × 1500 chars ≈ 3,750 tokens of context alone — that crowds out the answer entirely inside a 4096 window, and Ollama truncates the prompt (and logs a warning) once it exceeds num_ctx.

Verified live, not assumed. A deliberately oversized prompt (~30K tokens: a unique front marker sentence, a large filler block, a back marker, then the real question) was sent with num_ctx=4096 to two different models. The RUNNER column of ollama ps shows which engine Ollama used for each.

For the models tested here, the llama.cpp-served qwen3:4b on Ollama v0.40.7 (a stand-in, chosen only because Ollama runs it on llama.cpp rather than MLX) had its prompt cut down. prompt_eval_count came back as 2050, roughly half the context window, which leaves headroom for the answer. The Ollama server log (~/.ollama/logs/server.log on macOS) shows how:

level=WARN source=llama_server.go:334 msg="truncating input prompt" limit=2050 prompt=30861 keep=4 new=2050

It keeps a tiny fixed prefix (keep=4 tokens, effectively nothing) and discards everything else except the most recent limit tokens. The truncation keeps the tail of the prompt, not the head. The model confirmed it: its output mentioned the back marker (OTTER) and never the front marker (ZEBRA), which had been dropped. The front of an over-budget prompt is what gets dropped.

The MLX-served qwen3.5:2b (the model this lab uses) was not truncated by a near-identical prompt built the same way (30861 vs 30863 tokens). prompt_eval_count was 30863, no truncation warning appeared in the log, and the model found both markers (ZEBRA, OTTER). The runner is the likely reason, but the two runs used different models, so the cause is not proven. To test it, run one model on each runner and compare. Do not read this as permission to overfill the window: the behavior can change between Ollama releases, and it costs time and memory (that single request took about 44 seconds). The exact keep/limit split on the llama.cpp runner is also an implementation detail. So size top-k × chunk_size against num_ctx anyway, and watch for the "truncating input prompt" warning when your model runs on the llama.cpp runner. With the app's prompt shape (Context:\n{context}\n\nQuestion: {prompt}\n\nAnswer: — retrieved chunks first, the question last), truncation on that runner means an over-budget prompt loses your earliest retrieved chunks first while the question itself, sitting at the very end, is the last thing to go. Sizing top-k × chunk_size against your actual num_ctx before scaling either knob up is the check that catches this before it becomes a silent, hard-to-diagnose "the model ignored my context" bug — watch for the "truncating input prompt" warning in Ollama's server log as the tell.

---

5 — Failure modes: retrieval miss vs. generation miss

When a naive-RAG app answers wrong, there are two structurally different causes, and confusing them wastes debugging time:

• Retrieval miss — the wrong chunk (or no useful chunk) was retrieved. The model then answered correctly given what it was handed — it just wasn't handed the right thing. This is an embedding/chunking/top-k problem (§1–§3), not a model problem.
• Generation miss — the right chunk was retrieved (you can confirm this by reading "View Source Chunks" in the lab's UI or querying ChromaDB directly, §6), but the model still answered incorrectly, ignored part of the context, or hallucinated something the context didn't say. This is a prompting or model-capability problem.

Naive RAG makes retrieval misses more likely than they need to be because there is no re-ranking step. The pipeline retrieves once by raw vector distance and hands whatever comes back straight to the LLM — it never asks "is chunk #2, which scored slightly worse on embedding distance but is obviously more topically relevant, actually the better answer?" A production RAG system commonly adds a re-ranker (a smaller, more precise model that re-scores the top-N candidates from the vector search before choosing what to hand the LLM) specifically to catch cases where embedding similarity and true relevance diverge — two runbook sections can be semantically adjacent in embedding space (both are "ops procedures") while only one actually answers the question. This module's naive RAG has no such step; that gap is the class of gap M6's agentic retrieval addresses.

---

6 — Querying ChromaDB directly: seeing what Learning Mode summarizes

The Streamlit UI's Learning Mode panel already shows retrieved chunks, but it summarizes them — truncated previews, no raw distances. ChromaDB's own HTTP API gives you the unfiltered picture: the exact stored chunk text and the exact distance score behind every "Found N relevant chunks" line.

List what's actually in the collection the lab's app writes to:

curl -s http://localhost:${M5_CHROMA_PORT:-8000}/api/v2/tenants/default_tenant/databases/default_database/collections | python3 -c "
import json, sys
for c in json.load(sys.stdin):
    print(json.dumps({k: c[k] for k in ('name', 'dimension')} | {'space': c['configuration_json']['hnsw']['space']}, indent=2))
"

Chroma 1.x serves its HTTP API under /api/v2, scoped by tenant and database (default_tenant / default_database out of the box). The raw response is a long JSON document per collection, so the one-liner prints only the three fields that matter here.

Expected output

{
  "name": "documents",
  "dimension": 768,
  "space": "l2"
}

"space": "l2" (read from the collection's hnsw configuration) is ChromaDB confirming, straight from its own API, the default index metric claimed above — no override was ever set, so the collection came up on L2. "dimension": 768 confirms nomic-embed-text's embedding size.

If your documents collection is empty

The base lab only populates documents through a Streamlit UI upload — there's no headless ingest path in the app itself. If you tore the stack down with docker compose down -v (the lab's own teardown offers this) and brought it back up, the volume is gone and documents is empty, so the query below would return nothing. This guarded step checks first and only re-ingests if needed, replicating the app's own process_uploaded_file code path (UnstructuredMarkdownLoader + chunk_size=500, chunk_overlap=50 — same as §1) — not the UI:

docker exec genai-app mkdir -p /tmp/deepdive-docs
docker cp docs/acme-runbooks.md genai-app:/tmp/deepdive-docs/acme-runbooks.md
docker exec genai-app python3 -c "
import chromadb, os
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import UnstructuredMarkdownLoader
client = chromadb.HttpClient(host=os.environ['CHROMA_HOST'], port=int(os.environ['CHROMA_PORT']))
try:
    n = client.get_collection('documents').count()
except Exception:
    n = 0
if n > 0:
    print(f'documents collection already populated: {n} chunks. No re-seed needed.')
else:
    emb = OllamaEmbeddings(model='nomic-embed-text', base_url=os.environ['OLLAMA_BASE_URL'])
    vs = Chroma(client=client, collection_name='documents', embedding_function=emb)
    docs = UnstructuredMarkdownLoader('/tmp/deepdive-docs/acme-runbooks.md').load()
    chunks = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50).split_documents(docs)
    vs.add_documents(chunks)
    print(f're-seeded documents collection: {len(chunks)} chunks ingested.')
"

Expected output (tested against both states live — freshly wiped and already-populated):

re-seeded documents collection: 2 chunks ingested.

or, if the collection was already populated (the common case right after the lab):

documents collection already populated: 2 chunks. No re-seed needed.

Query it directly for a question, using the same embedding model the app uses, and see the raw distances behind the ranking (this one-liner runs inside the genai-app container, where langchain-chroma and the Ollama client are already installed):

docker exec genai-app python3 -c "
import chromadb, os
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma
emb = OllamaEmbeddings(model='nomic-embed-text', base_url=os.environ['OLLAMA_BASE_URL'])
client = chromadb.HttpClient(host=os.environ['CHROMA_HOST'], port=int(os.environ['CHROMA_PORT']))
vs = Chroma(client=client, collection_name='documents', embedding_function=emb)
for doc, score in vs.similarity_search_with_score('How do I restart the payments service?', k=3):
    print(f'{score:.4f}', doc.page_content[:80].replace(chr(10), ' '))
"

Expected output

0.6956 Acme Platform Runbooks  Payments service  To restart the Acme payments service,
1.0966 Checkout 503 errors  If the checkout page returns HTTP 503, the web tier is satu

Only two lines print even though k=3 was requested — this collection has exactly 2 chunks (§1), so ChromaDB hands back everything it has and stops; there is no third chunk to return. similarity_search_with_score returns raw (squared) L2 distance (§3) — lower is better, the opposite of a similarity percentage. This is the exact number ChromaDB's index is ranking on when the Streamlit app decides which chunks to hand the LLM; the UI never shows you this figure, only the resulting text.

Embedding norm — verifying the L2≈cosine claim from §3. The claim above (nomic-embed-text produces close-to-unit-norm vectors, so L2 and cosine rank chunks identically) is only true if it holds through this app's exact code path — langchain-ollama's OllamaEmbeddings, not just the raw Ollama API. Verify it directly:

docker exec genai-app python3 -c "
import os, math
from langchain_ollama import OllamaEmbeddings
emb = OllamaEmbeddings(model='nomic-embed-text', base_url=os.environ['OLLAMA_BASE_URL'])
vecs = emb.embed_documents([
    'How do I restart the payments service?',
    'Checkout 503 errors are caused by web tier saturation.',
    'Database backups are retained for 30 days.',
])
for i, v in enumerate(vecs):
    norm = math.sqrt(sum(x*x for x in v))
    print(f'vector {i}: dim={len(v)} L2norm={norm:.6f}')
"

Expected output

vector 0: dim=768 L2norm=1.000000
vector 1: dim=768 L2norm=1.000000
vector 2: dim=768 L2norm=1.000000

Confirmed, not assumed: three arbitrary sentences through langchain-ollama's OllamaEmbeddings all land at L2 norm ≈ 1.000000 (a last-digit wobble such as 1.000001 would be float rounding noise, not a real deviation). The L2≈cosine claim in §3 holds for this exact app code path — not a general Ollama guarantee, but true here, measured.

---

Before running the real experiment below, feel the trade-off yourself. This playground re-chunks the exact Acme corpus live as you move the dials, retrieves against the real captured distances from this course's lab, and tracks the context-budget arithmetic from §4 — including the orphan-heading trap the 150/0 variant produces.

🕹 RAG Retrieval PlaygroundOpen in new tab ↗

7 — Experiment: re-ingesting the corpus under different chunking strategies

Compare the lab's baseline chunking against two variants, without touching the running app or its documents collection — every variant gets its own ChromaDB collection name, so this runs against the same containers already up, no extra process or memory cost.

This experiment runs inside the genai-app container (that's where langchain-chroma and the Ollama client are already installed), but the corpus file only exists on your host, at labs/m5/docs/acme-runbooks.md — the image was built from app/ alone (see compose.yaml's build.context: ./app), so the container never received the docs/ folder that sits next to app/ in the lab directory. Copy it in explicitly first:

CORPUS="$(pwd)/docs/acme-runbooks.md"
mkdir -p ~/rag-deepdive-lab && cd ~/rag-deepdive-lab
docker exec genai-app mkdir -p /tmp/deepdive-docs
docker cp "$CORPUS" genai-app:/tmp/deepdive-docs/acme-runbooks.md
docker exec genai-app ls -la /tmp/deepdive-docs/

Expected output

total 12
drwxr-xr-x 2 root root    4096 Oct 21 10:49 .
drwxrwxrwt 1 root root    4096 Oct 21 10:49 ..
-rw-r--r-- 1  501 dialout  823 Jul  6 02:38 acme-runbooks.md

Write the ingest-and-compare script. The question set is embedded directly in the script (not a separate host file — the script only ever runs inside the container, so a file written next to it on the host would not be visible there). It re-ingests the corpus under three chunking configurations — the lab's own baseline (chunk_size=500, overlap=50), a smaller-chunks/no-overlap variant, and a larger-chunks/generous-overlap variant — each into its own named collection, and prints every result to stdout so the host can capture it without reaching back into the container's filesystem:

cat > compare_chunking.py << 'PYEOF'
import os
import chromadb
from langchain_ollama import OllamaEmbeddings, OllamaLLM
from langchain_chroma import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import UnstructuredMarkdownLoader

OLLAMA = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
CHROMA_HOST = os.getenv("CHROMA_HOST", "localhost")
CHROMA_PORT = int(os.getenv("CHROMA_PORT", "8000"))
DOCS = os.getenv("DOCS_PATH", "/tmp/deepdive-docs/acme-runbooks.md")

QUESTIONS = [
    "How do I restart the payments service?",
    "What happens if checkout is overloaded?",
    "How long are database backups retained?",
    "Who do I page for an unacknowledged incident?",
]

VARIANTS = {
    "baseline":  {"chunk_size": 500,  "chunk_overlap": 50},   # the lab's own values
    "variant-a": {"chunk_size": 150,  "chunk_overlap": 0},    # smaller, no overlap
    "variant-b": {"chunk_size": 1200, "chunk_overlap": 200},  # larger, generous overlap
}

# temperature=0 here (vs. the app's temperature=0.7) is deliberate: this experiment compares
# retrieval across chunking strategies, so we want the *generation* step as deterministic as
# possible. The live Streamlit app keeps 0.7 for more natural answers — expect its UI answers to
# vary more, run to run, than what this script prints.
emb = OllamaEmbeddings(model="nomic-embed-text", base_url=OLLAMA)
llm = OllamaLLM(model="qwen3.5:2b", base_url=OLLAMA, temperature=0, num_ctx=4096, reasoning=False)
client = chromadb.HttpClient(host=CHROMA_HOST, port=CHROMA_PORT)

for name, params in VARIANTS.items():
    collection = f"deepdive-{name}"
    docs = UnstructuredMarkdownLoader(DOCS).load()
    splitter = RecursiveCharacterTextSplitter(length_function=len, **params)
    chunks = splitter.split_documents(docs)
    vs = Chroma(client=client, collection_name=collection, embedding_function=emb)
    vs.add_documents(chunks)
    print(f"COLLECTION: {collection}")
    print(f"=== {name} (chunk_size={params['chunk_size']}, overlap={params['chunk_overlap']}) -> {len(chunks)} chunks ===")

    for q in QUESTIONS:
        hits = vs.similarity_search_with_score(q, k=3)
        top_doc, top_score = hits[0]
        context = "\n\n".join(d.page_content for d, _ in hits)
        prompt = f"Based on the following context, answer the question.\n\nContext:\n{context}\n\nQuestion: {q}\n\nAnswer:"
        answer = llm.invoke(prompt)
        print(f"  Q: {q}")
        print(f"    top distance: {top_score:.4f}  chunk: {top_doc.page_content[:70].strip()!r}")
        print(f"    answer: {answer.strip()[:120]}")
PYEOF
docker exec -i genai-app python3 - < compare_chunking.py | tee ~/rag-deepdive-lab/variant-collections.txt

The -i on docker exec is required — without it, docker exec never attaches its stdin to the container process, so piping the script in (< compare_chunking.py) silently sends nothing and the script exits immediately with no output. tee writes everything the container printed — collection names and per-question results — to both your terminal and the host file ~/rag-deepdive-lab/variant-collections.txt, which is what turns the container's stdout into a real artifact you can grep, diff, or fold into the table below.

Expected output

COLLECTION: deepdive-baseline
=== baseline (chunk_size=500, overlap=50) -> 2 chunks ===
  Q: How do I restart the payments service?
    top distance: 0.6956  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: To restart the Acme payments service, run the following command in your terminal:

`kubectl rollout restart deploy/payme
  Q: What happens if checkout is overloaded?
    top distance: 0.7753  chunk: 'Checkout 503 errors\n\nIf the checkout page returns HTTP 503, the web ti'
    answer: Based on the context provided, if the checkout page returns an HTTP 503 error (indicating saturation), the recommended a
  Q: How long are database backups retained?
    top distance: 0.7744  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: Based on the context provided, database backups are retained for **30 days**.
  Q: Who do I page for an unacknowledged incident?
    top distance: 0.9237  chunk: 'Checkout 503 errors\n\nIf the checkout page returns HTTP 503, the web ti'
    answer: Based on the context provided under the section "On-call escalation," if the on-call engineer is unacknowledged for 15 m
COLLECTION: deepdive-variant-a
=== variant-a (chunk_size=150, overlap=0) -> 11 chunks ===
  Q: How do I restart the payments service?
    top distance: 0.5145  chunk: 'To restart the Acme payments service, run: kubectl rollout restart dep'
    answer: To restart the payments service, run the following command:

`kubectl rollout restart deploy/payments -n prod`
  Q: What happens if checkout is overloaded?
    top distance: 0.7243  chunk: 'If the checkout page returns HTTP 503, the web tier is saturated. Scal'
    answer: If the checkout page returns HTTP 503 errors, it indicates that the web tier is saturated. To resolve this, you should s
  Q: How long are database backups retained?
    top distance: 0.3700  chunk: 'Database backups'
    answer: Based on the context provided, database backups are retained for **30 days**.
  Q: Who do I page for an unacknowledged incident?
    top distance: 0.7125  chunk: 'Page the on-call engineer via the #acme-oncall Slack channel. If unack'
    answer: Based on the context provided, you should page the **on-call engineer** via the **#acme-oncall Slack channel**. The text
COLLECTION: deepdive-variant-b
=== variant-b (chunk_size=1200, overlap=200) -> 1 chunks ===
  Q: How do I restart the payments service?
    top distance: 0.7514  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: To restart the Acme payments service, run the following command in your terminal:

`kubectl rollout restart deploy/payme
  Q: What happens if checkout is overloaded?
    top distance: 0.9770  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: If the checkout page returns HTTP 503 errors, it indicates that the web tier is saturated. In response, you should scale
  Q: How long are database backups retained?
    top distance: 0.8756  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: Based on the context provided, database backups are retained for **30 days**.
  Q: Who do I page for an unacknowledged incident?
    top distance: 1.0830  chunk: 'Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments'
    answer: Based on the context provided under the "On-call escalation" section, if an incident is not acknowledged within 15 minut

Fold the three variants' results into a comparison table (variant, chunk count, retrieved-chunk snippet, whether the retrieved context actually contained the answer, model's answer):

Variant	Chunk size / overlap	Chunk count	Retrieved top chunk (payments Q)	Contains the answer?	Model's answer
baseline	500 / 50	2	dist 0.6956 — "Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments…"	Yes	"To restart the Acme payments service, run the following command in your terminal: `kubectl rollout restart deploy/payme…"
variant-a	150 / 0	11	dist 0.5145 — "To restart the Acme payments service, run: kubectl rollout restart dep…"	Yes	"To restart the payments service, run the following command: kubectl rollout restart deploy/payments -n prod"
variant-b	1200 / 200	1	dist 0.7514 — "Acme Platform Runbooks\n\nPayments service\n\nTo restart the Acme payments…" (the whole corpus is one chunk)	Yes	"To restart the Acme payments service, run the following command in your terminal: `kubectl rollout restart deploy/payme…"

The three variants tell three different stories about the same corpus. variant-a (150 chars, no overlap) produces the tightest, lowest-distance match on the payments question (0.5145, the best of all three) because its 11 tiny chunks let one chunk be purely about the restart command — nothing else diluting the vector. It also isolates "Database backups" into its own chunk, dropping that question's distance to 0.3700, the single best match across the whole experiment — but that top chunk is a heading with no body text ('Database backups', nothing else), exactly the orphaned-fact failure mode §1 warns small chunks can produce; the answer only reads correctly here because k=3 also pulled in a neighboring chunk that actually contains the retention period. baseline (500/50) sits in the middle — 2 chunks, each covering roughly two runbook sections, distances in the 0.69–0.92 range. variant-b (1200/200) collapses the entire ~800-character corpus into a single chunk — at this chunk size on this small a corpus, there's nothing left to retrieve from; every question, regardless of topic, returns the same one chunk, and the highest distances in the table (0.75–1.08) show the vector working harder to match a chunk that's an average of four unrelated procedures. All three variants still produced grounded, correct answers here — this corpus is small enough that even variant-b's single do-everything chunk contains the whole answer space. On a larger corpus, variant-b's collapse-everything failure mode would start returning topically wrong chunks instead of just imprecise ones.

Judge the deterministic side of this table strictly — chunk counts and distance scores are exact, reproducible facts about this corpus and these parameters. Judge the model's prose answers by shape only (did it contain the right command, did it stay grounded in the retrieved text) — exact wording varies run to run even at temperature=0 on a small local model.

Teardown for this section only — drop the three variant collections created above; this does not touch the documents collection the running lab app depends on:

docker exec -i genai-app python3 -c "
import chromadb, os
client = chromadb.HttpClient(host=os.environ['CHROMA_HOST'], port=int(os.environ['CHROMA_PORT']))
for name in open('/dev/stdin').read().split():
    try:
        client.delete_collection(name)
        print('deleted', name)
    except Exception as e:
        print('skip', name, e)
" <<< "deepdive-baseline deepdive-variant-a deepdive-variant-b"

The teardown heredoc needs -i for the same reason as the ingest step above: <<< feeds the container's /dev/stdin inside the -c script, and without -i that stdin is never attached, so open('/dev/stdin').read() blocks or returns empty and no collection actually gets dropped.

Expected output

deleted deepdive-baseline
deleted deepdive-variant-a
deleted deepdive-variant-b

---

Teardown

Page-scoped only. The three deepdive-* collections were already dropped in §7. Remove the local working directory this page created:

rm -rf ~/rag-deepdive-lab

Leave the m5 stack and the documents collection exactly as the lab left them — this page never touched docker compose down, and the lab's own teardown is unaffected by anything run here.

Where you will use this

• Chunk size and overlap trade context completeness against embedding precision. Use it when: you're tuning a real corpus — a short FAQ wants small, near-atomic chunks; a runbook where procedures span paragraphs wants larger chunks or overlap generous enough that a fact never lands split with no context on either side.
• top-k is a context-budget decision, not a "more retrieval is safer" dial. Use it when: you're deciding how many chunks to retrieve for a new corpus — size it to how often the answer genuinely spans multiple chunks, not to a default that happens to work on someone else's docs.
• ChromaDB's default index metric is L2 distance, not cosine similarity — verify, don't assume. Use it when: you swap embedding models and retrieval quality drops; check whether the new model's vectors are normalized before blaming the model, since unnormalized vectors behave differently under L2 than under cosine.
• The embedding model — not the LLM — determines what gets retrieved, and embeddings from different models are not comparable. Use it when: you're planning an embedding-model upgrade — budget for a full re-ingest of the corpus, not a drop-in swap; old and new vectors in the same collection produce meaningless distances against each other.
• "RAG answered wrong" splits into retrieval miss vs. generation miss, and they need different fixes. Use it when: diagnosing a bad answer in production — check the actually-retrieved chunks (via the raw API, not the UI summary) before touching the prompt or the model; a retrieval miss needs a chunking/top-k/embedding fix, a generation miss needs a prompting fix.
• Naive RAG has no re-ranking step, so a semantically-adjacent-but-wrong chunk can outrank the actually relevant one. Use it when: you're deciding whether a corpus has outgrown naive RAG — if wrong-topic retrievals are common despite good chunking, that's the signal to add a re-ranking stage or move to an agentic retrieval loop (M6), not to keep tuning top-k.