MODULE 5 DEEP DIVE  ·  CONTAINERS FOR GENAI & AGENTIC AI

The knobs nobody explained

The lab's defaults were never explained — this deck opens them

chunking size & overlap chunk_size=500 top-k how many cards k=3 the metric L2 vs cosine nomic-embed-text context budget the ceiling num_ctx=4096 what actually changes

Gourav Shah  ·  School of DevOps & AI  ·  Deep Dive (Part 2)

M5-DD·01

Cut the index card too small and the fact loses its command

A chunk boundary mid-idea splits a fact from the step that follows it

the payments service needs a restart kubectl rollout restart deploy/ payments -n prod the gap between cards card 1: the fact card 2: the command
M5-DD·02

Cut it too large and four procedures blur into one average

An oversized chunk compresses unrelated procedures into one blurry vector

restart payments… checkout 503 errors… database backups… on-call paging… one crowded card, 4 topics 4 topics, 1 embedding a mediocre match for anything specific
M5-DD·03

Overlap photocopies the tail onto the next card

The tail of chunk 1 is copied onto chunk 2 — no fact is ever orphaned

runbook text (one long document) chunk 1 500 chars chunk 2 500 chars overlap band — last 50 chars of chunk 1, overlap=50 embed → vector embed → vector
M5-DD·04

This corpus split into exactly 2 chunks, and that's why

chunk_size=500 on this ~800-char corpus produces exactly 2 chunks

1 2 3 try "\n\n" — paragraph break (preferred) then "\n" — line break then " " — word then "" — hard cut (last resort) 2 chunks this corpus, chunk_size=500 ~800-char Acme corpus ~2 runbook sections per chunk
M5-DD·05

Top-k decides how many cards land on the model's desk

k=3 always retrieves the three nearest chunks, relevant or not

query "restart payments?" 1st nearest k=3, rank 1 2nd nearest k=3, rank 2 3rd nearest k=3, rank 3 model must read all 3, every query
M5-DD·06

k=3 requested, 2 returned — you're already retrieving everything

Only 2 chunks exist here, so k=3 can't filter anything at all

k=3 requested slot 1 — filled slot 2 — filled slot 3 — nothing to give 2 chunks exist chunk 1 — dist 0.6956 chunk 2 — dist 1.0968 no third chunk to return
M5-DD·07

Distance, not similarity — and lower means closer

ChromaDB ranks by distance, not similarity — smaller number wins

the librarian book A book B ruler measures the gap default squared-L2 (Euclidean) distance closer = smaller number
M5-DD·08

Same ranking here, but only because the vectors sit on the unit circle

On the unit circle, L2 distance and cosine angle agree exactly

vector a vector b L2 distance cosine angle = ‖a−b‖² = 2 − 2·cos(a,b) true only when ‖a‖=‖b‖=1 nomic-embed-text ≈ unit-norm
M5-DD·09

The embedder decides what "similar" means — the LLM never gets a vote

The LLM never sees the corpus until retrieval hands it a slice

corpus nomic-embed-text defines "similar" Chroma ranking retrieved chunks qwen2.5 never sees the rest
M5-DD·10

Two embedding models never share a coordinate system

Two embedding models never share a coordinate system

model A's space model B's space "restart payments" "restart payments" not comparable swap models = full re-ingest
M5-DD·11

Four pieces have to fit inside one 4096-token ceiling

Four pieces compete for one fixed 4096-token ceiling

num_ctx = 4096 tokens scaffold context Q answer headroom real ceiling, independent of qwen2.5:1.5b's native training context
M5-DD·12

At this lab's scale, the prompt uses under 15% of the ceiling

This prompt uses under 15% of the 4096-token ceiling

1 2 3 = + template scaffolding — ~20 tokens retrieved context: 3 chunks × ~500 chars — ~300–390 tokens user question — ~15–30 tokens prompt total — ~350–450 tokens (chunk_size=500, k=3) headroom for the answer — ~3,650–3,700 tokens
M5-DD·13

Ollama keeps the tail and drops the front, verified live

Verified live: Ollama keeps the tail and drops the front

keep=4 discarded — prompt=33,742 tokens total surviving tail — limit=2050 log line: "truncating input prompt" the question lives here prompt_eval_count=2050
M5-DD·14

Two answers wrong for two different reasons

Two wrong answers, two different root causes to check first

retrieval miss wrong chunk retrieved model reasons fine on what it was given fix: chunking / top-k / embedding generation miss right chunk retrieved model still answers wrong or hallucinates fix: the prompt or the model
M5-DD·15

Naive RAG never asks "is the 2nd-best chunk actually better?"

Naive RAG never re-ranks — top distance wins, straight to the model

query vector search raw distance only re-ranker — missing — LLM the gap M6's agentic retrieval fills
M5-DD·16

ChromaDB's own API confirms the metric, not a guess

ChromaDB's own API confirms the metric — no override was ever set

1 2 3 curl .../api/v1/collections "space": "l2" "dimension": 768 confirms §3's claim — not inferred, read straight from the API no override set in app/main.py
M5-DD·17

0.6956 and 1.0968 — the numbers behind "Found 2 relevant chunks"

The real distances behind "Found 2 relevant chunks"

Q 1 2 "How do I restart the payments service?" dist 0.6956 — payments runbook chunk dist 1.0968 — checkout 503 chunk only 2 print — k=3 requested, 2 chunks exist
M5-DD·18

Three sentences, three vectors, norm 1.000000 every time

Three vectors, norm 1.000000 every time — measured, not assumed

1 2 3 "restart payments service?" "checkout 503 errors…" "database backups…" L2norm=1.000000 L2norm=1.000001 — rounding L2norm=1.000000
M5-DD·19

Same corpus, three chunk sizes, three different stories

Same corpus, three chunk sizes, three very different outcomes

baseline 500 / overlap 50 2 chunks dist 0.6956 the middle ground variant-a 150 / overlap 0 11 chunks dist 0.5146 — best nothing dilutes the vector variant-b 1200 / overlap 200 1 chunk dist 0.75–1.08 — worst every question, same chunk
M5-DD·20

MODULE 5 DEEP DIVE  ·  TAKEAWAYS

On a corpus this small, even the worst chunking still works

! 1 2 3 4 5 small-corpus-masks-dilution — variant-b's collapse-everything failure would bite on a larger corpus (this one is 823 bytes) chunk size trades context completeness against embedding precision — size it to your corpus top-k is a context-budget decision, not a "more retrieval is safer" dial ChromaDB's default metric is L2, not cosine — verify, don't assume, when you swap embedders the embedder — not the LLM — determines what gets retrieved; re-embed the whole corpus on a swap "RAG answered wrong" splits into retrieval miss vs generation miss — diagnose before you fix

Next: apply these levers to a real corpus.  ·  Gourav Shah  ·  School of DevOps & AI

M5-DD·21