MODULE 3 DEEP DIVE  ·  CONTAINERS FOR GENAI & AGENTIC AI

The swap that made this worth doing

The lab's two-variable swap onto vLLM — this deck opens the machinery behind it.

paged memory the KV cache, room by room continuous batching reseat the moment one finishes the lab's flags dtype, swap-space, max-len, max-seqs why the swap paid off

Gourav Shah  ·  School of DevOps & AI  ·  Deep Dive (Part 2)

M3-DD·01

A naive server books the whole floor for a maybe

Naive KV-cache allocation reserves a contiguous worst-case span per request.

front desk one guest stay unknown an entire floor, reserved locked to this guest alone
M3-DD·02

PagedAttention runs the hotel by the room, not the floor

PagedAttention hands out rooms on demand — a ledger tracks who holds which.

front desk block table: A → room 2 B → room 4, 7 room 1 room 2 room 3 free room 4 room 5 room 6 one room at a time
M3-DD·03

Block tables map logical tokens to scattered physical blocks

Fixed-size KV blocks map logical position to scattered physical blocks.

Request A generating tokens Request B generating tokens A: block table logical → block 2 B: block table logical → blocks 1, 4 block 1 — B block 2 — A block 4 — B block 3 free physical KV blocks — 16 tokens each
M3-DD·04

98% reserved, unused, and still unavailable to anyone else

A 1024-token reservation for a 20-token answer — the rest sits empty, locked.

reserved: 1024 tokens hatched area sits empty — but locked to this request alone used: 20 tokens
M3-DD·05

40% free in aggregate, but no single gap big enough

Contiguity, not total free memory, is what naive allocation runs out of.

free — 12% free — 14% free — 14% 40% free in aggregate, none of it adjacent pending request needs one contiguous span bigger than any gap blocked
M3-DD·06

A static-batch table waits for the slowest guest to leave

A static batch reseats only once the whole dining room has finished.

table 1 finished table 2 finished table 3 finished table 4 still lingering front door CLOSED new customers, waiting
M3-DD·07

vLLM reseats the moment a table clears

The instant a table clears, a new party is seated — no batch boundary.

table 1 still eating table 3 still eating table 2 cleared → reseated now new party table 4 unrelated pace, unaffected front door OPEN
M3-DD·08

The scheduler checks every slot, every single step

Every decoding step, vLLM frees finished slots and admits a waiting request.

running batch — this scheduling iteration seq 1 — running seq 3 — running seq 4 — running seq 2 — stop token slot + blocks freed seq 9 — waiting seq 9 — admitted same slot, same step
M3-DD·09

Continuous batching helps throughput; static batching hurts short jobs twice

Static batching pads to the slowest job; continuous batching stays always-busy.

static batch longest sequence (500 tok) short job (1 tok) padded to match longest short job waits behind all nine — returns only when the whole batch finishes idle slots between batches continuous batch seq always doing work seq always doing work short job short job finishes and returns the instant it's done no idle slots, ever
M3-DD·10

Paging is what makes admitting a request every step affordable

Cheap paged allocation is what makes admitting a request every step affordable.

paged memory cheap, block-sized allocation continuous batching admits/evicts every step makes admission cheap enough to do every scheduling iteration
M3-DD·11

Two flags spend the same memory two different ways

Two flags — dtype and max-model-len × max-num-seqs — spend one shared budget.

--dtype float32 2x KV bytes/token vs bf16 --max-model-len × --max-num-seqs worst-case blocks reserved VLLM_CPU_ KVCACHE_SPACE this lab's default (1024) was worked backward from the 5 GB container cap
M3-DD·12

`--swap-space` is accepted, validated, and does nothing here

`--swap-space` is accepted and validated, but contributes zero blocks here.

GPU deployment VRAM full swap-space host RAM sequence survives — blocks swap to RAM and back instead of aborting real overflow tier this CPU build --swap-space 1 accepted, validated 0 blocks live /metrics: num_cpu_blocks="0" source: cpu_worker.py zeroes it dead weight, not a budget line
M3-DD·13

The engine log already did the arithmetic for you

This run's own startup log ties the KV budget straight to a concurrency number.

1 2 3 VLLM_CPU_KVCACHE_SPACE = 1 GiB the budget this lab sets → 1456 blocks of 16 tokens each confirmed in this run's own engine log → "Maximum concurrency for 1024 tokens: 22.75x" roughly 22 max-length sequences before the cache is full
M3-DD·14

Three gauges tell you which lever to pull

Running, waiting, and cache-usage gauges read together diagnose the bottleneck.

num_requests_running slots busy now num_requests_waiting queued, not admitted gpu_cache_usage_perc how full the pool is (CPU pool — GPU-named metric) waiting up, cache LOW → raise MAX_NUM_SEQS waiting up, cache ~100% → raise the KV cache budget
M3-DD·15

One developer at a time never creates the problem this machinery solves

Paging and batching only pay off once requests overlap and compete for memory.

serving a model to... one dev, one request at a time many concurrent users/services Ollama-class server is enough no batching pressure to relieve vLLM-class serving pays off paging + continuous batching only earn their cost with overlap
M3-DD·16

Ollama wins the scoreboard; vLLM wins the scaling curve

Ollama wins raw tok/s here; vLLM-CPU scales 3.13x to Ollama's 1.10x.

absolute speed (tok/s, sequential) vLLM-CPU ~8.4 tok/s Ollama ~33.2 tok/s Ollama wins here scaling: sequential → concurrent (3 prompts) vLLM-CPU 3.13x Ollama 1.10x vLLM-CPU wins here
M3-DD·17

MODULE 3 DEEP DIVE  ·  TAKEAWAYS

What to carry into a real deployment

1 2 3 4 5 size the KV budget as blocks-per-sequence × concurrent sequences, not "how long is my prompt" continuous batching admits/evicts every scheduling step — a short request never queues behind long ones read the /metrics gauges first — waiting-queue + full cache means more memory, not a higher concurrency cap --swap-space is a no-op on this CPU backend — verified against source and /metrics, don't assume GPU-style headroom ask whether requests overlap in time before reaching for vLLM — if not, Ollama-class serving is the right call

Next: apply these levers to a real concurrent deployment.  ·  Gourav Shah  ·  School of DevOps & AI

M3-DD·18