MODULE 3  ·  DAY 1

Production Serving with vLLM

Continuous batching, PagedAttention, and the same /v1 contract

vLLM many shots at once — heads never idle

Gourav Shah  ·  School of DevOps & AI  ·  Containers for GenAI & Agentic AI

M3·01

What you'll learn

Why vLLM is the production workhorse — and how to run it on any laptop.

12 34 Why continuous batching + PagedAttention win ~3x throughput Serve vLLM behind the same OpenAI-compatible /v1 as Ollama Run the CPU track on any laptop — NUMA patch, thread tuning The GPU track + quantization: AWQ · GPTQ · FP8 trade-offs

Same client, bigger engine — you swap the backend, your application code never notices.

M3·02

Ollama is great — until the crowd arrives

A home espresso machine pulls one perfect shot. Ask for a second mid-brew and you wait your turn.

requests waiting in line R4R3 R2R1 Ollama one shot at a time throughput stalls

Perfect for a developer at a laptop. Under concurrent load, finished slots sit idle and the queue backs up.

M3·03

Continuous batching — never let a head sit idle

The moment one shot finishes, the next order slides in — no waiting for the whole batch.

Static batch seq A — running seq B — done seq C — running idle · wasted whole batch waits for the laggard Continuous batch seq A — running seq D — slots in mid-flight seq C — running token-level scheduling · no idle slots

As soon as a sequence emits its final token and leaves, a waiting request takes its place — the single biggest reason vLLM wins under load.

M3·04

PagedAttention — virtual memory for the KV cache

Your OS never demands one giant contiguous block per program. It hands out small pages and maps them anywhere.

Naive · reserved slab A: used reserved · empty B: used reserved · empty 60–80% of VRAM reserved-but-empty Paged · pages on demand A·1B·1 A·2B·2 A·3free freefree lookup table maps pages anywhere waste < 4% — many more sequences fit

Near-zero memory waste means far more concurrent sequences fit — which is exactly what feeds continuous batching enough work to stay busy.

M3·05

The payoff: ~3x throughput under load

Same GPU, same per-shot time — but the heads never go idle, so far more coffees per hour.

1x naive server ~3x vLLM throughput

Continuous batching + PagedAttention together give roughly 3x the throughput of a naive server on the same hardware.

M3·06

Same contract, bigger engine

The wall socket from M2, again — the client speaks to the contract, not the engine behind it.

Your M2 client openai SDK · curl /v1 contract (OpenAI API) Ollama · :11434 dev laptop vLLM · :8009 production throughput

Swapping Ollama for vLLM is a one-line change — point OPENAI_BASE_URL at the new address. No code, no SDK, no image change.

M3·07

The CPU track — learn the machinery anywhere

Apple Silicon exposes no GPU to containers, so containerized vLLM runs on CPU. Slow on purpose.

Container · any laptop CPU openeuler/vllm-cpu:0.9.1 multi-arch · runs native on arm64 SmolLM2 135M / 360M / 1.7B /v1 server same OpenAI API Study the engine batcher · quantization at a walking pace

Throughput isn't the lesson on CPU — understanding the OpenAI server, the batcher, and quantization mechanics is.

M3·08

Why containers report 0 NUMA nodes

NUMA is the building's floor plan. A container is a furnished apartment inside it — the floor plan is abstracted away.

Host (building) NUMA 0NUMA 1 NUMA 2NUMA 3 Container sees numa_size = 0 cpu_count // 0 crashes one-line patch if numa_size > 0 else cpu_count

The signature teaching point: a surgical sed patch guards the division so an upstream image behaves in a containerized world.

M3·09

CPU tuning knobs that keep the laptop usable

A few environment dials do most of the work — set threads to part of your cores, not all.

OMP_NUM_THREADS the main dial · caps OpenMP threads set 2–4 (~50–75% of perf cores) VLLM_CPU_KVCACHE_SPACE GB reserved for the KV cache small (1 GB) keeps memory tight OPENBLAS / MKL = 1 keep BLAS single-threaded don't let it fight OpenMP for cores Why leave cores free? the OS & your apps stay responsive the machine won't thermally throttle

Multi-threaded BLAS and multi-threaded OpenMP thrash the cache and slow everything down — keep BLAS at one thread.

M3·10

CPU track vs GPU track

Learn the engine on any laptop; reach for the throughput payoff on an NVIDIA box.

CPU track (this module) GPU track (production) openeuler/vllm-cpu any laptop CPU no GPU flags goal: learn the engine vllm/vllm-openai · TGI alt NVIDIA GPU · safetensors Container Toolkit + --gpus all --ipc=host · ~3x throughput

Both speak the identical /v1 contract — your M2 client wouldn't change moving from CPU to GPU.

M3·11

GPU operational gotchas

Two flags and one sizing rule separate a working GPU server from cryptic crashes.

--gpus all needs NVIDIA Container Toolkit exposes host GPU + drivers to container else falls back to CPU --ipc=host shares /dev/shm multi-process attention needs it omit → crashes under load VRAM sizing 7B @ 16-bit ≈ 14 GB weights + KV-cache headroom 24 GB card, or quantize

--max-model-len and --max-num-seqs cap KV-cache size and concurrency — the usual dials to fit memory.

M3·12

Quantization — trade a little precision for a lot of room

Like a JPEG: compress weights from 16-bit floats to 4- or 8-bit ints. A little accuracy lost, the file shrinks dramatically.

AWQ · 4-bit GPTQ · 4-bit FP8 · 8-bit activation-aware best accuracy at 4-bit ≈ ¼ the VRAM of FP16 strong quality retention mature ecosystem many checkpoints on Hub 3 / 8-bit variants too slightly more loss than AWQ near-lossless newer GPUs (Hopper/Ada) needs HW FP8 support chosen for large deployments

Rule of thumb: 4-bit AWQ roughly quarters VRAM vs FP16 for a small accuracy cost — often how a 7B model fits a consumer 8 GB card.

M3·13

TO THE LAB

Same socket. Bigger engine.

Build patched CPU image Serve SmolLM2 on /v1 Point client one-line env swap the M2 client, unchanged, talks to the vLLM server you build

Serve SmolLM2 on CPU vLLM and hit the same /v1 your M2 client already speaks.

Next: Lab — Serve SmolLM2 on CPU vLLM  ·  Gourav Shah · School of DevOps & AI

M3·14