MODULE 3  ·  DAY 1

Production Serving with vLLM

Continuous batching + PagedAttention, same /v1 contract

vLLM many shots at once — heads never idle

Gourav Shah  ·  School of DevOps & AI  ·  Containers for GenAI & Agentic AI

M3·01

What you'll learn

Why vLLM is the production workhorse — and how to run it here.

12 34 Why continuous batching + PagedAttention win ~3x throughput Serve vLLM behind the same OpenAI-compatible /v1 as Ollama Run the CPU track on any laptop — NUMA patch, thread tuning The GPU track + quantization: AWQ · GPTQ · FP8 trade-offs
M3·02

Ollama is great — until the crowd arrives

A home machine pulls one shot; ask for a second and you wait.

requests waiting in line R4R3 R2R1 Ollama one shot at a time throughput stalls
M3·03

Continuous batching — never let a head sit idle

A finished slot refills mid-flight — no waiting for the whole batch.

Static batch seq A — running seq B — done seq C — running idle · wasted whole batch waits for the laggard Continuous batch seq A — running seq D — slots in mid-flight seq C — running token-level scheduling · no idle slots
M3·04

PagedAttention — virtual memory for the KV cache

Small pages on demand, mapped anywhere — like your OS pages RAM.

Naive · reserved slab A: used reserved · empty B: used reserved · empty 60–80% of VRAM reserved-but-empty Paged · pages on demand A·1B·1 A·2B·2 A·3free freefree lookup table maps pages anywhere waste < 4% — many more sequences fit
M3·05

The payoff: ~3x throughput under load

Same GPU, same per-shot time — heads never idle, far more coffees.

1x naive server ~3x vLLM throughput
M3·06

Same contract, bigger engine

The wall socket from M2, again — the client speaks to the contract.

Your M2 client openai SDK · curl /v1 contract (OpenAI API) Ollama · :11434 dev laptop vLLM · :8009 production throughput
M3·07

The CPU track — learn the machinery anywhere

No GPU passthrough on Apple Silicon, so this runs on CPU — slow on purpose.

Container · any laptop CPU openeuler/vllm-cpu:0.9.1 multi-arch · runs native on arm64 SmolLM2 135M / 360M / 1.7B /v1 server same OpenAI API Study the engine batcher · quantization at a walking pace
M3·08

Why containers report 0 NUMA nodes

NUMA is the floor plan; a container is an apartment inside it.

Host (building) NUMA 0NUMA 1 NUMA 2NUMA 3 Container sees numa_size = 0 cpu_count // 0 crashes one-line patch if numa_size > 0 else cpu_count
M3·09

CPU tuning knobs that keep the laptop usable

A few env dials do most of the work — thread cap, not all your cores.

OMP_NUM_THREADS the main dial · caps OpenMP threads set 2–4 (~50–75% of perf cores) VLLM_CPU_KVCACHE_SPACE GB reserved for the KV cache small (1 GB) keeps memory tight OPENBLAS / MKL = 1 keep BLAS single-threaded don't let it fight OpenMP for cores Why leave cores free? the OS & your apps stay responsive the machine won't thermally throttle
M3·10

CPU track vs GPU track

Learn the engine on any laptop; reach for throughput on an NVIDIA box.

CPU track (this module) GPU track (production) openeuler/vllm-cpu any laptop CPU no GPU flags goal: learn the engine vllm/vllm-openai · TGI alt NVIDIA GPU · safetensors Container Toolkit + --gpus all --ipc=host · ~3x throughput
M3·11

GPU operational gotchas

Two flags and one sizing rule separate a working server from cryptic crashes.

--gpus all needs NVIDIA Container Toolkit exposes host GPU + drivers to container else falls back to CPU --ipc=host shares /dev/shm multi-process attention needs it omit → crashes under load VRAM sizing 7B @ 16-bit ≈ 14 GB weights + KV-cache headroom 24 GB card, or quantize
M3·12

Quantization — trade a little precision for a lot of room

Like a JPEG for weights — 16-bit floats shrink to 4- or 8-bit ints.

AWQ · 4-bit GPTQ · 4-bit FP8 · 8-bit activation-aware best accuracy at 4-bit ≈ ¼ the VRAM of FP16 strong quality retention mature ecosystem many checkpoints on Hub 3 / 8-bit variants too slightly more loss than AWQ near-lossless newer GPUs (Hopper/Ada) needs HW FP8 support chosen for large deployments
M3·13

TO THE LAB

Same socket. Bigger engine.

Serve SmolLM2 on CPU vLLM behind the same /v1 contract

Build patched CPU image Serve SmolLM2 on /v1 Point client one-line env swap the M2 client, unchanged, talks to the vLLM server you build

Next: Lab — Serve SmolLM2 on CPU vLLM  ·  Gourav Shah · School of DevOps & AI

M3·14