MODULE 3  ·  DAY 1

Production Serving with vLLM

Continuous batching + PagedAttention, same /v1 API

Gourav Shah  ·  School of DevOps & AI  ·  Containers for GenAI & Agentic AI

M3·01

What you'll learn

Batching, the contract, the CPU track, quantization

12 34 Continuous batching + PagedAttention: roughly three times the throughput on a GPU Serve vLLM behind the same OpenAI-compatible /v1 as Ollama Run the CPU track on any laptop — NUMA patch, thread tuning The GPU track + quantization: AWQ · GPTQ · FP8 trade-offs
M3·02

1 · Why vLLM: from one cup at a time to a busy café

M3·03

Ollama is great, until the crowd arrives

One espresso shot at a time

requests waiting in line R4R3 R2R1 Ollama one shot at a time throughput stalls
M3·04

Continuous batching, never let a head sit idle

A finished slot refills mid-flight

Static batch seq A — running seq B — done seq C — running idle · wasted whole batch waits for the laggard Continuous batch seq A — running seq D — slots in mid-flight seq C — running token-level scheduling · no idle slots
M3·05

PagedAttention, virtual memory for the KV cache

Paging the KV cache like your OS pages RAM

Naive · reserved slab A: used reserved · empty B: used reserved · empty 60–80% of VRAM reserved-but-empty Paged · pages on demand A·1B·1 A·2B·2 A·3free freefree lookup table maps pages anywhere waste < 4% — many more sequences fit
M3·06

The payoff: roughly three times the throughput

On a GPU: same hardware, heads never idle

1× naive server 3× vLLM (GPU) throughput
M3·07

2 · Same contract, bigger engine

M3·08

Same contract, bigger engine

The wall socket idea from M2, again

Your M2 client openai SDK · curl /v1 contract (OpenAI API) Ollama · :11434 dev laptop vLLM · :8009 production throughput
M3·09

3 · The CPU track (what you'll run)

M3·10

The CPU track, learn the machinery anywhere

No GPU passthrough — slow on purpose

Container · any laptop CPU openeuler/vllm-cpu:0.9.1 multi-arch · runs native on arm64 SmolLM2 135M in the lab /v1 server same OpenAI API Study the engine batcher · quantization at a walking pace
M3·11

Why containers report 0 NUMA nodes

The container can’t see the floor plan

Host (building) NUMA 0NUMA 1 NUMA 2NUMA 3 Container sees numa_size = 0 cpu_count // 0 crashes one-line patch if numa_size > 0 else cpu_count
M3·12

CPU tuning knobs that keep the laptop usable

A few dials, not all your cores

OMP_NUM_THREADS the main dial · caps OpenMP threads set 2–4 (~50–75% of perf cores) VLLM_CPU_KVCACHE_SPACE GB reserved for the KV cache small (1 GB) keeps memory tight OPENBLAS / MKL = 1 keep BLAS single-threaded don't let it fight OpenMP for cores Why leave cores free? the OS & your apps stay responsive the machine won't thermally throttle
M3·13

4 · The GPU track (throughput — documented, not run here)

M3·14

CPU track vs GPU track

Learn here, scale on an NVIDIA box

CPU track (this module) GPU track (production) openeuler/vllm-cpu any laptop CPU no GPU flags goal: learn the engine vllm/vllm-openai · TGI alt NVIDIA GPU · safetensors Container Toolkit + --gpus all --ipc=host · ~3x throughput
M3·15

A GPU launch is one docker run

Read-only here: the command for a Linux GPU box

docker run --gpus all --ipc=host -p 8000:8000 \ vllm/vllm-openai:latest --model mistralai/Mistral-7B-Instruct-v0.3 GPU passes through shared memory for attention same /v1 API on :8000
M3·16

5 · Quantization in practice

M3·17

Quantization, trade a little precision for a lot of room

Like a JPEG, but for model weights

AWQ · 4-bit GPTQ · 4-bit FP8 · 8-bit activation-aware best accuracy at 4-bit ≈ ¼ the VRAM of FP16 strong quality retention mature ecosystem many checkpoints on Hub 3 / 8-bit variants too slightly more loss than AWQ near-lossless newer GPUs (Hopper/Ada) needs HW FP8 support chosen for large deployments
M3·18

6 · Operational gotchas

M3·19

GPU gotchas: toolkit, ipc, VRAM

Two flags, one sizing rule

--gpus all needs NVIDIA Container Toolkit exposes host GPU + drivers to container else falls back to CPU --ipc=host shares /dev/shm multi-process attention needs it omit → crashes under load VRAM sizing 7B @ 16-bit ≈ 14 GB weights + KV-cache headroom 24 GB card, or quantize
M3·20

CPU gotchas: two flags, a patch, a wait

What goes wrong on a laptop, and the fix

max-model-len caps the context and the KV cache too big: OOM kill max-num-seqs sequences batched at once drop one of the two NUMA patch divide by zero at startup not optional First run image + model download, then cached slow, not hung
M3·21

TO THE LAB

Same socket. Bigger engine.

Serve SmolLM2 on CPU vLLM, same /v1 API

Build patched CPU image Serve SmolLM2 on /v1 Point client one-line env swap the M2 client, unchanged, talks to the vLLM server you build

Next: Lab — Serve SmolLM2 on CPU vLLM  ·  Gourav Shah · School of DevOps & AI

M3·22