Lesson: Serving Local Models
Module goal: By the end of this module, you will know the main open model-serving engines out there, the OpenAI-compatible API contract that ties them all together, and the two wiring patterns that connect your containerized app to a local model. It doesn't matter if the model runs natively on the host or inside a container, both patterns work the same way.
Module slides
Walk through this short whiteboard deck to get the big picture before the hands-on lab. Or open it fullscreen if that works better for you.
1. Demo First: Docker Model Runner
Docker 4.40+ ships a built-in model runner. One command and you have a served model:
docker model run ai/gpt-oss
Docker downloads the quantized weights, serves them, and exposes the standard /v1 API. You don't need Ollama, you don't need a separate process, and there is nothing extra to install on the host. Docker Model Runner works on Docker Desktop and on Docker CE on Linux. It is fully open source. It picks the engine for you: llama.cpp by default, vLLM on NVIDIA GPUs (Linux and Windows with WSL2), and a vllm-metal backend on Apple Silicon. Compose can use it too. The top-level models: element is part of the Compose Specification, so it is not a Docker Desktop trick. For a quick demo, or for a team that already lives inside Docker's toolchain, this is the easiest way in. (ai/gpt-oss is a 20B model, so run it only on a machine with plenty of RAM. On our 16 GB reference laptop, stick with the smaller models from Section 4.)
This course takes a different path. We focus on runtime-agnostic open engines, that is, engines that don't care which container tool you use. Docker Model Runner is good to know, but the patterns ahead work the same way on Docker, Rancher Desktop, Podman, bare Linux, and Windows with WSL2. You don't need any Docker-specific tooling for them. The same principles apply to Docker Model Runner too. The engines below are the ones you'll actually wire up in the labs.
2. The Open Engines: Ollama, llama.cpp, LocalAI
Analogy: Imagine these engines as different brands of espresso machine: La Marzocco, Breville, DeLonghi. Each one is built differently on the inside, one uses a thermoblock, another a boiler, and each has its own pressure profile and control system. But they all pour into the same standard cup. That cup is the OpenAI API. You can swap the machine under the counter, and no barista needs to be retrained.
The three engines you'll encounter in this course:
| Engine | Best for | Container-ready? | Notes |
|---|---|---|---|
| Ollama | Local dev, one-command setup | Yes (with caveats on Mac) | The de-facto dev standard; on Apple Silicon it runs supported models on the MLX engine (Metal), elsewhere on llama.cpp |
| llama.cpp | Minimal footprint, CPU or Metal | Yes | The inference core inside Ollama; use directly for maximum control |
| LocalAI | Multi-backend OpenAI hub | Yes | One container routing to multiple backends: llama.cpp, Whisper, Stable Diffusion |
Ollama is the engine you already have running from M1. It's serving qwen3.5:2b on Metal at :11434. The lab builds a containerized client that calls it.
There is one detail worth knowing. Since Ollama 0.40, a Mac with Apple Silicon runs supported models on the MLX engine by default, and falls back to llama.cpp when a model is not supported. Everywhere else, Ollama uses llama.cpp. You can see which one is in use: load a model, then run ollama ps and look at the RUNNER column. On our reference laptop it shows mlx for qwen3.5:2b. (ollama show does not print the engine.) You need Ollama 0.40 or newer for this course. The reasoning switch-off in Section 6 also works on older releases.
llama.cpp is the inference engine inside Ollama, and the fallback when MLX is not available. It handles GGUF-format models, quantization, and hardware acceleration. You can run llama.cpp in its own container when you need something lighter than the full Ollama stack.
LocalAI is the right choice when you need one container that looks like OpenAI to every caller, and routes to whatever backend or hardware is available underneath. It's popular on Linux servers that run mixed hardware.
M3 adds a fourth engine, vLLM, for high-throughput batched inference on a GPU VM. That's coming up later. For now, for development, Ollama is the standard.
3. The OpenAI-Compatible Endpoint: The Universal Contract
Extending the wall-socket analogy from M1: Different countries wire their power plants differently, but a standard wall socket always delivers power the same way. You saw in M1 that Ollama exposes /v1/chat/completions. Here's what makes this useful: every engine in the table above exposes that exact same interface.
Two endpoints. That's the entire contract:
| Endpoint | What it does |
|---|---|
POST /v1/chat/completions | Generate a response from a list of messages |
GET /v1/models | List available models |
Your application code calls one URL. That URL is an environment variable. Swap the engine, change the variable:
# Dev — Ollama, native Mac, reached from inside a container:
OPENAI_BASE_URL=http://host.docker.internal:11434/v1
# Staging — LocalAI, Compose service:
OPENAI_BASE_URL=http://localai:8080/v1
# Production — vLLM, GPU VM:
OPENAI_BASE_URL=http://vllm-service:8000/v1
No code change. One variable.
The app talks to the contract. Which engine sits behind it is a deployment decision, not a code decision.
This is the one idea that makes the rest of the course click. The Acme Docs Assistant (M5), the Support Agent (M6), and the Incident Crew (M7) all speak this same contract. So when the team decides to move from a 2B dev model to a 13B production model, the application code does not change at all.
4. GGUF and Model Selection for Laptops
Analogy: Quantization is like compressing a photo. A camera RAW file is huge, and mathematically it's perfect. A JPEG at 80% quality is a fraction of the size. You can barely tell the difference for most purposes, and it loads instantly. GGUF is the JPEG for LLM weights: a compact, quantized format that llama.cpp (and therefore Ollama) loads and runs well on CPU and Metal, without needing CUDA. On Apple Silicon, Ollama can also use MLX-format weights for the same job.
Most Ollama tags are GGUF files at roughly Q4_K_M quantization, not the original float16 weights. The 2b part of a tag is the parameter count, and the quantization level is the compression setting. On Apple Silicon, qwen3.5:2b is the exception: Ollama pulls an MLX build in safetensors format at mxfp8 quantization (the M1 /api/tags capture shows this), which is why it takes 3.1 GB on disk, more than the 2.5 GB qwen3:4b GGUF.
Practical sizing rule: for a Q4 GGUF model, parameter count into 0.6 gives you roughly the RAM usage. The rule does not apply to MLX builds such as qwen3.5:2b. A 7B model costs roughly 4 GB. On 16 GB unified memory, with 8 GB reserved for the OS and app containers, you have about 4 to 5 GB for the model.
Model options at that budget:
| Model | Size on disk | Strength | Course use |
|---|---|---|---|
qwen3.5:2b | ~3.1 GB | Fast iteration | Labs (already pulled) |
qwen3:4b | ~2.5 GB | Better reasoning, still fast | Optional, not a drop-in (see below) |
| Qwen3 4B-8B | 2-5 GB | Latest reasoning series, strong tool use | Recommended beyond labs |
| Llama 3.2 3B | ~2 GB | Meta's compact instruction model | Good general tasks |
| Mistral 7B | ~4 GB | Strong instruction following | Popular production baseline |
| gpt-oss 20B | too big for 16 GB | OpenAI's open-weight release; great via Docker Model Runner on bigger machines | Demo only (Section 1) |
Stay with qwen3.5:2b for all the labs in this course. It keeps iteration fast and the machine responsive. qwen3:4b is not a drop-in swap: the build we tested ignores think: false and prints its reasoning inline, which breaks the labs that parse a short answer. The model-selection decision gets more interesting in M3, when you learn to serve a model with vLLM and need to think about the trade-off between throughput and model size.
5. Two Wiring Patterns, One App
From M1 you already know the Apple Silicon constraint: the hypervisor exposes no virtual GPU, so a container on Mac falls back to CPU. The model server must run natively to get Metal acceleration. That is why Pattern A is what you use on Mac. Pattern B is what the same compose.yaml looks like on Windows with WSL2 and NVIDIA, or on a Linux GPU VM.
The app code in both patterns is identical. A single environment variable, OPENAI_BASE_URL, switches the target:
- Pattern A (Mac):
OPENAI_BASE_URL=http://host.docker.internal:11434/v1 - Pattern B (GPU host):
OPENAI_BASE_URL=http://ollama:11434/v1
The compose.yaml handles the difference through an env file or a Compose override. Your Python client never sees it. You build once on your laptop, and the same image drops onto a Linux GPU VM with one config change.
This portability is the real payoff of that universal contract. The app treats the engine as an addressable service, that is, something it reaches by URL and nothing more. Whether that service runs as a native process, a container on the same host, or a remote VM, the application does not need to know or care.
6. Reasoning Models and the Universal Contract
Analogy: Imagine a student who works the problem on scratch paper before writing the answer on the exam sheet. A reasoning model (also called a thinking model) does the same: it reasons first and answers after. The scratch work is real thinking, but the exam gives only a fixed number of lines. If the student fills every line with scratch work, the answer box stays empty.
That is exactly what happens with qwen3.5:2b. It thinks by default. If you ask Ollama's /api/show endpoint, you would find thinking: {values: [false, true], default: true} in the response. The reasoning tokens count against max_tokens, which is the limit on how long the reply can be. We called /v1/chat/completions without a switch-off and got content: "" back, with completion_tokens: 4080. The model spent the whole 4,096 budget on reasoning and never reached the answer. With "reasoning_effort": "none" the same question came back in 11 tokens.
There are three switch-offs, one per layer:
- Native Ollama APIs (
/api/generate,/api/chat): pass"think": false. - The
/v1endpoint: pass"reasoning_effort": "none". For models that only have on and off, any other effort value means on. - langchain-ollama: set
reasoning=False.
Now here is an interesting thing. Two other options do not work on this model. Adding /no_think to the prompt still produces a thinking field and an empty answer. A Modelfile cannot set it either, because PARAMETER think is an unknown parameter.
Every lab in this course switches thinking off. The model is small, we want the same kind of output on every run, and we want the token budget to be spent on the answer. Leave it on when the task is hard, such as multi-step planning or debugging, and give it a generous max_tokens.
Summary
| Concept | The short version |
|---|---|
| Docker Model Runner | Docker-native OSS option (docker model run) on Desktop and Docker CE; llama.cpp, vLLM or vllm-metal underneath; same /v1 API |
| Open engines | Ollama (dev standard; MLX on Apple Silicon, llama.cpp elsewhere), llama.cpp (core, minimal), LocalAI (multi-backend hub) |
The /v1 contract | Two endpoints, one URL, swap engines by changing an env var, never code |
| GGUF + quantization | Compressed weights that fit laptop RAM; Q4 ≈ 0.6 × parameter count in GB |
| Model selection | qwen3.5:2b for labs; Qwen3, Llama 3.x, Mistral 7B for heavier tasks |
| Reasoning models | Thinking tokens eat max_tokens; switch off with reasoning_effort: "none" on /v1, think: false on native APIs |
| Two wiring patterns | Mac → native model via host.docker.internal; Windows/Linux → model container via Compose hostname |
In the lab you'll containerize a client that speaks this contract.