MODULE 2  ·  DAY 1

Serving Local Models

Open engines behind one universal endpoint

Ollama llama.cpp LocalAI /v1 endpoint Your app

Gourav Shah  ·  School of DevOps & AI  ·  Hands-on

M2·01

What you'll learn

Five ideas that make model serving portable and boring — in the best way.

12 34 5 See Docker Model Runner — then choose runtime-agnostic open engines Know when Ollama, llama.cpp, or LocalAI is the right engine Treat the OpenAI-compatible /v1 API as the universal contract Read GGUF + quantization; size a model to a 16 GB laptop Wire model-native (Mac) vs model-in-container (GPU host)

One app, many backends — the serving engine becomes a deployment choice, never a code change.

M2·02

The problem: every engine speaks differently?

If each serving engine had its own API, swapping one would mean rewriting the app. It doesn't have to.

Your app has to know each one? Engine A Engine B Engine C custom SDK, custom URL another SDK, another URL yet another SDK... custom glue for each

Three custom integrations is three ways to break. The fix is one shared contract — the rest of this module.

M2·03

The quick demo: Docker Model Runner

Docker 4.40+ serves a model in one command — good to know, but tied to Docker's toolchain.

one command docker model run ai/gpt-oss Pulls quantized weights Serves them locally Exposes the same /v1 API

Slickest on-ramp inside Docker — but this course goes runtime-agnostic, so the patterns run anywhere.

M2·04

Open engines: different machines, same cup

Espresso machines differ inside — boiler vs thermoblock — but they all pour into one standard cup.

Ollama dev standard, Metal llama.cpp the lean core LocalAI multi-backend hub OpenAI API the same standard cup

Swap the machine under the counter — no barista, and no app, needs retraining.

M2·05

Which engine, when

Three open engines, three jobs. You reach for the one that fits the moment.

Ollama llama.cpp LocalAI The de-facto dev standard — one-command setup Metal-accelerated natively on Apple Silicon The inference core inside Ollama — minimal footprint Run it directly when you want maximum control One container that looks like OpenAI to every caller Routes to many backends — popular on mixed-HW Linux

For development, Ollama is the standard. M3 adds vLLM for high-throughput batched serving on a GPU.

M2·06

The universal contract: the /v1 endpoint

A wall socket — swap the power station behind it and your appliance never notices.

Ollama vLLM LocalAI /v1 (OpenAI API) Your app / agent code never changes

Two endpoints — chat/completions and models. The app talks to the contract, not the engine.

M2·07

Swap engines by changing one variable

The endpoint is an environment variable. Dev, staging, production differ by one line — not code.

OPENAI_BASE_URL one variable your app reads DEV STAGING PRODUCTION Ollama, native Mac LocalAI, Compose vLLM, GPU VM host.docker.internal localai:8080 vllm-service:8000

Move a 1.5B dev model to a 13B production model — the application does not notice.

M2·08

GGUF: the JPEG of model weights

RAW is huge and perfect. JPEG at 80% is a fraction of the size and looks the same. GGUF is the JPEG for LLMs.

float16 RAW mathematically perfect huge — won't fit the laptop quantize Q4_K_M GGUF compact + fast sizing rule params × 0.6 ≈ GB of RAM 7B ≈ 4 GB at Q4

Ollama pulls GGUF, not float16 — so laptop CPU and Metal run it without CUDA.

M2·09

Picking a model for a 16 GB laptop

Discussed big, required small. Labs stay tiny; heavier models are there when you outgrow them.

MODEL SIZE (Q4) COURSE USE qwen2.5:1.5b ~1 GB Labs (default) qwen2.5:3b ~2 GB Optional upgrade Qwen3 4B–8B 2–5 GB Beyond labs Llama 3.2 3B ~2 GB General tasks Mistral 7B ~4 GB Prod baseline

Stay on qwen2.5:1.5b for every lab — fast iteration keeps the machine responsive. gpt-oss 20B is demo-only.

M2·10

Two wiring patterns, one app

The app code is identical. Only OPENAI_BASE_URL changes — native model vs model-in-container.

Pattern A · Mac App container Ollama NATIVE Metal GPU host.docker.internal :11434 Pattern B · GPU host App container Ollama CONTAINER NVIDIA GPU ollama:11434 Compose network

Build once on your laptop; the same image drops onto a Linux GPU VM with one config change.

M2·11

BIG IDEA

The engine is a deployment choice, not a code choice

any engine any hardware /v1 contract Your app — plugged in once never rewired

Next: in the lab you'll containerize a client that speaks this contract.

Continue to the Serving lab.  ·  Gourav Shah · School of DevOps & AI

M2·12