MODULE 2  ·  DAY 1

Serving Local Models

Open engines behind one universal endpoint

Ollama llama.cpp LocalAI /v1 endpoint Your app

Gourav Shah  ·  School of DevOps & AI  ·  Hands-on

M2·01

What you'll learn

Five ideas that make model serving portable and boring.

12 34 5 See Docker Model Runner — then choose runtime-agnostic open engines Know when Ollama, llama.cpp, or LocalAI is the right engine Treat the OpenAI-compatible /v1 API as the universal contract Read GGUF + quantization; size a model to a 16 GB laptop Wire model-native (Mac) vs model-in-container (GPU host)
M2·02

The problem: every engine speaks differently?

One API per engine means one rewrite per swap. It doesn't have to.

Your app has to know each one? Engine A Engine B Engine C custom SDK, custom URL another SDK, another URL yet another SDK... custom glue for each
M2·03

The quick demo: Docker Model Runner

One command serves a model — tied to Docker's toolchain, not runtime-agnostic.

one command docker model run ai/gpt-oss Pulls quantized weights Serves them locally Exposes the same /v1 API
M2·04

Open engines: different machines, same cup

Machines differ inside, but every espresso pours into one standard cup.

Ollama dev standard, Metal llama.cpp the lean core LocalAI multi-backend hub OpenAI API the same standard cup
M2·05

Which engine, when

Three open engines, three jobs — reach for the one that fits.

Ollama llama.cpp LocalAI The de-facto dev standard — one-command setup Metal-accelerated natively on Apple Silicon The inference core inside Ollama — minimal footprint Run it directly when you want maximum control One container that looks like OpenAI to every caller Routes to many backends — popular on mixed-HW Linux
M2·06

The universal contract: the /v1 endpoint

Swap the power station behind the socket — the appliance never notices.

Ollama vLLM LocalAI /v1 (OpenAI API) POST /chat/completions GET /models Your app / agent code never changes
M2·07

Swap engines by changing one variable

The endpoint is one environment variable — not a code change.

OPENAI_BASE_URL one variable your app reads DEV STAGING PRODUCTION Ollama, native Mac LocalAI, Compose vLLM, GPU VM host.docker.internal localai:8080 vllm-service:8000
M2·08

GGUF: the JPEG of model weights

RAW is huge and perfect; a JPEG at 80% looks the same and loads instantly.

float16 RAW mathematically perfect huge — won't fit the laptop quantize Q4_K_M GGUF compact + fast sizing rule params × 0.6 ≈ GB of RAM 7B ≈ 4 GB at Q4
M2·09

Picking a model for a 16 GB laptop

Big models get discussed; small ones get required for the labs.

MODEL SIZE (Q4) COURSE USE qwen2.5:1.5b ~1 GB Labs (default) qwen2.5:3b ~2 GB Optional upgrade Qwen3 4B–8B 2–5 GB Beyond labs Llama 3.2 3B ~2 GB General tasks Mistral 7B ~4 GB Prod baseline
M2·10

Two wiring patterns, one app

Same app code — only OPENAI_BASE_URL changes between the two.

Pattern A · Mac App container Ollama NATIVE Metal GPU host.docker.internal :11434 Pattern B · GPU host App container Ollama CONTAINER NVIDIA GPU ollama:11434 Compose network
M2·11

BIG IDEA

The engine is a deployment choice, not a code choice

Next: containerize a client that speaks this contract.

any engine any hardware /v1 contract Your app — plugged in once never rewired

Continue to the Serving lab.  ·  Gourav Shah · School of DevOps & AI

M2·12