MODULE 2  ·  DAY 1

Serving Local Models

Open engines behind one universal endpoint

Ollama llama.cpp LocalAI /v1 endpoint Your app

Gourav Shah  ·  School of DevOps & AI  ·  Hands-on

M2·01

What you'll learn

Five ideas that make model serving portable and boring.

12 34 5 See Docker Model Runner — then choose runtime-agnostic open engines Know when Ollama, llama.cpp, or LocalAI is the right engine Treat the OpenAI-compatible /v1 API as the universal contract Read GGUF + quantization; size a model to a 16 GB laptop Wire model-native (Mac) vs model-in-container (GPU host)
M2·02

1 · Demo First: Docker Model Runner

M2·03

The quick demo: Docker Model Runner

One command, but locked to Docker's toolchain.

one command docker model run ai/gpt-oss Pulls quantized weights Serves them locally Exposes the same /v1 API
M2·04

2 · The Open Engines: Ollama, llama.cpp, LocalAI

M2·05

Open engines: different machines, same cup

Different machines, same standard cup.

Ollama dev standard, Metal llama.cpp the lean core LocalAI multi-backend hub OpenAI API the same standard cup
M2·06

Which engine, when

Match the engine to the job.

Ollama llama.cpp LocalAI The de-facto dev standard — one-command setup Metal-accelerated natively on Apple Silicon The inference core inside Ollama — minimal footprint Run it directly when you want maximum control One container that looks like OpenAI to every caller Routes to many backends — popular on mixed-HW Linux
M2·07

3 · The OpenAI-Compatible Endpoint: The Universal Contract

M2·08

The problem: every engine speaks differently?

Three engines, three ways to break.

Your app has to know each one? Engine A Engine B Engine C custom SDK, custom URL another SDK, another URL yet another SDK... custom glue for each
M2·09

The universal contract: the /v1 endpoint

Swap what's behind the socket, not the plug.

Ollama vLLM LocalAI /v1 (OpenAI API) POST /chat/completions GET /models Your app / agent code never changes
M2·10

Swap engines by changing one variable

One variable, not a code change.

OPENAI_BASE_URL one variable your app reads DEV STAGING PRODUCTION Ollama, native Mac LocalAI, Compose vLLM, GPU VM host.docker.internal localai:8080 vllm-service:8000
M2·11

4 · GGUF and Model Selection for Laptops

M2·12

GGUF: the JPEG of model weights

RAW is huge, JPEG loads instantly.

float16 RAW mathematically perfect huge — won't fit the laptop quantize Q4_K_M GGUF compact + fast sizing rule params × 0.6 ≈ GB of RAM 7B ≈ 4 GB at Q4
M2·13

Picking a model for a 16 GB laptop

Small models for the labs, big ones for later.

MODEL SIZE (Q4) COURSE USE qwen2.5:1.5b ~1 GB Labs (default) qwen2.5:3b ~2 GB Optional upgrade Qwen3 4B–8B 2–5 GB Beyond labs Llama 3.2 3B ~2 GB General tasks Mistral 7B ~4 GB Prod baseline
M2·14

5 · Two Wiring Patterns, One App

M2·15

Two wiring patterns, one app

Same app code, one variable changes.

Pattern A · Mac App container Ollama NATIVE Metal GPU host.docker.internal :11434 Pattern B · GPU host App container Ollama CONTAINER NVIDIA GPU ollama:11434 Compose network
M2·16

BIG IDEA

The engine is a deployment choice, not a code choice

Next: containerize a client that speaks this contract.

any engine any hardware /v1 contract Your app — plugged in once never rewired

Continue to the Serving lab.  ·  Gourav Shah · School of DevOps & AI

M2·17