MODULE 2  ·  DAY 1

Serving Local Models

Open engines behind one universal endpoint

Ollama llama.cpp LocalAI /v1 endpoint Your app

Gourav Shah  ·  School of DevOps & AI  ·  Hands-on

M2·01

What you'll learn

Six ideas that make model serving portable and boring.

12 34 56 See Docker Model Runner — then choose runtime-agnostic open engines Know when Ollama, llama.cpp, or LocalAI is the right engine Treat the OpenAI-compatible /v1 API as the universal contract Read GGUF + quantization; size a model to a 16 GB laptop Wire model-native (Mac) vs model-in-container (GPU host) Switch thinking off so the token budget goes to the answer
M2·02

1 · Demo First: Docker Model Runner

M2·03

The quick demo: Docker Model Runner

Open source, but it needs Docker Desktop or CE.

one command docker model run ai/gpt-oss · 20B, too big for 16 GB Pulls quantized weights Serves them locally Exposes the same /v1 API
M2·04

2 · The Open Engines: Ollama, llama.cpp, LocalAI

M2·05

Open engines: different machines, same cup

Different machines, same standard cup.

Ollama dev standard, MLX on Mac llama.cpp the lean core LocalAI multi-backend hub OpenAI API the same standard cup
M2·06

Which engine, when

Match the engine to the job.

Ollama llama.cpp LocalAI The de-facto dev standard — one-command setup MLX engine on Apple Silicon (Ollama 0.40+) The inference core inside Ollama — minimal footprint Run it directly when you want maximum control One container that looks like OpenAI to every caller Routes to many backends — popular on mixed-HW Linux
M2·07

3 · The OpenAI-Compatible Endpoint: The Universal Contract

M2·08

The problem: every engine speaks differently?

Three engines, three ways to break.

Your app has to know each one? Engine A Engine B Engine C custom SDK, custom URL another SDK, another URL yet another SDK... custom glue for each
M2·09

The universal contract: the /v1 endpoint

Swap what's behind the socket, not the plug.

Ollama vLLM LocalAI /v1 (OpenAI API) POST /chat/completions GET /models Your app / agent code never changes
M2·10

Swap engines by changing one variable

One variable, not a code change.

OPENAI_BASE_URL one variable your app reads DEV STAGING PRODUCTION Ollama, native Mac LocalAI, Compose vLLM, GPU VM host.docker.internal localai:8080 vllm-service:8000
M2·11

4 · GGUF and Model Selection for Laptops

M2·12

GGUF: the JPEG of model weights

RAW is huge, JPEG loads instantly.

float16 RAW mathematically perfect huge — won't fit the laptop quantize Q4_K_M GGUF compact + fast sizing rule params × 0.6 ≈ GB of RAM 7B ≈ 4 GB at Q4 Exception: qwen3.5:2b on a Mac is an MLX mxfp8 build, 3.1 GB, so the rule does not apply
M2·13

Picking a model for a 16 GB laptop

Small models for the labs, big ones for later.

MODEL SIZE ON DISK COURSE USE qwen3.5:2b ~3.1 GB Labs (default) qwen3:4b ~2.5 GB Optional, not a drop-in Qwen3 4B–8B 2–5 GB Beyond labs Llama 3.2 3B ~2 GB General tasks Mistral 7B ~4 GB Prod baseline gpt-oss 20B too big for 16 GB Demo only (Section 1)
M2·14

5 · Two Wiring Patterns, One App

M2·15

Two wiring patterns, one app

Same app code, one variable changes.

Pattern A · Mac App container Ollama NATIVE Metal GPU host.docker.internal :11434 Pattern B · GPU host App container Ollama CONTAINER NVIDIA GPU ollama:11434 Compose network
M2·16

Reasoning models: thinking tokens first

Thinking tokens count against max_tokens.

Request qwen3.5:2b max_tokens 4,096 no switch sent: thinking is ON by default thinking: 4,080 tokens content: "" "reasoning_effort": "none" answer: 11 tokens, budget left over short answer same model, same question, same 4,096-token budget
M2·17

Analogy — scratch paper never handed in

Fill every line with scratch work, no answer.

Thinking on EXAM ANSWER SHEET Answer: (empty) scratch work used every line Thinking off scratch paper stays on the desk EXAM ANSWER SHEET Answer: Paris room left for the answer lines on the sheet = max_tokens · scratch work = thinking tokens · answer box = content
M2·18

Switching thinking off, per API

Three switches work; two look right but fail.

API LAYER SWITCH-OFF Native: /api/generate, /api/chat OpenAI-compatible: /v1/chat/completions langchain-ollama "think": false "reasoning_effort": "none" reasoning=False does NOT work on qwen3.5:2b /no_think in the prompt still thinks, empty answer Modelfile PARAMETER think unknown parameter
M2·19

BIG IDEA

The engine is a deployment choice, not a code choice

Next: containerize a client that speaks this contract.

any engine any hardware /v1 contract Your app — plugged in once never rewired

Continue to the Serving lab.  ·  Gourav Shah · School of DevOps & AI

M2·20