SCHOOL OF DEVOPS & AI  ·  MLOps / LLMOps

Containers for GenAI & Agentic AI

The Open-Source Way

AI runs identically on any OCI runtime

Gourav Shah  ·  2 Full Days  ·  Intermediate → Advanced  ·  Hands-on

01

One real system — built one step per module

Ship one real block per module — never tools in isolation.

model serve package RAG agent crew Platform shipped
02

Container-native, not Docker-native

Docker Desktop now costs money for large orgs — we use the open standard.

Docker-proprietary (2025) Docker Desktop (paid) Docker Model Runner Docker MCP Gateway Docker Scout Docker Offload (GPU) swap for Open-source & universal Colima / Rancher / Podman Ollama · vLLM · llama.cpp ToolHive (MCP gateway) Trivy · Syft · Cosign Any NVIDIA / cloud GPU
03

What you'll be able to do

By the end of two days — as a DevOps or app/AI developer.

12 34 5 Run any GenAI / agentic stack on any OCI runtime — no paid Docker Desktop Serve local LLMs with Ollama & vLLM behind one OpenAI-compatible API Handle the Apple-Silicon vs Windows GPU reality with the right pattern Build naive RAG → Agentic RAG → tool-using agent → multi-agent crew Package models as OCI artifacts, then secure & ship with open tools
04

Two connected use cases

A clean GenAI app and a real agentic system — connected at the tool boundary.

DAY 1 · GenAI Docs Assistant Naive RAG over Acme's runbooks Ollama · ChromaDB · Streamlit DAY 2 · Agentic Support Agent → Incident Crew single agent → multi-agent MCP tools · ToolHive · CrewAI used as a tool
05

The intelligence progression

Each AI pattern arrives exactly when it earns its keep.

Naive RAG Agentic RAG Tool-using agent Multi-agent crew simple & predictable ————————————→ capable & complex
06

The build ladder — one step per module

Each module adds exactly one rung to the same growing system.

M1 · Runtime + a model responding M2 · The OpenAI-compatible endpoint M3 · Endpoint scaled with vLLM M4 · Model versioned as OCI artifact M5 · Docs Assistant — naive RAG M6 · Declarative Agentic-RAG agent M7 · Incident Crew — multi-agent M8 · Hardened: guardrails · SBOM · sign Capstone · The platform shipped end-to-end ← DAY 1 · Serve & Package DAY 2 · RAG → Agentic → Ship → M3B (optional) · LoRA/QLoRA fine-tuning — a bonus rung
07

The one constraint that shapes every lab

Apple Silicon can't pass its GPU into a container — so we wire around it.

Your Mac (host) Model Server Ollama — NATIVE Metal GPU + unified memory Containers App Agent Tools Vector DB host.docker.internal :11434
08

One contract: the OpenAI-compatible endpoint

A wall socket: swap the power station, the appliance never notices.

Ollama vLLM llama.cpp /v1 (OpenAI API) Your app / agent code never changes
09

It all runs on a 16 GB laptop

A non-negotiable design rule — engineered lean on purpose, not by luck.

≈ 4–6 GB peak headroom on a 16 GB machine Small models 1B–4B, Q4 quantized discussed big, required small One shared endpoint every component + every crew agent hits the same server Build up, tear down compose mem/cpu caps; stop services a step no longer needs
10

The open-source tool map — one tool per job

A tidy stack: for each job, the tool we actually use in the labs.

CONTAINER RUNTIME OrbStack · Colima · Rancher · Podman SERVING Ollama (dev) · vLLM (CPU + GPU) PACKAGING KitOps / ModelKit · ORAS VECTOR STORE ChromaDB · Qdrant · pgvector MCP TOOLS ToolHive — MCP servers as containers AGENTS AGENTS.md / SOUL.md + Skills + MCP ORCHESTRATION CrewAI · LangGraph (when needed) SUPPLY CHAIN Trivy · Syft · Grype · Cosign
11

Write the stack — don't paste it

One growing compose.yaml, hand-authored service by service.

compose.yaml model (M2) vector DB (M5) embeddings + app (M5) MCP gateway (M6) agents / crew (M7) one block per module
12

Program at a glance

Two days, two halves — serve & package, then reason & ship.

DAY 1 Serve & Package the Model M1 · M2 · M3 · M4 steps 0 → 3 DAY 2 RAG → Agentic → Multi-Agent → Ship M5 · M6 · M7 · M8 + Capstone steps 4 → 8 Optional M3B (LoRA/QLoRA) extends the program toward 2.5 days
13

What to bring

All free and open source — a laptop and a bit of container fluency.

Knowledge Container basics: build, run, volumes, networks The Compose file format Git & GitHub; basic CI/CD ideas Command-line fluency (macOS or Windows) Basic Python helps — no ML background needed System A runtime: OrbStack / Colima / Rancher / Podman Apple Silicon or Windows 11 + WSL2 Ollama installed natively 16 GB RAM, 4 cores, 30 GB free disk VS Code + GitHub + a registry account
14

LET'S BUILD

Build it once. Run it anywhere.

Container-native, not Docker-native — the open standard is the through-line.

Colima · Rancher · OrbStack · Podman AI Shipped

Start with Setup → Prerequisites, then Module 1.  ·  Gourav Shah · School of DevOps & AI

15