MODULE 3B  ·  OPTIONAL  ·  DAY 1

Customizing Models with LoRA / QLoRA

Make the model yours — reproducibly, in a container

Base model frozen weights LoRA adapter a tiny trained layer that clips onto a big model

Gourav Shah  ·  School of DevOps & AI  ·  GPU-gated · Hands-on

M3B·01

What you'll learn

Four ideas: when to fine-tune, what LoRA does, which tool to use

12 34 When fine-tuning beats prompting or RAG — and when it doesn't What LoRA and QLoRA actually do to a model's weights Which toolchain: Axolotl on NVIDIA, MLX-LM on Apple Silicon Why a frozen training container is the reproducibility unit
M3B·02

The problem: a behaviour gap

No prompt reliably fixes this. That's not a knowledge gap.

Generalist + prompt "usually" valid JSON occasionally breaks format wrong style · wrong dialect fine-tune Specialist always valid JSON consistent format & style learned from 50–5000 examples
M3B·03

Prompt vs RAG vs fine-tune

Three tools, three gaps. Reach for the cheapest one that closes yours.

Prompting the gap: instruction a better system prompt fixes it RAG the gap: knowledge reason over your private docs (M5/M6) Fine-tune the gap: behaviour new style, dialect, reliable structure
M3B·04

LoRA — sticky notes on a textbook

Can't rewrite the expensive textbook — so add Post-it notes in the margins

base weights (frozen) W billions of params + A B two small low-rank matrices (trained) W + A×B at inference
M3B·05

QLoRA — squeeze the base to 4-bit

Quantize the frozen base to 4-bit and a 7B fits one consumer GPU

base in 16-bit too big for one GPU quantize to 4-bit 4-bit base (frozen) adapter hi-precision 7B fits ~24 GB
M3B·06

The open-source toolchain

A handful of mature stacks cover almost every fine-tuning job

NVIDIA (containerized) Axolotl — YAML-driven, ships a Docker image Unsloth — 2x faster, ~60% less VRAM LLaMA-Factory — broad model coverage all wrap Hugging Face TRL / PEFT Apple Silicon (native) MLX-LM — runs on Metal + Neural Engine unified memory: no copy CPU↔GPU fine-tune 3B–7B on 16–32 GB RAM no CUDA · no bitsandbytes needed
M3B·07

The same GPU reality — again

Native on Apple Silicon, containerized on NVIDIA — never the reverse

Apple Silicon Mac MLX-LM — NATIVE Metal · Neural Engine · unified memory container = CPU only bitsandbytes / CUDA won't install NVIDIA (WSL2 / cloud VM) Axolotl — CONTAINER QLoRA · CUDA · bitsandbytes NVIDIA GPU --gpus all
M3B·08

The frozen container is the experiment

Scripts rot — Python, PEFT, bitsandbytes drift. An OCI image does not.

a bare script deps drift over time defaults change silently "works on my machine" freeze it pinned image + YAML every dep version locked config in git · tag in GHCR reproduce it 6 months later
M3B·09

What you produce: a tiny adapter

A 50–200 MB adapter directory — two small files, ready to serve

LoRA adapter (~50–200 MB) adapter_model.safetensors adapter_config.json Merge → one portable file Hot-load → Ollama · vLLM adapter Package → ModelKit artifact (M4) mlx_lm.fuse · peft merge base shared, adapter swapped per request base ref + adapter shipped together
M3B·10

The pipeline, end to end

Fine-tuning is one rung — it feeds serving and packaging you know

fine-tune M3B adapter tiny serve M2 / M3 package M4 ship GHCR the adapter flows into the same system you've been building
M3B·11

TO THE LAB

Two tracks, one destination: a working adapter

Track A MLX-LM native · Apple Silicon Track B Axolotl container · NVIDIA LoRA adapter measurable behaviour change

Fine-tune the behaviour · keep the base · ship the tiny adapter

Optional module — do it if your cohort needs model customization.  ·  Gourav Shah · School of DevOps & AI

M3B·12