MODULE 3B  ·  OPTIONAL  ·  DAY 1

Customizing Models with LoRA / QLoRA

Make the model yours, reproducibly, in a container

Base model frozen weights LoRA adapter a tiny trained layer that clips onto a big model

Gourav Shah  ·  School of DevOps & AI  ·  GPU-gated · Hands-on

M3B·01

What you'll learn

12 34 When fine-tuning beats prompting or RAG — and when it doesn't What LoRA and QLoRA actually do to a model's weights Which toolchain: Axolotl on NVIDIA, MLX-LM on Apple Silicon Why a frozen training container is the reproducibility unit
M3B·02

1 · When to fine-tune — and when not to

M3B·03

The problem: a behaviour gap

A behaviour gap, not a knowledge gap

Generalist + prompt "usually" valid JSON occasionally breaks format wrong style · wrong dialect fine-tune Specialist always valid JSON consistent format & style learned from 50–5000 examples
M3B·04

Prompt vs RAG vs fine-tune

Pick the cheapest tool that closes the gap

Prompting the gap: instruction a better system prompt fixes it RAG the gap: knowledge reason over your private docs (M5/M6) Fine-tune the gap: behaviour new style, dialect, reliable structure
M3B·05

2 · LoRA — sticky notes on a textbook

M3B·06

LoRA, sticky notes on a textbook

Freeze the textbook, add sticky notes

base weights (frozen) W billions of weights + A B two small low-rank matrices (trained) W + A×B at inference
M3B·07

QLoRA, squeeze the base to 4-bit

Quantize the base so a 7B fits one GPU

base in 16-bit too big for one GPU quantize to 4-bit 4-bit base (frozen) adapter hi-precision 7B fits ~24 GB
M3B·08

3 · The toolchain

M3B·09

The open-source toolchain

One toolchain, split by hardware

NVIDIA (containerized) Axolotl — YAML-driven, ships a Docker image Unsloth — 2x faster, ~60% less VRAM LLaMA-Factory — broad model coverage all wrap Hugging Face TRL / PEFT Apple Silicon (native) MLX-LM — runs on Metal + Neural Engine unified memory: no copy CPU↔GPU fine-tune 3B–7B on 16–32 GB RAM no CUDA · no bitsandbytes needed
M3B·10

4 · The GPU reality

M3B·11

The same GPU reality, again

Native on Mac, containerized on NVIDIA

Apple Silicon Mac MLX-LM — NATIVE Metal · Neural Engine · unified memory container = CPU only bitsandbytes / CUDA won't install NVIDIA (WSL2 / cloud VM) Axolotl — CONTAINER QLoRA · CUDA · bitsandbytes NVIDIA GPU --gpus all
M3B·12

5 · Reproducibility — the frozen container is the experiment

M3B·13

The frozen container is the experiment

Scripts rot, a pinned image doesn't

a bare script deps drift over time defaults change silently "works on my machine" freeze it pinned image + YAML every dep version locked config in git · tag in GHCR reproduce it 6 months later
M3B·14

6 · What you produce — and where it fits

M3B·15

What you produce: a tiny adapter

One tiny adapter, three ways to use it

LoRA adapter (~50–200 MB) adapter_model.safetensors adapter_config.json Merge → one portable file Hot-load → Ollama · vLLM adapter Package → ModelKit artifact (M4) mlx_lm.fuse · peft merge base shared, adapter swapped per request base ref + adapter shipped together
M3B·16

The pipeline, end to end

One more step in the pipeline you own

fine-tune M3B adapter tiny serve M2 / M3 package M4 ship GHCR the adapter flows into the same system you've been building
M3B·17

TO THE LAB

Two tracks, one destination: a working adapter

Track A MLX-LM native · Apple Silicon Track B Axolotl container · NVIDIA LoRA adapter measurable behaviour change

Fine-tune the behaviour, keep the base

Optional module — do it if your cohort needs model customization.  ·  Gourav Shah · School of DevOps & AI

M3B·18