MODULE 3B  ·  OPTIONAL  ·  DAY 1

Customizing Models with LoRA / QLoRA

Make the model yours — reproducibly, in a container

Base model frozen weights LoRA adapter a tiny trained layer that clips onto a big model

Gourav Shah  ·  School of DevOps & AI  ·  GPU-gated · Hands-on

M3B·01

What you'll learn

Four ideas — when to fine-tune, what LoRA does, which tool, and why the container matters.

12 34 When fine-tuning beats prompting or RAG — and when it doesn't What LoRA and QLoRA actually do to a model's weights Which toolchain: Axolotl on NVIDIA, MLX-LM on Apple Silicon Why a frozen training container is the reproducibility unit

The output is a small LoRA adapter that plugs straight into serving (M2/M3) and packaging (M4).

M3B·02

The problem: a behaviour gap

Sometimes no prompt reliably makes the model behave the way you need. That's not a knowledge gap.

Generalist + prompt "usually" valid JSON occasionally breaks format wrong style · wrong dialect fine-tune Specialist always valid JSON consistent format & style learned from 50–5000 examples

Fine-tuning teaches new behaviour — style, dialect, reliable output structure — not new facts.

M3B·03

Prompt vs RAG vs fine-tune

Three tools, three different gaps. Reach for the cheapest one that closes your gap.

Prompting the gap: instruction a better system prompt fixes it RAG the gap: knowledge reason over your private docs (M5/M6) Fine-tune the gap: behaviour new style, dialect, reliable structure

Knowledge problem → RAG. Reasoning problem → better prompt or bigger base. Behaviour problem → fine-tune.

M3B·04

LoRA — sticky notes on a textbook

You can't rewrite the expensive textbook. So you add Post-it notes in the margins instead.

base weights (frozen) W billions of params + A B two small low-rank matrices (trained) W + A×B at inference

Freeze the base, train two tiny matrices, add them at inference. The adapter is ~1–3% of the model.

M3B·05

QLoRA — squeeze the base to 4-bit

LoRA's frozen base still hogs memory. Quantize it to 4-bit and a 7B fits one consumer GPU.

base in 16-bit too big for one GPU quantize to 4-bit 4-bit base (frozen) adapter hi-precision 7B fits ~24 GB

QLoRA = LoRA + a 4-bit frozen base — half the memory again, with the adapter kept high-precision.

M3B·06

The open-source toolchain

A handful of mature stacks cover almost every fine-tuning job — pick by your hardware.

NVIDIA (containerized) Axolotl — YAML-driven, ships a Docker image Unsloth — 2x faster, ~60% less VRAM LLaMA-Factory — broad model coverage all wrap Hugging Face TRL / PEFT Apple Silicon (native) MLX-LM — runs on Metal + Neural Engine unified memory: no copy CPU↔GPU fine-tune 3B–7B on 16–32 GB RAM no CUDA · no bitsandbytes needed

Axolotl's YAML config is the interface; the pinned Docker image is what makes the run reproducible.

M3B·07

The same GPU reality — again

On Apple Silicon you fine-tune natively. On NVIDIA you fine-tune in a container. Never the reverse.

Apple Silicon Mac MLX-LM — NATIVE Metal · Neural Engine · unified memory container = CPU only bitsandbytes / CUDA won't install NVIDIA (WSL2 / cloud VM) Axolotl — CONTAINER QLoRA · CUDA · bitsandbytes NVIDIA GPU --gpus all

A 7B QLoRA fits ~24 GB. On Mac, MLX is the accelerated path — CUDA tools simply don't run in a Mac container.

M3B·08

The frozen container is the experiment

Scripts rot — Python, PEFT, and bitsandbytes all drift. An OCI image does not.

a bare script deps drift over time defaults change silently "works on my machine" freeze it pinned image + YAML every dep version locked config in git · tag in GHCR reproduce it 6 months later

YAML in git + image tag in your registry = an immutable, repeatable experiment record.

M3B·09

What you produce: a tiny adapter

The run emits a 50–200 MB adapter directory — two small files, ready to serve.

LoRA adapter (~50–200 MB) adapter_model.safetensors adapter_config.json Merge → one portable file Hot-load → Ollama · vLLM adapter Package → ModelKit artifact (M4) mlx_lm.fuse · peft merge base shared, adapter swapped per request base ref + adapter shipped together

One adapter, three exits — merge for portability, hot-load for flexibility, package to ship.

M3B·10

The pipeline, end to end

Fine-tuning is one rung — it feeds serving and packaging you already know.

fine-tune M3B adapter tiny serve M2 / M3 package M4 ship GHCR the adapter flows into the same system you've been building

Nothing new downstream — a fine-tune just adds a custom-behaviour adapter to the pipeline you own.

M3B·11

TO THE LAB

Two tracks, one destination: a working adapter

Track A MLX-LM native · Apple Silicon Track B Axolotl container · NVIDIA LoRA adapter measurable behaviour change

Fine-tune the behaviour · keep the base · ship the tiny adapter.

Optional module — do it if your cohort needs model customization.  ·  Gourav Shah · School of DevOps & AI

M3B·12