MODULE 3B  ·  CONTAINERS FOR GENAI & AGENTIC AI

The flags you didn't touch

The flags mlx_lm.lora ran — and what each one controls.

rank / alpha notepad width NF4 / QLoRA 4-bit base loss curves lr, overfit, batch chat template the silent trap one training run

Gourav Shah  ·  School of DevOps & AI  ·  Deep Dive (Part 2)

M3B-DD·01

`r` is how wide the correction notepad is

Rank is the notepad's width — more lines, more correction capacity.

frozen textbook base weights, untouched r = 4 four lines to scribble in r = 64 sixty-four lines — more paper to carry
M3B-DD·02

Two thin matrices instead of one fat one

Two thin matrices replace one fat update — params scale with r.

W (d × d) frozen, untouched A (d × r) trainable B (r × d) trainable A × B r=4 → r=64 is a 16x jump
M3B-DD·03

`alpha/r` is a ratio, not two knobs

alpha/r is the real dial — not alpha or r alone.

r = 8, alpha = 16 narrower notepad alpha / r = 2 r = 16, alpha = 32 wider notepad alpha / r = 2 same effective step size
M3B-DD·04

Not every layer gets a notepad

target_modules decides which layers get a notepad clipped on.

one transformer layer q_proj notepad clipped default target v_proj notepad clipped default target k_proj untouched output proj untouched feed-forward untouched what the model attends to what it retrieves
M3B-DD·05

Dropout stops the notepad memorizing example order

Dropout zeroes a random adapter path each step.

adapter in path A — active path B path C zeroed this step adapter out
M3B-DD·06

16 buckets, placed where the weights actually live

NF4 places its 16 buckets where the weight distribution is dense.

shelf 16 buckets — dense near zero, sparse at the tails parts (weights) cluster where the buckets are dense
M3B-DD·07

Double quantization: quantizing the quantizer's own numbers

Double quantization compresses NF4's own scaling constants too.

NF4 4-bit weights the main quantization per-block scale constants needed to place each bucket second pass: quantize the constants to 8-bit
M3B-DD·08

Storage in 4-bit, math in 16-bit

4-bit at rest, bf16 for the math — the adapter stays fp16 throughout.

at rest — 4-bit NF4 frozen base weights this is the memory saving dequant during compute — bf16/fp16 forward / backward pass higher precision result flows on adapter A, B stays fp16 always
M3B-DD·09

The learning rate is the optimizer's step size

Learning rate is the optimizer's step size toward the loss floor.

loss floor (minimum) lr too low barely moves in 50 iters right-sized lr lands on the floor lr too high oscillates past the floor
M3B-DD·10

Falling train loss can still mean the adapter is memorizing

Val loss stalling while train falls is the memorization signature.

healthy train and val fall together overfitting val stalls while train keeps falling
M3B-DD·11

Bigger batch costs memory; more accumulation steps don't

Gradient accumulation buys a bigger effective batch at flat memory.

bigger --batch-size one big forward/backward pass memory climbs into the red zone gradient accumulation several micro-batches, accumulated memory stays flat
M3B-DD·12

Same JSON shape, wrong token boundaries — and it still trains

A chat-template mismatch trains silently — no error, no warning.

train.jsonl messages: [...] your shape tokenizer template expects <|im_start|> your shape ≠ this mismatch loss still falls training "succeeds" no error, no warning
M3B-DD·13

One fixed prompt set, reused every time you change a knob

Compare variants on the same fixed held-out prompts, every time.

fixed prompt set held-out, reused baseline rank 4 high lr compare same prompts, every run
M3B-DD·14

Missing knowledge or missing behavior — the fork before you fine-tune

Knowledge gap leads to RAG. Behaviour gap leads to prompt, then fine-tune.

model gets it wrong missing knowledge missing behavior RAG retrieve, then prompt can a better prompt fix it? check first yes, few-shot works no prompting fine-tune
M3B-DD·15

Same lab, three adapters, one comparison table

One shared dataset, three adapters — only rank or lr changes.

train.jsonl 8 rows, identical baseline rank 8, lr 1e-5 variant A rank 4, lr 1e-5 variant B rank 8, lr 1e-4
M3B-DD·16

Cutting rank in half measurably raises the floor

Rank 4 lands at loss 0.449 vs rank 8's 0.200 — same data, same lr.

baseline — rank 8 3.721 0.200 (val 0.148) variant A — rank 4 3.721 0.449 (val 0.374)
M3B-DD·17

10x the learning rate, and no instability shows up

10x lr converged faster and lower — 0.054 vs baseline's 0.200.

baseline 0.200 10x lr 0.054 (val 0.028) 3.721 instability needs a harder, noisier dataset to show up
M3B-DD·18

MODULE 3B DEEP DIVE  ·  TAKEAWAYS

What to carry into a real adapter

1 2 3 4 5 alpha/r is a ratio — rescale alpha when you bump rank NF4 is a designed quantization, not a lossy hack — trust it val loss stalling while train falls is memorization, not success verify the chat template matches before touching rank or lr reach for gradient accumulation before a bigger batch size

Next: apply these knobs to your own corporate-docs adapter.  ·  Gourav Shah  ·  School of DevOps & AI

M3B-DD·19