Skip to main content

Deep Dive: Agent Knobs Under the Hood

The lab ran the Incident Crew end to end and watched it produce two outcomes: APPROVED for a runbook-backed fix, REJECTED/escalate when the relevance gate found nothing to work with. It never asked why each of the four agents runs at the temperature it runs at, why the pipeline is one pass through four functions instead of a loop that can retry, or why the gates that decide APPROVED vs REJECTED live in Python string checks instead of another LLM call asking "is this okay?" This page opens crew.py itself: the exact temperature each agent gets and why, why this crew has no iteration cap because it was never built to loop, how a finding moves from one agent to the next as a plain string, where the code-level gates actually sit, and what happens when you turn one off. It closes with a same-prompt, sequential comparison across three knob variants.

▶ Module 7 Deep Dive — Agent KnobsOpen fullscreen ↗
Where this picks up

You need the m7 stack up — ChromaDB and the built crew image. This works whether the stack is currently running or was torn down after the lab; up.sh is idempotent, so re-running it is safe.

Every command on this page is run from the root of your course-repo clone (the directory containing labs/, site/, planning/ — wherever you ran git clone for this course). Paths below are always relative to that root, exactly like the lab page.

cd labs/m7
bash up.sh

Expected output

m7 ready: chromadb healthy, crew image built.

1 — Temperature, per agent, not per crew​

The lab treated qwen3.5:2b as one model behind four roles. It is — but each role calls that model with a different temperature, and that difference is deliberate, not incidental.

Analogy: think of temperature as how much a musician improvises around a melody. At low temperature the musician plays the melody exactly as written, every time — reliable, boring, correct. At high temperature the musician takes liberties: more expressive, sometimes brilliant, sometimes off-key. You want the string section reading sheet music at low temperature during a recording take. You want the jazz soloist's improvised break at higher temperature — that's where the interesting ideas live. An incident-response crew is almost entirely string section: the moments where a wrong note is expensive far outnumber the moments where creativity helps.

Reading crew.py's four llm() call sites directly, the actual values are:

AgentCall siteTemperatureWhy
Triagellm(f"Incident: {incident}", profile("triage"), temperature=0)0Classification — one correct AREA / SEV label exists per incident; no benefit to variety
Investigator (relevance gate)llm(..., profile("investigator"), temperature=0).upper().startswith("YES")0A yes/no safety-relevant decision gets fed straight into an if — variety here means flaky gating
Fixerllm(f"Incident: {incident}\nRunbook passage:\n{runbook}\n\n...", profile("fixer"))0.2 (default)No explicit temperature= argument — falls through to llm()'s default parameter. Some room to phrase the recommendation naturally, but still low: the command itself has to come verbatim from the runbook
Reviewerllm(f"Incident: {incident}\nProposed fix:\n{fix}\n\nReview it.", profile("reviewer"), temperature=0)0Safety gate — APPROVED/REJECTED is a binary decision the code branches on; it cannot be allowed to drift

Three of four agents run at temperature=0. The one exception — the Fixer — doesn't even set the argument explicitly; it inherits llm()'s function default of 0.2 (see the signature, def llm(prompt, system, temperature=0.2):). That's not an oversight to fix — it's the right default for the one call in the pipeline that generates prose around a fact (the command) rather than deciding a fact. Nothing in this crew calls llm() above 0.2. This crew has no brainstorming role; every agent's job is to be right, not interesting, which is why the whole pipeline clusters at the low end of the temperature range.

Three of the four calls pin temperature=0 explicitly. The Fixer is the only one that leaves room for phrasing — and even that room is small.

Raising a classification or gating agent's temperature does not make it "more creative" in a useful sense — it makes its correctness less repeatable. §6 measures this directly rather than asserting it: the same incident, run three times at temperature=0 vs temperature=0.9 on the Triage call, to see what actually moves and what doesn't on a model this size.

2 — Iteration control: why this crew has no loop to bound​

The lesson's Mermaid diagram showed a straight pipeline: Triage → Investigator → Fixer → Reviewer. Reading crew.py's run() function confirms it literally is that — four sequential calls, no while, no retry, no "try again with a different approach" branch anywhere in the file. This section is about why that absence is a design decision worth naming, not an accident.

Analogy: picture someone who doesn't find the right answer on their first web search, so they re-google the same question with slightly different wording — and keeps doing that, tweaking a word each time, hoping the next search nails it. Nothing stops them; there's no rule saying "you get three tries." An agent given a tool and no iteration cap can do exactly this: call the tool, look at the result, decide it's not good enough, call the tool again with slightly different arguments, and repeat — burning time and cost with no guarantee the next call is any better than the last, and no built-in stopping point except a wall-clock timeout somewhere upstream.

That failure mode belongs to a different shape of agent than this crew: a ReAct-style loop, where a single agent reasons, calls a tool, observes the result, and decides whether to answer or retry — potentially many times per task.

A ReAct-style loop reasons → calls → observes → decides, and can cycle back to "reason" again. Without an explicit iteration cap, that cycle only stops when something outside the loop (a timeout, an API error) forces it to.

The Incident Crew has none of this. retrieve() is called exactly once by the Investigator — there is no "the first result looked weak, try a different query" branch. If ChromaDB's nearest-neighbour search returns a bad match, the relevance gate (§4) catches it by saying NO to that one candidate; the code does not respond by retrying the query with different phrasing. Each of the four agents gets exactly one shot: one llm() call each (the Investigator's relevance gate is a second, separate llm() call, not a retry of the first — it's checking the retrieved candidate, not re-querying).

This is a bounded-latency, bounded-cost design choice, not a limitation the crew happens to have. A single incident report costs exactly 5 model calls, always: 1 Triage + 1 retrieval-embedding + 1 relevance-gate + (0 or 1) Fixer + (0 or 1) Reviewer. You can predict the p99 latency and the p99 cost of running this crew because there is no code path where it runs 10 times instead of 1. An unbounded ReAct loop trades that predictability for the ability to self-correct — worth it when a task genuinely benefits from "try again, smarter," expensive when a bug or a bad prompt sends the loop searching for an answer that was never going to be found, burning tokens and wall-clock time until something external kills it. In production this shows up as a very concrete SLO question: an on-call responder wants a crew that answers in a bounded, known time, every time — even if the answer is sometimes "REJECTED, escalate" — over a crew that might occasionally find a better answer after ten retries but might just as easily spin for two minutes on an incident it was never going to solve.

3 — Delegation: what actually moves between agents​

The lesson called this a "sequential pipeline." Reading run() line by line shows exactly what crosses each hop, and it's worth being precise about it, because the shape of that handoff is what makes small-model multi-agent work reliably at all.

Trace the actual variables run() passes forward:

incident (the raw string argument)
  -> Triage:        llm(incident) -> triage            (printed, but NOT passed to Investigator)
  -> Investigator:   retrieve(cid, incident) -> candidate  (queries with the ORIGINAL incident text, not triage's output)
                      llm(candidate, incident) -> relevant (bool)
                      candidate or "" -> runbook
  -> Fixer:          llm(incident, runbook) -> fix
  -> Reviewer:        llm(incident, runbook, fix) -> verdict

The detail worth catching: retrieve() queries ChromaDB with the raw incident string, not with Triage's output. Triage's AREA: ... | SEV: ... | ... line is printed to the console for the human reading the transcript, but run() never feeds it into the Investigator's retrieval call or its prompt. Triage's actual job in this pipeline is display and (implicitly) an early opportunity to notice a garbled incident report — it does not narrow what the Investigator searches for. If you wanted Triage's classification to steer retrieval (e.g. filter ChromaDB by AREA), that's a real extension, but it isn't what this crew does today.

Every other handoff is a plain Python string, passed as an f-string argument into the next llm() call — no structured object, no JSON schema, no message history the receiving agent can inspect beyond what's interpolated into its prompt. This is "structured" only in the loose sense that each string has a known shape by convention (the Reviewer's prompt is built assuming fix looks like the Fixer's profile says it will) — nothing in the code validates that shape before using it. Compare this to a framework like CrewAI, where delegation can be dynamic: an agent can decide at runtime to hand a sub-task to a different agent, or a manager agent can route work based on the content of a request, rather than a human-authored run() function hard-coding the order. This crew's declarative pipeline is the opposite of dynamic delegation — the order Triage → Investigator → Fixer → Reviewer is fixed in the Python source, not decided by any agent at runtime.

That fixed order is a strength for this task, not a missing feature. Two dynamic-delegation failure modes it sidesteps entirely:

  • Context dilution. Each agent's prompt here is exactly the fields it needs — the Fixer never sees the Triage output, the Reviewer never sees the raw retrieval candidate that got rejected. In a framework where every agent shares a growing conversation history, later agents can end up reasoning over context that has nothing to do with their actual job, and a small model's limited attention gets spent parsing irrelevant history instead of doing its one task.
  • Error cascade. Because each hop passes forward only what the previous stage decided to output, a mistake at one stage is visible and inspectable at the next — the Reviewer sees the Fixer's exact fix string, not a summarized or re-interpreted version of it. In a framework where a manager agent re-phrases or re-summarizes what a sub-agent found before passing it on, an early misunderstanding can get "smoothed over" in the rephrasing and become invisible to everything downstream — the later agent inherits the error without ever seeing the original evidence that would have let it catch it.

The cost of delegation depth is the same in both designs: every hop is another LLM call, and every LLM call is latency and, if you're paying for tokens, cost. This crew already pays for four hops on every run (five, counting the relevance-gate's separate call). A framework that adds dynamic re-routing — "let me ask a different agent to double-check this" — adds hops on top of that, and each added hop is a further chance for the crew to drift off the original incident.

4 — Guardrail placement: where the code actually gates, not the prompt​

The lesson called the relevance gate "not optional" and the Reviewer "the crew's most important member." This section is about where in the pipeline those checks physically sit in the code, and why that placement — not just their existence — is what makes them trustworthy.

Both gates in this crew are Python string checks on an LLM's output, not a second LLM call asked to police the first LLM call's own answer:

relevant = llm(..., profile("investigator"), temperature=0).upper().startswith("YES")
verdict.upper().startswith('APPROVED')

Neither gate trusts the model's output as-is. Each one takes a string the model produced and runs it through an unambiguous, deterministic Python check — .startswith("YES"), .startswith('APPROVED') — before that answer is allowed to change what happens next. A model that responds "Yes, I believe this passage is relevant because..." still passes the gate (.upper().startswith("YES") matches the leading YES); a model that hedges with "Possibly, but I'm not certain" fails it, correctly, because the code isn't parsing intent — it's checking a prefix. This is why a code-level gate beats asking the model "and are you sure that's safe?" in a follow-up prompt: a follow-up prompt is still just another probabilistic text generation that can itself say the wrong thing. A string check on a constrained output format cannot be talked out of its answer.

Mapped onto the pipeline, the two gates sit at two different points:

Gate 1 sits after a tool call, before the Fixer acts — it validates a retrieval result before anything downstream trusts it. Gate 2 sits at final-answer time, before a human sees it — it validates the Fixer's proposed action before it reaches the person who might run it.

Neither gate sits at the third possible point: before a tool call, validating arguments going in. This crew doesn't need one there, because its one tool call (retrieve()) takes the unmodified incident text as input — there's no LLM-constructed query string or LLM-chosen argument that could be malformed before the call happens. That third gate type earns its keep in agents that let the model choose what to call and with what arguments — validate the arguments before the tool executes, not just the result after. This crew's Investigator has no such freedom: it always calls the same retrieval function the same way.

This module's gates are about behavior — tuning what a specific agent does on a specific input, verified by running it. That's a different concern from governance — auditing, logging, and policy enforcement across a whole fleet of agent deployments, which is M8's territory. A useful way to keep the two apart: this page asks "does this one gate correctly reject a bad Kafka retrieval right now," M8 asks "can I prove, after the fact, that every deployed crew has a gate like this one at all, and that nobody quietly removed it." Where you'd add a third layer beyond this crew's two: input validation (reject a malformed or empty incident string before Triage ever runs), and an explicit output schema for the Reviewer's verdict (today, verdict.upper().startswith('APPROVED') trusts the model to put APPROVED or REJECTED at the very start of its response — a model that instead opens with an explanation before the verdict would silently fail the gate and read as REJECTED by default, which is a safe failure direction here, but worth knowing about before you change the Reviewer's profile).

5 — Observing what each agent actually receives​

There's no debug flag or verbose environment variable in crew.py — grep confirms it (no DEBUG, no --verbose, no log-level env var anywhere in the file). The observation tooling this crew has is docker compose run output plus the structured stage markers already in the code — [TRIAGE], [INVESTIGATOR], [FIXER], [REVIEWER], OUTCOME: — every one of which is a plain print() in run(), not a logging framework. That's sufficient here because every stage's input and output is fully visible in that transcript already; there's no hidden internal reasoning step the prints don't surface.

What the prints do not show is the exact prompt text sent to the model — triage, candidate, fix, and verdict are the outputs of each llm() call, not the full prompt that produced them. To see the actual prompt an agent receives, read the profile file it's built from directly — that plus the f-string in run() (§3, "what actually moves") is the complete prompt, since nothing else gets concatenated in:

cat labs/m7/crew/profiles/investigator.md

Expected output

# Investigator

**Role:** Incident investigator. Given a triaged incident, you use the **Acme runbook knowledge base**
(agentic RAG) to find the relevant runbook. Report the single most relevant runbook passage verbatim.
If no runbook covers it, say `NO RUNBOOK FOUND`. Do not invent procedures. Do not run commands — you
only gather the relevant runbook for the Fixer.

That's the entire system argument the Investigator's llm() calls receive — the prompt argument is the f-string built in run() (the incident text, plus the candidate passage and the relevance question on the gate call). Between the profile file and run()'s f-strings, you can reconstruct the exact prompt any agent saw for any run without needing a debug flag the code doesn't have.

6 — Experiment: knob variants, one at a time, sequential​

Everything below runs sequentially, one variant at a time — never two docker compose run crew invocations in parallel. This machine budgets one crew container (~15 MB) plus the shared ChromaDB container (~15 MB) at a time; running variants back to back keeps the stack at or under 4 GB total, almost all of it the one shared model. It also matters for a second reason: on qwen3.5:2b, small-model output varies run to run even at the same temperature, so overlapping variants would make it impossible to tell whether a difference in output came from the knob you changed or from ordinary small-model noise.

Small-model variance — read before judging the results

qwen3.5:2b-class models respond to temperature and prompt changes less predictably than larger models. The comparisons below are illustrative of the mechanism — what changing a knob does to a pipeline's stability — not a universal recipe for what temperature to pick. Judge the deterministic side of every result strictly, but note that "deterministic" here is conditional on the gates being intact: with the relevance gate in place (Baseline, Variant A), the OUTCOME: marker and which stages ran or short-circuited are exact, reproducible facts about this run — Investigator and Reviewer both stay pinned at temperature=0, so the gate's YES/NO and the final APPROVED/REJECTED don't move even when Triage's temperature does. Variant B removes that gate, and §6's Variant B section shows directly what happens to the OUTCOME: marker once it's gone: it stays stable, but on the wrong answer. Judge the prose — the Triage summary sentence, the Reviewer's stated reason — by shape only in every variant: whether it stayed on topic and matched the expected disposition, not its exact wording. If you swap in a larger model later, expect the outcome markers to stay just as stable at low temperature, but expect the prose to vary less at a given temperature than it does here — a bigger model needs less determinism-by-temperature to stay on-message. One caveat: qwen3:4b on Ollama 0.40.7 ignores think: false and prints its reasoning inline, which breaks the YES/NO gate (see the lab's Troubleshooting), so pick a model that honours the switch-off.

Baseline: the lab's own run, as printed​

This is Step 4 from the lab, unmodified — included here so the variants below have something concrete to diff against without flipping back to the lab page.

cd labs/m7
docker compose run --rm crew "The checkout page is returning HTTP 503 errors for all users."

Expected output

 Container chromadb Running
 Container m7-crew-run-3d9f87ee9d45 Creating
 Container m7-crew-run-3d9f87ee9d45 Created
[crew] Acme Incident Crew: Triage -> Investigator -> Fixer -> Reviewer (4 profiles, one shared model: qwen3.5:2b)

======================================================================
INCIDENT: The checkout page is returning HTTP 503 errors for all users.
======================================================================

[TRIAGE]      AREA: web/checkout | SEV: SEV1 | Checkout API returning 503 errors.

[INVESTIGATOR] ## Checkout 503 errors
If the checkout page returns HTTP 503, the web tier is saturated. Scale it up:
`kubectl scale deploy/web --replicas=5 -n prod`. Then check the load balancer
health in the Acme dashboard.

[FIXER]       Scale up the web tier by running `kubectl scale deploy/web --replicas=5 -n prod`.

[REVIEWER]    APPROVED: Command matches runbook and is a non-destructive scaling action.

======================================================================
OUTCOME: APPROVED — ready for a human to apply

On a small model, exact wording varies run to run — judge by the stage markers and the OUTCOME: line, the same guidance the lab itself gives.

Variant A: Triage temperature raised (0 → 0.9), 3 repeats​

crew.py has no environment-variable override for temperature — it's a literal temperature=0 in the Triage call site. To change it without editing the tracked source, copy crew.py, patch the copy, and rebuild the image from a scratch build context that swaps in the patched file. This keeps the original file in the repo untouched and gives you a one-line revert (delete the copy, rebuild from the unmodified source).

REPO_ROOT="$(pwd)"
mkdir -p ~/crew-deepdive-lab && cd ~/crew-deepdive-lab
cp "$REPO_ROOT/labs/m7/crew/crew.py" ./crew-hot-triage.py
sed -i.bak 's/llm(f"Incident: {incident}", profile("triage"), temperature=0)/llm(f"Incident: {incident}", profile("triage"), temperature=0.9)/' ./crew-hot-triage.py
diff "$REPO_ROOT/labs/m7/crew/crew.py" ./crew-hot-triage.py

(Run this from your repo root — REPO_ROOT="$(pwd)" captures it before the cd below, so every later command in this section can find the tracked source again without you retyping the path.)

Expected output

73c73
<     triage = llm(f"Incident: {incident}", profile("triage"), temperature=0)
---
>     triage = llm(f"Incident: {incident}", profile("triage"), temperature=0.9)

(diff exits non-zero when it finds a difference — that non-zero exit is the expected, correct result here, not a failure.)

Build a one-off image from this patched copy, reusing the lab's own build context so the profiles and Dockerfile don't have to be duplicated:

cd "$REPO_ROOT/labs/m7"
cp ~/crew-deepdive-lab/crew-hot-triage.py crew/crew.py.deepdive-hot-triage
docker build -t acme-incident-crew:hot-triage \
  --build-arg CREW_FILE=crew.py.deepdive-hot-triage \
  -f - . << 'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY crew/ ./crew/
COPY docs/ ./docs/
ARG CREW_FILE=crew.py
RUN cp crew/${CREW_FILE} crew/crew.py
ENTRYPOINT ["python", "crew/crew.py"]
EOF
rm crew/crew.py.deepdive-hot-triage

Expected output

#10 unpacking to docker.io/library/acme-incident-crew:hot-triage
#10 unpacking to docker.io/library/acme-incident-crew:hot-triage 0.3s done
#10 DONE 1.3s

Run it 3 times against the same incident, sequentially, logging each transcript:

cd "$REPO_ROOT/labs/m7"
for i in 1 2 3; do
  docker run --rm --network m7_default \
    -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
    -e CHROMA_HOST=chromadb -e CHROMA_PORT=8000 \
    -e LLM_MODEL=qwen3.5:2b -e EMBEDDING_MODEL=nomic-embed-text \
    acme-incident-crew:hot-triage \
    "The checkout page is returning HTTP 503 errors for all users." \
    | tee ~/crew-deepdive-lab/variant-a-run${i}.log
done

Expected output

=== run 1 ===
[crew] Acme Incident Crew: Triage -> Investigator -> Fixer -> Reviewer (4 profiles, one shared model: qwen3.5:2b)

======================================================================
INCIDENT: The checkout page is returning HTTP 503 errors for all users.
======================================================================

[TRIAGE]      AREA: web/checkout | SEV: SEV2 | Checkout API returning 503 errors

[INVESTIGATOR] ## Checkout 503 errors
If the checkout page returns HTTP 503, the web tier is saturated. Scale it up:
`kubectl scale deploy/web --replicas=5 -n prod`. Then check the load balancer
health in the Acme dashboard.

[FIXER]       Run the following command to scale up the web tier:

```bash
kubectl scale deploy/web --replicas=5 -n prod
```

[REVIEWER]    APPROVED: Command matches runbook and is a non-destructive scaling action.

======================================================================
OUTCOME: APPROVED — ready for a human to apply

=== run 2 ===
[crew] Acme Incident Crew: Triage -> Investigator -> Fixer -> Reviewer (4 profiles, one shared model: qwen3.5:2b)

======================================================================
INCIDENT: The checkout page is returning HTTP 503 errors for all users.
======================================================================

[TRIAGE]      AREA: web/checkout | SEV: SEV1 | Payment gateway outage impacting all checkout attempts.

[INVESTIGATOR] ## Checkout 503 errors
If the checkout page returns HTTP 503, the web tier is saturated. Scale it up:
`kubectl scale deploy/web --replicas=5 -n prod`. Then check the load balancer
health in the Acme dashboard.

[FIXER]       Run the following command to scale up the web tier:

```bash
kubectl scale deploy/web --replicas=5 -n prod
```

[REVIEWER]    APPROVED: Command matches runbook and is a non-destructive scaling action.

======================================================================
OUTCOME: APPROVED — ready for a human to apply

=== run 3 ===
[crew] Acme Incident Crew: Triage -> Investigator -> Fixer -> Reviewer (4 profiles, one shared model: qwen3.5:2b)

======================================================================
INCIDENT: The checkout page is returning HTTP 503 errors for all users.
======================================================================

[TRIAGE]      AREA: web/checkout | SEV: SEV2 | Checkouts returning 503 errors

[INVESTIGATOR] ## Checkout 503 errors
If the checkout page returns HTTP 503, the web tier is saturated. Scale it up:
`kubectl scale deploy/web --replicas=5 -n prod`. Then check the load balancer
health in the Acme dashboard.

[FIXER]       Run the following command to scale up the web tier:

```bash
kubectl scale deploy/web --replicas=5 -n prod
```

[REVIEWER]    APPROVED: Command matches runbook and is a non-destructive scaling action.

======================================================================
OUTCOME: APPROVED — ready for a human to apply

All three runs landed on OUTCOME: APPROVED — the structural decision held steady across a 0 → 0.9 temperature jump. The Triage prose did move. On this model the format held (one terse AREA: ... | SEV: ... line every time, no run-on or multi-section write-up), but the content drifted: the severity came back as SEV2, SEV1 and SEV2 for the same incident, and run 2 invented a cause the incident never mentioned (Payment gateway outage) for what is a checkout web-tier problem. That is the small-model-variance note above made concrete: the classification label is not a stable fact at high temperature. The crew's overall behavior — retrieve the checkout runbook, propose the scale command, get it APPROVED — never wavered, because Triage's output is printed but never fed forward (§3), and Investigator, Fixer and Reviewer all stayed at their original low/zero temperatures. Raising just the Triage temperature made Triage noisier without making the crew's final decision any less reliable — evidence for why temperature is a per-role knob, not a per-crew one (§1).

Variant B: guardrail-off — bypass the relevance gate​

Patch a second copy so the relevance gate always passes, regardless of what the model says — this simulates a guardrail that got silently disabled (a real production failure mode: a feature flag left in a bypass state, a refactor that accidentally always sets relevant = True). The gate's whole decision collapses to one unambiguous suffix, .startswith("YES") — replacing just that suffix with or True is a one-line, unambiguous patch that doesn't depend on matching the multi-line f-string above it:

cd ~/crew-deepdive-lab
cp "$REPO_ROOT/labs/m7/crew/crew.py" ./crew-no-gate.py
python3 -c "
import pathlib
p = pathlib.Path('crew-no-gate.py')
src = p.read_text()
needle = '.upper().startswith(\"YES\")'
assert src.count(needle) == 1, f'expected exactly 1 match, found {src.count(needle)}'
p.write_text(src.replace(needle, '.upper().startswith(\"YES\") or True  # DEEP-DIVE: gate bypassed on purpose'))
"
diff "$REPO_ROOT/labs/m7/crew/crew.py" ./crew-no-gate.py

Expected output

81c81
<                    profile("investigator"), temperature=0).upper().startswith("YES")
---
>                    profile("investigator"), temperature=0).upper().startswith("YES") or True  # DEEP-DIVE: gate bypassed on purpose

Build and run it against the Kafka incident — the one the baseline crew correctly escalates because no runbook covers it:

cd "$REPO_ROOT/labs/m7"
cp ~/crew-deepdive-lab/crew-no-gate.py crew/crew.py.deepdive-no-gate
docker build -t acme-incident-crew:no-gate \
  --build-arg CREW_FILE=crew.py.deepdive-no-gate \
  -f - . << 'EOF'
FROM python:3.12-slim
WORKDIR /app
COPY crew/ ./crew/
COPY docs/ ./docs/
ARG CREW_FILE=crew.py
RUN cp crew/${CREW_FILE} crew/crew.py
ENTRYPOINT ["python", "crew/crew.py"]
EOF
rm crew/crew.py.deepdive-no-gate
docker run --rm --network m7_default \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -e CHROMA_HOST=chromadb -e CHROMA_PORT=8000 \
  -e LLM_MODEL=qwen3.5:2b -e EMBEDDING_MODEL=nomic-embed-text \
  acme-incident-crew:no-gate \
  "The Kafka event streaming cluster has stopped processing messages." \
  | tee ~/crew-deepdive-lab/variant-b-kafka-no-gate.log

Captured output — one real run

[crew] Acme Incident Crew: Triage -> Investigator -> Fixer -> Reviewer (4 profiles, one shared model: qwen3.5:2b)

======================================================================
INCIDENT: The Kafka event streaming cluster has stopped processing messages.
======================================================================

[TRIAGE]      AREA: database | SEV: 1 | Kafka event stream processing halted

[INVESTIGATOR] ## Payments service
To restart the Acme payments service, run:
`kubectl rollout restart deploy/payments -n prod`.
The payments service depends on the Postgres primary in the `prod` namespace.

[FIXER]       Restart the payments deployment to resolve the dependency issue.

```bash
kubectl rollout restart deploy/payments -n prod
```

[REVIEWER]    APPROVED: Command is a non-destructive, runbook-backed remediation for restarting a dependent service.

======================================================================
OUTCOME: APPROVED — ready for a human to apply

With the gate bypassed, runbook is whatever ChromaDB's nearest-neighbour search returned for the Kafka query — the payments runbook chunk, retrieved and printed by [INVESTIGATOR] above with total confidence and no hint that it's the wrong section. The baseline crew (§ Baseline above, and the lab's own Step 5) correctly says NO RUNBOOK FOUND for this exact incident, because the relevance gate answers NO when asked whether the payments passage addresses a Kafka failure. With the gate always returning True, the Fixer treats the mismatched passage as confirmed-relevant and quotes its payments command. We ran this exact image six times against the Kafka incident. Every run retrieved the payments chunk, every Fixer proposed the payments restart, and every Reviewer answered APPROVED.

That is the sharper finding, and it differs from what a weaker Reviewer would show. The Reviewer here is given the runbook passage along with the fix, so it checks the command against that passage — and the command does match it. The Reviewer is doing its job and still approves a payments restart for a Kafka outage, because the thing it compares against is the wrong runbook. A reviewer can only be as right as the evidence handed to it. Removing the gate does not make the crew flaky here; it makes the crew consistently, confidently wrong, and the APPROVED marker now certifies a mismatch. The gate, not the Reviewer, is what stopped this. That is the finding.

Across the six runs the command never changed: every run quoted the payments restart. Small models can still vary their wording around it, which is why you judge by the command and the marker, not the prose. This is why you test a guardrail by removing it and watching, not by assuming what the downstream agents will do.

The tracked crew.py itself was never touched — only the temporary crew.py.deepdive-* copies inside the build contexts (removed by the rm crew/crew.py.deepdive-* line right after each build) and the two extra local images (removed together in the section teardown below). Confirm the source tree is clean:

git -C "$REPO_ROOT" status --short labs/m7/crew/crew.py

Expected output

(no output — a clean `git status --short` on this path prints nothing)

Comparison table​

VariantKnob changedWall timeTool/model callsOUTCOME markerConsistency vs. baseline
Baselinenone (Triage=0, gate=0, Fixer=0.2, Reviewer=0)~9.5s5 (fixed, per §2)APPROVED—
A (×3 repeats)Triage temperature 0 → 0.9~6.6–8.6s per run5 per runAPPROVED, all 3 runsStructural outcome stable; Triage stayed one terse line but its content drifted: SEV2, SEV1, SEV2 for the same incident, and run 2 invented a "payment gateway outage" the report never mentioned
BRelevance gate bypassed (relevant = True, always)~4.5s (3.9–9.7s over 6 runs)5 (same count — gate bypass changes the decision, not the call count)APPROVED in 6 of 6 runs — pinned to the wrong answerRetrieves the wrong (payments) runbook every time, the Fixer quotes the payments restart, and the Reviewer, comparing it to that same wrong passage, approves it. The gate, not the Reviewer, is what made the Kafka incident escalate in the Baseline

Wall times are one machine, and the first (Baseline) run also paid for loading the model, so read them as the same order of magnitude, not a benchmark.

Judge the deterministic side of this table strictly, and note what "deterministic" is conditional on. In Baseline and Variant A, the relevance gate is intact, so the OUTCOME: line and which stages ran are exact, reproducible facts about each run (per the small-model-variance note above) — the gate pins the verdict regardless of what Triage's temperature does upstream. In Variant B, the gate itself is the thing removed, and the table row above documents what breaks: the OUTCOME: marker is still stable, but it is stable on the wrong answer. Judge the Triage/Fixer/Reviewer prose in every variant by shape — did it stay on-topic, did it reach a defensible disposition — not by exact wording, which will vary between the three Variant A repeats even though nothing else changed between them. Wall times above are single-machine, single-run measurements on this laptop, included to show the same order of magnitude across variants (temperature and a gate bypass do not change the crew's call count, so wall time stays roughly flat) — not a timing benchmark to reproduce exactly.

Teardown for this section only — remove the two deep-dive-only images; this does not touch the tracked acme-incident-crew:latest image the lab itself built:

docker rmi acme-incident-crew:hot-triage acme-incident-crew:no-gate 2>/dev/null || true

Expected output

Untagged: acme-incident-crew:hot-triage
Deleted: sha256:bba6a0cebfdbf3382d0a1ab9f4ce32961430439947705fb62e7f9d2a34a734cf
Untagged: acme-incident-crew:no-gate
Deleted: sha256:a980d19975f15730511c5dfcff4c9023b14be6a34be44f0592f126bd2f2aea51

Teardown​

Page-scoped only. Remove the local working directory this page created:

rm -rf ~/crew-deepdive-lab

Leave the m7 stack as the lab's own teardown (labs/m7/down.sh) expects to find it — this page never ran docker compose down itself, and the two extra images it built (hot-triage, no-gate) are already removed above. If you're done with the module entirely, run the lab's own teardown:

cd labs/m7 && bash down.sh

Expected output

 Container chromadb Stopping
 Container chromadb Stopped
 Container chromadb Removing
 Container chromadb Removed
 Network m7_default Removing
 Network m7_default Removed
Where you will use this
  • Temperature is a per-agent-role decision, not a per-crew setting. Use it when: you're tuning a real multi-agent pipeline — pin classification and safety-gate calls at 0, leave headroom (0.2–0.4) only on the one or two calls that generate prose around an already-determined fact, the way this crew's Fixer does.
  • A sequential, single-pass pipeline trades self-correction for predictable cost and latency — that's a deliberate choice, not a missing feature. Use it when: you're bounding an agent's cost/latency for an SLO — a fixed-hop pipeline like this crew gives you an exact worst-case call count; an unbounded ReAct loop needs an explicit iteration cap before it's safe to put in front of the same SLO.
  • Delegation between agents is only as safe as what actually crosses each hop — check what's really being passed, not what the diagram implies. Use it when: debugging a multi-agent pipeline that "isn't using" information you thought it had — as this crew shows, Triage's classification is displayed but never fed into the Investigator's query; the same gap can exist silently in a framework crew.
  • A code-level gate (.startswith("YES"), .startswith("APPROVED")) beats a prompt-level ask because it can't be talked out of its answer. Use it when: you're deciding where to add a safety check in a new pipeline — put it after the tool call that produced risky input, and again at final-answer time before a human sees the result; a follow-up prompt asking the model to double-check itself is not a substitute for either.
  • A code-level gate, not low temperature by itself, is what pins the outcome marker run to run. Variant A raised Triage's temperature and the OUTCOME: marker held steady — because the gate was still there to pin it. Variant B left every temperature untouched and removed only the gate. Without the gate the marker stayed stable, APPROVED in 6 of 6 runs, but it certified the wrong runbook. Use it when: you're evaluating whether a crew's output is reliable enough to trust. Don't read a stable outcome marker as a correct one until you've confirmed there's still a code-level gate between the model's opinion and the marker; a stable-looking pipeline with no gate can be consistently, confidently wrong.