Hermes Agent Anatomy: Brain, Skills, Tools, and Guardrails

Reference Document

This is the "textbook" version of what you build hands-on in Modules 7-10. It explains WHY the Hermes agent architecture works the way it does — the reasoning behind the design, not the steps for operating it. Bookmark this and return to it when you hit unexpected agent behavior, design a new agent, or onboard a teammate.

Companion labs: Module 7 — Agent Skills | Module 8 — Tool Integration | Module 10 — Domain Agents

1. What Is an Agent? Plain English.

1.1 The Difference Between a Chatbot and an Agent

A chatbot waits for your question, produces one response, and stops. Every interaction is stateless — the bot does not carry intent forward between messages, it does not take initiative, and it cannot alter anything in the world beyond producing text.

An agent is different in one fundamental way: it has goals, not just responses. Given an objective, an agent runs a loop:

receive goal
  → plan what tools to use
    → execute a tool
      → observe the result
        → update the plan
          → execute the next tool
            → ... (repeat until goal reached or stuck)

That loop is the core capability that separates a chatbot from an agent. The agent does not just answer "what queries are slow?" — it runs psql -c "SELECT * FROM pg_stat_statements", reads the output, decides which queries to investigate deeper, runs EXPLAIN ANALYZE on each candidate, cross-references with CloudWatch CPU metrics, and produces a structured diagnosis. Each step observes the environment and informs the next.

1.2 Why DevOps Practitioners Should Care

You already think in loops. An SRE incident response is:

observe alert → gather metrics → form hypothesis → test fix → verify result → close or escalate

An agent implements exactly this loop — autonomously, in code, 24/7. The agent does not replace the engineer's domain expertise. It runs the engineer's expertise (encoded in SKILL.md) on demand, at scale, without paging the on-call at 2am for a task that has a known procedure.

The reason this matters now: modern LLMs crossed the threshold for structured diagnostic tasks in well-scoped domains. An RDS slow-query diagnosis is bounded enough that an LLM with the right context makes the same decision a senior DBA makes, repeatedly and without fatigue.

1.3 The ReAct Pattern: Observe, Think, Act

Under the hood, the Hermes agent loop implements the ReAct pattern (Reasoning + Acting):

Observe: The agent receives tool output (CloudWatch metrics, psql query results, kubectl describe output).
Think: The LLM reasons about what the output means, whether it confirms or contradicts a hypothesis, and what to do next. This reasoning happens inside the model's context window.
Act: The agent emits a tool call. The tool runs. Output comes back. The loop continues.

The loop terminates when the agent decides it has reached its goal (diagnosis complete, remediation applied, escalation triggered) or when it hits the max_turns limit from config.yaml.

1.4 Why Four Components? Why Not Just the LLM?

Early agent frameworks tried to use a single LLM to do everything: reason about infrastructure, remember procedures, execute commands, and self-govern. This fails for predictable reasons:

The pure-LLM agent problem:

Hallucination at the tool layer: Without constrained tool access, the LLM invents commands ("run db-heal --auto" — no such command exists). The agent loop spins, the environment does not change, and the agent hallucinates a fix that never ran.
Context drift: Over long loops, the model forgets the original objective and starts solving the wrong problem. Without external skills anchoring the procedure, the reasoning drifts.
Scope creep: Without behavioral constraints, the agent attempts more than it should. A DBA agent that can run DDL will eventually run DDL when it thinks a schema change would help — even without human review.
No auditability: A pure-LLM agent has no observable decision points.

The four-component model solves each failure mode with a dedicated layer:

Failure Mode	Component That Solves It
Hallucinated commands	Tools (only registered tools can be called)
Context drift	Skills (SKILL.md anchors the procedure)
Scope creep	Guardrails (SOUL.md + DANGEROUS_PATTERNS)
No auditability	Registry + approval audit log

2. The Four-Component Architecture

2.1 Brain: The LLM Reasoning Engine

The Brain is any LLM that supports tool calling (function calling). In Hermes, the Brain is configured in config.yaml under the model key:

model:
  default: "anthropic/claude-haiku-4"
  provider: "auto"

The Brain receives, at each turn of the agent loop:

The system prompt (SOUL.md + any platform context)
All loaded skills for this session
The conversation history so far (previous messages + tool outputs)
The list of available tool schemas (JSON function definitions)

From this context, the Brain produces either:

A tool call (name + arguments) → dispatched to the registry
A final response (no tool call) → loop terminates, response returned to user

The Brain never directly accesses the file system, network, or database. It only emits decisions. The execution layer (Tools) carries out those decisions.

Why LLM model choice matters: For structured diagnostic tasks, a smaller model (Haiku-tier) is sufficient and preferred — lower cost, lower latency, shorter context window requirements. The Brain does not need to "know" everything; it needs to reason over the context it is given. The context (SKILL.md) does the heavy lifting.

2.2 Skills: Encoded Domain Expertise

A Skill is a SKILL.md file loaded into the agent's context at session start. It is procedural memory — the operational playbook the agent follows for a specific diagnostic or remediation task.

Skills are discovered from the skills/ directory inside the agent's profile:

~/.hermes/profiles/track-a/
  config.yaml
  SOUL.md
  skills/
    dba-rds-slow-query/
      SKILL.md

The Hermes skill format aligns with the agentskills.io spec (December 2025) — a cross-platform standard used across 30+ agent frameworks. Skills authored for Hermes can be shared with other compatible platforms without modification.

A Skill's two zones enforce a fundamental separation of concerns:

Scripts Zone (Phase 1): Deterministic. CLI commands only. Exact expected output. No reasoning. No decisions. Run this, get that. Every command is reproducible and auditable.
Agents Zone (Phase 2): Reasoning. IF/THEN/ELSE logic on the data collected in Phase 1. No new data collection. The reasoning is bounded by numeric thresholds and named diagnosis strings — not open-ended "investigate further."

This separation prevents a class of agent failure called "mid-loop data discovery": the agent starts reasoning, realizes it needs more data, collects more data mid-reasoning, the new data changes the reasoning direction, and the loop spirals.

2.3 Tools: The Execution Layer

Tools are the only channel through which the agent affects or observes the external world. In Hermes, tools are registered in tools/registry.py using a singleton ToolRegistry instance.

Each tool registration specifies:

name: the function name the LLM uses to call it
toolset: the logical group this tool belongs to (terminal, file, web, skills, mcp)
schema: the JSON Schema defining inputs the LLM must provide
handler: the Python function that executes the actual work
check_fn: a function that returns True/False based on whether this tool is available in the current environment

The platform_toolsets.cli key in config.yaml determines which toolsets are enabled for this agent:

platform_toolsets:
  cli: [terminal, file, web, skills]   # Track A: full access

platform_toolsets:
  cli: [web, skills]   # Fleet coordinator: no terminal

A tool not in the enabled toolsets is not passed to the LLM — the Brain never knows it exists.

2.4 Guardrails: Two-Layer Safety

Hermes implements a two-layer safety system. The layers are independent — both must be satisfied for a command to execute.

Layer 1 — Behavioral (SOUL.md NEVER rules):

The SOUL.md file defines the agent's identity, scope, and behavioral constraints. The NEVER DO section encodes domain-specific prohibitions:

NEVER execute ALTER TABLE, CREATE INDEX, or any DDL without explicit human approval
NEVER recommend VACUUM FULL during business hours
NEVER mask an ambiguous root cause

These rules are part of the Brain's context. When the Brain generates a plan, it applies these rules during reasoning — before emitting a tool call.

Layer 2 — Mechanical (DANGEROUS_PATTERNS in tools/approval.py):

Even if the Brain decides to run a command (violating a SOUL.md rule), tools/approval.py intercepts every terminal command before execution. The DANGEROUS_PATTERNS list contains regex patterns for commands with catastrophically bad outcomes:

DANGEROUS_PATTERNS = [
    (r'\bDROP\s+(TABLE|DATABASE)\b', "SQL DROP"),
    (r'\bDELETE\s+FROM\b(?!.*\bWHERE\b)', "SQL DELETE without WHERE"),
    (r'\brm\s+-[^\s]*r', "recursive delete"),
    # ... 30+ patterns
]

When a pattern matches, the command does not execute. The approval mode (from config.yaml) determines what happens:

manual: blocks and waits for a human to type [o]nce / [s]ession / [a]lways / [d]eny
smart: an auxiliary LLM reviews the flagged command and auto-approves low-risk matches
off: no blocking (not recommended for production)

3. The Hermes Implementation

3.1 How the Agent Loop Works in run_agent.py

The main agent loop lives in run_agent.py inside the AIAgent class. The loop follows this structure:

System prompt construction: Loads SOUL.md, platform context, and all enabled skills into the system prompt via agent/prompt_builder.py.
Tool schema injection: Queries tools/registry.py for all registered tools whose toolsets match platform_toolsets.cli. These schemas are passed to the LLM in the tools parameter of every API call.
LLM call: Sends the conversation history + tool schemas to the configured model. The model returns either a final message or a list of tool_calls.
Tool dispatch: For each tool_call, the registry's dispatch() method routes the call to the correct handler. If the tool is terminal_tool, the command passes through check_all_command_guards() in tools/approval.py first.
Result injection: Tool outputs are appended to the conversation as tool role messages, then the loop calls the LLM again with the updated history.
Termination: The loop ends when the LLM returns a message with no tool_calls, or when max_turns (from config.yaml) is reached.

The loop is not magic. It is a while loop that calls an API. The agent's intelligence comes from the context (steps 1-2), not from special logic in the loop itself.

3.2 How Skills Are Discovered and Loaded

At agent startup, Hermes scans the skills/ directory, reads each SKILL.md file, and prepends their content to the system prompt. The Brain sees the complete skill procedure as part of its initial context — not retrieved on demand, but present from turn 1.

The session startup sequence:

Load config.yaml — determines model, toolsets, approval mode
Load SOUL.md — agent identity and behavioral constraints
Scan skills/ directory — load all SKILL.md files
Initialize tool registry with enabled toolsets
First LLM call with full context: SOUL.md + all skills + tool schemas

3.3 How config.yaml Controls Capabilities

model:
  default: "anthropic/claude-haiku-4"   # which Brain

platform_toolsets:
  cli: [terminal, file, web, skills]    # which Tools

approvals:
  mode: manual                           # Guardrail behavior
  timeout: 300

command_allowlist: []                    # pre-approved patterns (empty = nothing permanent)

agent:
  max_turns: 30                          # loop termination

The platform_toolsets.cli list is the single most impactful config decision. A coordinator agent with cli: [web, skills] cannot execute commands no matter what it decides — the terminal tool is never registered in its schema.

4. DevOps Agent Walkthroughs

4.1 Aria — Track A Database Health Agent

Component	Implementation	File
Brain	`anthropic/claude-haiku-4`	`config.yaml` line 7
Skills	`dba-rds-slow-query/SKILL.md`	`skills/` directory
Tools	`terminal, file, web, skills`	`config.yaml` platform_toolsets
Behavioral guardrails	NEVER DDL, NEVER VACUUM FULL in hours	`SOUL.md` NEVER DO section
Mechanical guardrails	DANGEROUS_PATTERNS: DROP TABLE, DELETE without WHERE	`tools/approval.py`
Governance level	L2: manual approval for all flagged commands	`config.yaml` approvals.mode

Why Aria has no DDL capability (behavioral layer): Aria's SOUL.md states: "NEVER execute ALTER TABLE, CREATE INDEX, or any DDL without explicit human approval." Even if Aria diagnoses that a missing index is causing 80% of slow queries, it cannot create the index. It reports the finding, proposes the exact CREATE INDEX statement, and escalates to a human DBA for review and execution.

The mock/live duality: Aria uses mock-psql and mock-aws wrappers (in course/infrastructure/wrappers/). When HERMES_LAB_MODE=mock, the wrappers intercept psql and aws calls and return pre-baked JSON fixtures from course/infrastructure/mock-data/. Aria does not know whether it is in mock or live mode — it runs the same commands either way.

4.2 Fleet Coordinator — No Terminal, Delegation Only

platform_toolsets:
  cli: [web, skills]   # No terminal
delegation:
  max_iterations: 30
  default_toolsets: ["terminal", "file", "web", "skills"]

Why no terminal? A coordinator's job is to route work to domain specialists, not execute domain commands directly. The coordinator is kept "hands-off the keyboard" by design — it cannot run commands because its role is orchestration, not execution.

The delegation tool: When the coordinator needs domain work done, it calls delegate_task — a tool that spins up a subagent using one of the installed specialist profiles. The specialist inherits the coordinator's context, runs its SKILL.md procedure, and returns a structured result.

5. Quick Reference

Four-Component Summary

Component	Role	Hermes File	Config Key
Brain (LLM)	Reasoning and planning — decides what to do next	`run_agent.py` (AIAgent class)	`model.default` in `config.yaml`
Skills (SKILL.md)	Domain knowledge — tells the Brain HOW to do domain work	`skills/*/SKILL.md` in profile	Auto-loaded from `skills/` dir
Tools	Execution — the only way to affect the outside world	`tools/registry.py` + tool files	`platform_toolsets.cli` in `config.yaml`
Guardrails	Safety — two layers: behavioral (SOUL.md) + mechanical (DANGEROUS_PATTERNS)	`SOUL.md` + `tools/approval.py`	`approvals.mode` in `config.yaml`

Agent Types and Typical Configurations

Agent Type	Platform Toolsets	Skills	Approval Mode	Example
Domain specialist	`[terminal, file, web, skills]`	1-3 domain skills	`manual` (L2) or `smart` (L3)	Track A, B, C agents
Fleet coordinator	`[web, skills]`	None (uses delegation)	`manual`	`fleet-coordinator`
Read-only advisor	`[web, skills]`	Domain skills	`manual`	L1 governance pattern
Autonomous executor	`[terminal, file, web, skills]`	Domain skills + allowlist	`smart` + `command_allowlist`	L4 governance

Common Failure Modes and Diagnosis

Symptom	Likely Cause	Check
Agent runs same tool 5+ times in a loop	Skill missing "when to stop" criterion	Agents Zone decision tree — does every branch terminate?
Agent proposes commands it could run directly	SOUL.md NEVER rule not strong enough, or missing	Review SOUL.md NEVER DO section
Agent ignores DANGEROUS_PATTERNS and runs dangerous command	`approvals.mode: off` set in config	Set `mode: manual` or `mode: smart`
Coordinator executing commands instead of delegating	`terminal` in coordinator's `platform_toolsets.cli`	Remove `terminal` from coordinator config
Mock mode produces wrong output	Wrong `HERMES_LAB_SCENARIO` or wrappers not in PATH	Verify `export PATH="$(pwd)/wrappers:$PATH"` and `HERMES_LAB_SCENARIO=clean\|messy`
Agent loop terminates too early	`max_turns` too low for multi-step skill	Increase `agent.max_turns` in `config.yaml`

Governance Levels at a Glance

Level	Name	Terminal	Approval Mode	Allowlist	Profile
L1	Assistive	No	manual (no commands to flag)	Empty	Read-only advisory agents
L2	Advisory	Yes	manual	Empty	Domain specialist agents (course default)
L3	Proposal	Yes	smart	Empty	Agents with smart approval for reduced fatigue
L4	Semi-autonomous	Yes	smart	Populated	Trusted production agents with known-safe commands

Key Vocabulary

Term	Definition
SOUL.md	Agent identity file — role, behavior rules, escalation policy. Loaded at startup.
SKILL.md	Machine-readable runbook — when to use, how to collect data (Scripts Zone), how to reason (Agents Zone), never do, escalation
Scripts Zone	Phase of a skill that runs deterministic CLI commands. No reasoning. No decisions.
Agents Zone	Phase of a skill that applies IF/THEN/ELSE reasoning to collected data. No new CLI commands.
DANGEROUS_PATTERNS	Regex list in `tools/approval.py`. Any matching command is blocked pending approval.
platform_toolsets	Config key controlling which tool categories are available to this agent.
HERMES_LAB_MODE	Env var that controls mock vs live routing in wrapper scripts.
ReAct loop	Observe → Think → Act → Observe pattern implemented in `run_agent.py` AIAgent class.
delegation	Fleet coordinator capability: routing work to specialist subagents instead of executing directly.
command_allowlist	Permanent approval list for pre-approved command patterns (L4 governance).
max_turns	Loop termination limit from `config.yaml`; prevents runaway agents
agentskills.io	Cross-platform SKILL.md specification (Dec 2025) — skills portable across 30+ frameworks

1. What Is an Agent? Plain English.​

1.1 The Difference Between a Chatbot and an Agent​

1.2 Why DevOps Practitioners Should Care​

1.3 The ReAct Pattern: Observe, Think, Act​

1.4 Why Four Components? Why Not Just the LLM?​

2. The Four-Component Architecture​

2.1 Brain: The LLM Reasoning Engine​

2.2 Skills: Encoded Domain Expertise​

2.3 Tools: The Execution Layer​

2.4 Guardrails: Two-Layer Safety​

3. The Hermes Implementation​

3.1 How the Agent Loop Works in run_agent.py​

3.2 How Skills Are Discovered and Loaded​

3.3 How config.yaml Controls Capabilities​

4. DevOps Agent Walkthroughs​

4.1 Aria — Track A Database Health Agent​

4.2 Fleet Coordinator — No Terminal, Delegation Only​

5. Quick Reference​

Four-Component Summary​

Agent Types and Typical Configurations​

Common Failure Modes and Diagnosis​

Governance Levels at a Glance​

Key Vocabulary​