Agentic systems fail differently than traditional software. A conventional service fails loudly — an exception, a 500, a stuck queue. An agent fails plausibly: it calls the wrong tool with confident parameters, retries a destructive action because the error message looked transient, or completes a task in a way that is technically correct and operationally wrong. Guardrails are how you convert plausible failure into contained failure.
After deploying agent systems across insurance claims triage, supplier onboarding, and IT service management, we've converged on a five-layer guardrail architecture. Each layer exists because we watched a specific class of failure get past the layer above it.
Layer 1: Tool permissioning, not prompt pleading
The most common guardrail mistake is putting constraints in the system prompt — "never issue refunds above $500" — and hoping the model complies. Prompts are advisory. Permissions are structural.
Every tool an agent can call should be registered with a manifest that declares its risk class, side effects, and argument constraints:
{
"tool": "issue_refund",
"risk": "write.financial",
"constraints": {
"amount": { "max": 500, "currency": "USD" },
"requires_case_id": true
},
"idempotency_key": "case_id"
}
The executor — the deterministic code that actually invokes tools — validates arguments against this manifest before the call leaves the process. If the agent proposes a $2,000 refund, the call is rejected at the executor, the rejection is fed back into the agent's context as a structured error, and the model can re-plan within bounds. The constraint holds even if the prompt is jailbroken, because the constraint was never in the prompt.
Idempotency keys matter more than people expect. Agents retry. If issue_refund isn't idempotent on case_id, a timeout followed by a retry double-pays the customer. We treat every write tool without an idempotency contract as a production blocker.
Layer 2: Plan validation before execution
Single-step validation catches bad calls; it doesn't catch bad sequences. An agent can pass every per-call check while assembling a harmful plan — exporting a customer list (allowed), composing an email (allowed), sending it externally (allowed) — that in combination exfiltrates data.
For multi-step workflows we require the agent to emit its plan as a typed artifact before executing anything:
PLAN
1. fetch_claim(claim_id=C-88412) [read]
2. query_policy(policy_id from step 1) [read]
3. compute_settlement(...) [pure]
4. issue_refund(amount from step 3) [write.financial]
5. notify_customer(...) [write.comms]
A rules engine evaluates the plan as a whole: no external communication after bulk reads, no more than one financial write per task, no writes to systems outside the task's declared scope. Plans that violate a rule are rejected with the specific rule cited, which — usefully — teaches the model to propose compliant plans on the next attempt. In our claims deployment, plan-level rejection caught 3.1% of proposed plans that per-call checks would have allowed.
Layer 3: Budgets — tokens, time, money, and actions
Agents loop. Without hard budgets, a confused agent will burn tokens re-reading the same document or retry a failing API forty times. Every task gets four budgets enforced by the runtime, not the model:
- Token budget — total context consumed across all steps (we typically start at 150K per task).
- Wall-clock budget — tasks that run past their SLA escalate rather than grind on.
- Action budget — a cap on tool calls, with a separate, much smaller cap on write-class calls (often 3–5).
- Cost budget — dollar ceiling combining inference and downstream API costs.
Exhausting any budget doesn't kill the task silently; it transitions the task to a needs_review state with the full trace attached. Budget exhaustion is a signal, and in practice it's one of our best regression detectors: when a model upgrade increases average action counts by 20%, budgets surface it on day one.
Layer 4: Human checkpoints where reversibility ends
The right question for human-in-the-loop design isn't "how risky is this action?" but "how reversible is it?" Reading is reversible. Drafting is reversible. Sending, paying, deleting, and signing are not.
We classify every tool as reversible or irreversible, and irreversible actions above a materiality threshold require an approval token that only a human reviewer can mint. Crucially, the approval UI shows the reviewer the agent's reasoning trace and the exact proposed call, not a summary. Summaries hide the failure. The reviewer approves the call, and the executor verifies the approval token matches the call's hash — so the agent can't get approval for one action and execute a different one.
Approval fatigue is real, so thresholds should be dynamic. In our supplier-onboarding system, agents earn autonomy per workflow: after 500 approved actions of a given type with a false-positive rate under 1%, the threshold for that action type relaxes. Autonomy is granted by evidence, not by launch-day optimism.
Layer 5: Post-hoc audit and replay
Every task produces an append-only trace: model versions, prompts, retrieved context, tool calls, arguments, results, and approvals — hashed and stored for replay. This is table stakes for regulated industries, but it's also an engineering asset. When an agent misbehaves, we replay the trace against a candidate fix (new prompt, new model, new rule) and confirm the failure disappears before deploying. Our evaluation suite is, in large part, a curated library of past failures.
Two metrics we track on every agentic deployment: intervention rate (what fraction of tasks required human correction — should trend down) and silent deviation rate (tasks completed "successfully" that audits later flagged — should be near zero and is the number your risk committee actually cares about).
What this costs, and why it's worth it
The guardrail stack adds latency (plan validation adds 1–3 seconds), engineering effort (tool manifests are real work), and inference overhead (structured re-planning after rejections). In our deployments the overhead runs 10–15% of total system cost.
Against that: an insurance client's agent system processes 11,000 claims-adjacent tasks monthly with a 0.4% intervention rate, and in eighteen months of operation the guardrail layers have blocked every incident-class failure before it reached a customer — including two that originated from upstream data corruption, not model error. That last point is the argument in miniature: guardrails don't just protect you from the model. They protect you from everything the model touches.
Start with Layer 1 and Layer 5 — permissioning and audit — on your first agent deployment. Add plan validation when workflows exceed three steps. You will not regret the discipline; you will only regret its absence, and by then it will be in an incident report.
WRITTEN BY
Daniel Reyes
Chief Technology Officer
Daniel leads Ilmora's engineering organization of 400+ engineers across four continents. A former distributed-systems architect, Daniel writes about platform architecture, agentic systems, and the engineering discipline behind reliable AI in production.