Real‑Time Governance: Keeping Agent Fleets Reliable at Scale
The blog post highlights challenges in managing large-scale agent systems ("agent fleets") in production versus controlled demos. While demos operate under curated conditions, production fleets face unpredictable tasks, varied inputs, and prolonged runtime, leading to hidden failures. Examples include agents repeatedly retrying tasks, excessive token costs, or undetected errors in responses. These issues go unnoticed until they escalate, such as costly outages or customer-facing mistakes. The post contrasts demo reliability with real-world scalability, showing that unmonitored fleets risk financial loss, reliability gaps, and reputational damage. Real-time governance is critical to dynamically track performance, detect anomalies, and enforce constraints (e.g., cost, accuracy) across uncontrolled, high-volume operations. Without such oversight, scaling agent fleets becomes inherently risky, as traditional testing fails to replicate live complexities.
An agent fleet fails in ways a single script never does. One agent calls the same tool twelve times chasing a task it can't finish. Another burns four dollars in tokens on a request that should cost four cents. A third returns a confident, wrong answer, and the mistake sits unnoticed until a customer forwards the transcript to your CEO.
These failures don't show up in a demo. A demo runs once, under your eyes, on a task you picked. Production runs the fleet unattended, on tasks you didn't pick, for as long as the loop lets it.
The comparison
Fig: Ungoverned fleet versus a governed fleet with runtime controls
An ungoverned fleet has no step limit, no budget, and no check on the output before it reaches a user. A governed fleet wraps every run in three controls: a step counter that stops a loop before it spins forever, a budget tracker that halts a run before it burns real money, and a judge that scores the output before anyone sees it.
Catching loops before they compound
Most infinite loops are an agent calling the same tool with the same arguments, over and over, because the result never changed and nothing told it to stop. Two checks catch almost every case: a hard step limit, and a repeat detector on the exact tool call signature.
class LoopGuard:
def __init__(self, max_steps=8, max_repeats=2):
self.max_steps = max_steps
self.max_repeats = max_repeats
self.steps = 0
self.call_signatures = []
def check(self, tool_name, tool_args):
self.steps += 1
if self.steps > self.max_steps:
raise StepLimitExceeded(self.steps)
signature = (tool_name, json.dumps(tool_args, sort_keys=True))
repeats = self.call_signatures.count(signature)
if repeats >= self.max_repeats:
raise LoopDetected(signature, repeats)
self.call_signatures.append(signature)
Call check before every tool invocation, so the guard can stop the loop before the call happens. Catching a runaway pattern on attempt three costs a few cents. Catching it on attempt three hundred costs a pager alert.
Tracking cost before the invoice does
Token usage is the metric most teams check after the bill arrives and rarely during the run itself. A budget object attached to each run fixes that: every model call reports its usage, and the run halts the moment it crosses a limit set before the task started.
class RunBudget:
def __init__(self, run_id, max_tokens, max_cost_usd):
self.run_id = run_id
self.max_tokens = max_tokens
self.max_cost_usd = max_cost_usd
self.tokens_used = 0
self.cost_usd = 0.0
def record(self, usage):
self.tokens_used += usage.total_tokens
self.cost_usd += usage.total_tokens * PRICE_PER_TOKEN
if self.tokens_used > self.max_tokens or self.cost_usd > self.max_cost_usd:
raise BudgetExceeded(self.run_id, self.tokens_used, self.cost_usd)
Set the limit per task type, not as a single global constant. A customer support reply and a research summary don't cost the same to produce, and one ceiling either starves the expensive task or lets the cheap one run wild.
Grading output with a second model
A step limit and a budget stop an agent from running too long or too expensively. Neither checks whether the answer is correct. That's a separate job, and the cleanest way to do it at scale is a second model grading the first one against a rubric.
JUDGE_PROMPT = """You are grading an agent's response against its task.
Score correctness, completeness, and safety from 0 to 1 each.
Return JSON: {"score": float, "reasons": string, "pass": boolean}
"""
def judge(task, response, threshold=0.8):
verdict = judge_model.complete(
system=JUDGE_PROMPT,
user=f"Task: {task}\nResponse: {response}",
response_format="json",
)
verdict["pass"] = verdict["score"] >= threshold
return verdict
Run the judge on every response in a high-stakes flow, and on a sample in a high-volume one. Log every score with the transcript that produced it. A month of judge scores turns into a dataset that shows which prompts drift, which tools produce weak answers, and which agents need a rewrite before a customer finds the gap first.
Governance is the product
Cost, safety, and unpredictability are the reasons a fleet of agents doesn't get budget approval. A step limit, a spending cap, and a judge in the loop turn each of those risks into a number on a dashboard, and numbers are what get a fleet approved.
Responses (0)
Sign in to commentNo responses yet.