An agent fleet fails in ways a single script never does. One agent calls the same tool twelve times chasing a task it can't finish. Another burns four dollars in tokens on a request that should cost four cents. A third returns a confident, wrong answer, and the mistake sits unnoticed until a customer forwards the transcript to your CEO.

These failures don't show up in a demo. A demo runs once, under your eyes, on a task you picked. Production runs the fleet unattended, on tasks you didn't pick, for as long as the loop lets it.


The comparison

Fig: Ungoverned fleet versus a governed fleet with runtime controls

An ungoverned fleet has no step limit, no budget, and no check on the output before it reaches a user. A governed fleet wraps every run in three controls: a step counter that stops a loop before it spins forever, a budget tracker that halts a run before it burns real money, and a judge that scores the output before anyone sees it.


Catching loops before they compound


Most infinite loops are an agent calling the same tool with the same arguments, over and over, because the result never changed and nothing told it to stop. Two checks catch almost every case: a hard step limit, and a repeat detector on the exact tool call signature.

class LoopGuard:
    def __init__(self, max_steps=8, max_repeats=2):
        self.max_steps = max_steps
        self.max_repeats = max_repeats
        self.steps = 0
        self.call_signatures = []

    def check(self, tool_name, tool_args):
        self.steps += 1
        if self.steps > self.max_steps:
            raise StepLimitExceeded(self.steps)
        signature = (tool_name, json.dumps(tool_args, sort_keys=True))
        repeats = self.call_signatures.count(signature)
        if repeats >= self.max_repeats:
            raise LoopDetected(signature, repeats)
        self.call_signatures.append(signature)

Call check before every tool invocation, so the guard can stop the loop before the call happens. Catching a runaway pattern on attempt three costs a few cents. Catching it on attempt three hundred costs a pager alert.


Tracking cost before the invoice does


Token usage is the metric most teams check after the bill arrives and rarely during the run itself. A budget object attached to each run fixes that: every model call reports its usage, and the run halts the moment it crosses a limit set before the task started.

class RunBudget:
    def __init__(self, run_id, max_tokens, max_cost_usd):
        self.run_id = run_id
        self.max_tokens = max_tokens
        self.max_cost_usd = max_cost_usd
        self.tokens_used = 0
        self.cost_usd = 0.0

    def record(self, usage):
        self.tokens_used += usage.total_tokens
        self.cost_usd += usage.total_tokens * PRICE_PER_TOKEN
        if self.tokens_used > self.max_tokens or self.cost_usd > self.max_cost_usd:
            raise BudgetExceeded(self.run_id, self.tokens_used, self.cost_usd)

Set the limit per task type, not as a single global constant. A customer support reply and a research summary don't cost the same to produce, and one ceiling either starves the expensive task or lets the cheap one run wild.


Grading output with a second model


A step limit and a budget stop an agent from running too long or too expensively. Neither checks whether the answer is correct. That's a separate job, and the cleanest way to do it at scale is a second model grading the first one against a rubric.

JUDGE_PROMPT = """You are grading an agent's response against its task.
Score correctness, completeness, and safety from 0 to 1 each.
Return JSON: {"score": float, "reasons": string, "pass": boolean}
"""

def judge(task, response, threshold=0.8):
    verdict = judge_model.complete(
        system=JUDGE_PROMPT,
        user=f"Task: {task}\nResponse: {response}",
        response_format="json",
    )
    verdict["pass"] = verdict["score"] >= threshold
    return verdict

Run the judge on every response in a high-stakes flow, and on a sample in a high-volume one. Log every score with the transcript that produced it. A month of judge scores turns into a dataset that shows which prompts drift, which tools produce weak answers, and which agents need a rewrite before a customer finds the gap first.

Governance is the product

Cost, safety, and unpredictability are the reasons a fleet of agents doesn't get budget approval. A step limit, a spending cap, and a judge in the loop turn each of those risks into a number on a dashboard, and numbers are what get a fleet approved.