Keeping Agent Loops From Burning Your API Budget

I’ll draft the article directly, with concrete engineering tactics, a comparison table, and practical examples. I’m keeping it grounded in real loop behavior and cost controls, and I’ll make sure it ends with actionable takeaways.## The Fastest Way to Melt an API Budget

I once watched a “simple” coding agent turn a 400-token user request into a 28,000-token invoice line item in under two minutes.

The pattern was painfully familiar: the agent called a search tool, summarized results, re-read its own summary, asked the same tool again with slightly different wording, then expanded context until every turn carried the whole conversation, half the repo, and a trail of failed plans. Nothing was wrong with the model. The loop design was wrong.

That’s the core problem with agentic systems: the cost doesn’t scale with user prompts, it scales with loop behavior. A single bad loop can multiply:

• input tokens from ever-growing context,
• output tokens from verbose “thinking”,
• tool spend from repeated calls,
• and latency from serial retries.

If you’re building agents in production, the real job is not “make the model smarter.” It’s “make the loop cheaper, shorter, and harder to run away.”

Why Agent Loops Explode Costs

A normal chat turn has one input and one output. An agent loop has a sequence like this:

1. user asks a question,
2. model plans,
3. model calls a tool,
4. tool returns data,
5. model reads the data,
6. model decides next step,
7. repeat.

Every iteration adds new tokens to the next iteration’s prompt. If you keep appending full tool outputs and previous thoughts, the prompt grows roughly like a staircase.

In practice, that produces three cost multipliers:

• Context accumulation: each step carries prior steps forward.
• Tool chatter: the same tool gets called with slightly different arguments.
• Runaway retries: the model keeps “trying one more thing” when it’s already done enough.

A common gotcha is that token waste is usually invisible until you look at per-run traces. The final answer may be concise, but the hidden chain can be 10x larger than the user-visible output.

A simple cost model

If a loop runs N steps, each with:

• C tokens of carried context,
• I tokens of tool output injected back in,
• O tokens of model output,

then a rough total is:

total_tokens ≈ N * (C + I + O)

That’s optimistic, because C usually grows as the loop continues. In a real agent, the later turns are often the most expensive.

The Control Levers That Actually Matter

There are a few places where you can apply pressure. The trick is to combine them.

The most effective production setups use all of them.

1) Cap Steps Aggressively

This is the simplest and most underrated fix: if you don’t set a hard maximum, you are trusting the model to stop itself.

I usually start with:

• 3–5 steps for narrow tasks,
• 6–8 steps for moderate tasks,
• more only when there’s a strong reason.

A good cap is not just a safety valve. It forces better decomposition. If the agent only gets four moves, it tends to:

• ask for the right tool earlier,
• avoid exploratory detours,
• and converge faster on a usable answer.

Example loop controller

MAX_STEPS = 5

for step in range(MAX_STEPS):
    action = model.next_action(state)

    if action.type == "final":
        return action.final_answer

    observation = run_tool(action.tool_name, action.arguments)
    state = update_state(state, action, observation)

raise RuntimeError("Agent stopped after step cap")

A common gotcha: step caps should apply to the whole plan, not each subroutine independently. Otherwise you move the explosion from the top-level agent into nested helpers.

2) Trim Context Every Turn

Most agents carry too much history. They don’t need every prior tool response, especially if the response was large and only one detail mattered.

In practice, I’ve had the best results with a layered memory policy:

• Keep verbatim: user request, current objective, final decisions, unresolved constraints.
• Summarize: prior reasoning, branch attempts, long tool outputs.
• Drop: dead-end hypotheses, duplicate tool results, old intermediate drafts.

What to retain

A compact state object works better than raw transcript replay:

{
  "goal": "Find the failing endpoint and propose a fix",
  "constraints": [
    "No schema changes",
    "Must preserve backward compatibility"
  ],
  "known_facts": [
    "500s started after deploy 1842",
    "Only /v1/search is affected"
  ],
  "open_questions": [
    "Is the new cache layer returning stale payloads?"
  ],
  "step": 3
}

This is much cheaper than feeding the model 20 turns of “thinking out loud.”

Practical trimming pattern

Use three buffers:

• working_set: current turn, tool outputs, active plan.
• summary_memory: compressed history.
• archive: anything you may need for audits, but not for prompt context.

The agent only sees working_set + summary_memory. When working_set grows too large, summarize it into summary_memory.

The gotcha here is summary drift. If your summarizer is lossy, it can erase constraints and cause the agent to repeat mistakes. I like to preserve structured facts separately from free-form summaries so the model doesn’t have to infer them from prose.

3) Use Cheaper Sub-Agents for Cheap Work

Not every step deserves your best model.

If the task is:

• extracting fields from JSON,
• classifying a ticket,
• checking whether a file changed,
• or generating a short search query,

then a smaller, cheaper model is often enough. Save the expensive model for planning, synthesis, and ambiguity.

Common split

• Primary agent: reasons about the task, decides when to stop.
• Sub-agent: does narrow work like search, extraction, or normalization.
• Verifier: checks outputs for format, completeness, or obvious mistakes.

A practical architecture:

User
  -> Orchestrator (strong model)
      -> Search agent (cheap model)
      -> Extract agent (cheap model)
      -> Final synthesis (strong model)

Why this works

The primary model stays focused on control flow instead of doing everything itself. The sub-agent can be prompt-constrained and token-capped hard.

Example prompt for a search sub-agent:

Return only:
1. the best search query,
2. 3 likely source categories,
3. no explanation.

That kind of narrow contract keeps outputs short and predictable.

If your platform gives you affordable multi-model access, this is where it pays off: one strong model for orchestration, one cheaper model for repetitive sub-tasks, and no reason to burn premium tokens on routine parsing.

4) Cache Tool Results Like You Mean It

Agent loops often pay for the same information multiple times:

• repeated file reads,
• repeated database lookups,
• repeated web searches,
• repeated embedding or retrieval calls.

Caching tool outputs is one of the highest-ROI changes you can make.

Cache by semantic key, not raw text

A raw prompt cache misses if the wording changes slightly. Better keys are based on normalized intent:

def cache_key(tool_name, args):
    normalized = normalize_args(args)
    return f"{tool_name}:{hash(normalized)}"

Example cache entry:

{
  "tool": "repo_search",
  "query": "auth middleware rate limit bug",
  "result": {
    "files": ["src/middleware/auth.py", "src/api/routes.py"],
    "ts": 1710001123
  }
}

When caching helps most

• Repository inspection agents,
• document search,
• stable knowledge bases,
• deterministic transforms,
• repeated retrieval in multi-turn workflows.

Where caching can hurt

• rapidly changing data,
• user-specific results,
• anything with time-sensitive freshness requirements.

The common gotcha is caching tool outputs but not exposing freshness to the agent. If the model believes it got live data when it got cached data, it may make bad decisions. Include timestamps or validity windows in the tool metadata.

5) Put Spend on the Dashboard

If you only track latency and success rate, you’ll miss the real failure mode: expensive success.

You want per-run observability for:

• input tokens,
• output tokens,
• tool calls,
• tool latency,
• total spend,
• step count,
• stop reason.

Minimal event schema

{
  "run_id": "r_8f2c",
  "step": 4,
  "model": "gpt-5.5",
  "input_tokens": 1842,
  "output_tokens": 221,
  "tool_name": "repo_search",
  "tool_latency_ms": 312,
  "estimated_cost_usd": 0.087,
  "stop_reason": "step_cap"
}

What to watch in practice

• Tokens per successful task: the cleanest cost signal.
• Tokens per step: catches prompt bloat.
• Repeated tool calls: usually a planning bug.
• Stop reason distribution: if many runs end by cap, your agent is underplanning.

One of the most useful graphs is cost by task type. Some categories will naturally be expensive. That’s fine. What you want to catch is the task that should be cheap but isn’t.

6) Stop Runaway Loops Early

A loop should stop for more reasons than “I hit the step cap.”

Useful stop conditions include:

• the model has produced a final answer,
• the same tool call repeats with no new information,
• the confidence or utility delta is below threshold,
• the task is solved according to a verifier,
• the plan has stagnated.

Loop breaker example

seen_calls = set()

for step in range(MAX_STEPS):
    action = model.next_action(state)
    signature = (action.tool_name, canonicalize(action.arguments))

    if signature in seen_calls:
        return fail("repeated tool call with no new information")
    seen_calls.add(signature)

    observation = run_tool(action.tool_name, action.arguments)
    state = update_state(state, action, observation)

    if verifier_says_done(state):
        return finalize(state)

This catches a very common failure mode: the model “wants to be helpful” and just rephrases the same query.

Another useful guardrail: utility gating

If a step doesn’t change the state enough, don’t keep going. That can be as simple as:

• no new files found,
• no new entities extracted,
• no improvement in answer confidence,
• no novel evidence added.

The exact metric depends on the task, but some notion of marginal gain is essential.

What Actually Happens When You Tune This Well

Once you tighten the loop, the agent usually gets better in ways that are easy to miss at first:

• It becomes less chatty and more decisive.
• It asks for the right tool sooner.
• It produces fewer “I’m still investigating” cycles.
• It becomes cheaper and faster because fewer turns means less context and less latency.

The important trade-off is that hard caps and aggressive trimming can make the system brittle on genuinely hard tasks. So don’t pretend the agent is “solving” problems it is really just timing out on. Surface that honestly in logs and user-facing messages.

My rule of thumb: optimize for bounded competence over infinite wandering.

A Practical Production Setup

If I were shipping an agent tomorrow, I’d start with:

• MAX_STEPS = 5
• structured state, not full transcript replay
• summary memory after each turn
• cheap sub-agents for search/extraction
• tool-result cache with freshness metadata
• per-run token and cost logging
• repeated-call detector
• explicit stop reasons

That setup is not glamorous, but it’s what keeps the bill from growing faster than the product.

Practical Takeaways

• Cap steps first; it’s the cheapest protection against runaway loops.
• Trim context aggressively and keep structured state separate from summaries.
• Use cheaper sub-agents for narrow tasks instead of spending premium tokens everywhere.
• Cache repeatable tool outputs, but expose freshness so the model doesn’t trust stale data blindly.
• Log spend per run, per step, and per tool call so cost regressions are visible immediately.
• Stop loops on repetition, stagnation, or verified completion — not just on exhaustion.
• Design for bounded competence: good enough, fast, and predictable beats clever-but-expensive every time.