Engineering Lower Token Usage Without Hurting Quality

Last month I reviewed a customer-support agent that was spending 18,000–42,000 input tokens per ticket before it wrote a single word. The actual user problem was usually under 300 tokens. The rest was “just in case” context: full conversation history, every internal policy page retrieved by keyword search, JSON tool results copied verbatim, and a 900-token system prompt that had accreted over six months.

The fix was not “use a smaller model” or “lower max_tokens and hope.” We cut median token usage by about 68% while keeping answer quality stable by doing boring engineering work: pruning context, retrieving narrower evidence, summarizing state, enforcing output schemas, and measuring tokens honestly.

This article is about that kind of work. Not prompt golf. Not tricks that make the model worse. Just practical ways to spend fewer tokens without making your system dumber.

Token usage is an architecture problem

Most teams treat token cost as a model-pricing problem. In practice, it is usually an information-routing problem.

A typical production LLM call has four token buckets:

• Instruction tokens: system prompt, developer prompt, formatting rules, safety constraints.
• Context tokens: chat history, retrieved docs, tool outputs, database records.
• Reasoning/task tokens: the actual user request and intermediate task details.
• Output tokens: the generated response, JSON object, SQL query, summary, or tool call.

When token usage gets out of control, context and output are usually the culprits. The model is not expensive because the user asked a hard question. It is expensive because the application stuffed the entire world into the prompt and then allowed the model to ramble.

A useful first exercise is to log token usage by section, not just total request size:

{
  "request_id": "ticket_81291",
  "model": "sonnet-4.6",
  "tokens": {
    "system": 742,
    "history": 3890,
    "retrieval": 12640,
    "tool_results": 2110,
    "user": 184,
    "output": 931
  },
  "latency_ms": 4820
}

Once you see this breakdown, the optimization path becomes obvious. You do not need to debate whether Claude Opus 4.8, Sonnet 4.6, GPT-5.5, Gemini 3, Fable 5, or an open model like Qwen or DeepSeek is “best” in the abstract. You first need to stop sending 12,000 irrelevant retrieval tokens.

Start with a token budget per route

I like assigning budgets by product route, not globally. A legal contract review, coding agent, and FAQ bot should not share the same token assumptions.

Example budget for a support assistant:

The point is not that these numbers are universally correct. They are intentionally modest starting constraints. The useful behavior is that every stage must justify its tokens.

A common gotcha: if you only set max_tokens for output, your input can still balloon silently. Output limits prevent long answers; they do not stop retrieval stuffing.

Prune context before retrieval, not after

Many systems retrieve documents using the latest user message but still include the full conversation history in the final prompt. That means old tangents, previous failed tool calls, and stale assumptions remain in context.

Instead, maintain a compact working state:

{
  "user_goal": "Get refund for duplicate annual subscription charge",
  "known_facts": [
    "User paid twice on 2025-01-08",
    "Invoice IDs: inv_1042 and inv_1077",
    "Account email: user@example.com"
  ],
  "constraints": [
    "Refund policy allows duplicate-charge refunds within 90 days",
    "User prefers email follow-up"
  ],
  "open_questions": [
    "Whether one charge has already been reversed"
  ],
  "last_updated_turn": 12
}

Then your final call includes:

• The compact state.
• The latest user message.
• Only relevant retrieved policy snippets.
• Maybe the last one or two conversational turns if tone continuity matters.

Do not blindly include every prior message. Chat history is not memory; it is a transcript. Memory is the distilled state needed to continue the task.

In practice, I use three layers:

1. Recent turns: last 1–4 turns, depending on task sensitivity.
2. Running summary: stable facts and decisions.
3. External retrieval: docs, database records, tool outputs.

That usually beats raw transcript stuffing because the model sees less noise and fewer conflicting instructions.

Retrieval should be selective, not decorative

Retrieval-augmented generation often starts as a quality improvement and becomes a token leak. The failure mode is easy to spot: top-k is set to 10 or 20, chunks are 1,000 tokens each, and the final prompt contains a pile of vaguely related documents.

Better retrieval is usually cheaper retrieval.

A practical retrieval pipeline:

def build_context(query, user_state):
    search_query = rewrite_for_search(query, user_state)

    candidates = vector_search(
        query=search_query,
        top_k=30,
        filters={"product": user_state["product"]}
    )

    reranked = rerank_cross_encoder(
        query=search_query,
        docs=candidates,
        top_k=6
    )

    selected = []
    token_budget = 1800

    for doc in reranked:
        snippet = extract_relevant_span(doc.text, query=search_query, max_tokens=300)
        if token_count(selected) + token_count(snippet) <= token_budget:
            selected.append({
                "doc_id": doc.id,
                "title": doc.title,
                "text": snippet
            })

    return selected

The important details:

• Retrieve more candidates than you plan to include.
• Rerank before generation.
• Extract relevant spans instead of passing full chunks.
• Enforce a retrieval token budget.
• Keep document IDs so the model can cite or reference evidence internally.

Chunk size matters. If your chunks are too small, the model loses context. If they are too large, every match drags in irrelevant text. For product docs and policy pages, I often start around 300–600 tokens per chunk with headings preserved, then tune based on failure cases.

What actually happens when you stuff too much retrieval context? The model often becomes less precise. It tries to reconcile adjacent but irrelevant policies, picks up obsolete caveats, or answers the wrong version of the question. Token reduction can improve quality because it removes distractors.

Summarization is compression with accountability

Summarization can save huge amounts of context, but sloppy summaries are dangerous. If a summary drops a constraint, your agent may confidently do the wrong thing.

I prefer structured summaries over prose:

{
  "task": "Debug failing OAuth callback in staging",
  "environment": {
    "service": "auth-api",
    "branch": "release-2025-02-14",
    "runtime": "node 20"
  },
  "confirmed": [
    "Callback returns 302 to /login",
    "Provider sends code and state",
    "State validation passes"
  ],
  "suspected": [
    "Session cookie is not persisted after callback"
  ],
  "rejected": [
    "Provider credentials are invalid",
    "Redirect URI mismatch"
  ],
  "next_steps": [
    "Inspect Set-Cookie attributes in staging response",
    "Compare SameSite and Secure flags with production"
  ]
}

This is more token-efficient and more reliable than:

The user is debugging OAuth and has tried various things…

A good summarization step should preserve:

• User goal.
• Hard facts.
• Decisions already made.
• Constraints and preferences.
• Failed attempts.
• Open questions.
• IDs, filenames, timestamps, and other exact handles.

Do not summarize away exact values unless they are truly irrelevant. “The user has an invoice issue” is cheap but useless. “Duplicate charge: inv_1042 and inv_1077 on 2025-01-08” is the kind of compression you want.

For long-running agents, I also store summary versions. If the model makes a bad summary, you need the ability to inspect when the state drifted.

Stop output rambling with schemas

A surprising amount of token waste is output-side. The prompt says “be concise,” but the model writes a preamble, caveats, a recap, and an offer to help.

For machine-consumed outputs, use schemas. Do not ask for “a JSON response” casually; define the shape and reject anything else.

Example classification call:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["category", "priority", "needs_human", "reason"],
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "bug", "account", "how_to", "other"]
    },
    "priority": {
      "type": "string",
      "enum": ["low", "normal", "high", "urgent"]
    },
    "needs_human": {
      "type": "boolean"
    },
    "reason": {
      "type": "string",
      "maxLength": 160
    }
  }
}

For user-facing responses, give a format with a hard ceiling:

Write the answer in this structure:

- First sentence: direct answer, max 25 words.
- Then 2-4 bullets with concrete steps.
- Do not include a greeting.
- Do not include "let me know if you need anything else."
- Stop after the final bullet.

This works better than “keep it short” because it gives the model a shape.

A common gotcha: low max_tokens can truncate useful output mid-thought. A schema or constrained format reduces output length while preserving completeness. Truncation is a last resort, not a formatting strategy.

Use stop sequences carefully

Stop sequences are useful when the completion has a natural boundary. For example, if the model is generating one section in a templated document, you can stop at the next section marker.

{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "user",
      "content": "Generate only the Risk Summary section for this review..."
    }
  ],
  "max_tokens": 500,
  "stop": ["\n## Mitigations", "\n## Appendix"]
}

Stop sequences are less useful for open-ended assistant replies because they can cut off valid content. They also behave differently across APIs and model families, especially when tool calling or structured outputs are involved. Treat them as boundary guards, not a primary compression technique.

Good uses:

• Generating one section of a known template.
• Preventing the model from continuing into examples.
• Stopping at a delimiter in batch processing.
• Constraining legacy completion-style prompts.

Risky uses:

• Stopping on common punctuation.
• Stopping on words that may appear in normal answers.
• Using stop sequences instead of a proper output schema.
• Assuming identical behavior across Claude, GPT, Gemini, and open-model gateways.

Split tasks across cheaper models

Not every step needs your strongest model. In production systems, I often route sub-tasks to smaller or cheaper models:

This is where multi-model access is operationally useful. You may want Sonnet 4.6 for most agentic writing, GPT-5.5 for a tricky reasoning path, Gemini 3 for certain multimodal or long-context workflows, Fable 5 when the 1M context window is genuinely needed, and open models like Llama, Qwen, DeepSeek, MiniMax, or Kimi for controlled internal tasks. The engineering point is routing, not brand loyalty.

The limitation: orchestration overhead is real. Every extra model call adds latency, failure modes, logging complexity, and evaluation surface area. If a cheap classifier saves 300 tokens but adds 600 ms and another retry path, it may not be worth it.

A pattern I like:

if route == "simple_faq":
    model = "haiku-4.5"
elif route == "standard_support":
    model = "sonnet-4.6"
elif route == "legal_or_financial_exception":
    model = "opus-4.8"
else:
    model = "sonnet-4.6"

The exact names will change. The policy should remain: use the cheapest model that reliably satisfies the task’s quality bar.

Measure tokens in and out honestly

Token optimization fails when teams only look at average cost per request. You need distributions, broken down by route and stage.

Track at least:

• Input tokens by prompt section.
• Output tokens.
• Retrieved document count and retrieval tokens.
• Model name and version.
• Latency percentiles.
• Cache hit rate, if using prompt or context caching.
• Retry count.
• User-visible quality signals.
• Human escalation or correction rates.

A useful dashboard shows p50, p90, and p99. Token bugs often live at the tail.

Example log line:

{
  "route": "support.draft_answer",
  "model": "sonnet-4.6",
  "input_tokens": 3820,
  "output_tokens": 514,
  "retrieved_docs": 4,
  "retrieval_tokens": 1320,
  "latency_ms": 3100,
  "cache_hit": false,
  "quality": {
    "user_thumb": "up",
    "human_edited": false
  }
}

Be careful with “we reduced tokens by 40%” claims. Did retries increase? Did escalations increase? Did users ask more follow-up questions because answers became too terse? Did the cheaper model produce more invalid JSON, forcing repair calls?

The honest metric is not tokens per call. It is tokens per successful task.

Long context is not permission to be lazy

Large context windows are genuinely useful. Fable 5’s 1M context class, Gemini’s long-context strengths, and the long-context modes across frontier models enable workflows that were awkward a few years ago: repository-wide code analysis, large document review, multi-hour transcripts, dense research packets.

But long context is a capability, not a default architecture.

Long-context calls can be appropriate when:

• The answer depends on many distant parts of the input.
• Retrieval would miss cross-document relationships.
• The task is exploratory and you do not yet know what matters.
• The cost of losing a detail is higher than the cost of reading more.

They are usually wasteful when:

• The user asks a narrow factual question.
• A few source snippets are sufficient.
• The context contains duplicate or obsolete information.
• The model is being used as a database filter.

A common pattern is to use long context for offline analysis and produce compact artifacts for online serving. For example, analyze a full policy corpus once, then generate a structured policy map, embeddings, summaries, and test cases. The runtime assistant should not reread the whole corpus for every ticket.

A practical optimization workflow

When I inherit a token-heavy LLM system, I usually do this in order:

1. Instrument first
   Add token accounting by section. Do not optimize blind.

2. Cap retrieval context
   Set a hard retrieval token budget. Rerank and extract spans.

3. Replace transcript stuffing with state
   Keep recent turns plus structured memory.

4. Constrain outputs
   Use schemas for machine outputs and explicit formats for user-facing outputs.

5. Route sub-tasks
   Move classification, rewriting, and summarization to cheaper models when reliable.

6. Evaluate quality at task level
   Compare successful resolutions, not just per-call cost.

7. Watch the tail
   Investigate p95 and p99 token spikes. They often reveal broken retrieval or runaway history.

Here is a simple guardrail I have used in services:

TOKEN_LIMITS = {
    "support.classify": {"input": 800, "output": 80},
    "support.retrieve": {"input": 500, "output": 0},
    "support.draft": {"input": 4500, "output": 700},
    "support.summarize": {"input": 1800, "output": 300},
}

def assert_token_budget(route, prompt_sections, max_output_tokens):
    input_tokens = {
        name: count_tokens(text)
        for name, text in prompt_sections.items()
    }

    total_input = sum(input_tokens.values())
    limits = TOKEN_LIMITS[route]

    if total_input > limits["input"]:
        raise TokenBudgetError(route, total_input, input_tokens)

    if max_output_tokens > limits["output"]:
        raise TokenBudgetError(route, max_output_tokens, {"output": max_output_tokens})

Failing fast is better than discovering a week later that one bad retrieval query spent your budget on irrelevant PDFs.

Practical takeaways

• Token reduction should start with visibility: log input and output tokens by prompt section, route, and model.
• Retrieval beats stuffing when it is selective: rerank, extract spans, and enforce a context budget.
• Summaries should be structured state, not vague prose. Preserve exact facts, IDs, constraints, and failed attempts.
• Output schemas reduce rambling more safely than simply lowering max_tokens.
• Stop sequences are useful boundary guards, but they are not a substitute for structured outputs.
• Route simple sub-tasks to cheaper models only when the added latency and failure modes are worth it.
• Judge savings by tokens per successful task, not tokens per individual call.
• Long-context models are powerful, but they do not remove the need for information architecture.