LLM costs rarely matter in a pilot and frequently matter at scale. A prompt that grew to include every instruction anyone thought of, a retrieval step that passes twenty passages where five would do, an agent that loops a few extra times per task: each looks minor in development and becomes a real line item at production volume. As we noted in LLMs in production, the cheapest model that meets the accuracy bar is usually the right one.
This post covers the levers that reliably reduce spend without degrading quality, and the one prerequisite that makes them safe to pull: an eval suite that tells you when quality drops.
Measure cost per task, not per token
Token prices are the wrong unit for decisions. What the business cares about is the cost of a resolved ticket, a processed invoice or an answered question, including retries, tool calls, retrieval, reranking and the share of cases escalated to a human. Instrument every model call with the feature, tenant and task it belongs to.
{
"ts": "2026-10-26T10:42:17Z",
"feature": "support_triage",
"tenant": "acme-eu",
"task_id": "tkt-88213",
"model": "small-general-v3",
"route_reason": "default",
"input_tokens": 1840,
"cached_input_tokens": 1500,
"output_tokens": 96,
"latency_ms": 640,
"outcome": "auto_resolved"
}With that record you can answer the questions that matter: which feature drives spend, what a task costs end to end, and whether cache hit rates and routing decisions behave as expected. Without it, cost reviews become arguments about the monthly invoice.
Lever 1: Send fewer tokens
- Trim the prompt. Long system prompts accumulate. Remove instructions that evals show make no difference.
- Retrieve less, better. A reranker and a relevance threshold usually let you send a handful of strong passages instead of many weak ones. See RAG for enterprise documents.
- Keep tool outputs small. Agents pay for every tool result they read on every subsequent step. Return only the fields the task needs.
- Cap output length and ask for structured output where possible. Output tokens are typically priced higher than input tokens.
- Summarise or truncate long conversation history instead of resending everything each turn.
Lever 2: Caching
Prompt (prefix) caching
Most major providers offer some form of prompt caching: when consecutive requests share an identical prefix, the cached portion is processed at reduced cost and latency. Mechanics differ by provider (automatic or explicit, minimum lengths, cache lifetime), but the design rule is the same: put stable content first and variable content last.
[cache-friendly order]
1. System instructions (identical for every request)
2. Tool definitions (identical, deterministic order)
3. Reference material / few-shot (identical per feature or tenant)
4. Conversation history (grows, but prefix stays stable)
5. Current user input (changes every request)A timestamp or request ID inserted near the top of a prompt silently defeats caching for everything after it. Check the cached_input_tokens field in your logs rather than assuming.
Response caching
If identical inputs recur, such as the same classification request or the same FAQ, cache the final response keyed on the normalised input, the prompt version and the model. Be cautious with semantic caching (returning a cached answer for a similar question): it can serve a confidently wrong answer to a question that differs in one important detail, and it must respect permissions so one user's answer is never served to another who lacks access to the sources.
Lever 3: Route by difficulty
Not every request needs the most capable model. A common pattern is to send everything to a smaller, faster model by default and escalate to a larger one when the task is known to be hard, or when the small model's output fails validation or reports low confidence.
{
"feature": "support_triage",
"default_model": "small",
"escalate_to": "large",
"escalate_when": [
{
"input_tokens_gt": 6000
},
{
"language_not_in": [
"en",
"hi"
]
},
{
"validation_failed": true
},
{
"tag_in": [
"legal",
"complaint_escalation"
]
}
],
"max_output_tokens": 300,
"prompt_version": "triage-v14"
}Routing changes quality, so every routing rule should be validated against the eval suite before it ships. Our post on evaluating LLM applications shows how to compare configurations on the same test set, sliced by category, so you can see where the small model is good enough and where it is not.
Lever 4: Batch what isn't interactive
Document backfills, nightly classification, report generation and eval runs do not need an answer in seconds. Many providers offer batch APIs that accept large sets of requests with a longer turnaround at a discount, and they avoid competing with interactive traffic for rate limits. Splitting workloads into interactive and batch paths is often the largest single saving available to a team that has been running everything synchronously.
Lever 5: Budgets and limits
Agents and user-facing chat can generate unbounded usage, whether through loops, very long sessions or abuse. Enforce budgets in code:
- Per task: maximum steps, tokens and wall-clock time for an agent run, with escalation when exceeded. See AI agents in the enterprise.
- Per user or tenant: daily token budgets and rate limits, with clear behaviour when reached.
- Per feature: spend alerts against a forecast, so an unexpected spike pages someone before the invoice arrives.
def check_budget(tenant: str, feature: str, est_tokens: int) -> None:
used = usage_store.tokens_today(tenant, feature)
limit = budgets.get(tenant, feature)
if used + est_tokens > limit:
metrics.incr("budget_block", tenant=tenant, feature=feature)
raise BudgetExceeded(f"{feature} daily budget reached for {tenant}")
if used + est_tokens > 0.8 * limit:
alerts.warn_once(tenant, feature, "80% of daily LLM budget used")Self-hosting is a cost decision too
Running open-weight models on your own infrastructure can make sense for very high, steady volumes or for strict data-residency needs (see AI data privacy and governance). It replaces per-token cost with GPU capacity, operations staff and the work of keeping models current. Compare total cost per task, including utilisation, before committing; idle GPUs are expensive.
Key takeaways
- Instrument cost per task, per feature and per tenant before optimising.
- Send fewer tokens: tighter prompts, better retrieval, smaller tool outputs, capped outputs.
- Structure prompts for prefix caching, and verify cache hits in logs.
- Route easy work to smaller models and batch anything non-interactive, gated by evals.
- Enforce budgets in code for agents, users and tenants.
HashTech reviews LLM cost and architecture as part of its AI enablement work, alongside an opportunity assessment when the question is whether a use case pays off at all.
