Cheaper tokens, bigger bills
Nearly every enterprise AI program eventually lands in the same meeting. Someone from finance opens the cloud invoice, points at the AI line item, and asks why it tripled in a quarter when the price per million tokens fell twice in the same period.
Both facts hold. Frontier model pricing has fallen steadily since 2023, and Gartner projects that running inference on a trillion-parameter model will cost at least 90 percent less by 2030 than it did in 2025. Gartner's August 2026 forecast also puts the cost of inference per agentic workflow on a path to more than quintuple through 2028. The firm named the pattern the inference paradox: better unit economics escalate total spend instead of containing it.
The distance between those two curves is where AI budgets die. The cause is not waste in the ordinary sense. It is what agents do, how often they do it, and how much context each step drags along.
An agent is not a chatbot with better manners
A single chatbot exchange is cheap. One turn in, one answer out, roughly a thousand tokens, a fraction of a cent. An agent that plans a task, retrieves context, calls tools, reads the results, reasons about its own output, and tries again when the result is wrong can run between ten cents and a dollar per completed task. Gartner's March 2026 analysis found agentic models consume five to thirty times more tokens per task than a standard chatbot.
That multiplier is the entire story. A pilot runs on a handful of curated tasks with an engineer watching. Production runs the same loop thousands of times a day on untidy inputs, with retries nobody budgeted for. Pilot economics bear almost no relationship to production economics, and most teams learn the difference from the first full-scale invoice rather than from a capacity plan.
Where the tokens actually go
Intuition is a poor guide. A January 2026 study from Concordia University, "Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering," ran thirty software tasks through the ChatDev multi-agent framework on GPT-5 and measured the breakdown. Reasoning tokens alone ranged from 17,280 to 40,000 per task. Code review consumed an average of 59.4 percent of all tokens. Writing the initial code took 8.6 percent.
The split is worth a second look. Generating the code, the step most teams think of as the product, is a rounding error in the bill. The expensive part is the machinery around it: the agent re-reading files, checking its own work, and re-reasoning over context it has already seen. A cost program that obsesses over generation quality while ignoring the review and retry loops is optimizing the wrong 8.6 percent.
Three levers that move the number
Route by task, not by habit
Most enterprise traffic is not frontier-model work. Classification, extraction, routing, and short summarization run comfortably on a small model. Genuinely hard synthesis and long multi-step reasoning need a large one, and those calls are rare next to the rest. A routing split that sends the bulk of steps to a lightweight tier and reserves the frontier model for difficult calls routinely cuts blended cost by around 40 percent, with heavier savings on neatly routable workloads. The price gap explains why: a frontier model can cost twenty to fifty times more per token than a small one. The common failure is not a wrong routing decision here or there. It is leaving every step on the expensive default because nobody measured which steps actually needed it.
Fix the cache boundary before tuning the cache
Prompt caching is the highest-leverage change available for repeated context, because providers bill cached reads at a fraction of the input price. Rates vary by provider. On Claude, cache reads run at roughly 2.5 to 10 percent of the standard input rate, and a 2026 cross-provider study measured 41 to 80 percent lower API costs on long-horizon agent tasks once caching was enabled.
Caching pays off only when the cached prefix stays stable between calls. ProjectDiscovery's security agent Neo runs twenty to forty or more model steps per task on top of a twenty-thousand-token system prompt. Its initial cache hit rate was 7 percent, because the system prompt carried working memory that changed as the agent worked, invalidating the cacheable prefix on nearly every step. Moving the dynamic memory out of the system prompt and into a user turn lifted the hit rate without changing what the model could see. Deriv, the trading platform, reported an 85.8 percent cache hit rate on its production assistant, which cut its input-token cost by 77 percent.
Ordering matters as much as switching caching on. Put the stable material first, meaning tool definitions, system instructions, and reference documents, and the volatile material last. One caveat worth knowing: naive full-context caching can raise latency instead of lowering it, so set the boundary deliberately rather than caching everything by default.
Cap the loop
An agent without a budget will retry until something else stops it. Per-task spend limits, iteration ceilings, and circuit breakers that halt a workflow after a set number of retries turn a runaway into a logged failure. A single misbehaving agent can burn through a day of budget before anyone notices, because token-spend alerts typically fire after the fact. The control belongs in the runtime, not in a dashboard somebody checks on Mondays.
Retrieval is where input tokens quietly pile up
Embedding spend is usually small change. OpenAI's text-embedding-3-small charges roughly two cents per million tokens, and Cohere's Embed-4 runs about twelve cents per million. Vector storage adds a monthly line, not a crisis. Where the money actually goes is the model call that follows retrieval. A RAG pipeline that pulls five documents and pastes them into the prompt pays for every retrieved token on every query, and stale or over-broad retrieval makes that worse.
Two techniques attack this. Query-aware compression trims retrieved documents down to what the question actually needs. Microsoft Research's LLMLingua demonstrated up to twenty times prompt compression with minimal quality loss on the studied tasks, and FrugalGPT showed that cascading a cheap model before an expensive one can match stronger-model accuracy at a fraction of the cost. Semantic caching, which reuses the response to a semantically similar query instead of calling the model again, has posted production hit rates in the 20 to 40 percent range. None of these are exotic. They are prompt and pipeline discipline that most teams skip because the pilot never demanded them.
The counterintuitive part: fewer tokens can cost more
Compression sounds like free money. It often is not. PointFive benchmarked four Claude Code configurations across 103 tasks and 2,908 executed runs, and measured the result against the actual bill rather than a token counter. The configuration that cut the most tool-output tokens, 38.4 percent of them, ended up with a 6.8 percent higher pooled billed cost.
The reason is ordinary rather than mysterious. A compressor edits what the agent can see in the middle of a task. Strip the wrong context and the agent takes more turns, calls more tools, and re-derives information it used to have for free. Token reduction and cost reduction are different metrics, and they can move in opposite directions. This is the discipline that separates real AI cost work from theater: measure every change against the invoice, never against a proxy such as token count or compression ratio.
What to measure instead
The metric that survives a conversation with a CFO is cost per successful task. It folds in retries, failed runs, human corrections, and the tokens an agent burns looping on a problem it cannot solve. A team can halve cost per token and still watch cost per outcome climb, and only the second number appears on the invoice.
Four numbers are worth tracking every month: total inference spend by use case, cache hit rate, cost per successful task broken out by workload tier, and the savings actually realized from each routing or tier change. The last one closes the loop, because a projected saving that never shows up in the bill is a modeling error rather than a win.
A 30-day sequence for teams already in production
- Start with instrumentation. Per-task cost and cache hit rate come first, because most teams cannot name their cost per task and every later decision leans on that baseline.
- Reorder the prompts so the stable prefix leads and the volatile context trails. Record the hit rate before and after.
- Build the routing table. Classify the real step distribution, move the simple majority to a cheaper tier, then confirm quality holds on a frozen evaluation set.
- Add per-task budget caps and retry ceilings, and log every trip. A cap that never fires within a month is set too high, and one that fires constantly is hiding a quality problem.
Cheap tokens do not fix expensive architecture
Gartner's projection that inference will become roughly a hundred times more cost-efficient by 2030 is probably right, and it will not rescue a badly designed agent system, for the same reason cheap electricity does not make an inefficient factory profitable. Falling unit costs enable more ambitious agents, which consume disproportionately more tokens, which pushes total spend higher. Gartner itself warns that providers will not pass the full reduction through to buyers. The durable advantage goes to teams whose loops, memory, and routing are sound enough that the token bill stays a predictable line item.
Inference cost is an architecture problem, and it rewards the same discipline as any other production system: measure the real thing, change one variable at a time, and decline to celebrate a saving you cannot see in the bill. Teams that treat it that way end up with agents that hold up in production. Teams that treat it as a procurement question get another surprising invoice.
