Why Organizations Waste 40% of LLM Spend Without Governance
Organizations usually do not waste AI budget because the invoice is wrong. They waste it because cost control happens after the request has already been sent to the provider. When every workload defaults to an expensive model, repeated prompts are answered from scratch, and no team has a hard budget boundary, 20 to 40 percent of spend can disappear without improving output quality. Keeptrusts closes that gap by putting wallets, reserve and settle, provider routing, caching, dashboards, and exports directly in the execution path.
Use this page when
- Your AI bill is rising faster than usage or business value.
- Multiple teams are using LLMs, but nobody can explain which workloads deserve premium models and which do not.
- You need a practical explanation of why governance reduces cost without forcing teams to stop shipping.
Primary audience
- Primary: Technical Leaders
- Secondary: Technical Engineers, FinOps owners
The problem
The 40 percent waste number is not a single failure. It is usually the combination of four predictable leaks.
First, organizations treat every request like a premium reasoning problem. A support summary, a ticket classification task, and a long-form planning prompt are often all sent to the same top-tier model because the application hardcodes one default. That means cheap, repetitive work is billed at premium rates. The model is not the problem. The routing decision is.
Second, teams pay again for work they already bought. A customer support assistant may answer the same refund, shipping, or password-reset question thousands of times per month. An internal enablement bot may summarize the same policy or pricing guidance for every new seller. Without caching at the gateway, every repeated prompt becomes another upstream charge. When that repeated traffic is spread across several applications, the waste is hard to see until the invoice lands.
Third, budgets are often advisory instead of operational. Finance sees monthly spend after the fact. Engineering managers may get notified when a team is near a target, but nothing stops another batch of requests from leaving the building. If a retry loop, prompt experiment, or surge in customer traffic hits a premium model, the organization keeps spending while people argue over whether the traffic was legitimate.
Fourth, ownership is vague. A shared provider account creates one large bill but no clear budget owner. Without team-scoped wallets, billing budgets, dashboards, and monthly exports, cost discussions collapse into anecdotes. One manager insists support is the largest consumer. Another blames engineering experiments. Finance cannot close the books quickly because the attribution data is missing.
This is why unmanaged LLM spend feels irrational. The problem is not just price per token. The problem is that model choice, budget enforcement, and reuse are disconnected from the runtime path.
The solution
Keeptrusts turns spend control into an execution-time discipline.
Reserve and settle is the anchor. Before the gateway forwards a request, it estimates the cost and reserves that amount against the effective wallet scope. After the provider responds, it settles the reservation to the actual cost. That changes cost control from retrospective accounting into a pre-dispatch decision. If no eligible wallet has enough balance, the request is held or queued for review instead of becoming another surprise line item.
Wallets provide the hard boundary. Billing budgets provide the soft warning. Teams can run with both: a billing budget at 80 or 95 percent to warn the owner, then wallet enforcement to stop the spend from running past the approved limit.
Provider routing removes premium-model waste. Keeptrusts can route traffic across multiple targets so the organization is not stuck paying one vendor's highest rate for every task. The point is not to chase the cheapest number blindly. The point is to create a default path where simple tasks land on cheaper capable models and only expensive work uses expensive capacity.
Caching removes duplicate upstream calls. For exact matches or semantically similar questions, the gateway can return the stored answer immediately. In Keeptrusts documentation, cache hits settle at zero cost in billing dashboards, and cache hits do not debit wallet balance. That matters because the savings show up twice: lower provider cost and more remaining wallet capacity for work that actually needs inference.
Dashboards and exports make the savings credible. Spend becomes visible by team, provider, model, and period. Exports give finance and operations a clean artifact for monthly review, instead of forcing someone to rebuild attribution from raw logs.
Implementation
The practical rollout is straightforward: give teams wallets, enable cost tracking, route across more than one provider target, and add cache for repetitive workloads.
pack:
name: cost-discipline
version: 1.0.0
enabled: true
cache:
enabled: true
mode: exact
ttl_seconds: 3600
max_entries: 20000
namespace: support-prod
providers:
routing:
strategy: usage_based
targets:
- id: openai-mini
provider: openai:chat:gpt-5.4-mini-mini
secret_key_ref:
env: OPENAI_API_KEY
- id: openai-premium
provider: openai:chat:gpt-5.4-mini
secret_key_ref:
env: OPENAI_API_KEY
consumer_groups:
- name: support
api_key: kt_cg_support_prod
wallet_team_id: team_support
- name: engineering
api_key: kt_cg_engineering_prod
wallet_team_id: team_engineering
cost_tracking:
enabled: true
wallet_enforcement: true
budget_alerts:
- threshold_percent: 80
action: notify
- threshold_percent: 95
action: notify
- threshold_percent: 100
action: block
That config does three things that matter financially. It routes requests through multiple targets instead of one premium default. It enables exact cache for repeat prompts. It ties consumer groups to team wallets so cost attribution and enforcement happen at the same boundary.
Then add soft limits and reporting:
kt spend budget create --name "Support Monthly Cap" --limit 8000 --period monthly
kt spend budget create --name "Engineering Monthly Cap" --limit 15000 --period monthly
kt export-jobs create --type events --format csv --date-from 2026-05-01 --date-to 2026-05-31
The budgets give owners warning before they hit the wall. The export gives finance a month-end artifact with spend evidence that can be sorted by team, model, and provider. For the control-chain view of how these runtime decisions fit into Keeptrusts, use Policies Overview alongside the spend documentation.
Results and impact
Consider a company spending $80,000 per month on LLM traffic across support, engineering, and internal operations.
If 30 percent of that traffic is repetitive support or enablement work, and exact or semantic caching avoids even a quarter of those upstream calls, roughly $6,000 in monthly spend disappears without reducing service quality.
If another 35 percent of the traffic is classification, extraction, summarization, or short drafting work that does not need a premium model, provider routing can shift that volume to a lower-cost target. Even a conservative 30 percent blended reduction on that slice saves about $8,400 per month.
Now add budget discipline. Without reserve and settle, a few retry storms, open-ended experiments, or unowned shared tools can easily create 10 to 15 percent overspend because requests are still allowed after the team has effectively crossed its intended budget. If hard wallet limits eliminate even $10,000 of that monthly overrun, the total improvement approaches $24,000 per month, or 30 percent of the original bill.
It is not hard to reach 40 percent when the starting point is chaotic: one premium default model, no cache, and no hard budget enforcement. The ROI is not magic. It comes from removing three concrete failure modes: paying premium rates for simple tasks, paying repeatedly for the same answer, and paying after a budget should already have stopped the request.
The secondary impact is organizational. Once dashboards show spend by team and exports support monthly review, budget owners stop arguing about whose intuition is right. They start making operational decisions: move ticket classification to a cheaper model, increase cache coverage for FAQ traffic, or top up a wallet only when the workload is justified.
Key takeaways
- The biggest source of LLM waste is usually not token price. It is missing runtime controls.
- Reserve and settle changes spend management from an invoice problem to a request-time decision.
- Provider routing and caching reduce waste for two different reasons: one improves model selection, the other removes duplicate upstream calls.
- Wallets, billing budgets, dashboards, and exports are what make the savings enforceable, visible, and auditable.