An inference bill records a model call. It does not necessarily record the cost of producing an accepted answer.
A production answer may involve retrieval, context assembly, several inference calls, tool execution, verification, retries, human review, and operational recovery. Some of those steps have a direct price. Others consume compute, staff time, or service capacity. Latency is another consequence of the path, though it is not automatically a monetary cost.
The useful unit of measurement is therefore rarely “cost per model call.” For many production workflows, it is cost per accepted outcome.
Figure 1 — The actual cost path
Caption: Tool results can return to context assembly and inference. Retries may repeat some of that work. Human review and failure handling add their own labor and elapsed time.
Accessible description: The path begins with a task and identity, then retrieves information, prepares context, and runs inference. A tool call can return new information for another context and inference cycle. The result is verified, then accepted, sent to a human when review is required, retried, or escalated. Each repeated path can add cost and latency.
Follow the work, not just the invoice
Token usage is visible and often straightforward to price. It is only one line in the account.
Retrieval may require query processing, ranking, data transfer, and access to a search or database service. Indexing and storage are shared costs that happen before any single answer; they still belong in a system-level cost model, usually allocated across the workloads they support.
Context preparation also consumes resources. A pipeline may filter records, apply access rules, resolve entities, remove duplicates, assemble evidence, and serialize it for inference. The resulting context affects both the work done before the model call and the input sent to it. Tool results can enlarge later context, so a multi-step path may pay again to process information it just retrieved.
Inference cost depends on where and how the model runs. An external API may charge for input and output usage or another service unit. A locally hosted model moves the accounting toward compute capacity, utilization, energy, and operations. These approaches cannot be compared by token price alone. The relevant comparison uses the same workload, success criteria, and service expectations.
Every branch can change the total
A tool call can add an API charge, a database query, a paid search request, data transfer, or time spent waiting for a downstream service. It can also return information that triggers another model call.
Verification has a cost too. A schema validator or deterministic policy check may use existing compute. A second model used to judge the first answer adds another inference step. Human review adds active work, and may introduce queueing or interruption. These are different measures: staff time is a labor cost; review wait contributes to elapsed latency.
Retries make the gap between “price per call” and “price per outcome” especially clear. A failed attempt may have already paid for retrieval, context assembly, inference, and a tool call. A retry may repeat all or part of that path. A fallback may use another model or provider. A failure that reaches an operator may require investigation, correction, reprocessing, or rollback.
Those steps should be counted when they occur. Their possible impact should be tracked separately when it cannot be reliably expressed as money.
Research on model selection has made the same broader point from a different direction. FrugalGPT explores shorter prompts, model approximation, and model cascades as ways to change the relationship between cost and task performance. Its findings come from its evaluated tasks; they are not a guarantee that a cascade will reduce the cost of any production workflow. [1]
A January 2026 preprint comparing one-shot answers with tool-using plans across two task settings found that the value of planning and tools varied by task. In one setting, tools improved accuracy while substantially increasing latency; in another, added planning did not provide consistent accuracy gains. The benchmark is small and specific, but it illustrates why an end-to-end path needs measurement. [2] A 2025 preprint on budget-aware tool use likewise treats tokens and tool calls as separate resource costs and finds that simply allowing more tool calls does not ensure better performance. [3]
Cost and latency belong next to each other
A workflow’s elapsed time can be written as:
Total latency = queueing + retrieval + context preparation + inference + tools + verification + retries + human wait
That expression is not a cost formula. Waiting time does not always create a direct charge. It can, however, affect a service-level objective, occupy worker capacity, delay a decision, or increase the time needed to recover from a failure.
The distinction matters when optimizing. A design that reduces API spend but adds several slow tool calls may be unsuitable for an interactive workflow. A slower path may be acceptable for a batch process if it improves evidence quality or reduces human rework. The trade-off depends on the service being delivered.
Account for the accepted outcome
A practical task-level ledger can use this structure:
Task cost = retrieval + context preparation + inference calls + tool calls + verification + human work + recovery
Then allocate shared costs such as infrastructure, indexing, storage, observability, and maintenance to the relevant workload.
To calculate cost per accepted outcome, include the attributable cost of failed attempts and divide by the number of outcomes that actually met the acceptance criteria:
Cost per accepted outcome = total attributable cost / accepted outcomes
An answer that was generated but rejected is not a completed outcome. A workflow that required a person to repair the result should count that work. This denominator discourages optimization that makes individual calls cheaper while making successful work less likely or more labor-intensive.
The ledger should preserve enough detail to explain where costs came from: workflow or task type, model and runtime, input and output usage, retrieval activity, tool calls, verification results, retries, and human review. Avoid recording sensitive content merely to improve cost attribution. The aim is to make resource use observable while keeping data handling within policy.
CARM’INK’s engineering view
For CARM’INK, this belongs to Inference Engineering: selecting and operating a path that meets the task’s quality, latency, governance, and cost requirements.
That path includes more than the model. Semantic retrieval and context preparation contribute to it. So do tools, validation, controlled transitions, human review, and recovery. An external model price and a locally provisioned runtime have different cost structures; neither is inherently cheaper for every workload.
The engineering decision should compare complete paths against the same accepted outcome. It should also make retries bounded, tool use observable, verification proportionate to risk, and human review explicit where the workflow requires it.
The cost of an AI answer is the cost of the work required to produce an answer the system and its users can actually accept.
References
- Lingjiao Chen, Matei Zaharia, and James Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance”, published in Transactions on Machine Learning Research, 2024.
- Subha Ghoshal and Ali Al-Bustami, “When Do Tools and Planning Help LLMs Think? A Cost- and Latency-Aware Benchmark”, arXiv preprint, 6 January 2026.
- Tengxiao Liu et al., “Budget-Aware Tool-Use Enables Effective Agent Scaling”, arXiv preprint, 21 November 2025.