Serving keeps a model available. Inference engineering makes each invocation fit its task, evidence, operational limits, and authority.
A model endpoint can be available, fast, and correctly provisioned while the application still fails.
Imagine an assistant that turns a maintenance note into a proposed work order. The endpoint returns within its latency target and the response parses as JSON. Yet the system may have selected a model that performs poorly on terse notes, supplied stale asset context, retried until the request exceeded its cost budget, or sent a draft directly to a write operation that requires approval.
The model server may be healthy. The inference system is not doing its job.
Serving is a subsystem
Model serving is essential engineering. It covers how model weights are loaded and executed, how requests are scheduled and batched, and how memory and accelerators are used to meet throughput and latency goals. Work such as ORCA’s iteration-level scheduling addresses exactly this serving problem for autoregressive generation. Its contribution is meaningful within that boundary; it does not by itself decide whether the application selected the right model, supplied the right evidence, or used the result appropriately. (Yu et al., OSDI 2022)
Inference engineering takes the request as its unit of design. It asks what the system must do before, during, and after tokens are generated so that the result is usable under the application’s constraints.
In CARM’INK’s Semantic Intelligence perspective, inference engineering is the design, integration, measurement, and control of model inference across the application lifecycle. It includes model and runtime selection; context preparation; routing; structured output; fallback behavior; latency and cost budgets; observability; and controlled execution. Model serving is one important layer within that wider discipline.
Give the request an operating contract
A prompt describes part of a request. An operating contract makes the surrounding requirements explicit: which evidence is allowed, how a model is selected, what form the answer must take, what limits apply, and what the system is permitted to do with the result.
The specimen below is illustrative pseudo-configuration. It is not a vendor format, an existing CARM’INK schema, or a runnable configuration.
task: prepare_work_order_draft
context:
source_ids: [maintenance_note, asset_record]
source_version: required
access_scope: request_scoped
inference:
model_policy: task_and_quality_requirements
runtime_policy: deployment_constraints
selection_reason: recorded
output:
schema: work_order_draft
validate_before_return: true
budgets:
deadline: request_specific
maximum_model_calls: bounded
cost_limit: request_specific
fallback:
on_timeout: return_status_without_write
execution:
authority: prepare_only
approval_required_before_write: true
trace:
record: [stage_timings, selected_model, context_version,
output_validation, fallback_reason]These fields are not paperwork around the model call. They define how the call is allowed to behave and give the surrounding system enough information to evaluate that behavior later.
Selection is an engineering decision
“Use the strongest model for every request” can waste time and money. “Use the cheapest model” can make the task unreliable. Model and runtime selection should be based on the task’s required quality, context length, output constraints, latency target, deployment conditions, and the cost of an error.
Routing research frames this as a quality–cost trade-off. RouteLLM, for example, evaluates routers that choose between models using preference data and reports results on its selected benchmarks. Those results establish that routing can be worth testing; they do not guarantee the same trade-off for another task, model pool, language, or risk profile. (Ong et al., 2024, revised 2025)
Routing also adds another decision system that can fail. A 2026 EACL evaluation found category-specific weaknesses in the routers it tested, including safety-relevant misrouting behavior. The result is bounded to that study’s systems and evaluation, but it makes a practical point: evaluate router behavior across the task and risk categories that matter, and provide a safe path when confidence or policy checks fail. (Kassem, Schölkopf, and Jin, EACL 2026)
The runtime belongs in the same evaluation. A model’s advertised capability is not the whole result: tokenization, quantization, context limits, hardware availability, batching, and concurrency affect what the deployed system can actually deliver. Compare candidate configurations on representative tasks, and record which model and runtime handled each request. A route that saves cost but obscures its decision is difficult to govern.
Context and output shape the result
Context preparation determines what the model can use: selected source material, relevant structure, versions, permissions, and task-specific constraints. Context engineering and inference engineering meet here. The first organizes information for a task; the second incorporates that prepared context into a measured, controlled execution path.
Output constraints are part of the inference design too. If another service consumes the answer, a schema can make the expected fields explicit and allow the application to reject malformed results before they travel downstream. But structural validity is not factual correctness, adequate evidence, or permission to act. Research on schema-constrained generation continues to find meaningful challenges in producing valid structured outputs across diverse schemas. (Lu et al., ACL 2025)
Treat validation as a boundary with a defined response. A missing required field might trigger one bounded repair attempt or return a review state. A contradictory source should not be “fixed” by asking the model to produce more confident prose. The system should preserve enough provenance to show which context and validation rules were used.
A fallback changes the behavior
Fallback is often treated as a retry with a different endpoint. That is too broad. A retry may handle a transient transport failure. A fallback may change the model, reduce the task, return a partial result, or route to a human. Each choice changes the contract.
Make fallback behavior explicit and bounded. Set a total deadline and a maximum number of calls. Preserve the reason for changing paths. Validate the fallback result using the same applicable output and policy checks. If the source context is missing or access is denied, changing models does not repair that problem. If the fallback cannot preserve the requested guarantees, return a clear incomplete or review-required state instead of silently lowering them.
This makes resilience inspectable. It also limits retry amplification, hidden changes in quality, and unexpected cost growth.
Optimize the whole request, not just token speed
Latency is the time a user or downstream process waits for a usable result. It may include queueing, context retrieval and preparation, model calls, structured-output validation, retries, approval, and tool execution. Time to first token and time per output token help explain model-server behavior, but they do not describe the complete operation.
Cost has a similar boundary. Include the work required to prepare and validate context, the number and size of model calls, retries, and any tools used. A cheaper first call can become an expensive workflow if it routinely triggers repair calls or human rework. The meaningful comparison is cost and latency against the task outcome and the level of assurance required.
Observability should follow those boundaries. Record the selected model and runtime, context version or identifiers, routing reason, stage timings, validation outcome, fallback path, and execution result. Protect sensitive prompts and source content; tracing should not become an unexamined copy of private data.
OpenTelemetry’s GenAI semantic conventions provide examples of separating model-server measures—such as request duration and time to first token—from client operations and workflow duration. The conventions are currently marked as in development, so teams should treat them as evolving guidance and verify their version before adopting specific names. (OpenTelemetry GenAI metrics conventions)
Generation is not authorization
A well-formed output is still only model output. The application must determine whether it can be used, who or what can execute the next operation, and what evidence or approval that operation requires.
Keep that authority outside the model’s generated text. Enforce permissions in the application or tool boundary; validate inputs again before writes; use idempotency and audit records for consequential actions; and require human review when the impact or uncertainty warrants it. The specific controls depend on the application’s risk. NIST’s AI Risk Management Framework describes governance as cross-cutting and risk management as continuous across the AI system lifecycle; it provides a useful general basis for treating execution controls as part of system design, rather than as a final prompt instruction. (NIST AI RMF Core)
A working definition
For CARM’INK, inference engineering is the engineering of the complete path from a task and its prepared context to a validated, observable, and appropriately controlled model-assisted outcome. It includes serving, while extending beyond serving to the choices and safeguards that make inference fit the task.
The central design question is not only whether the model can return tokens quickly. It is whether the system can produce the right kind of result, under known limits, with a traceable path and an authorized next step.
References
- Yu, G.-I. et al. “Orca: A Distributed Serving System for Transformer-Based Generative Models.” 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), 2022. USENIX paper and presentation
- Ong, I. et al. “RouteLLM: Learning to Route LLMs with Preference Data.” arXiv:2406.18665, 2024; revised 2025. Paper
- Kassem, A. M., Schölkopf, B., and Jin, Z. “How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities.” Proceedings of EACL 2026, 7496–7507. Paper
- Lu, Y. et al. “Learning to Generate Structured Output with Schema Reinforcement Learning.” Proceedings of ACL 2025, 4905–4918. Paper
- OpenTelemetry. “Semantic Conventions for Generative AI Metrics.” Status: Development. Specification source
- National Institute of Standards and Technology. “AI RMF Core,” excerpt from Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023. NIST resource