A model is replaceable only when the system around it knows what must remain unchanged.
Moving inference from a local GPU to a CPU runtime, or from local execution to an external provider, can look like a deployment choice. In a production system, it is also a change to the execution environment, the data boundary, the failure modes, and sometimes the behavior users observe.
The architectural question is therefore not simply, “Can this model be swapped?” It is: Which parts of the system are allowed to change when the inference runtime changes?
For CARM’INK SEE, the answer begins with a separation. Semantic structure and operational logic belong to the system. A model and its runtime are selected capabilities within it.
Portability is a boundary, not a promise
A common request format can hide provider-specific APIs. It cannot make two models behave identically.
Different runtimes may vary in their supported input formats, context limits, structured-output behavior, latency, resource requirements, and failure modes. An external provider also changes where inference occurs and which data crosses the system boundary. A compatible endpoint is useful; it is not evidence of equivalent behavior or equivalent governance.
The boundary should make those differences explicit. Application logic constructs a task using the system’s own concepts, context, constraints, and output requirements. A runtime adapter translates that request into the selected inference implementation and normalizes its response. Provider-specific request objects and response fields stay inside the adapter.
This is a familiar pattern in systems engineering. ONNX Runtime, for example, exposes an execution-provider interface for hardware-specific execution, including CPU and GPU options. Its documentation also makes clear that an execution provider handles the portions it supports; the abstraction does not erase implementation differences. [1] Google Cloud’s reference architecture offers a related pattern at the serving layer: a unified frontend routes requests to backends hosted on-premises or by different providers, and incompatible APIs require a translation layer. [2]
The architectural lesson is broader than either example: a stable seam gives a system room to change; it does not make the things behind the seam interchangeable.
Keep meaning and execution in different places
A production AI system has at least two kinds of responsibility.
Semantic responsibility belongs to the application and its knowledge layer: what its entities mean, how they relate, which sources support a claim, and what context is relevant to a task.
Operational responsibility belongs to the governed system: which transitions are valid, which tools may be called, which identities may act, and what must be recorded or escalated.
Inference contributes a third capability: it interprets a prepared request and returns a candidate result. That result can help resolve ambiguity or produce structured output. It should not silently redefine the organization’s concepts, grant a permission, or advance a critical state.
This separation matters most when the model changes. If provider-specific JSON is embedded throughout business logic, a new runtime becomes a distributed rewrite. If policy exists only in a prompt, changing models may alter how that policy is interpreted. If authorization is inferred from generated text, an output has been allowed to cross into a role it should not own.
The system should instead make the operational decision in its own controlled path. The model can return a recommendation or a structured candidate. Deterministic validation and the authorized workflow decide what happens next.
Three execution modes, one semantic contract
CARM’INK SEE treats the runtime as an explicit configuration choice. The semantic architecture and application-level generation contract remain in place while the selected execution mode changes.
| Mode | Generation | Embeddings | Boundary to evaluate |
|---|---|---|---|
| GPU | Local | Local | Hardware and runtime dependencies; workload capacity |
| CPU | Local sovereign execution, without NVIDIA dependencies | Local | Model fit, latency, and throughput on the target CPU |
| WRAPPER | External | Local CPU | Data transfer, provider behavior, API translation, and external service conditions |
The table describes placement, not a performance ranking. Actual latency, cost, and quality depend on the workload, model, hardware, and configuration; they need to be measured for the deployment.
In all three modes, application code should speak the same internal request contract. The selected adapter owns the translation to its runtime. Embeddings stay local across the modes, and their index and dimensional contracts must remain compatible. Matching vector dimensions alone does not prove that two embedding spaces mean the same thing: changing an embedding model may require re-embedding and evaluation rather than simply reusing existing vectors.
The mode is selected through authoritative configuration. A transition from local execution to an external wrapper changes the data boundary, so it must be a deliberate operational choice. CARM’INK SEE does not silently fall back from sovereign inference to external inference when local execution fails. A failure should be visible and handled by an explicitly governed path.
Make runtime changes reviewable
The goal is not to make model changes frictionless. It is to make their effects legible.
Before a runtime change is accepted, the system should be able to answer:
- Does the request still carry the intended semantic context and provenance?
- Does the selected model meet the output contract for this task?
- Are validation, permissions, and workflow transitions enforced outside generated output?
- Does telemetry identify the configured mode, runtime, model version, and failure status without exposing sensitive content unnecessarily?
- Have latency, cost, and quality been evaluated under the target workload?
- If execution fails, is the response an explicit failure, a governed retry, or a separately approved route?
These checks treat a model change as an engineering change with observable consequences. A generic interface helps contain the change. Contract tests and operational review determine whether the new implementation is suitable.
The system also needs to record which runtime produced a result. Without that provenance, an investigation may see the generated text but not the conditions that shaped it: model version, execution mode, configuration, or adapter behavior. Observability is part of the portability design because it makes differences diagnosable after deployment.
The property to preserve
For CARM’INK SEE, Semantic Intelligence is the architectural identity: the system’s structured meaning, context, and relationships. Inference Engineering is a capability within that architecture: the controlled selection and execution of models and runtimes.
That distinction gives flexibility a precise meaning. A deployment can move between local GPU, local CPU, and external-wrapper inference without teaching every part of the application a new provider dialect. But the move remains subject to capability checks, data-boundary decisions, and validation of behavior.
The useful measure of model flexibility is not how many providers appear in a configuration file. It is whether the system can change where or how inference runs while keeping meaning explicit, operational authority governed, and the consequences of the change observable.
References
- ONNX Runtime, “Execution Providers”. Documentation describing its execution-provider interface and hardware-specific execution options.
- Google Cloud Architecture Center, “Networking for AI inference model serving on all backends”. Last reviewed 20 May 2026; describes a unified frontend for model backends hosted on-premises or by different providers, including API translation for incompatible backends.