Production AI includes the knowledge, context, rules, state, tools, identity, authority, inference, observability, and people that determine what a model output can mean and do.
A model can be evaluated in isolation. A production AI outcome cannot.
A model is a computational component. A production system includes the information and mechanisms that shape its inputs and responses, the permissions and interfaces around it, and the people responsible for its use.
This distinction changes what a benchmark can tell us. A score describes a model under a particular test setup. It does not establish what knowledge a deployed system can access, which context it supplies, what policy applies, whose identity makes the request, what state persists, which tools can change external records, or how people detect and handle failure.
Machine-learning research has long examined this wider boundary. Sculley and colleagues describe technical debt arising from data dependencies, configuration, feedback loops, boundary erosion, and undeclared consumers. Amershi and colleagues’ study of machine-learning software teams identifies data discovery and versioning, model reuse, and model entanglement as engineering challenges. These works predate today’s generative systems; their relevance here is the systems-engineering lesson, not a claim that they evaluated every modern AI deployment. (Sculley et al., NeurIPS 2015; Amershi et al., ICSE-SEIP 2019)
What surrounds the model
| System element | What it determines |
|---|---|
| Knowledge | Which sources and semantic relationships are available, and how their provenance is retained |
| Context | Which evidence and constraints are selected for this task |
| Policy | What is required, permitted, prohibited, or escalated |
| State | Which prior events or decisions persist, and under what validity conditions |
| Identity | Which person, service, or agent is making the request |
| Authority | Which operations that identity may perform |
| Tools | Which external systems can be read or changed |
| Inference | Which model, runtime, and route produce the response |
| Observability | What evidence lets people inspect behavior and reconstruct outcomes |
| People | Who defines the objective, handles exceptions, and owns consequences |
These elements interact. Identity without scoped authority can grant excessive power. State without provenance can be mistaken for current fact. A tool that writes directly from generated output can turn a proposal into an unreviewed action. A trace that omits context and policy versions may be unable to explain why a result occurred.
The model generates a response. The system gives it context, authority, consequences, and accountability.
Evaluate the deployed boundary
Model evaluation remains necessary. It answers questions about a model’s behavior under stated conditions. System evaluation asks whether the integrated application behaves acceptably when evidence is incomplete or outdated, policies change, routing selects a different model, a tool is unavailable, an identity lacks permission, output validation fails, or a human must intervene.
The NIST AI Risk Management Framework treats governance as a cross-cutting function and organizes risk work around the AI system’s context and lifecycle. That framing supports a practical design principle: define responsibility and controls across the system that is used, not only around the model that generates. The framework is voluntary guidance; CARM’INK’s system boundary is an engineering interpretation, not a NIST definition. (NIST AI RMF Core)
CARM’INK’s position
For CARM’INK, Semantic Intelligence is concerned with how meaning is represented, selected, preserved, and used across an AI system. Inference matters, but its output only becomes operationally meaningful within the surrounding knowledge, context, policies, state, identities, permissions, tools, observability, and human responsibilities.
The design object is therefore the governed path from evidence to inference to outcome. That path must make its sources and constraints legible, its authority explicit, and its consequential behavior reviewable.
The model is not the system. The system is what we deploy, govern, observe, and answer for.
References
- Sculley, D. et al. “Hidden Technical Debt in Machine Learning Systems.” Advances in Neural Information Processing Systems 28, 2015, pp. 2503–2511. Google Research publication page
- Amershi, S. et al. “Software Engineering for Machine Learning: A Case Study.” 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. DOI
- National Institute of Standards and Technology. “AI RMF Core,” excerpt from Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023. NIST resource