The Missing Layer Between Enterprise Data and AI Agents

Agents can retrieve enterprise documents. They still need help understanding what those documents mean to the organization, for this task, at this moment.

An enterprise rarely stores its knowledge in one vocabulary. A customer relationship system, an ERP, a contract repository, a policy portal, and a ticketing platform may describe the same organization, obligation, or event in different ways. Each has its own identifiers, fields, definitions, and update cycle.

People often reconcile those differences through experience. An AI agent has no dependable way to do that unless the system provides one. If every agent must reconstruct organizational meaning from raw documents for every task, the architecture is asking inference to repeat work that should already be made explicit.

This is the gap between enterprise data and agent inference. It calls for a semantic and context layer: a way to resolve meaning, select relevant evidence, and represent what is known before a model is asked to reason.

Retrieval answers only part of the problem

Retrieval-augmented generation connects a model to external information. The original RAG work combined a model’s parametric memory with a retrievable, non-parametric memory, enabling access to information outside the model’s parameters.[1]

That is an important capability, but a retrieved passage is not yet organizational context. Retrieval can find a document containing the words “supplier renewal.” It may not establish which supplier record corresponds to the document, whether the clause remains in force, which policy version governs, or whether the request falls within the agent’s permitted scope.

Graph-based retrieval can help with some of these problems. For example, the GraphRAG research describes using an entity graph and community summaries to answer corpus-wide questions; its reported gains apply to a class of global sense-making tasks, not to every enterprise agent problem.[2] A graph or a vector index can improve what a system finds. Neither alone defines which facts are authoritative, current, relevant, or usable in a specific task.

The distinction is between finding information and preparing a defensible context for inference.

Define the layer by what it hands to inference

“Semantic and context layer” describes an architectural function, not a requirement to buy or build one specific product. It mediates between enterprise sources and the inference runtime. Its job is to turn scattered records into task-relevant, source-linked context while preserving ambiguity instead of concealing it.

Context element	What it contributes
Entity references	Identifiers, aliases, types, and links that help distinguish one organization, person, asset, or obligation from another
Claims and relations	Facts and connections, each tied to a source and its relevant time
Provenance	Where information came from and how it was derived
Definitions and policy references	The terms and rules that frame the task
Conflicts and gaps	Information that disagrees, is missing, or may be out of date
Selection rationale	Why specific information was included for this task

Provenance is particularly important when information crosses system boundaries. W3C PROV-O provides a vocabulary for representing provenance, including entities, activities, agents, and relationships among them.[3] An implementation does not have to adopt PROV-O wholesale to benefit from the principle: a claim should remain traceable to its source and history.

The layer can be federated. It does not need to copy every enterprise document into a central repository or replace systems of record. It may resolve identifiers, maintain semantic mappings, retrieve records from their owners, and assemble a context package for a particular task. The important property is that inference receives meaning and evidence in a form the system can inspect—not that all data lives in one place.

A context contract

A context package should be explicit enough for the runtime to use and for an engineer to inspect. For example, a system might pass a typed object like this:

ContextPacket {
  purpose,
  entities,
  claims: [{ value, source, observedAt, validDuring }],
  definitions,
  unresolved,
  coverage,
  contextVersion
}

This is a schematic contract, not a universal schema. Its design makes several useful distinctions visible:

• A claim has a source and time; it is not just text in a prompt.
• A definition can be distinguished from a particular instance of a fact.
• An unresolved conflict can be passed to inference as a limitation, not silently settled by retrieval ranking.
• The context version allows the system to identify which assembled view informed a response.

The context layer should also record why a particular source was included and what relevant information was excluded. That does not make the result infallible. It makes the selection reviewable and gives the inference runtime a better basis for calibrated answers, abstention, or escalation.

Meaning and authority have different jobs

A semantic layer can help resolve which records relate to a request and identify relevant policy references. It should not be treated as the authorization decision itself.

There are two distinct questions:

1. What information is relevant and available for this task?
2. Is this actor or agent authorized to perform this operation now?

Access controls must govern the context returned to the agent. Then consequential actions should pass through a separate, enforceable decision point at execution time. Open Policy Agent’s agent-tool-calling example illustrates this pattern: the agent harness sends a proposed tool call to a policy decision point and enforces the result before running the tool.[4]

Keeping these responsibilities distinct prevents “the model saw it” from becoming “the model may act on it.” It also allows a system to change an action policy without pretending that the underlying business meaning has changed.

The failure modes become easier to see

Without a mediation layer, each agent or workflow tends to develop its own interpretation of enterprise terminology. One process may treat a record as authoritative because it is recent; another may privilege a signed document; a third may retrieve a cached summary. The resulting answers can sound consistent while relying on different evidence.

A semantic and context layer does not eliminate disagreements among sources. It makes them representable. If a system finds two conflicting dates, it can carry both claims, their provenance, and the conflict forward. If the source of truth is unknown, it can mark that gap. The model can then explain the uncertainty or request a decision instead of inventing an organizational consensus.

This also changes how teams can evaluate agent systems. Alongside answer quality, they can measure whether the system resolves entities correctly, preserves source links, identifies stale or conflicting information, respects context access boundaries, and abstains when evidence is insufficient. Those measures test the information path into inference, not just the model’s output.

The architectural position

The missing layer is not another prompt template and not a universal ontology that attempts to encode every corner of an organization. It is a governed mediation function that connects enterprise meaning to task-specific context.

CARM’INK’s research position is that Semantic Intelligence begins before generation: the system must make entities, relationships, provenance, time, and uncertainty legible before inference can use them well. Retrieval remains useful. Models remain useful. Their work becomes more reliable when they receive a coherent account of what the organization knows, how it knows it, and what remains unresolved.

Enterprise agents should not have to rediscover organizational meaning from raw documents on every task. The architecture should carry that meaning forward—source-linked, time-aware, scoped to the task, and separate from the authority to act.

References

1. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, Advances in Neural Information Processing Systems 33, 2020.
2. Darren Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization”, 2024.
3. W3C, “PROV-O: The PROV Ontology”, W3C Recommendation, 30 April 2013.
4. Open Policy Agent, “Agent Tool Calling”.