Human-in-the-Loop Is an Architecture, Not a Button

Human oversight works when the system gives people defined authority throughout a workflow—not when it asks them to approve an opaque result at the end.

PROPOSED → APPROVED → EXECUTED

This three-state sequence is often presented as human-in-the-loop control. It leaves out the questions that determine whether the control works: What exactly is being approved? Which evidence supports it? Who has authority to decide? What happens if the context changes before execution?

A button can record a decision. It cannot, by itself, make that decision timely, informed, binding, or enforceable.

The engineering problem is therefore larger than adding a review step. Human-in-the-loop design must shape the system’s states, transitions, permissions, escalation paths, evidence, and operational responsibilities.

Oversight has to enter before the final step

Automation can affect different parts of a workflow: acquiring information, analyzing it, selecting a decision, and implementing an action. Those functions can be automated to different degrees; the person’s role may change at each stage.[2] A system that lets an agent perform all four functions and gives a human only a final confirmation has chosen a particular allocation of authority. It has not necessarily chosen a useful one.

NIST’s AI Risk Management Framework treats human roles and oversight as dependent on the system and its context. It distinguishes human-AI configurations and notes that some systems may require human oversight while others may not.[1] The design question is not “Where should every workflow insert an approval button?” It is “Where must a person be able to understand, change, stop, or own a consequential decision?”

That answer depends on the task’s impact, the quality of available evidence, the system’s permissions, and what recovery is possible if something goes wrong.

Map human authority across the workflow

An oversight plan should identify the human role at each point where the system’s direction or consequences can change.

Workflow point	Human contribution	System responsibility
Task definition	Set purpose, boundaries, or exceptions	Record the task scope and permitted data and actions
Context assembly	Correct entity, meaning, or source selection	Show provenance, freshness, gaps, and conflicts
Interpretation	Challenge a conclusion or add missing context	Present relevant evidence and uncertainty
Action selection	Choose, reject, or revise an option	Keep proposals separate from executable actions
Execution	Authorize a defined operation when required	Enforce permissions and recheck current conditions
Outcome review	Stop, correct, or escalate an unexpected result	Verify the outcome, preserve the record, and support recovery

The table does not mean that a human must intervene at every row. Low-impact, well-bounded work may proceed automatically. Higher-impact actions, unclear evidence, policy conflicts, or exceptions may require a person to make or confirm a decision.

The essential requirement is that the workflow’s behavior changes according to the human’s defined role. If a person can only read a summary after the system has acted, that is monitoring or audit. It is not authorization of the earlier action.

Treat approval as a state transition

A governed workflow should represent review as a meaningful state with controlled transitions. For example:

PREPARED
  → VALIDATED
  → NEEDS_REVIEW
  → APPROVED or REJECTED
  → REVALIDATED
  → EXECUTED
  → VERIFIED

The transitions matter as much as the labels:

• A failed validation should block or redirect the action.
• A reviewer’s rejection should result in no execution.
• A changed target, parameter, source record, or policy should invalidate the earlier approval.
• An expired approval should return the action to review or hold it.
• A failed postcondition should trigger a defined recovery or escalation path.

Approval should be bound to the action that will actually occur. At minimum, the system should record the reviewer’s identity and decision, the action and target, the relevant parameters, the evidence or context version, the applicable policy version, and the time and expiry of the approval.

This makes the approval an authorization for a specific operation under specific conditions—not a general signal that “the agent seems fine.”

Route uncertainty; do not hide it behind a score

Uncertainty should influence workflow routing. But a model’s confidence score alone is not a reliable basis for deciding whether a human must review an action.

A more useful design considers several signals:

• conflicting or missing evidence;
• stale source data;
• ambiguity in the request or target;
• disagreement between independent checks;
• a policy exception;
• an action whose effects are difficult to reverse;
• a tool or system state the agent cannot verify.

When such conditions arise, the system can pause, request clarification, present alternatives, or escalate to a person with the right expertise and authority. The goal is not to maximize human intervention. It is to make uncertainty visible and ensure that it leads to an intentional next step.

Give reviewers evidence and usable choices

Human review is only as strong as the information and actions the system gives the reviewer. A concise review surface should show:

• what the agent proposes to do, including the target and parameters;
• the evidence and sources that support the proposal;
• unresolved conflicts, missing information, and freshness;
• which deterministic checks passed or failed;
• the likely consequence of the action and the available recovery path.

The reviewer also needs meaningful options: approve, reject, request more information, escalate, or take control. A default “approve” button beside a large opaque summary encourages a confirmation ritual, not informed oversight.

Human-AI interaction design has long treated system behavior and interface design as engineering concerns. Amershi and colleagues’ guidelines, for example, were developed and evaluated as guidance for designing human-AI interaction.[3] For governed agents, the equivalent principle is practical: give a person enough context and control to perform the assigned role, without flooding them with irrelevant information or routine alerts.

Keep permissions and responsibility explicit

A review decision does not replace the system’s permission checks. The execution boundary should still verify that the requester, agent, and reviewer have the required authority for the specific operation. It should also check that the approved action remains within scope immediately before execution.

Responsibility needs the same clarity. The system should identify who can approve which classes of action, who handles escalations, who can stop a workflow, and who owns recovery after an incident. NIST calls for human-AI roles and responsibilities to be clearly differentiated; an approval event should therefore be part of a defined accountability structure, not a substitute for one.[1]

The audit record should preserve the action proposed, evidence presented, decision made, state changes, execution result, and any later correction. That record supports review and learning. It does not transfer all responsibility to the person who clicked a button.

Test the control, not the button

Approval counts alone say little about whether oversight is effective. A high approval rate could reflect accurate automation; it could also reflect poor evidence, alert fatigue, or a review process that makes rejection difficult.

Evaluate the control through the workflow:

• Are approvals invalidated when the action or evidence changes?
• Can a reviewer reject, pause, or escalate without friction?
• Do uncertain or conflicting cases reach the right person?
• Are actions blocked when authority or preconditions are missing?
• Can the system detect whether an action achieved its intended postcondition?
• Are review workload and intervention outcomes monitored over time?

A useful design test is to remove the approval button from a workflow diagram. If nothing else changes—no state is held, no permission is checked, no person receives evidence, and no action is blocked—then the workflow has a button, but not an effective human control.

Put human authority where the system needs it

Human-in-the-loop is not a requirement to make people approve every model output. It is an architectural commitment to place meaningful human authority at the points where goals, evidence, risk, or actions require it.

CARM’INK’s governed-agentic position follows from this: inference can propose and act within defined boundaries, while human roles, deterministic checks, and controlled transitions govern consequential decisions. The human is not an external fuse attached to an otherwise autonomous workflow. The human is a participant whose authority is represented in the system.

References

1. National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023, especially Appendix C.
2. Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens, “A Model for Types and Levels of Human Interaction with Automation”, IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 30(3), 286–297, 2000.
3. Saleema Amershi et al., “Guidelines for Human-AI Interaction”, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019.