Adaptive human oversight: beyond the HITL checkbox
David Baum 17 min read The comforting fiction#
“Don’t worry, there’s a human in the loop.”
This sentence has become the default reassurance for every AI deployment. It sounds prudent. Responsible. Auditable. And for traditional AI systems (a credit scoring model, a content recommendation engine) it was largely true. A human could review the output, override if necessary and document why.
AI agents break this model in three ways.
First, agents act at a speed and volume that makes per-decision human review physically impossible. A single agent processing customer support tickets may handle hundreds of interactions per hour. A procurement agent may evaluate and execute dozens of purchase orders per day. Inserting a human approval gate on every action doesn’t create oversight. It creates a bottleneck that either kills the automation’s value or, more commonly, trains humans to click “approve” without reading.
Second, agents blur risk boundaries within a single workflow. The same agent that books a conference room (trivial) may also share a file with an external partner (sensitive) and update a CRM record (consequential), all in one session. A binary HITL/no-HITL classification can’t handle this. The oversight model needs to be dynamic, adapting to the risk level of each action within the workflow.
A 2025 JRC study of 1,400 professionals found no tendency for operators to follow fair AI recommendations over unfair ones. Organizational culture and company interests consistently overrode individual oversight judgment. The Responsible AI Foundation’s March 2026 synthesis found humans in governance roles provided correct oversight roughly half the time, driven by learned carelessness, organizational pressure and rubber-stamping under time constraints.
Third, the empirical evidence on human oversight effectiveness is sobering. A 2025 study by the European Commission’s Joint Research Centre, cited in the European Data Protection Supervisor’s TechDispatch #2/2025, investigated how effectively human oversight can override biased AI decisions. The study encompassed field experiments with 1,400 professionals. The finding: there was no tendency for operators to follow fair AI recommendations over unfair ones. Organizational culture and company interests consistently overrode individual oversight judgment.
The Responsible AI Foundation’s March 2026 synthesis of oversight research puts it more bluntly: humans in governance roles provided correct oversight roughly half the time. The primary causes weren’t incompetence. They were “learned carelessness” after long periods of error-free operation, motivation to comply with organizational goals rather than responsible AI principles and the well-documented tendency to rubber-stamp automated decisions under time pressure.
This doesn’t mean human oversight is useless. It means that where you place humans in the workflow matters far more than whether you place them there at all.
Three modes, not two#
The industry conversation frames human oversight as binary: human-in-the-loop or not. That framing is wrong. In practice, there’s a spectrum with three distinct modes, each appropriate for different risk levels.
Human-in-the-loop (HITL) requires the agent to pause and wait for explicit human approval before executing an action. The system presents the proposed action, the human reviews it with full context and the action proceeds only after approval. This is appropriate for high-risk decisions: financial disbursements above a threshold, access grants to restricted data, actions with legal or regulatory consequences and any irreversible operation affecting customers or external parties.
Human-on-the-loop (HOTL) allows the agent to act autonomously while a human monitors outputs and can intervene after the fact. The agent executes within defined guardrails and the human receives alerts when actions approach policy boundaries or when anomalies are detected. This works for medium-risk scenarios where speed matters but mistakes are reversible: CRM record updates, internal communications, routine report generation, standard procurement below a dollar threshold.
Human-out-of-the-loop (HOOTL) grants the agent full autonomy for routine operations, with periodic audit review rather than real-time monitoring. The agent operates independently, logs all actions and a human reviews aggregated logs on a scheduled cadence. This is appropriate for low-risk, high-volume tasks: public-facing FAQ responses with approved answer sets, internal data formatting, meeting summarization and other operations where the blast radius of an error is contained to a single user or workflow.
The critical insight, articulated clearly by Strata Identity’s 2026 guide on practicing HITL, is that agents blur these boundaries within a single workflow. An agent that books a flight (HOOTL), then negotiates a vendor contract (HITL), then sends a follow-up email to the vendor (HOTL) requires different oversight levels at different steps in the same session. A static, one-size-fits-all oversight classification doesn’t work. The oversight model must be dynamic, policy-driven and enforceable through identity controls at the agent level.
Without an enforcement layer (a system that can pause the agent, route the approval request to the right person and block execution until the approval is received) oversight checkpoints are advisory at best. This is why tools like an agent registry that tracks every agent’s risk tier and oversight mode are foundational to making adaptive oversight operational.
The adaptive oversight architecture#
Static oversight fails because agent risk isn’t static. The adaptive oversight model routes each agent action to the appropriate oversight mode in real-time, based on the action’s risk profile. Here’s how it works.
Risk-based routing#
Every agent action is evaluated against the same dimensions used in risk classification (Pillar 2): what data does this action touch? What authority level does it exercise? What’s the blast radius if it fails? The action, not the agent, determines the oversight mode.
A single agent can have actions routed to all three modes within one session. The customer support agent answering a product FAQ (HOOTL) gets routed to HITL the moment it needs to issue a refund above a configured threshold. The routing is governed by policy-as-code (Pillar 3), not by manual tagging.
Confidence-based escalation#
When the agent’s confidence in its own output drops below a threshold or when the underlying model signals uncertainty, the action escalates to a higher oversight mode regardless of its default classification. A summarization task normally handled in HOOTL mode gets escalated to HOTL if the model’s output quality score falls below a configured floor. This prevents the agent from silently producing low-quality work in a mode where no human is watching.
Progressive automation#
New agents don’t start in production with full autonomy. They earn it. The progressive automation model works like this:
Phase 1: Audit mode. Every action requires human approval. The human reviews 100% of outputs. This phase establishes the behavioral baseline and validates that the agent’s actions align with policy.
Phase 2: Supervised mode. The agent acts autonomously for action types that demonstrated accuracy in Phase 1. Actions outside the validated set still require approval. The human reviews a statistical sample of autonomous actions.
Phase 3: Monitored mode. The agent operates autonomously for all validated action types. The human monitors exception alerts and reviews aggregated dashboards. New action types still require approval until validated.
Promotion between phases requires sustained performance: error rates below 2% for 30 consecutive days, no policy violations in the review period and no drift events. Any safety incident triggers immediate demotion to Audit mode pending investigation.
Time-boxed decisions#
Human approval requests have configurable timeouts. If no human responds within the configured window:
- For reversible actions: Auto-approve and flag for post-hoc review. The action proceeds, but it’s marked in the audit log as “auto-approved due to timeout” and added to the next review queue.
- For irreversible actions: Auto-block and alert the escalation chain. The agent notifies the requester that the action is pending and routes the approval to the next reviewer.
This prevents the approval queue from becoming a dead end while preserving safety for high-stakes actions.
Mapping to risk tiers#
The adaptive model connects directly to the four-tier risk classification from Pillar 2:
| Risk tier | Default oversight | Cert expiry | What it means in practice |
|---|---|---|---|
| Tier 1 (Low) | HOOTL | Annual review | Agent runs freely; logs reviewed on a scheduled cadence |
| Tier 2 (Medium) | HOTL | 365 days | Agent runs autonomously; human monitors dashboards and exception alerts |
| Tier 3 (High) | HITL for sensitive actions, HOTL for routine | 180 days | Human approval gates at defined checkpoints; routine actions monitored |
| Tier 4 (Critical) | HITL for all consequential actions | 90 days | Dual-reviewer requirement for irreversible actions; continuous monitoring |
The automation bias problem, and how to design around it#
Even when humans are in the loop, they don’t always provide effective oversight. The reason is automation bias: the documented tendency to over-rely on automated recommendations, especially when the system has a track record of being correct.
The AI Act creates an asymmetric responsibility: providers must enable awareness of automation bias, but the bias itself manifests in the deployer’s context, where the provider has no control over workload, time pressure or organizational incentives.
Johann Laux’s February 2025 paper Automation Bias in the AI Act, published in the European Journal of Risk Regulation, examines the legal implications of this problem.
The EU AI Act is the first regulation to explicitly mention automation bias. Article 14(4)(b) requires that human overseers “remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system.” But as Laux argues, the Act creates an asymmetric responsibility: providers must enable awareness of automation bias, but the bias itself manifests in the deployer’s context, where the provider has no control over workload, time pressure or organizational incentives.
The EDPS TechDispatch study confirms this in practice. The JRC researchers found that operators prioritized company interests over fairness even when AI recommendations were demonstrably biased. The organizational context (incentive structures, performance metrics, time pressure) shaped oversight behavior more than individual training or awareness.
Design countermeasures#
You can’t train away automation bias, but you can design around it:
Research from the University of Duisburg-Essen (Kraemer et al., November 2025) proposes two metrics enterprise teams should track: Relative Positive AI Reliance (RAIR) measures how often humans correctly adopt accurate AI suggestions. Relative Positive Self-Reliance (RSR) measures how often humans correctly reject inaccurate ones. A healthy oversight system shows high scores on both. A system suffering from automation bias shows high RAIR but low RSR: humans accept everything.
Failure exposure during onboarding. The EDPS recommends that oversight training include “a sample of cases of system failures.” Direct exposure to incorrect AI outputs during training reduces (though doesn’t eliminate) automation bias in subsequent operation. Don’t train overseers only on happy paths.
Friction at the right moments. For Tier 3+ actions, require the human to articulate why they’re approving, not just click a button. A free-text rationale field, even if brief, forces the cognitive engagement that a one-click approval doesn’t. Log the rationale as audit evidence.
Periodic challenge prompts. Inject synthetic “test” actions that are deliberately incorrect. If the human approves them, flag the approval as a training event and route the action to a secondary reviewer. This combats the “learned carelessness” that develops after long error-free periods.
Reviewer rotation. Don’t assign the same human to oversee the same agent indefinitely. Rotate reviewers on a 30-day cadence. Fresh eyes catch patterns that habituated reviewers miss.
The aviation parallel. Crew Resource Management made aviation oversight a trained discipline. Pilots practice judgment under pressure in simulators before they’re trusted with passengers. Enterprise AI oversight needs the same rigor: not just training on what the agent does, but structured practice on when to escalate, when to override and when to shut down.
EU AI Act Article 14: what compliance requires#
Article 14 of the EU AI Act establishes six capabilities that high-risk AI systems must provide to human overseers. With full enforcement of high-risk system rules beginning August 2, 2026, these move from aspirational to mandatory.
The six capabilities, paraphrased from Article 14(4):
- Understand capabilities and limitations. The overseer must be able to properly understand what the system can and cannot do and monitor its operation for anomalies
- Awareness of automation bias. The overseer must remain aware of the tendency to over-rely on AI outputs
- Correct interpretation. The overseer must be able to correctly interpret the system’s output, using available tools and methods
- Override authority. The overseer must be able to disregard, override or reverse the system’s output in any particular situation
- Ability to not use. The overseer must be able to decide not to use the system at all
- Ability to halt. The overseer must be able to interrupt or stop the system’s operation
The regulation defines what overseers must be able to do, but says nothing about how to make those capabilities useful in practice. That gap between legal text and operational reality is where most compliance efforts stall and where legal scholarship offers the sharpest analysis.
Capability 2 is particularly telling. The Act requires that overseers “remain aware of the tendency to over-rely on AI outputs,” a behavioral mandate written into law. But awareness of automation bias doesn’t prevent it. The research is clear on that point. Structural safeguards (time-boxed decisions, forced justification, auto-block after deadlines) are the only interventions that reliably reduce rubber-stamping.
The obligations fall primarily on providers to create the technical conditions for oversight, not on deployers to perform it. The Act prescribes oversight capabilities (the infrastructure) rather than oversight performance (the practice).
Melanie Fink’s analysis of Article 14 (SSRN, February 2025) highlights a critical nuance: the obligations fall primarily on providers to create the technical conditions for oversight, not on deployers to perform it. The Act prescribes oversight “capabilities” (the infrastructure) rather than oversight “performance” (the practice). This means compliance requires demonstrable technical mechanisms (pause points, override controls, halt switches), not just documentation that a human was nominally assigned.
The adaptive oversight architecture maps directly to these requirements. Risk-based routing provides the pause points (capability 1, 4). Progressive automation creates the learning period for understanding capabilities (capability 1). Confidence-based escalation ensures the system surfaces uncertainty (capability 3). Time-boxed decisions with auto-block preserve halt authority (capability 6). And the entire framework is auditable: every routing decision, every approval, every override is logged with actor, timestamp and rationale.
Article 14(3) states that oversight measures “shall be commensurate with the risks, level of autonomy and context of use.” But the Act provides no definition of what “commensurate” means in practice. The adaptive oversight model fills this gap: risk-based routing ensures that oversight intensity scales with the risk of each specific action, not just the risk classification of the agent as a whole. This is the strongest defensible position for an auditor asking “how did you determine the appropriate level of oversight?”
The oversight decision tree#
For each agent action, run through this decision sequence:
Step 1: Is the action reversible? If no, route to HITL regardless of other factors. Irreversible actions (sending external communications, executing financial transactions, granting access to restricted data) always require human approval.
Step 2: What data does the action touch? If Restricted (PII, PHI, payment data), route to HITL. If Confidential, HOTL minimum. If Internal or Public, proceed to Step 3.
Step 3: What’s the blast radius? If External (customer-facing, partner-facing), route to HITL or HOTL depending on reversibility. If Organization-wide, HOTL. If Team or Individual, HOOTL.
Step 4: What’s the agent’s track record? If the agent is in Phase 1 (Audit mode), HITL for everything. If Phase 2 (Supervised), follow the routing from Steps 1-3 but sample autonomous actions for review. If Phase 3 (Monitored), follow the routing and review exceptions only.
Three worked examples#
Expense approval agent (Tier 2, Medium risk): Routine expense reports under $500 from known employees get HOOTL with weekly audit review. Expenses over $500 or from contractors get HOTL with real-time dashboard monitoring. Expenses over $5,000 or flagged for policy anomaly get HITL with mandatory approval before disbursement.
Customer support agent (Tier 3, High risk): Standard FAQ responses using approved answer set get HOOTL. Responses involving account-specific data or actions (refunds, credits, escalations) get HOTL with exception alerts. Any response that modifies customer billing, closes an account or involves a complaint gets HITL with human approval before the action is executed.
Clinical triage agent (Tier 4, Critical risk): All patient-facing routing decisions get HITL with clinician approval. Internal workflow steps (scheduling, record retrieval) get HOTL with anomaly detection. No actions in HOOTL mode. Every action is either actively monitored or explicitly approved.
Oversight is a system, not a checkbox#
The question isn’t whether to have human oversight. The EU AI Act mandates it for high-risk systems. The question is how to make it effective: how to place humans where they add judgment rather than rubber stamps, how to route decisions to the right oversight mode based on actual risk and how to design systems that resist the automation bias that makes oversight decay over time.
The adaptive model provides the architecture: risk-based routing, progressive automation, confidence-based escalation, time-boxed decisions and design countermeasures for automation bias. Connect it to your risk classification (Pillar 2), enforce it through policy-as-code (Pillar 3) and log everything for your audit trail (Pillar 7).
Start with your Tier 3 and Tier 4 agents, the ones where oversight failures have real consequences. Map their action types. Define the routing rules. Train the reviewers on failure cases, not just happy paths. And track your RAIR and RSR metrics to know whether your oversight is working or just checking a box. A compliance certification pipeline that ties oversight mode to risk tier and auto-expires certifications when oversight requirements are unmet turns this from a governance diagram into an enforceable system.
Sources and further reading#
More in governance
The lethal trifecta: governing the three capabilities you can't remove
Five days in January 2026, four AI productivity tools shipped indirect prompt injection vulnerabilities. Same pattern in every case. Simon Willison named it in June 2025: private data access plus untrusted content plus external communication equals data exfiltration. You can't remove any leg without breaking the agent. Governance has to shift from prevention to containment.
David Baum The confused deputy problem: lessons from Meta's AI agent Sev-1
On March 18 2026, a Meta AI agent exposed restricted company and user data for two hours. The credentials were valid. The governance was not.
David Baum Guardian agents: when AI governs AI
Humans can't watch every agent, every action, every second. Guardian agents (dedicated policy engines that monitor, audit and enforce governance on other agents at runtime) resolve the tension between machine-speed execution and meaningful oversight.
David Baum