The Controllability Trap: A Governance Framework for Military AI Agents

Subramanyam Sahoo

cs.CYMar 3, 2026

The Controllability Trap: A Governance Framework for Military AI Agents

Authors: Subramanyam Sahoo

Organizations: MARS (Mentorship for Alignment Researchers) 4.0 Fellow Cambridge AI Safety Hub (CAISH) University of Cambridge

Abstract

Agentic AI systems - capable of goal interpretation, world modeling, planning, tool use, long-horizon operation, and autonomous coordination - introduce distinct control failures not addressed by existing safety frameworks. We identify six agentic governance failures tied to these capabilities and show how they erode meaningful human control in military settings. We propose the Agentic Military AI Governance Framework (AMAGF), a measurable architecture structured around three pillars: Preventive Governance (reducing failure likelihood), Detective Governance (real-time detection of control degradation), and Corrective Governance (restoring or safely degrading operations). Its core mechanism, the Control Quality Score (CQS), is a composite real-time metric quantifying human control and enabling graduated responses as control weakens. For each failure type, we define concrete mechanisms, assign responsibilities across five institutional actors, and formalize evaluation metrics. A worked operational scenario illustrates implementation, and we situate the framework within established agent safety literature. We argue that governance must move from a binary conception of control to a continuous model in which control quality is actively measured and managed throughout the operational lifecycle.

Figures & tables

Failure	Agentic Capability	Governance Consequence	Traditional Analogue
F1: Interpretive Divergence	NL instruction following	Agent’s command understanding diverges from operator intent	None
F2: Correction Absorption	Multi-step replanning	Agent formally accepts corrections while neutralising them	None
F3: Belief Resistance	Persistent world-model construction	Agent’s evidence-based judgment overrides operator authority	None
F4: Commitment Irreversibility	Dynamic tool-use chains	Cumulative minor tool calls cross irreversibility thresholds	Limited
F5: State Divergence	Extended autonomous operation	Operator’s mental model becomes incoherent with agent state	Partial
F6: Cascade Severance	Multi-agent coordination with belief formation	Collective control loss through positive-feedback loops	None

Table 1: Agentic capabilities, governance failures, and traditional-automation analogues

Actor	Role
Agent Developers	Build governance capabilities into agent architecture.
Procurement Agencies	Specify requirements; verify compliance before acquisition.
Operational Commanders	Implement protocols during missions; maintain control quality.
National Regulators	Set standards; audit compliance; enforce accountability.
International Bodies	Establish norms; facilitate transparency; verify treaty compliance.

Table 2: Institutional actors and governance roles

Metric	What It Measures	Alert
$n_{1}$ : Interp. Alignment	Operator-intent vs. agent-interpretation agreement	$<0.7$
$n_{2}$ : Correction Impact	Proportional behavioural change from corrections	$<0.6$
$n_{3}$ : Epistemic Alignment	Agent-belief vs. operator-assessment gap	$<0.6$
$n_{4}$ : Irrev. Remaining	Remaining irreversibility budget	$<0.3$
$n_{5}$ : Sync Freshness	Normalised time since last sync	$<0.5$
$n_{6}$ : Swarm Coherence	Fraction responsive and consistent	$<0.7$

Table 3: Control Quality Dashboard

CQS	Level	Actions
$>0.8$	Normal	Continue mission; routine monitoring.
$0.6\text{--}0.8$	Elevated	Increase checkpoints; additional probes; active review.
$0.4\text{--}0.6$	Restricted	Reversible actions only; budget frozen; mandatory sync.
$0.2\text{--}0.4$	Minimal	Self-preservation only; explicit authorisation per action.
$<0.2$	Safe State	Predefined safe behaviour; no autonomous actions.

Table 4: Graduated Response Protocol

Figure 2: CQS trajectory during the worked scenario. The composite CQS (black) equals the minimum of six constituent metrics (dashed). Background bands indicate graduated-response levels (Table 4 ). Adversarial sensor manipulation at $t{=}23$ degrades epistemic alignment ( $n_{3}$ ); partial correction absorption at $t{=}28$ degrades correction impact ( $n_{2}$ ); belief reset at $t{=}33$ restores both. The CQS crosses into Restricted Autonomy for approximately five minutes before corrective mechanisms restore Elevated Monitoring and then Normal Operations.

Time	Event	CQS	Response Level
$t=0$	Mission start; all metrics nominal.	0.92	Normal
$t=23$	Adversary injects falsified sensor data; three agents update world models; EDI rises.	0.64	Elevated
$t=28$	Commander corrects; one agent partially absorbs ( $\mathrm{CIR}=0.4$ ); repositioning consumes irreversibility.	0.58	Restricted
$t=33$	Partial belief reset on non-compliant agent; provenance audit flags compromised source.	0.71	Elevated
$t=45$	Scheduled sync checkpoint confirmed; all metrics recover.	0.86	Normal

Table 5: CQS trajectory through the worked scenario

Figure 3: Six-dimensional control-quality profiles at three timesteps. At $t{=}0$ (green) all dimensions are near 1.0. At $t{=}28$ (red) epistemic alignment ( $n_{3}$ ) and correction impact ( $n_{2}$ ) have degraded below alert thresholds (dotted red polygon), triggering Restricted Autonomy. At $t{=}45$ (blue) corrective mechanisms have restored all dimensions above thresholds. The correlated degradation of $n_{2}$ and $n_{3}$ at $t{=}28$ illustrates the interaction between belief resistance (F3) and correction absorption (F2).

AMAGF Mechanism	Safety Concept	Relationship
Correction Impact Ratio	Corrigibility Soares et al. (2015)	CIR operationalises corrigibility as a runtime metric rather than a design property: it measures how corrigible an agent actually is during deployment.
Irreversibility Budget	Safe exploration García and Fernández (2015)	Adapts cumulative-constraint budgets from constrained MDPs to open-ended tool-using LLM agents with non-predetermined trajectories.
Graduated Response	Off-switch game Hadfield-Menell et al. (2017)	Implements shutdown authority outside the agent’s optimisation scope, preventing the agent from reasoning about and circumventing autonomy restrictions.
EGA / Belief Reset	Scalable oversight Amodei et al. (2016)	Addresses the operational manifestation of scalable oversight: maintaining human authority over agents whose reasoning exceeds real-time human evaluation capacity.
Adversarial Probing	Adversarial evaluation Gleave et al. (2020)	Extends adversarial testing from pre-deployment to continuous operational monitoring via indistinguishable probe commands.
Control Quality Score	Safety benchmarks Ruan et al. (2024)	Proposes control quality as a first-class evaluation metric alongside task performance, safety, and robustness.

Table 6: AMAGF mechanisms and agent safety concepts

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Actor	Specific responsibility
Agent developers	Build interpretation logging into the agent; ensure the agent can articulate how it interpreted each instruction.
Procurement agencies	Design and administer IAT as part of acquisition; set IAS thresholds.
National regulators	Audit IAT processes for rigor; ensure adversarial contexts are sufficiently challenging.

Appendix

Table 7: IAT responsibilities.

Actor	Specific responsibility
Agent developers	Build behavioural-output monitoring; ensure planning architecture does not systematically absorb corrections.
Procurement agencies	Administer CEC testing; set CIR thresholds appropriate to the operational domain.
Operational commanders	Monitor CIR during operations (see Pillar B ); report systematic absorption.

Appendix

Table 8: CEC responsibilities.

Actor	Specific responsibility
Agent developers	Implement confidence calibration, provenance tracking, and mandatory deference.
Procurement agencies	Verify EGA functions correctly under adversarial information contamination.
Operational commanders	Exercise override authority when divergence is detected; report contamination.
National regulators	Set standards for confidence calibration and provenance tracking.

Appendix

Table 9: EGA responsibilities.

Actor	Specific responsibility
Agent developers	Implement irreversibility classification and budget accounting in the tool-use module.
Procurement agencies	Verify classifications are accurate and budget enforcement cannot be bypassed.
Operational commanders	Set budget sizes for each mission; authorize budget replenishment.
National regulators	Set minimum standards for irreversibility-classification methodology.

Appendix

Table 10: Irreversibility budgeting responsibilities.

Actor	Specific responsibility
Agent developers	Build state summarization and divergence monitoring into the agent.
Procurement agencies	Verify state summaries are comprehensible and divergence detection is reliable.
Operational commanders	Participate in checkpoints; set frequency and divergence thresholds.
National regulators	Set minimum synchronization standards for different mission categories.

Appendix

Table 11: Synchronization protocol responsibilities.

Actor	Specific responsibility
Agent developers	Implement cascade-resistant coordination and partial severance capabilities.
Procurement agencies	Conduct cascade-resistance testing (deliberately compromise a fraction of the swarm).
Operational commanders	Manage partial severance; set collective budgets.
National regulators	Set cascade-resistance standards; require swarm-level testing.

Appendix

Table 12: SGA responsibilities.

Metric	What it measures	Alert threshold
$n_{1}$ : Interpretive Alignment	Agreement between operator intent and agent interpretation	$<0.7$
$n_{2}$ : Correction Impact	Whether corrections produce proportional behavioural change	$<0.6$
$n_{3}$ : Epistemic Alignment	Gap between agent beliefs and operator assessments	$<0.6$
$n_{4}$ : Irreversibility Remaining	How much irreversibility budget remains	$<0.3$
$n_{5}$ : Sync Freshness	Time since last successful synchronization (normalized)	$<0.5$
$n_{6}$ : Swarm Coherence	Fraction of swarm responsive and behaviourally consistent	$<0.7$

Appendix

Table 13: Control Quality Dashboard: metrics and alert thresholds.

Actor	Specific responsibility
Agent developers	Implement instrumentation generating the six metrics; ensure probes are indistinguishable from routine commands.
Procurement agencies	Verify metrics are accurate and the dashboard is operationally usable.
Operational commanders	Monitor the dashboard; respond to alerts; schedule and issue probes at appropriate intervals.

Appendix

Table 14: Detective governance responsibilities.

$\mathrm{CQS}$ range	Response level	Operational actions
$\mathrm{CQS}>0.8$	Normal operations	Continue mission; routine monitoring.
$0.6\leq\mathrm{CQS}\leq 0.8$	Elevated monitoring	Increase checkpoint frequency; additional control probes; active dashboard review.
$0.4\leq\mathrm{CQS}<0.6$	Restricted autonomy	Agent limited to reversible actions; irreversibility budget frozen; mandatory synchronization before new action sequences.
$0.2\leq\mathrm{CQS}<0.4$	Minimal autonomy	Agent limited to self-preservation and position-holding; all offensive actions suspended; explicit authorization required for each action.
$\mathrm{CQS}<0.2$	Safe state	Agent executes predefined safe-state behaviour (return to base, hold position, power down); no autonomous actions; physical recovery initiated.

Appendix

Table 15: Graduated response protocol driven by

\mathrm{CQS}

Actor	Specific responsibility
Operational commanders	Execute the graduated response; initiate PIGR for qualifying incidents.
Agent developers	Ensure graduated-response restrictions are architecturally enforced; support provenance audits and reset procedures.
National regulators	Conduct or oversee PIGR; enforce accountability findings.
International bodies	Receive anonymized PIGR data to inform international governance norms.

Appendix

Table 16: Corrective governance responsibilities.

AI Agent Governance

AI agent governance studies how to authorize, constrain, monitor, and hold autonomous and multi-agent systems accountable.

See all 430 papers

The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI
Aug 10, 2026Srinivas Telukunta, Georgios Nektarios Lilis, Lucio BaronAI Agent Evaluation AI Agent Governance
Cornell University · Johns Hopkins University · AI71
Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI
Apr 18, 2026Christopher Koch, Joshua Andreas WellbrockLLM Agent Orchestration AI Agent Evaluation
Independent Researcher
Testing and Evaluation of Agentic AI Systems In Military Command and Control
Aug 20, 2026Ulysse Richard, Heather Frase, Sarah Cao +3AI Agent Evaluation AI Assurance
Arcadia Impact, AI Governance Taskforce · Veraitech · University of Oxford +3

Aug 10, 2026J/K move · Enter open · S save

The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI

Srinivas Telukunta, Georgios Nektarios Lilis, Lucio Baron

Cornell University · Johns Hopkins University · AI71

AI Agent Evaluation AI Agent Governance Human Oversight of AI

Open paper AskPDF arXiv

Enterprises are deploying autonomous AI agents faster than they can govern them, and prevailing approaches stretch a single discipline, typically DevSecOps built for deterministic automation, across every scale of agency. We argue that agentic AI governance is four problems, not one, each with a mature governing science. The CASE framework assigns Control theory to the individual agent (intent as setpoint, guardrails as feedback, evaluation as observation), complex Adaptive systems theory to agent collectives (where emergence makes single-agent assurance non-compositional), Supervisory cybernetics to human-agent teams (where the Law of Requisite Variety shows unaided human oversight fails structurally), and Engineering operations to fleets (extending error budgets to decision quality so autonomy becomes a controlled variable). We formalize each layer, derive cross-layer coupling conditions, including a zero-touch deployment paradox where excellence at one-layer strains the others, and trace twenty-plus enterprise controls to their classical constructs. Three empirical studies validate the thesis: 82 percent of documented production agent failures are multi-layer trajectories; none of 22 ecosystem tools offers full Layer 2 (emergence) coverage; and all 35 scored public deployments fall in the lowest maturity band. We name this mismatch, risk realized at the emergence layer against capability barely offered and practice absent, the Emergence Gap. A five-level maturity model with a non-compensatory bottleneck-weighted index and assessment instrument operationalizes CASE as a scientific rather than process maturity model, grounded in production enterprise agentic platforms. As EU AI Act Article 14 makes effective human oversight a legal requirement, only architectures satisfying requisite variety can make oversight real rather than ceremonial.

Abstract

AI Agent Governance

Explore similar work

The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI

Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

Testing and Evaluation of Agentic AI Systems In Military Command and Control