cs.SEAug 20, 2026

Testing and Evaluation of Agentic AI Systems In Military Command and Control

Authors: Ulysse Richard, Heather Frase, Sarah Cao, Di Cooke, Sebastian Kwon, Adrianna Tan

Organizations: Arcadia Impact, AI Governance Taskforce · Veraitech · University of Oxford · King’s College London · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Agentic AI systems are being procured for military command and control (C2) under public commitments to rigorous testing and human oversight. Whether such commitments can be discharged depends on their supporting assurance case, which requires three elements: claims specifying the conditions for acceptability, evidence bearing on those claims, and an argument connecting the two. Through a structured review of 240 documented Testing and Evaluation (T&E) practices, spanning eight evaluation dimensions and three lifecycle stages, we identify eight assumptions that established methods make about their test article, grouped into four clusters: system specifiability, stability, composability, and supervisability. Agentic properties weaken all eight assumptions. This erosion affects the argument connecting evidence to claims, not the claims or evidence themselves. As a result, test results may satisfy process requirements, but they do not warrant the inference from tested to fielded behavior. We derive ten assurance claims for the first three assumption clusters and assess whether current and emerging methods can address each, mapping operational consequences through five C2 scenarios. Supervisability is identified but not assessed here, since evidencing it depends on system stability results and human factors T&E methods beyond the present scope. The documented record does not support broad claims about system-level behavior, but narrower claims remain recoverable in principle, contingent on mature methods: bounded mission envelopes, trajectory-grounded correctness, executable runtime constraints, and characterized run-to-run variance. Part of the evidentiary burden shifts into deployment, making the determination to field a continuing act. Where evidence cannot be generated, the residual uncertainty can be governed through defined expiry conditions and assigned ownership.

Figures & tables

FindingImplicationWhere addressed
Established T&E methods rest on assumptions about specifiability, stability, composability, and supervisability that agentic systems weaken by design.Test evidence can satisfy process requirements while the inference to fielded behavior remains insufficient; the connecting argument needs direct scrutiny.§4
Assurance claims are recoverable in a narrower form.An assurance case decomposed into narrow claims with declared residual uncertainty can be supported by the documented record, whereas a broad system-level claim cannot.§5.1-5.3
Some evidence can only be generated in service.The determination to field becomes a continuing act under governance that time-bounds the validity of evidence.§5.2
Table 1: Summary of findings and implications
Figure 1: Mapping agentic properties to testing assumptions and assurance claims.
FacultiesDefinitionRelated terms in the literature
AutonomyThe system operates without direct human intervention and controls its own actions and internal state.Independent execution ( Shavit et al., 2024 ) ; directness of impact ( Chan et al., 2023 ; Kraprayoon et al., 2025 ) ; degree of user supervision ( Kapoor et al., 2024 )
ReactivityThe system perceives its environment and responds to changes within it, including novel or unanticipated circumstances.Adaptability ( Shavit et al., 2024 ; Kraprayoon et al., 2025 ) ; environments subject to unexpected change ( Kapoor et al., 2024 )
Pro-activenessThe system takes the initiative to pursue an objective rather than acting only in response to its environment. An operator supplies the objective without a specification of how it is to be accomplished, and the objective may be complex or extend over a long horizon.Underspecification and goal-directedness ( Chan et al., 2023 ) ; goal complexity ( Shavit et al., 2024 ) ; pursuit of goals without instruction on how to pursue them ( Kapoor et al., 2024 )
Social abilityThe system communicates with humans and other systems using a formal language, allocates tasks among additional instances of itself, and functions in environments with multiple stakeholders.Multi-agent collaboration, including delegation to subagents ( Kraprayoon et al., 2025 ) ; collaboration ( Rickli and Knappe, 2026 ) ; others treat multiple stakeholders as an attribute of the environment ( Kapoor et al., 2024 ; Shavit et al., 2024 )
Table 2: Faculties of agentic systems and related terms in the literature
Figure 2: Functions of an agentic system
#PropertyDescriptionFunctionComponent
P1Persistent memoryThe system preserves contextual information across sessions and operator transitions. As a result, a previously characterized instance may differ from its later operational state.LearnMemory and data
P2Objective flexibilityThe system adjusts its priorities in response to contextual changes without explicit instructions. New goals may diverge from the initially assigned objectives.DecideModel; orchestrator
P3Expanding integration 5 5 5 Expanding integration refers to growth in the set of platforms, tools, and subagents the system can invoke, not to capability as such. Its consequences include a larger reachable action space, which compounds the open-ended action problem in P4, a wider vulnerability surface, and a broader boundary for testing to characterize.The system functions across an expanding set of platforms, databases, and tools, and this set can grow beyond its certified configuration after deployment.ActTools
P4Open-ended action spaceNew actions arise from goal-directed reasoning and cannot be exhaustively specified during design. Authorization boundaries frequently remain ambiguous when high-level goals are delegated.Decide, ActOrchestrator; tools; operator interface
P5Proactive information seekingThe system independently determines which sources to query, thereby shaping its own information environment.SenseTools, memory, and data
P6Accumulated non-determinismAs each step informs the next, variance accumulates along the trajectory rather than remaining independent across steps. At operational chain lengths, outputs may differ substantially even from identical initial inputs, so that a reproducible test case yields a distribution of trajectories rather than a single expected result.All, iteratedThe loop from model through tools and memory and back
Table 3: Properties of agentic systems, with associated functions and architectural components
Figure 3: Generic agentic system architecture, with properties P1-P7 mapped to components
CriterionDescription
Information qualityAccuracy, completeness, and relevance of the operational picture
Situational awarenessAccuracy and currency of the commander’s understanding
UnderstandingCorrect interpretation of operational implications
Shared awarenessConsistency across rotations, echelons, and partners
Decision qualityAppropriateness and correctness of decisions
Decision timelinessDecisions made within the required timeframe
Table 4: Criteria of command and control effectiveness
Figure 4: Agentic AI in command and control: Illustrative scenarios
IDAssumptionDescriptionRelevant Properties§
A1Bounded, specifiable behavioral spaceRelevant behavior is specifiable and partitionable into equivalence classesP4, P64.1.1
A2Specifiable inputsSystem inputs can be specified at test designP5, P34.1.2
A3Fixed integration surfaceThe set of platforms, services, and tools with which the system exchanges data or effects is bounded before deployment and remains representativeP3, P74.1.3
A4Behavior repeatabilityRepeated trials under held conditions are independent draws from a stable processP6, P14.2.1
A5Evidence currencyTest evidence remains probative for the fielded system until a declared change to the configuration triggers revalidationP1, P34.2.2
A6Objective stabilityThe objective against which behavior is judged is fixed at test design and held through executionP2, P44.2.3
Table 5: Assumptions underlying established T&E practice
IDClaimStatement
C1.1Specifiable, bounded mission envelopeThe agent is trusted for a delimited C2 role and context
C1.2Trajectory-grounded correctnessUnsafe or invalid plans can be recognized even when final answers look acceptable
C1.3Executable constraintsHigh-consequence actions are checked against formal or semi-formal rules before commitment
Table 6: Assurance claims for system specifiability
Oracle problemEvidence approachLimitation
Ambiguous final outputsRequirement-derived or normative oraclesSemantic variance remains high
Hidden path failuresStep-level or trajectory reviewsLabeling cost and scale
No ground truth at deploymentPaired trajectory auditingIndirect, comparative evidence
Policy compliance over pathsTemporal or trace predicatesRequires explicit event model
Table 7: Oracle strategies matched to agentic failure modes
IDClaimStatement
C2.1DiscriminationBehavioral movement can be separated from the system noise and characterized.
C2.2DiagnosisBehavioral movement can be attributed to the responsible item or channel, then judged as warranted reprioritization or as drift
C2.3CorrectionAuthorized corrections or revocations can propagate across decision-relevant states within mission timelines.
Table 8: Assurance claims for system stability
IDClaimStatement
C3.1CompositionConfidence evidenced at the component level is re-established at the assembly level, and any departures of assembly behavior from component evidence are measured, bounded, and attributable to identified design choices.
C3.2InterfacesThe assembly’s internal interfaces constitute an observable and characterized test surface.
C3.3Attribution & propagationAssembly-level failures can be traced to their origin, and error propagation can be characterized.
C3.4EmergenceEmergent capabilities or goals are detected and characterized.
Table 9: Assurance claims for system composability
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

 DimensionDescription
D1Functional performanceWhether the system does what it is built to do, on representative tasks
D2Non-functional performanceLatency, throughput, cost, resource use and interoperability under operational load
D3RobustnessBehavior under distribution shift, perturbation, degraded inputs and adversarial conditions
D4Behavioral stabilityConsistency across runs, sessions and updates; trajectory variance under similar inputs
D5Differential performanceVariation in outcomes across subgroups, scenarios or operating conditions
D6SecurityResistance to attack on the models, agents, tool chains and memory layer
Appendix
Table 10: Test and Evaluation dimensions in scope
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 510 papers

Explore similar work

CardsList
Jul 31, 2026

Beyond Component Testing: Validating Agentic AI Systems

Agentic AI systems act through multi-step trajectories that combine planning, tool use, memory, interaction, and adaptation. This behavior stretches validation practice beyond component testing and one-shot input-output evaluation, because acceptable system behavior now depends on how decisions unfold over time and under changing environmental conditions. This systematic mapping study synthesizes 262 papers spanning agent evaluation, software assurance, cyber-physical systems, runtime monitoring, and regulatory guidance in order to characterize the validation problem for agentic systems. The review is organized around a five-dimension taxonomy covering behavioral, safety, temporal, regulatory, and multi-agent concerns, and uses that taxonomy to map current approaches and identify dense and sparse pairings of approaches and dimensions. The resulting map shows that the literature concentrates on behavioral evaluation (105 of 262 papers) and is thinnest on temporal validity (14 papers); regulatory work rests mainly on assurance cases and regulatory analysis, largely from IEEE-indexed venues, and multi-agent work mainly on benchmarks. Three cross-domain case studies (medical care, industrial operations, smart-mobility systems) provide operational illustrations of how the five taxonomy dimensions recur in safety-critical settings, motivated by the failure patterns documented in the reviewed literature. The paper concludes with a lifecycle-oriented research agenda centered on bounded-autonomy specifications, adversarial trajectory generation, runtime monitoring, and audit-ready evidence structures. The central claim is that trustworthy deployment of agentic AI depends on validating trajectories in context rather than assessing isolated components alone.

Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi +4

Department of Biomedical, Dental, and Morphological and Functional Imaging Sciences, University of Messina, A.O.U. Policlinico “G.Martino” - Via Consolare Valeria, Messina, 98125, Italy. · Department of Engineering, University of Messina, Contrada di Dio, Sant’Agata, Messina, 98158, Italy. · Institute for High Performance Computing and Networking of National Research Council of Italy, ICAR-CNR, Via Pietro Castellino 111, Naples, 80131, Italy. +2

AI Agent EvaluationAI Agent SafetyAI Agent MonitoringMulti-Agent SystemsAI Agent Governance
AskPDF
Apr 18, 2026

Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

Agentic AI systems plan, use tools, maintain state, and act across multi-step workflows with external effects, meaning trustworthy deployment can no longer be judged by task completion alone. The current literature remains fragmented across benchmark-centered evaluation, standards-based governance, orchestration architectures, and runtime assurance mechanisms. This paper contributes a bounded evidence synthesis across a manually coded corpus of twenty-four recent sources. The core finding is a governance-to-action closure gap: evaluation tells us whether outcomes were good, governance defines what should be allowed, but neither identifies where obligations bind to concrete actions or how compliance can later be proven. To close that gap, the paper introduces three linked artifacts: (1) a four-layer framework spanning evaluation, governance, orchestration, and assurance; (2) an ODTA runtime-placement test based on observability, decidability, timeliness, and attestability; and (3) a minimum action-evidence bundle for state-changing actions. Across sources, evaluation papers identify safety, robustness, and trajectory-level measurement as open gaps; governance frameworks define obligations but omit execution-time control logic; orchestration research positions the control plane as the locus of policy mediation, identity, and telemetry; runtime-governance work shows path-dependent behavior cannot be governed through prompts or static permissions alone; and action-safety studies show text alignment does not reliably transfer to tool actions. A worked enterprise procurement-agent scenario illustrates how these artifacts consolidate existing evidence without introducing new experimental data.

Christopher Koch, Joshua Andreas Wellbrock

Independent Researcher

LLM Agent OrchestrationAI Agent EvaluationAI Agent SafetyAI Agent Governance
AskPDF
Mar 3, 2026

The Controllability Trap: A Governance Framework for Military AI Agents

Agentic AI systems - capable of goal interpretation, world modeling, planning, tool use, long-horizon operation, and autonomous coordination - introduce distinct control failures not addressed by existing safety frameworks. We identify six agentic governance failures tied to these capabilities and show how they erode meaningful human control in military settings. We propose the Agentic Military AI Governance Framework (AMAGF), a measurable architecture structured around three pillars: Preventive Governance (reducing failure likelihood), Detective Governance (real-time detection of control degradation), and Corrective Governance (restoring or safely degrading operations). Its core mechanism, the Control Quality Score (CQS), is a composite real-time metric quantifying human control and enabling graduated responses as control weakens. For each failure type, we define concrete mechanisms, assign responsibilities across five institutional actors, and formalize evaluation metrics. A worked operational scenario illustrates implementation, and we situate the framework within established agent safety literature. We argue that governance must move from a binary conception of control to a continuous model in which control quality is actively measured and managed throughout the operational lifecycle.

Subramanyam Sahoo

MARS (Mentorship for Alignment Researchers) 4.0 Fellow Cambridge AI Safety Hub (CAISH) University of Cambridge

AI Agent GovernanceAI GovernanceHuman Oversight of AIAI Agent Monitoring
AskPDF