Model-Free Runtime Measurement of Task-Relevant Information
Organizations: Faculty of Engineering and Applied Science, Cranfield University, Cranfield, MK43 0AL, UK.
Abstract
Autonomous systems increasingly monitor physical kinematics, environmental states and task completion outcomes. While existing work provides methods to plan information seeking actions and interpret acquired observations, current approaches do not monitor whether the live process of information gathering is functioning properly. Consequently, such systems can remain operationally active without resolving outstanding mission requirements. To address this limitation, we introduce Acquisition-State Telemetry (AST), a model free architecture that evaluates ongoing information acquisition from observable evidence relative to declared task requirements. Specifically, a domain interface maps raw observations into task resolution coordinates and normalized comparative representations, which are processed by a fixed causal monitor. This monitor emits two continuous real time signals that quantify (i) prospective progress across unresolved requirements and (ii) informational friction generated when recurrent evidence yields negligible informational gain. We evaluated a single, unadjusted monitor configuration across eight diverse domains including search and rescue dialogue, spatial grid exploration, hierarchical capability acquisition, continuous robotic manipulation, and probabilistic intent estimation. We show that across these domains the friction channel distinguished unproductive stagnation from purposeful exploration prior to task completion, while remaining muted during post completion activity. Furthermore, in closed loop reinforcement learning experiments under early termination constraints, we demonstrate that penalizing informational friction allowed agents to sustain high mission success, whereas unpenalized policies and baselines with matched operational costs failed to complete required tasks.
Figures & tables
| Cond. | Policy variant | Categories resolved | Total knowledge | Full-res. rate | Mean friction ( ) | Stall burden (%) |
| A | Baseline | 7.67 [7.00, 8.00] | 0.840 [0.766, 0.878] | 0.93 [0.80, 1.00] | 0.009 [0.002, 0.022] | 0.00 [0.00, 0.00] |
| Coupled-Blind | 8.00 [8.00, 8.00] | 0.874 [0.872, 0.877] | 1.00 [1.00, 1.00] | 0.004 [0.003, 0.005] | 0.00 [0.00, 0.00] | |
| AST-Obs | 7.80 [7.40, 8.00] | 0.854 [0.808, 0.879] | 0.97 [0.90, 1.00] | 0.007 [0.003, 0.016] | 0.00 [0.00, 0.00] | |
| Full-AST | 8.00 [8.00, 8.00] | 0.887 [0.885, 0.890] | 1.00 [1.00, 1.00] | 0.002 [0.002, 0.002] | 0.00 [0.00, 0.00] | |
| AST-Pen | 8.00 [8.00, 8.00] | 0.887 [0.884, 0.890] | 1.00 [1.00, 1.00] | 0.002 [0.002, 0.002] | 0.00 [0.00, 0.00] | |
| B | Baseline | 2.37 [1.67, 3.10] | 0.348 [0.275, 0.426] | 0.07 [0.00, 0.17] | 0.119 [0.101, 0.132] | 26.53 [21.34, 31.98] |
| Study | Substrate | Acquisition structure | Primary evidence | Focus |
| 1 — Controlled anchor | SAR | Scripted efficient and recurrent questioning regimes | Monitor sensitivity to controlled behavioral regimes; invariance to automated scorer substitution | Cross-domain reuse |
| 2 — Human interaction | MultiWOZ [ 35 ] | Predominantly atomic slot filling | Next-slot progress prediction under minimal baseline friction | Reuse; task structure |
| Map Task [ 36 ] | Incremental collaborative grounding | Multi-horizon landmark prediction and sustained friction during iterative clarification | Reuse; task structure | |
| AnnoMI [ 37 ] | Reflective motivational interviewing | Friction correlation with epistemic throughput and independence from clinical rapport ratings | Reuse; task structure | |
| 3 — Non-linguistic | MiniGrid [ 5 ] | Incremental 2D spatial exploration | Discrimination of trajectory loops in incomplete sectors versus post-goal loitering | Reuse; friction selectivity |
| Crafter [ 38 ] | Discrete achievement unlock tree | Detection of injected no-op loops and prerequisite-guided unlock forecasting | Reuse; task structure |
| Condition | AUC | Precision @ | Recall @ | Span coverage | ||
| MiniGrid I1 (residual stall) | 0.882 | 0.742 | 0.745 | 0.800 | 0.334 | 0.103 |
| MiniGrid I2 (post-goal loiter) | 0.715 | 0.455 | 0.467 | 0.500 | 0.179 | 0.106 |
| Crafter (injected no-op loop) | 0.767 | 0.853 | 0.698 | 0.952 | 0.509 | 0.133 |
| MetaWorld I1 (pre-grasp orbit) | 0.983 | 0.800 | 1.000 | 1.000 | 0.999 | 0.051 |
| MetaWorld I2 (post-goal holding) | 0.499 | 0.213 | 0.036 | 0.050 | 0.039 | 0.041 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Substrate | Progress interpretation | Friction interpretation | ||||
| SAR | Task-critical missing-person facts | Witness/source responses to queries | Expert completeness per fact; five-model LLM ensemble in the robustness check | Response embedding | Residual acquisition value across mission-critical facts | Repeated or degraded witness evidence associated with unresolved facts |
| AnnoMI | Projected client-state dimensions | Client statements within MI sessions | Manual rubric annotation of client disclosures | Client utterance embedding | Residual acquisition value across projected client-state dimensions | Repeated low-gain client evidence associated with unresolved dimensions |
| MultiWOZ | Transactional booking slots | User utterances following system prompts | Belief-state slot completeness changes | User-turn text embedding | Residual acquisition value across required booking slots | Repeated constraints while required slots remain unresolved |
| Map Task | Unique session-specific landmarks | Giver statements following follower queries | Manual rubric annotation of landmark grounding | Giver utterance embedding | Residual acquisition value across landmark coordinates | Repeated spatial descriptions while landmark grounding remains unresolved |
| MiniGrid | grid-sector partition | -step spatial visitation windows | Fraction of walkable tiles revealed | Unit-norm visitation and action histogram | Residual acquisition value across spatial sectors | Recurrent local visitation associated with incomplete spatial coverage |
| Crafter | 22-element capability achievement tree | Action, inventory, and achievement logs | Fraction of unlock prerequisites satisfied | Unit-norm action–inventory–milestone signature | Residual acquisition value across achievement prerequisites | Low nominal friction under predominantly atomic updates; sustained pre-resolution recurrence remains detectable |
| Regime | Human Expert Scorer | Automated LLM Ensemble Scorer | ||||
| Stall burden (%) | Mean | Categories resolved | Stall burden (%) | Mean | Categories resolved | |
| S1 | ||||||
| S2 | ||||||
| S3 | ||||||
| Diagnostic metric pair | Spearman | -value |
| Stall burden vs. information gain / turn | ||
| Mean friction ( ) vs. information gain / turn | ||
| Stall burden vs. total knowledge resolved | ||
| Mean friction ( ) vs. total knowledge resolved |
| Substrate | PE B (Ours) | Recency Baseline | Uniform Guessing | ||||||
| MiniGrid | 0.253 | 0.326 | 0.373 | 0.603 | 0.624 | 0.651 | 0.114 | 0.155 | 0.235 |
| Crafter (clean) | 0.226 | 0.380 | 0.544 | 0.077 | 0.178 | 0.228 | 0.090 | 0.129 | 0.211 |
| Crafter (with loops) | 0.132 | 0.119 | 0.100 | 0.118 | 0.147 | 0.086 | 0.118 | 0.046 | 0.071 |
| MetaWorld (clean) | 0.906 | 0.977 | 0.985 | 0.531 | 0.360 | 0.333 | 0.333 | 0.320 | 0.294 |
| MetaWorld (with loops) | 0.643 | 0.819 | 0.893 | 0.685 | 0.573 | 0.593 | 0.274 | 0.313 | 0.390 |
| Scenario | Initial MAP errors | Confident wrong starts | True-hypothesis resolution | post-change | mid-epoch |
| S1 | 20/40 | 4/40 | 0.014 | 0.051 | |
| S3 | 60/60 | 42/60 | 0.075 | 0.130 |
| Symbol | Type / Domain | Operational Definition |
|---|---|---|
| Interface Tuple | Task binding . | |
| Finite Set | Schema coordinates representing task dimensions. | |
| Observation Space | Evidence space emitted by the monitored process. | |
| Element | Evidence observation received at exchange . | |
| Requirement targeted by the preceding action. The marker denotes an unattributed exchange. | ||
| Function | Task-resolution map projecting evidence onto schema resolution. |
AI Agent Monitoring
AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.
See all 296 papers