Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
A multimodal agent must decide whether visual evidence is worth acquiring, which candidate or resolution to query, and when further interaction can no longer change its decision. We study this problem under a strict free-metadata/paid-observation boundary, stochastic vector costs, and a frozen execution trace, and treat it as decision-certifiable active evidence acquisition. A Counterfactual Evidence Interaction Graph (CEIG) represents unary evidence value, signed second-order interaction, and directed conditional harm while maintaining explicit score enclosures for unresolved interactions; a decision-weighted enclosure objective tightens the bounds that can change action identity. Gap-directed refinement evaluates only interactions that can displace the incumbent acquisition and stops once an ε-decision certificate shows that unresolved terms cannot change the selected action beyond a declared tolerance. The certificate is model-relative: it verifies policy-consistent optimality under the pinned learned score, and semantic quality is measured separately through post-freeze labels. Paired continuation rollouts learn the downstream value of paid previews, an online operational cost filter is separated from independent finite-policy-family trajectory certification, and an acquisition monitor with an independently implemented verifier binds every certificate to the frozen trace. On 640 cases split into 384/128/128 training, validation, and test cases, the controller reaches task score 0.842, grounding F1 0.816, evidence recall 0.873, 1.74 accepted calls, 286.5K visual tokens, and 742ms p95 latency. Dense replay of issued certificates shows 0.992 action agreement and a 0.988 certificate rate while evaluating 19.6 instead of 191.4 expensive interaction terms per decision.
Figures & tables
method
unit
pair.
budget/risk
cert.
oracle
A/R
replay
audit
semantic
ActiveEvidence
whole image
yes
call cap / traj. risk
yes
post-freeze
yes
cross-impl.
0.973
held-out quality
VisionThink
whole image
N/R
N/R
N/R
not explicit
partial
N/R
N/R
task benchmarks
AdaptVision
crop/resolution
N/R
N/R
N/R
not explicit
partial
N/R
N/R
task benchmarks
AdaTooler-V
visual tools
N/R
N/R
N/R
not explicit
partial
N/R
N/R
task benchmarks
DART
visual tools
N/R
N/R
N/R
not explicit
partial
N/R
N/R
task benchmarks
ToolScope
multi-tool
N/R
N/R
N/R
not explicit
partial
N/R
N/R
task benchmarks
Table 1. Contract fields reported by the closest adaptive visual-acquisition methods: pairwise evidence interactions (pair.), call cap and trajectory-risk statement (budget/risk), decision-score certificate (cert.), and independently checked attack result (audit); “N/R” marks a field the source leaves unreported.
Figure 1. Overview of ActiveEvidence. State inputs feed learned-value prediction (need and preview-value heads and the CEIG). Decision-focused computation filters infeasible actions, keeps the incumbent S∗=St(bt) and challenger bounds [L,U]=[ℓt,ut] , and resolves the widest unresolved edge of the top challenger until maxU−S∗≤ϵ certifies the action. Previews and acquisitions re-score the state, STOP and ABSTAIN terminate, and all events are frozen and independently verified before evaluation. A left-to-right pipeline with five panels above a verification band. State inputs (request, free metadata such as ID, type, and dimension, paid evidence that contains pixels, and the budget) enter learned-value prediction, where offline supervision trains the need and preview-value heads and a CEIG graph over candidates A to F has positive, negative, and harm edges. Decision-focused computation applies a cost filter to obtain feasible actions, keeps an incumbent score S-star and challenger bounds L to U, selects the challenger with the largest U, resolves its widest unresolved edge, and repeats. The epsilon-decision certificate checks that max U minus S-star is at most epsilon. Actions are PREVIEW and ACQUIRE, which loop back to update and re-score the state, and the terminal STOP and ABSTAIN. The verification band passes attempted, rejected, and accepted events through trace freeze, an independent verifier, and evaluation.
method
task
ground.
evidence
coverage
calls
V tok.
p95 ms
viol.
Text-only
0.687
0.421
0.000
0.844
0.00
0.0K
218
0.000
Always-on
0.817
0.789
0.948
0.969
4.93
808.6K
1687
0.126
Random
0.731
0.596
0.623
0.875
2.91
476.8K
1034
0.071
Static metadata
0.764
0.674
0.704
0.891
2.43
398.7K
889
0.052
Adaptive vision
0.806
0.758
0.818
0.914
2.32
381.4K
917
0.047
Weighted metadata
0.782
0.711
0.749
0.898
2.24
367.9K
846
0.043
Table 2. Held-out semantic quality and resource use on the 128-case test split under the shared Qwen2.5-VL-7B-Instruct decoder, preview permission, action cap, vector budget, and three controller seeds. Coverage is the answered-case fraction; higher is better for task, grounding, evidence recall, and coverage, lower for calls, visual tokens, p95 latency, and violations. Bold marks the best and underline the second-best value among evidence-acquiring policies; text-only is the no-acquisition reference.
Figure 2. Task score against visual tokens per case (K) for Table 2 : (a) all methods and (b) acquisition methods enlarged. Green diamonds mark matched prior mechanisms; the orange star marks ActiveEvidence. Two task-score versus token plots show all eighteen comparison methods and a zoom of the acquisition methods. Matched baselines are green diamonds and ActiveEvidence is an orange star.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
policy
activ.
attempt.
accept.
V tok.
lat. (ms)
ID hit
exhaust.
text only
0.000
0.00
0.00
0.00
1.00
0.000
0.000
always-on
1.000
2.24
1.63
13.00
4.25
0.500
0.625
static selector
1.000
2.01
1.50
13.00
4.25
0.625
0.500
active evidence
0.625
1.42
1.03
10.00
3.50
0.500
0.375
Appendix
Table 3. Mechanism-level and row-audit quantities under the metadata-only contract. Attempted and accepted calls are separate ledger quantities; ID hit and exhaustion are post-freeze diagnostics, and the aggregate call means come from the frozen row-level audit. Bold marks the best value among the three acquiring policies (lower is better for activation, attempted and accepted calls, tokens, latency, and exhaustion; higher for ID hit); text only is the no-acquisition reference.
study
evidence type
retained result and boundary
Mechanism screen
metadata fixture (proxy)
Eight metadata-only records: active selection activates 0.625 of rows, uses selected-image count 1.250, and reaches image-ID proxy 0.500. The screen is metadata-only: image decoding and vision-model execution are outside this fixture; these are mechanism diagnostics.
Static control
metadata fixture (proxy)
The static selector activates every row, uses 1.625 metadata-ranked images, and reaches image-ID proxy 0.625. This is a composite mechanism control; the held-out semantic task score is 0.764.
Field provenance
provenance record
TextConf-v0.3, VisualQuality-v0.4, and TextImageRel-v0.5 bind visible inputs and access records; ECE for ci,qij,rij is 0.041/0.036/0.039. Phase-specific oracle-access counts are reported in Appendix D.6 .
Matched policy controls
held-out test
The matched surface defines task/grounding/calls/V-tokens/p95 for the controls in Table 2 ; all control rows are joined under the same held-out receipt.
Held-out visual quality and cost
held-out test
The pinned Qwen2.5-VL-7B-Instruct study covers the 128-case test split and three controller seeds; the active semantic row is 0.842/0.816/0.873 with 1.74 calls, 286.5K visual tokens, and 742ms p95 latency.
Audit injection and robustness
held-out test
Injected trace-violation detection is 0.973 with clean-trace FPR 0.012; post-execution mutation detection ranges from 0.944–0.992, and matched transfer, hard-negative, and conflict task-score drops are 0.021, 0.028, and 0.034.
Appendix
Table 4. Experiment ledger and evidence boundary. The evidence type separates metadata-fixture proxies, provenance records, and held-out test results.
Evidence receipt
Supported interpretation
Manifest SHA-256 and 12480/12471 image hashes
Asset identity and byte-level matching
Accepted-call means 1.63/1.50/1.03 and exhaustion 0.625/0.500/0.375
Final-checkpoint vector-budget accounting
Field ECE 0.041/0.036/0.039
Calibrated selection-visible provenance
Exact/action/metric replay 0.985/0.972/0.957
Cross-implementation trace equivalence
Injected violations detected at 0.973
Audit-stress detection under the declared threat model
Eight-row fixture and held-out semantic table
Mechanism unit testing is separated from task quality
Appendix
Table 5. Evidence receipts and their supported interpretation.
stress
protocol
task
V tok.
audit outcome
candidate-pool size
small/medium/large fixed pools
0.833/0.842/ 0.846
231.7 /286.5/338.2K
0.969/0.973/0.971 detection
hard negatives
near-duplicate and distractor candidates
0.814
302.8K
0.976 detection; 0.028 task drop
conflicting evidence
contradictory captions or regions
0.808
310.6K
0.979 detection; 0.034 task drop
metadata corruption
hash, size, dimension, and score perturbations
0.832
293.4K
0.981 detection
budget shifts
call/token/latency vectors
0.823 (worst shift)
272.9K
0.029 violation; 0.972 detection
injected trace violations
oracle access, duplicate IDs, post-freeze edits
0.841
287.2K
0.973 detection; 0.012 clean FPR
Appendix
Table 6. Robustness and trace-audit stress matrix on the 128-case held-out split. Task is higher-is-better and visual tokens (V tok.) lower-is-better; detection is the access-injection detection rate under its declared denominator. Bold marks the best value across these stress settings.
policy
attempted
accepted
final Bcall
Ei
always-on
2.24
1.63
0.37
0.625
static selector
2.01
1.50
0.50
0.500
active evidence
1.42
1.03
0.97
0.375
Appendix
Table 7. Oracle-free row-level budget audit. Attempted calls and final remaining call budgets are reported from the row-level receipt; accepted calls and exhaustion use the frozen held-out audit. Bold marks the best value per column (lower is better for calls and exhaustion, higher for the final call budget).
gate
ranker
stopper
activ. ↓
selected ↓
ID proxy ↑
quality ↑
off
static
fixed
1.000
1.625
0.625
0.776
on
static
fixed
0.625
1.250
0.625
0.792
off
active
fixed
1.000
1.625
0.500
0.817
on
active
fixed
0.625
1.250
0.500
0.829
off/on
random
fixed
0.750
1.375
0.375
0.731
on
learned
learned
0.625
1.125
0.625
0.842
Appendix
Table 8. Predeclared mechanism controls for gate and ranker attribution on the eight-record fixture. Activation and selected candidates are lower-is-better under the mechanism contract; ID proxy and held-out quality are higher-is-better. Bold marks the best value per column.
policy
p95 (ms)
VRAM (GB)
abstain
fail.
disp.
text only
218
11.4
0.156
0.042
0.006
always on
1687
18.7
0.031
0.051
0.008
static selector
889
15.1
0.102
0.036
0.007
random selector
1034
15.6
0.117
0.049
0.009
active evidence
742
15.8
0.078
0.023
0.005
Appendix
Table 9. Held-out operational latency, memory, abstention, and failure outcomes under the same pinned contract. Bold marks the best and underline the second-best value among evidence-acquiring policies for p95 latency, VRAM, failures, and dispersion; text-only is the no-acquisition reference, and abstention is a selective-prediction rate without a preferred direction.
model
int. MAE ↓
sign AUC ↑
comp. ↑
redund. ↓
conflict ↓
Unary only
0.184
0.611
0.642
0.176
0.139
Latent pairwise
0.119
0.786
0.747
0.132
0.094
Bundle regression
0.142
0.738
0.709
0.147
0.112
No re-scoring
0.083
0.887
0.822
0.104
0.063
No uncertainty calibration
0.077
0.901
0.839
0.096
0.058
Full CEIG
0.071
0.914
0.861
0.081
0.047
Appendix
Table 10. Direct attribution of explicit interaction supervision on the frozen test split. Arrows indicate the preferred direction. Bold marks the best and underline the second-best value per column.
objective
task ↑
int. MAE ↓
dense agr. ↑
cert. ↑
edges ↓
regret ↓
No tightness
0.838
0.069
0.964
0.923
34.2
0.012
Uniform tightness
0.840
0.073
0.978
0.956
26.8
0.007
Decision-weighted (default)
0.842
0.071
0.992
0.988
19.6
0.003
Appendix
Table 11. Attribution of certificate-oriented CEIG training on the frozen test split. Arrows give the preferred direction; the three rows differ only in the training objective. Bold marks the best and underline the second-best value per column.
controller
nom. ↓
+20% ↓
+50% ↓
mixed ↓
task ↑
CVaR ↓
Nominal hard filter
0.061
0.148
0.319
0.274
0.788
0.151
Nominal filter + dual price
0.045
0.109
0.238
0.194
0.801
0.108
Conformal feasible set
0.025
0.052
0.113
0.079
0.814
0.058
Full validation-calibrated controller
0.018
0.024
0.041
0.029
0.823
0.032
Appendix
Table 12. Budget safety under execution-cost shifts. Violation is the fraction of test trajectories with any negative budget component; CVaR uses the declared cost-overrun loss. Bold marks the best and underline the second-best value per column.
Figure 3. Budget safety under execution-cost shifts for the four controllers of Table 12 : (a) violation rate per shift condition and (b) worst-shift task score against cost CVaR. A heatmap gives violation rates for four controllers under four cost conditions, followed by a scatter of worst-shift task score against cost CVaR. Full-controller heatmap values are bold.
policy
ncert
vθ
Uθ↓
approved
confirm. viol. ↓
Point-cost filter
500
31
0.082
no
0.061
Adaptive upper-cost filter
500
13
0.041
yes
0.025
Complete-policy certification
500
8
0.028
yes
0.018
Appendix
Table 13. Trajectory-risk certification for the frozen policy family. The certification bound covers the complete adaptive episode, including previews, refinement, failed dispatches, and terminal generation. Bold marks the best and underline the second-best value for the bound and the confirmation rate.
attack
cases
detect. ↑
false acc. ↓
impact ↓
Oracle field before freeze
200
0.992
0.004
0.061
Candidate ID or hash substitution
200
0.986
0.006
0.047
Rejected attempt removed
200
0.971
0.010
0.013
Cost under-reporting
200
0.958
0.016
0.028
Feasible action omitted
200
0.962
0.014
0.036
Action score modified
200
0.968
0.010
0.043
Appendix
Table 14. Decision and trace certificate stress test. Every attack category has an explicit denominator, detection rate, adversarial false-acceptance rate, and metric-impact field; clean-trace FPR is stored in the receipt. Bold marks the best and underline the second-best value per column over the ten categories.
Figure 4. Certificate attacks of Table 14 : (a) detection rate and (b) adversarial false-acceptance rate and metric impact per category. Dashed lines mark the overall detection rate (0.967) and overall metric impact (0.042). Two aligned dot plots list ten attack categories. Detection rates lie between 0.944 and 0.992. False-acceptance rates lie between 0.004 and 0.020, and metric impacts between 0.013 and 0.068.
attack
cases
reject. ↑
wrong ↓
unres. ↓
chg. ↓
Evaluation-label access before freeze
200
198
1
1
1
Undeclared pixel preview
200
195
3
2
2
Forged cost plus action-score edit
200
192
5
3
3
Omitted feasible action plus ledger edit
200
191
6
3
4
Post-freeze chain reconstruction
200
189
7
4
5
Appendix
Table 15. Access-boundary and coordinated certificate attacks. Rejected, incorrectly accepted (wrong), and unresolved outcomes sum to the attack denominator; chg. counts incorrectly accepted traces that change a reported metric. Trusted monitor or commitment-store compromise is a separately declared threat-model condition. Bold marks the best and underline the second-best value per column.
rule
agr. ↑
regret ↓
cert. ↑
edges ↓
ctrl. ms ↓
ver. ms ↓
Dense evaluation
1.000
0.000
1.000
191.4
184.6
38.2
Fixed- L neighbors
0.938
0.024
0.914
48.0
56.3
14.7
Random
0.912
0.038
0.875
32.5
49.1
12.5
Widest envelope
0.969
0.011
0.953
28.7
46.8
11.9
Top challenger (default)
0.992
0.003
0.988
19.6
41.8
9.4
Per-compute priority
0.994
0.002
0.991
16.8
38.5
8.7
Appendix
Table 16. Gap-directed refinement rules with identical model weights. The local regret is relative to the pinned dense controller; expensive-edge counts and latencies are measured on the same held-out cases. Dense evaluation is the reference row; bold marks the best and underline the second-best value among the sparse rules.
Figure 5. Refinement rules of Table 16 : (a) dense agreement and certificate rate, (b) expensive edges, and (c) controller and independent-verifier time. Per-compute priority retains high consistency with the least computation among sparse rules. Three aligned panels compare six inference rules on decision consistency, expensive interactions, and execution time. Gap-directed priority evaluates 16.8 edges with controller and verifier times 38.5 and 8.7 milliseconds.
pool
task ↑
calls ↓
ctrl. ms ↓
memory GB ↓
4
0.821
1.48
5.6
15.0
8
0.833
1.59
7.9
15.2
16
0.838
1.67
11.8
15.4
32
0.842
1.74
18.6
15.8
64
0.846
1.81
31.4
16.1
Appendix
Table 17. Candidate-pool scaling with sparse CEIG inference on the same test cases. Bold marks the best value per column.
Figure 6. Candidate-pool scaling of Table 17 : (a) task score, (b) accepted calls, (c) controller time, and (d) peak memory. Separate axes retain each metric’s units. Four panels plot task score, calls, controller milliseconds, and memory in gigabytes against pool sizes four through sixty-four.
pool
actions
pairs
eval. edges
fraction ↓
cert. ↑
p95 ms ↓
4
16
6
4.8
0.800
0.996
8.4
8
46
28
9.2
0.329
0.994
12.3
16
154
120
14.6
0.122
0.991
17.8
32
562
496
19.6
0.040
0.988
26.5
64
2146
2016
24.5
0.012
0.982
44.2
Appendix
Table 18. Action-space materialization under the frozen controller and unchanged bundle cap. Bold marks the smallest evaluated fraction, the highest certificate rate, and the lowest p95 controller time.
Figure 7. Action-space materialization of Table 18 : (a) declared actions, pair edges, and evaluated expensive edges per decision on a logarithmic scale and (b) certificate rate against candidate-pool size. Panel a plots three counts against pool sizes 4 to 64 on a log scale: declared actions rise from 16 to 2146 and pair edges from 6 to 2016, while evaluated edges rise from 4.8 to 24.5. Panel b shows the certificate rate decreasing from 0.996 to 0.982.
supervision
unary calls
pair calls
preview rollouts
GPU-h ↓
Nbreak↓
Unary targets only
7488
0
0
3.8
580
CEIG pair targets
7488
14750
0
11.2
940
Paired preview and decision targets
7488
14750
3072
14.8
1160
Certificate-oriented full training
7488
14750
3072
16.4
1290
Appendix
Table 19. Offline supervision cost and online amortization. All rows use the same answer backbone and report the charged label-generation units. Bold marks minimum training time and break-even queries; call counts describe supervision.
preview rule
task ↑
net benefit ↑
harmful ↓
V tok. ↓
regret ↓
No preview
0.816
0.000
0.000
0.0K
0.089
Fixed schedule
0.828
0.047
0.241
118.6K
0.063
Immediate-utility gate
0.832
0.076
0.158
84.3K
0.049
Unpaired return regression
0.836
0.098
0.106
73.1K
0.038
Paired decision-value supervision
0.842
0.124
0.043
62.4K
0.021
Appendix
Table 20. Preview-value attribution under matched information, access, and supervision budgets. Harmful previews have negative paired net benefit below a predeclared tolerance. Bold marks the best and underline the second-best value among preview-using rules; no preview is the reference row.
preview rule
perm. sens. ↓
pre AUC (0.5 ideal)
post AUC ↑
Unpaired return regression
0.016
0.508
0.724
Paired decision-value supervision
0.007
0.502
0.816
Appendix
Table 21. Pre-preview leakage test for the two learned preview rules. Pre and post AUROC are measured before and after the preview is purchased; permutation sensitivity has ideal value zero and pre-preview AUROC has chance value 0.5. Bold marks the better rule per column.
target
zero-shot drop ↓
recalibr. drop ↓
ECE ↓
viol. ↓
Smaller VLM, same domain
0.009
0.000
0.044
0.019
Second VLM family
0.018
0.001
0.051
0.023
New task domain
0.027
0.002
0.059
0.028
Appendix
Table 22. Transfer of the acquisition controller. Drops are task-score decreases under zero-shot and recalibration-only transfer; lower is better in every column, and bold marks the best value per column.
decision
issued
dense agr. ↑
disagr. ↓
regret ↓
Acquisition
222
0.991
0.009
0.003
Preview
108
0.991
0.009
0.002
STOP
118
0.992
0.008
0.002
ABSTAIN
10
1.000
0.000
0.000
All decisions
458
0.992
0.008
0.003
Appendix
Table 23. Dense-replay audit by decision type. Bold marks the best typewise agreement, disagreement, and regret; the pooled row is descriptive.
stratum
N
task ↑
calls ↓
V tok. ↓
95% CI
Domain 1
32
0.845
1.72
283.4K
[0.824, 0.866]
Domain 2
32
0.839
1.76
289.8K
[0.818, 0.860]
Domain 3
32
0.846
1.71
281.2K
[0.825, 0.867]
Domain 4
32
0.838
1.77
291.6K
[0.817, 0.859]
Low candidate count
64
0.837
1.54
254.3K
[0.821, 0.853]
High candidate count
64
0.847
1.94
318.7K
[0.831, 0.863]
Appendix
Table 24. Stratified held-out performance of the full controller. Intervals use the same hierarchical bootstrap as the primary analysis. Bold marks the best value within each domain, candidate-count, or interaction group.
Figure 8. Stratified held-out task scores of Table 24 with hierarchical-bootstrap 95% intervals. The shaded band and vertical line mark the aggregate interval [0.830,0.854] and the aggregate score 0.842. A forest plot lists four domains, two candidate-count strata, and two interaction strata with point estimates between 0.831 and 0.856 and their 95 percent intervals, over a shaded aggregate band from 0.830 to 0.854.
Language-model agents increasingly solve complex tasks through parallel reasoning and asynchronous information gathering. Controlled replay across two model families and three benchmarks shows that delivery timing redirects search decisions made before all results arrive, despite identical initial evidence. Acquisition disagreement reaches up to 24 percentage points above same-order repeat variation. This dependence persists after all initial evidence arrives and through a second search, turning temporary differences in what agents have seen into lasting differences in what they acquire. An acquisition-regret bound with a matching construction shows that invariance at a fixed early arrival count can limit the use of evidence already received. We introduce Confluent Evidence Acquisition (CEA), which instead releases local reasoning when each subquestion's evidence is complete and allocates shared search independently of completion order. For a fixed plan and stable conditional response law, we prove joint invariance of decisions, acquisitions, and answers. With question-only plans across three benchmarks, CEA brings acquisition disagreement to within 1.0 percentage point of repeat variation and achieves accuracy comparable to complete-input controls, improving over early commitment by up to 12.6 percentage points. Replacing CEA's fixed allocation with first-ready allocation under contention adds 21.6 percentage points of acquisition disagreement, while local query-target disagreement remains near repeat variation. Timed comparisons show that CEA reaches decisions earlier and responds faster than the same program waiting for all initial evidence.
Rule-based reasoning, as in eligibility checks and contract reviews, requires language models to assess evidence against individual conditions and combine their judgments under explicit rules. Errors in evidence assessment can leave a decision unchanged, but misinterpreting or overlooking decision-critical evidence can reverse it. Identifying such evidence allows more capable models to focus on checking the corresponding condition judgments, supporting accurate and safe decisions. Recognizing the evidence's criticality requires understanding how evidence affects a condition judgment and how that judgment affects the decision. To achieve the goal, we propose a INTERvention-based imPACT learning framework (InterPact), which enables counterfactual verification of evidence criticality in rule-governed decisions. Specifically, its evidence intervention constructor generates training pairs for a propagation verifier by editing case facts with a frozen language model while holding rules and non-target conditions fixed. Human-reviewed labels record the resulting condition and decision changes, while complete state-to-decision mappings supervise consequences beyond the observed edit. During training, the verifier weights learned conditional decision predictions by evidence-based condition probabilities through a fixed composition operation, propagating decision-change supervision into the base model. At inference, the trained base model directly judges criticality from the original case and target evidence, without human or stronger-model supervision. On single-case evidence criticality verification over adapted rule-governed decision cases, InterPact achieves 68.28% accuracy, outperforming all six baselines. These results support learned decision sensitivity as a basis for prioritizing evidence checks.
Haoyang Zhang, Jianpeng Zhao, Qi Hao +1
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China
As agents take on long-horizon tasks, users shift from making individual decisions to overseeing autonomous execution. Yet the volume of agent activity and the fragmentation of supporting evidence make it difficult to determine which decisions warrant user verification. We study monitors that identify consequential decisions and locate evidence to help users assess their implications. We introduce AgentMonBench, a software-engineering benchmark comprising three subsets that cover two complementary dimensions: alignment between requirements and behavior, and awareness of consequential autonomous decisions for verification. To support these judgments, we propose the Evidence-Grounded Behavior Graph (EBG), a training-free method that groups source-linked evidence into behaviors and organizes their relationships into a graph. EBG presents task-oriented views of this graph to help monitors interpret behavior in context. Experiments across eight models show that EBG improves decision identification and evidence localization in most settings compared with direct access to the original context. Further experiments show that EBG's evidence-localization gains persist across input scales and hyperparameter settings, while real-world applications illustrate its practical value for human oversight.
Zhongxiang Sun, Jiahao Yan, Hongkang Zhao +5
Gaoling School of Artificial Intelligence, Renmin University of China · Kuaishou Technology