Agent decision models select actions in a single forward pass from inputs that combine environmental observations with the agent's own record of previous actions. When an action fails silently, these sources can conflict: the record reports success, but the environment shows that the step did not take effect. We ask which source determines the next decision. Across Android screen traces, ALFWorld, and WebShop, we construct paired states that independently vary the observation, the logged action, and an explicit success claim. In all 12 configurations of four decision models and three domains, records have a larger average effect on commitment decisions than observations; after claimed-but-failed steps, the models still stop, advance, or purchase in 53-100% of cases. Yet the relevant evidence is often represented internally: linear probes on intermediate hidden states distinguish successful from failed steps on held-out apps, tasks, and goals (AUROC 0.77-1.00), and activation patching reveals increasing influence from records in later layers. We use this signal in EVIGATE, an intermediate-layer veto that requires no additional forward pass and adds 0.04-2.0% serving latency. Under injected silent failures, EVIGATE improves task success by 18.7 points in live ALFWorld agents, 5.0 in WebShop, and 5.1 in a phone simulator, and in WebShop and the simulator it also outperforms random vetoes at the same rate. Its effectiveness depends on representative calibration states: gates calibrated on offline counterfactual states do not transfer reliably to live agent trajectories.
Figures & tables
Domain
Terminal decision
Silent failure
States
Phones
complete
screen unchanged
3,200
ALFWorld
proceed
“Nothing happens.”
1,475
WebShop
buy now
options not selected
1,500
Table 1: The three domains: real Android screens from AndroidControl Li et al. (2024) (245 apps), unseen ALFWorld tasks, and WebShop goals. Request formats follow Ling et al. (2026b) .
Model
Wrong
Obs.
Logged
Claim
Phones
decider-4b
95%
+.05
+1.10
+.80
Kev-4B
96%
+.09
+.20
+.32
decider-12b
84%
+.21
+1.10
+.56
hosted Jev
88%
+.61
+2.31
+.95
Embodied (ALFWorld)
Table 2: Which input moves the decision. Wrong : how often the model still stops, proceeds or buys after a failed step whose record claims success ( Logged+claim ). Obs. , Logged , Claim : how much the observation, the logged step and the explicit claim raise logp of that option. Bold : the largest of the three. In every row the bold record effect exceeds the observation effect with a paired 95% CI above zero (Appendix Table 7 ).
All
Claimed
Model
Head
Probe
Head
Probe
Phones
decider-4b
.63
.77
.51
.74
Kev-4B
.65
.80
.55
.79
decider-12b
.69
.80
.61
.77
hosted Jev
.75
–
.68
–
Table 3: Detecting whether the step worked (AUROC on held-out apps, tasks or goals; 0.5 is chance). Head : the model’s own output. Probe : a linear probe on its hidden state. All : all test states. Claimed : only states whose record claims success, so that only the observation differs. Jev is closed-weight and cannot be probed.
Figure 2: Activation patching on decider-4b: the share of each input’s effect on logp of the terminal option that is restored by copying the decision-position state from the paired state at a single layer (mean ± 95% CI). The observation is present by mid-depth; the logged step and the claim arrive in the last layers. We keep pairs whose logp differs by more than 0.2. In WebShop the page barely moves the decision ( +0.02 , Table 2 ), so there is no observation effect to locate and only the logged-click curve ( n=100 ) is shown.
Figure 3: EviGate on an illustrative phone step. The record says Wi-Fi is on; the screen shows it is off. The output follows the record and proposes stop . The gate reads the decision-position state at a middle layer ℓ of the same forward pass, before record-dependent influence dominates the last layers, and vetoes. It acts only on terminal options. Probabilities are illustrative.
Model
Own score
Second q.
EviGate
Wrong
Phones
decider-4b
.59
.60
.71
72% → 38%
Kev-4B
.60
.61
.72
58% → 38%
decider-12b
.63
.62
.73
33% → 33%
hosted Jev
.66
.69
–
–
Embodied (ALFWorld)
Table 4: Gating terminal decisions on held-out groups: balanced accuracy (0.5 is chance). Own score : the model’s own probability of the terminal option, with a tuned threshold. Second q. : an added yes/no question asking whether the step worked. Wrong : failed steps that still end in a terminal decision, with the own score → with EviGate .
Condition
Phones ∗
ALFWorld
WebShop
No gate
77.2
42.5
8.5
Second question
83.2
61.2
7.0
EviGate , fit on offline contrast sets
–
58.2
2.5
EviGate , fit on target-domain states
82.4
61.2
13.5
random veto, matched rate
77.6
58.2
4.5
EviGate − no gate
Table 5: Task success (%) of live agents; same tasks and seeds in every row. Random veto: vetoes at random, about as often as the EviGate row above it (rates in the note below). Bottom: paired differences with 95% CIs over tasks.
Fine-tuned on
Test domain
Base
phones
all 3
all 3
1k each
all phones
Phones
Wrong
91%
47%
89%
47%
AUROC
.50
.84
.67
.82
ALFWorld
Wrong
90%
91%
0%
0.3%
AUROC
.91
.92
1.00
1.00
Table 6: Fine-tuning decider-4b on paired states. Wrong : failed steps whose record claims success that still end in a terminal decision. AUROC : the output’s detection of the failure on claimed states. Tests use held-out apps, games and goals. Last rows: accuracy on two of decider’s held-out general tasks.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Logged − Obs.
Claim − Obs.
Obs. ∣ claim
Phones
decider-4b
+1.05 [0.94, 1.17]
+0.74 [0.69, 0.80]
+0.00 [ − 0.01, 0.01]
Kev-4B
+0.11 [0.09, 0.14]
+0.23 [0.22, 0.25]
+0.05 [0.04, 0.06]
decider-12b
+0.89 [0.68, 1.09]
+0.34 [0.23, 0.46]
+0.03 [ − 0.02, 0.09]
hosted Jev
+1.70 [1.33, 2.10]
+0.34 [0.22, 0.47]
+0.14 [0.11, 0.18]
ALFWorld
Appendix
Table 7: Paired 95% bootstrap intervals for the effects in Table 2 (differences in logp of the terminal option). Decision points: phones 640 (decider-4b, Kev-4B) or 200 (decider-12b, Jev); ALFWorld 295; WebShop 300.
Model
Observation
Logged step
Claim
decider-4b
+0.05 [.02,.08]
+1.10 [1.00,1.21]
+0.80 [.74,.85]
decider-2b †
+0.02 [ − .02,.05]
+0.48 [.38,.58]
+0.79 [.72,.87]
Kev-4B
+0.09 [.07,.10]
+0.20 [.18,.22]
+0.32 [.31,.33]
Kev-0.8B †
+0.00 [.00,.01]
+0.02 [.01,.02]
+0.21 [.20,.22]
Laya †
−0.04 [ − .13,.06]
−0.01 [ − .06,.04]
+0.32 [.27,.38]
Kev-9B †
+0.34 [.28,.41]
+0.27 [.22,.32]
+0.24 [.22,.27]
Appendix
Table 8: Paired effects on logp(complete) on phones, with 95% bootstrap confidence intervals. For models up to 4B, explicit claims exert a larger mean effect than screen evidence. The larger models differ: Kev-9B weights the observed outcome more than either record component individually, whereas decider-12b remains most sensitive to the logged action. Rows marked † use a 1,000-state subset.
Output
Probe
Model
All
Claimed
All
Claimed
Gap
Text only
decider-4b
.63
.51
.77
.74
+.14 [.11,.17]
.59
Kev-4B
.65
.55
.80
.79
+.15 [.12,.17]
.59
decider-2b †
.59
.50
.65
.61
+.06 [.02,.09]
.59
Kev-0.8B †
.59
.50
.64
.59
+.04 [ − .01,.09]
.59
Laya †
.52
.50
.62
.60
–
–
Appendix
Table 9: Detecting whether the phone step worked (AUROC on held-out apps), as in Table 3 . Gap : probe minus output on all states, with 95% CI. Text only : a classifier on the screen text alone.
Model
Model’s choice
Own score
Second q.
Contrastive
EviGate
decider-4b
.59
.59
.60
.60
.71
Kev-4B
.59
.60
.61
.61
.72
decider-2b
.57
.55
.59
.56
.64
Kev-0.8B
.53
.56
.58
.56
.61
Laya
.50
.53
.55
.49
.59
Kev-9B
.66
.66
.68
.65
.75
Appendix
Table 10: Gating the phone stop decision on held-out apps (balanced accuracy), as in Table 4 , for all seven models and one more detector. Model’s choice : no gate. Own score and Second q. : as in Table 4 . Contrastive : contrastive decoding against a pass without the screen Shi et al. (2024) . Bottom: failed steps that still end in complete , over both failed variants ( Logged and Logged+claim , 1,280 states); Table 2 counts Logged+claim only, hence its higher rates.
Model
Observation
Logged step
Claim
Output
Probe
Qwen3.5-4B-Base
+.02
+.15
+.51
.59
.65
Qwen3.5-4B (chat)
+.05
+.28
+.42
.61
.72
Kev-4B (LoRA)
+.09
+.21
+.32
.66
.77
decider-4b (full fine-tuning)
+.01
+1.12
+.78
.62
.75
Appendix
Table 11: The bias is already present in the base model: one Qwen3.5-4B backbone before (top) and after (bottom) decision tuning, 1,000 phone states. Left: paired effect on logp(complete) . Right: AUROC for detecting the failure from the output and from a probe.
Figure 4: Layerwise decoding of action outcomes. (a) Phone probe AUROC across relative depth, compared with the decision model’s output (grey band). (b) Model-output AUROC (hollow points) and probe AUROC (filled points) across domains; dashed grey indicates the input-text baseline. In ALFWorld and WebShop, standardized failure messages also allow a text-only classifier to reach AUROC 1.00.
Figure 5: How many labels the phone probe needs (held-out apps). × : the output’s AUROC. Ten labelled episodes already beat the output for the decision-tuned models.
ALFWorld
WebShop
Normal pass
59%
99.7%
Record removed from layer 12
26%
93%
Record removed from layer 16
50%
99.5%
Record removed from layer 20
59%
99.7%
EviGate
0.5%
0%
Appendix
Table 12: Failed steps that still get a terminal decision when the record is removed from layer ℓ onward (decider-4b), compared with EviGate . Failed steps are both failed variants ( Logged and Logged+claim : ALFWorld 590, WebShop 600 states); Table 2 counts Logged+claim only.
Phones
ALFWorld
WebShop
Failed pass
Good vetoed
Failed pass
Good vetoed
Failed pass
Good vetoed
No gate
78%
0%
59%
0%
99.7%
0%
Record removed from the input
2%
93%
9.5%
13%
6.0%
7.5%
EviGate
25%
27%
0.5%
0%
0%
0.5%
Appendix
Table 13: Removing the record from the input vs. EviGate (decider-4b, paired states). Failed pass : failed steps that still end in a terminal decision. Good vetoed : steps that worked but whose terminal decision is blocked. Failed and good steps include both variants with and without the claim (phones: the 1,000-state subset, 400 failed and 400 good states; ALFWorld 590 and 590; WebShop 600 and 600).
Layer
Phones
ALFWorld
WebShop
obs / logged / claim
obs / logged / claim
obs / logged / claim
28
23 / 6 / 5
7 / 4 / 2
5 / 1 / 3
32
66 / 42 / 38
50 / 31 / 22
46 / 10 / 22
36
82 / 68 / 67
70 / 50 / 40
69 / 32 / 44
42
95 / 85 / 85
87 / 74 / 67
88 / 63 / 73
Appendix
Table 14: Activation patching in decider-12b (48 layers). Each cell is the share (%) of an input’s effect on the decision that is restored by patching the decision-position state at that layer. Obs = the screen or observation; logged = the logged step; claim = the false success claim. Before layer 26 every share is at most 2%.
States
Batch
Engine
+ EviGate
Overhead
T3A ( ∼ 2.7–3.4k tokens)
1
363.4
368.3
+1.35%
8
3870.2
3873.8
+0.09%
32
19503.6
19512.3
+0.04%
Short ( ∼ 280 tokens)
1
39.7
40.5
+2.01%
8
214.9
217.8
+1.32%
32
829.3
831.4
+0.26%
Appendix
Table 15: Decision latency (p50, ms) of decider-4b in its production engine, with and without EviGate (30 iterations after warm-up). The p99 latency of the base engine is within 0.3–0.7% of p50 for T3A states and 0.7–4.5% for short states, so overheads below about 1% are within measurement variability.
Condition
Success
Gain
False stops
Calls / episode
Calls / success
No gate
77.2
–
13.9
3.80
4.92
Random veto, second question’s rate
77.5
+0.3 [ − 0.3, 0.8]
13.3
3.99
5.14
Random veto, EviGate ’s rate
77.6
+0.4 [0.0, 0.8]
13.3
3.98
5.12
Contrastive decoding
77.1
−0.1 [ − 9.7, 7.9]
8.2
8.93
11.58
Second question
83.2
+6.0 [1.7, 11.2]
7.8
5.07
6.10
LLM judge (9B)
81.5
+4.3 [1.7, 7.6]
9.3
5.04
6.18
Appendix
Table 16: Phone simulator, all conditions (24 tasks × 30 seeds, decider-4b, EviGate with the balanced threshold). Gain : paired change in success vs. no gate, with 95% CI over tasks. False stops : episodes that stop before the goal is reached.
N
1
2
4
8
16
Oracle
9.8
16.9
25.9
35.8
44.0
Random attempt
9.8
14.1
17.2
19.8
21.2
Decision model’s own judgement
9.8
15.2
20.4
24.9
28.0
Probe on its hidden state
9.8
16.5
24.3
32.8
40.0
Appendix
Table 17: Best-of- N success (%) on 50 WebShop goals, using 16 sampled agent trajectories per goal. Each non-oracle strategy selects among candidate attempts using the indicated ranking criterion; results average over 300 random subsets per goal. Strategies choose only among attempts that end in a purchase, and a goal counts as failed when none of its N attempts buys; this is why even random selection improves with N . Confidence intervals resample goals.
Learned world models are becoming essential to general-purpose agents: by predicting action consequences, they support planning and decision-making while reducing reliance on costly trial and error. This reliance creates a fundamental ambiguity: when a world-model-guided decision fails, the trajectory alone may not reveal whether the agent's decision rule or the world model caused the loss. We formalize this failure-attribution problem as a counterfactual decomposition of return loss and prove that its components are not identifiable from passive interaction, even for finite-horizon planners. This obstruction motivates Dual-Frontier, a learning principle that admits a world-model-guided decision only when its predicted advantage exceeds a certified bound on decision-relevant world-model error; otherwise, evidence is allocated to world-model verification. Action-conditioned value bounds and a closed-loop extension guarantee non-decreasing return for admitted decisions. Calibrated gates and simultaneous confidence sequences support adaptive evidence reuse, with sufficient and necessary verification bounds. Controlled learned-model experiments validate the predicted failure modes and certification behavior, while cross-backbone tool-use benchmarks instantiate the same verify-then-promote rule in realistic agent world-model pipelines, consistently improving decision quality and reliability.
Huatai Zhu, Qiang Chen, Ziqian Kou +5
Central South University · The Hong Kong University of Science and Technology · Xiangjiang Laboratory +2
Successful agent execution need not identify which future product improvement its user would value. We present a decision-specific audit that maps a declared observation channel and product-value contrast to compatible intervals and witness populations. Its foundations are established identification and decision theory; the contribution is an executable measurement workflow and a controlled study of its limits. A frozen experiment makes 4,800 requests to two pinned model snapshots on shared synthetic tasks. All 36 conservative primary intervals remain unresolved despite different execution accuracy. An exploratory 2,400-call follow-up records supplied preferences and resolves three of nine comparisons per model. A deterministic extractor resolves seven of nine without model calls or calibration observations, exposing unnecessary uncertainty introduced by model-generated reports. A further 14,400 controlled multinomial simulations distinguish structural ambiguity from weak identification and finite calibration precision. We propose a source-labeled decision receipt and provide an offline viewer for inspecting the audit. These results motivate preserving decision-relevant structured input and diagnosing why a decision is unresolved before collecting more telemetry. The study contains no human participants or real customer outcomes. Full proofs, raw model provenance, controlled experiments, and reproducible analyses accompany the report.
AI agents depend on repeated decisions to accomplish tasks, with large language models (LLMs) commonly serving as their core decision makers. The recent emergence of Jev offers an opportunity to make these decisions faster and at lower cost. However, it remains unclear whether such local savings can be translated into efficient and reliable task completion. In this paper, we investigate this efficiency transfer across five agents in two representative application settings, embodied tasks in ALFWorld and mobile app use in AndroidWorld. Across 4,500 task executions, we jointly evaluate task effectiveness, decision-level and task-level resource use, and execution behavior. Experimental results show that Jev integration substantially reduces decision cost and latency across all five agents, with task-level gains varying by workflow. Task success remains close to baseline levels on ALFWorld and declines more substantially in two mobile agents. Auxiliary reasoning and additional interactions absorb decision savings, while failed recovery and premature completion limit successful delivery. Mean cost per successful outcome increases in one agent even as spending per attempt falls. These findings highlight the importance of preserving progress, supporting recovery, and verifying completion when integrating efficient decision models into agent workflows. Their practical value should be assessed through successful task delivery, recovery behavior, and resources per delivered outcome.
Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1
Nanjing University · Technical University of Munich