Claim Laundering: When Agent Decision Models Trust Records Over Reality

Tina Khezresmaeilzadeh; Seyedarmin Azizi; Erfan Baghaei Potraghloo; Mehdi Kamal; Massoud Pedram

cs.AIOct 19, 2026

Claim Laundering: When Agent Decision Models Trust Records Over Reality

Authors: Tina Khezresmaeilzadeh, Seyedarmin Azizi, Erfan Baghaei Potraghloo, Mehdi Kamal, Massoud Pedram

Organizations: University of Southern California, Los Angeles, CA, USA.

Abstract

Agent decision models select actions in a single forward pass from inputs that combine environmental observations with the agent's own record of previous actions. When an action fails silently, these sources can conflict: the record reports success, but the environment shows that the step did not take effect. We ask which source determines the next decision. Across Android screen traces, ALFWorld, and WebShop, we construct paired states that independently vary the observation, the logged action, and an explicit success claim. In all 12 configurations of four decision models and three domains, records have a larger average effect on commitment decisions than observations; after claimed-but-failed steps, the models still stop, advance, or purchase in 53-100% of cases. Yet the relevant evidence is often represented internally: linear probes on intermediate hidden states distinguish successful from failed steps on held-out apps, tasks, and goals (AUROC 0.77-1.00), and activation patching reveals increasing influence from records in later layers. We use this signal in EVIGATE, an intermediate-layer veto that requires no additional forward pass and adds 0.04-2.0% serving latency. Under injected silent failures, EVIGATE improves task success by 18.7 points in live ALFWorld agents, 5.0 in WebShop, and 5.1 in a phone simulator, and in WebShop and the simulator it also outperforms random vetoes at the same rate. Its effectiveness depends on representative calibration states: gates calibrated on offline counterfactual states do not transfer reliably to live agent trajectories.

Figures & tables

Domain	Terminal decision	Silent failure	States
Phones	complete	screen unchanged	3,200
ALFWorld	proceed	“Nothing happens.”	1,475
WebShop	buy now	options not selected	1,500

Table 1: The three domains: real Android screens from AndroidControl Li et al. (2024) (245 apps), unseen ALFWorld tasks, and WebShop goals. Request formats follow Ling et al. (2026b) .

Phones
Model	Wrong	Obs.	Logged	Claim
decider-4b	95%	$+.05$	$\mathbf{+1.10}$	$+.80$
Kev-4B	96%	$+.09$	$+.20$	$\mathbf{+.32}$
decider-12b	84%	$+.21$	$\mathbf{+1.10}$	$+.56$
hosted Jev	88%	$+.61$	$\mathbf{+2.31}$	$+.95$
Embodied (ALFWorld)

Table 2: Which input moves the decision. Wrong : how often the model still stops, proceeds or buys after a failed step whose record claims success ( Logged+claim ). Obs. , Logged , Claim : how much the observation, the logged step and the explicit claim raise

\log p

of that option. Bold : the largest of the three. In every row the bold record effect exceeds the observation effect with a paired 95% CI above zero (Appendix Table 7 ).

	All		Claimed
Model	Head	Probe	Head	Probe
Phones
decider-4b	.63	$\mathbf{.77}$	.51	$\mathbf{.74}$
Kev-4B	.65	$\mathbf{.80}$	.55	$\mathbf{.79}$
decider-12b	.69	$\mathbf{.80}$	.61	$\mathbf{.77}$
hosted Jev	.75	–	.68	–

Table 3: Detecting whether the step worked (AUROC on held-out apps, tasks or goals; 0.5 is chance). Head : the model’s own output. Probe : a linear probe on its hidden state. All : all test states. Claimed : only states whose record claims success, so that only the observation differs. Jev is closed-weight and cannot be probed.

Figure 2: Activation patching on decider-4b: the share of each input’s effect on

\log p

of the terminal option that is restored by copying the decision-position state from the paired state at a single layer (mean

\pm

95% CI). The observation is present by mid-depth; the logged step and the claim arrive in the last layers. We keep pairs whose

\log p

differs by more than 0.2. In WebShop the page barely moves the decision (

+0.02

, Table 2 ), so there is no observation effect to locate and only the logged-click curve (

n=100

) is shown.

Panel of Figure 3 — Figure 3: EviGate on an illustrative phone step. The record says Wi-Fi is on; the screen shows it is off. The output follows the record and proposes stop . The gate reads the decision-position state at a middle layer $\ell$ of the same forward pass, before record-dependent influence dominates the last layers, and vetoes. It acts only on terminal options. Probabilities are illustrative.

Model	Own score	Second q.	EviGate	Wrong
Phones
decider-4b	.59	.60	$\mathbf{.71}$	72% $\to$ 38%
Kev-4B	.60	.61	$\mathbf{.72}$	58% $\to$ 38%
decider-12b	.63	.62	$\mathbf{.73}$	33% $\to$ 33%
hosted Jev	.66	.69	–	–
Embodied (ALFWorld)

Table 4: Gating terminal decisions on held-out groups: balanced accuracy (0.5 is chance). Own score : the model’s own probability of the terminal option, with a tuned threshold. Second q. : an added yes/no question asking whether the step worked. Wrong : failed steps that still end in a terminal decision, with the own score

\to

with EviGate .

EviGate $-$ no gate
Condition	Phones ∗	ALFWorld	WebShop
No gate	77.2	42.5	8.5
Second question	$\mathbf{83.2}$	$\mathbf{61.2}$	7.0
EviGate , fit on offline contrast sets	–	58.2	2.5
EviGate , fit on target-domain states	82.4	$\mathbf{61.2}$	$\mathbf{13.5}$
random veto, matched rate	77.6	58.2	4.5

Table 5: Task success (%) of live agents; same tasks and seeds in every row. Random veto: vetoes at random, about as often as the EviGate row above it (rates in the note below). Bottom: paired differences with 95% CIs over tasks.

			Fine-tuned on
Test domain		Base	phones	all 3	all 3
				1k each	all phones
Phones	Wrong	91%	$\mathbf{47\%}$	89%	$\mathbf{47\%}$
	AUROC	.50	$\mathbf{.84}$	.67	.82
ALFWorld	Wrong	90%	91%	$\mathbf{0\%}$	0.3%
	AUROC	.91	.92	$\mathbf{1.00}$	$\mathbf{1.00}$

Table 6: Fine-tuning decider-4b on paired states. Wrong : failed steps whose record claims success that still end in a terminal decision. AUROC : the output’s detection of the failure on claimed states. Tests use held-out apps, games and goals. Last rows: accuracy on two of decider’s held-out general tasks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Phones
Model	Logged $-$ Obs.	Claim $-$ Obs.	Obs. $\mid$ claim
decider-4b	$+1.05$ [0.94, 1.17]	$+0.74$ [0.69, 0.80]	$+0.00$ [ $-$ 0.01, 0.01]
Kev-4B	$+0.11$ [0.09, 0.14]	$+0.23$ [0.22, 0.25]	$+0.05$ [0.04, 0.06]
decider-12b	$+0.89$ [0.68, 1.09]	$+0.34$ [0.23, 0.46]	$+0.03$ [ $-$ 0.02, 0.09]
hosted Jev	$+1.70$ [1.33, 2.10]	$+0.34$ [0.22, 0.47]	$+0.14$ [0.11, 0.18]
ALFWorld

Appendix

Table 7: Paired 95% bootstrap intervals for the effects in Table 2 (differences in

\log p

of the terminal option). Decision points: phones 640 (decider-4b, Kev-4B) or 200 (decider-12b, Jev); ALFWorld 295; WebShop 300.

Model	Observation	Logged step	Claim
decider-4b	$+0.05$ [.02,.08]	$\mathbf{+1.10}$ [1.00,1.21]	$+0.80$ [.74,.85]
decider-2b †	$+0.02$ [ $-$ .02,.05]	$+0.48$ [.38,.58]	$\mathbf{+0.79}$ [.72,.87]
Kev-4B	$+0.09$ [.07,.10]	$+0.20$ [.18,.22]	$\mathbf{+0.32}$ [.31,.33]
Kev-0.8B †	$+0.00$ [.00,.01]	$+0.02$ [.01,.02]	$\mathbf{+0.21}$ [.20,.22]
Laya †	$-0.04$ [ $-$ .13,.06]	$-0.01$ [ $-$ .06,.04]	$\mathbf{+0.32}$ [.27,.38]
Kev-9B †	$\mathbf{+0.34}$ [.28,.41]	$+0.27$ [.22,.32]	$+0.24$ [.22,.27]

Appendix

Table 8: Paired effects on

\log p(\texttt{complete})

on phones, with 95% bootstrap confidence intervals. For models up to 4B, explicit claims exert a larger mean effect than screen evidence. The larger models differ: Kev-9B weights the observed outcome more than either record component individually, whereas decider-12b remains most sensitive to the logged action. Rows marked † use a 1,000-state subset.

	Output		Probe
Model	All	Claimed	All	Claimed	Gap	Text only
decider-4b	.63	.51	$\mathbf{.77}$	$\mathbf{.74}$	$+.14$ [.11,.17]	.59
Kev-4B	.65	.55	$\mathbf{.80}$	$\mathbf{.79}$	$+.15$ [.12,.17]	.59
decider-2b †	.59	.50	$\mathbf{.65}$	$\mathbf{.61}$	$+.06$ [.02,.09]	.59
Kev-0.8B †	.59	.50	$\mathbf{.64}$	$\mathbf{.59}$	$+.04$ [ $-$ .01,.09]	.59
Laya †	.52	.50	$\mathbf{.62}$	$\mathbf{.60}$	–	–

Appendix

Table 9: Detecting whether the phone step worked (AUROC on held-out apps), as in Table 3 . Gap : probe minus output on all states, with 95% CI. Text only : a classifier on the screen text alone.

Model	Model’s choice	Own score	Second q.	Contrastive	EviGate
decider-4b	.59	.59	.60	.60	$\mathbf{.71}$
Kev-4B	.59	.60	.61	.61	$\mathbf{.72}$
decider-2b	.57	.55	.59	.56	$\mathbf{.64}$
Kev-0.8B	.53	.56	.58	.56	$\mathbf{.61}$
Laya	.50	.53	.55	.49	$\mathbf{.59}$
Kev-9B	.66	.66	.68	.65	$\mathbf{.75}$

Appendix

Table 10: Gating the phone stop decision on held-out apps (balanced accuracy), as in Table 4 , for all seven models and one more detector. Model’s choice : no gate. Own score and Second q. : as in Table 4 . Contrastive : contrastive decoding against a pass without the screen Shi et al. (2024) . Bottom: failed steps that still end in complete , over both failed variants ( Logged and Logged+claim , 1,280 states); Table 2 counts Logged+claim only, hence its higher rates.

Model	Observation	Logged step	Claim	Output	Probe
Qwen3.5-4B-Base	$+.02$	$+.15$	$+.51$	.59	.65
Qwen3.5-4B (chat)	$+.05$	$+.28$	$+.42$	.61	.72
Kev-4B (LoRA)	$+.09$	$+.21$	$+.32$	.66	.77
decider-4b (full fine-tuning)	$+.01$	$+1.12$	$+.78$	.62	.75

Appendix

Table 11: The bias is already present in the base model: one Qwen3.5-4B backbone before (top) and after (bottom) decision tuning, 1,000 phone states. Left: paired effect on

\log p(\texttt{complete})

. Right: AUROC for detecting the failure from the output and from a probe.

Figure 4: Layerwise decoding of action outcomes. (a) Phone probe AUROC across relative depth, compared with the decision model’s output (grey band). (b) Model-output AUROC (hollow points) and probe AUROC (filled points) across domains; dashed grey indicates the input-text baseline. In ALFWorld and WebShop, standardized failure messages also allow a text-only classifier to reach AUROC 1.00.

Figure 5: How many labels the phone probe needs (held-out apps).

\times

: the output’s AUROC. Ten labelled episodes already beat the output for the decision-tuned models.

	ALFWorld	WebShop
Normal pass	59%	99.7%
Record removed from layer 12	26%	93%
Record removed from layer 16	50%	99.5%
Record removed from layer 20	59%	99.7%
EviGate	0.5%	0%

Appendix

Table 12: Failed steps that still get a terminal decision when the record is removed from layer

\ell

onward (decider-4b), compared with EviGate . Failed steps are both failed variants ( Logged and Logged+claim : ALFWorld 590, WebShop 600 states); Table 2 counts Logged+claim only.

	Phones		ALFWorld		WebShop
	Failed pass	Good vetoed	Failed pass	Good vetoed	Failed pass	Good vetoed
No gate	78%	0%	59%	0%	99.7%	0%
Record removed from the input	2%	93%	9.5%	13%	6.0%	7.5%
EviGate	25%	27%	0.5%	0%	0%	0.5%

Appendix

Table 13: Removing the record from the input vs. EviGate (decider-4b, paired states). Failed pass : failed steps that still end in a terminal decision. Good vetoed : steps that worked but whose terminal decision is blocked. Failed and good steps include both variants with and without the claim (phones: the 1,000-state subset, 400 failed and 400 good states; ALFWorld 590 and 590; WebShop 600 and 600).

Layer	Phones	ALFWorld	WebShop
	obs / logged / claim	obs / logged / claim	obs / logged / claim
28	23 / 6 / 5	7 / 4 / 2	5 / 1 / 3
32	66 / 42 / 38	50 / 31 / 22	46 / 10 / 22
36	82 / 68 / 67	70 / 50 / 40	69 / 32 / 44
42	95 / 85 / 85	87 / 74 / 67	88 / 63 / 73

Appendix

Table 14: Activation patching in decider-12b (48 layers). Each cell is the share (%) of an input’s effect on the decision that is restored by patching the decision-position state at that layer. Obs = the screen or observation; logged = the logged step; claim = the false success claim. Before layer 26 every share is at most 2%.

States	Batch	Engine	+ EviGate	Overhead
T3A ( $\sim$ 2.7–3.4k tokens)	1	363.4	368.3	$+1.35\%$
	8	3870.2	3873.8	$+0.09\%$
	32	19503.6	19512.3	$+0.04\%$
Short ( $\sim$ 280 tokens)	1	39.7	40.5	$+2.01\%$
	8	214.9	217.8	$+1.32\%$
	32	829.3	831.4	$+0.26\%$

Appendix

Table 15: Decision latency (p50, ms) of decider-4b in its production engine, with and without EviGate (30 iterations after warm-up). The p99 latency of the base engine is within 0.3–0.7% of p50 for T3A states and 0.7–4.5% for short states, so overheads below about 1% are within measurement variability.

Condition	Success	Gain	False stops	Calls / episode	Calls / success
No gate	77.2	–	13.9	3.80	4.92
Random veto, second question’s rate	77.5	$+0.3$ [ $-$ 0.3, 0.8]	13.3	3.99	5.14
Random veto, EviGate ’s rate	77.6	$+0.4$ [0.0, 0.8]	13.3	3.98	5.12
Contrastive decoding	77.1	$-0.1$ [ $-$ 9.7, 7.9]	8.2	8.93	11.58
Second question	83.2	$+6.0$ [1.7, 11.2]	7.8	5.07	6.10
LLM judge (9B)	81.5	$+4.3$ [1.7, 7.6]	9.3	5.04	6.18

Appendix

Table 16: Phone simulator, all conditions (24 tasks

\times

30 seeds, decider-4b, EviGate with the balanced threshold). Gain : paired change in success vs. no gate, with 95% CI over tasks. False stops : episodes that stop before the goal is reached.

$N$	1	2	4	8	16
Oracle	9.8	16.9	25.9	35.8	44.0
Random attempt	9.8	14.1	17.2	19.8	21.2
Decision model’s own judgement	9.8	15.2	20.4	24.9	28.0
Probe on its hidden state	9.8	16.5	24.3	32.8	40.0

Appendix

Table 17: Best-of-

N

success (%) on 50 WebShop goals, using 16 sampled agent trajectories per goal. Each non-oracle strategy selects among candidate attempts using the indicated ranking criterion; results average over 300 random subsets per goal. Strategies choose only among attempts that end in a purchase, and a goal counts as failed when none of its

N

attempts buys; this is why even random selection improves with

N

. Confidence intervals resample goals.

AI Agent Reliability

AI agent reliability studies whether tool-using systems act correctly, consistently, safely, and recoverably across changing tasks and environments.

See all 339 papers

Dual-Frontier: When Can an Agent Trust Its World Model?
Sep 22, 2026Huatai Zhu, Qiang Chen, Ziqian Kou +5World Model-Based Planning Model-Based Planning
Central South University · The Hong Kong University of Science and Technology · Xiangjiang Laboratory +2
The Delegation Blind Spot: Auditing Product Decisions from Agent Choices
Sep 22, 2026Shivam GuptaAI Agent Auditing Partial Identification
Independent Research
When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows
Oct 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1LLM Agent Workflow Optimization LLM Agent Reliability
Nanjing University · Technical University of Munich

Sep 22, 2026J/K move · Enter open · S save

Dual-Frontier: When Can an Agent Trust Its World Model?

Huatai Zhu, Qiang Chen, Ziqian Kou, Wenhao Li+4

Central South University · The Hong Kong University of Science and Technology · Xiangjiang Laboratory+2

World Model-Based Planning Model-Based Planning World Model Evaluation Counterfactual Learning Model-Based RL

Open paper AskPDF arXiv

Learned world models are becoming essential to general-purpose agents: by predicting action consequences, they support planning and decision-making while reducing reliance on costly trial and error. This reliance creates a fundamental ambiguity: when a world-model-guided decision fails, the trajectory alone may not reveal whether the agent's decision rule or the world model caused the loss. We formalize this failure-attribution problem as a counterfactual decomposition of return loss and prove that its components are not identifiable from passive interaction, even for finite-horizon planners. This obstruction motivates Dual-Frontier, a learning principle that admits a world-model-guided decision only when its predicted advantage exceeds a certified bound on decision-relevant world-model error; otherwise, evidence is allocated to world-model verification. Action-conditioned value bounds and a closed-loop extension guarantee non-decreasing return for admitted decisions. Calibrated gates and simultaneous confidence sequences support adaptive evidence reuse, with sufficient and necessary verification bounds. Controlled learned-model experiments validate the predicted failure modes and certification behavior, while cross-backbone tool-use benchmarks instantiate the same verify-then-promote rule in realistic agent world-model pipelines, consistently improving decision quality and reliability.

Abstract

AI Agent Reliability

Explore similar work

Dual-Frontier: When Can an Agent Trust Its World Model?

The Delegation Blind Spot: Auditing Product Decisions from Agent Choices

When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows