Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

Yubo Li; Ramayya Krishnan; Rema Padman

cs.AIOct 13, 2026

Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

Authors: Yubo Li, Ramayya Krishnan, Rema Padman

Organizations: Carnegie Mellon University

Abstract

System One models choose an agent's next action by scoring a supplied set of options in a single pass. This makes them fast and cheap, but when they choose wrongly there is nothing to inspect: did the model misread what the agent has seen, or read it correctly and still act wrongly? We answer behaviorally. On a frozen benchmark of agent decision states from simulated environments, customer-service policies, tool use, web navigation and coding, we pair every next-action question with factual probes about the same state, each answered independently. Across seventeen fast systems and a reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form. These errors are not noise; they gather on the same states across systems of every kind. Where the probed facts provably determine the action, models that read every fact still fail to combine them, and their own answers, composed by code, repair the failure. On real states, what is missing is often not a fact but the next sub-goal.

Figures & tables

Figure 1: Paired measurement on a real ALFWorld state. Left: the goal, the history and all 31 admissible actions; the gold action follows only from what the trace reveals. Middle: the decision and each factual probe are answered independently on the same state (JEV’s answers: every probe right, action wrong). Right: crossing all facts correct (

U

) with action correct (

D

) gives four outcomes; this state is facts right, action wrong.

				Latency (ms)
System	Model	Readout	Input limit	p50	p95
System One models, hosted API
JEV 1.13 ( TypeSafe AI, 2026 )	hosted	native distribution	32k	205.7	290.5
OpenAI Decisions ( OpenAI, 2026a )	hosted (public beta)	native distribution (two decimals)	none documented	150.6 ∗	197.6 ∗
System One models, open weights
Clef-27B ( Cloudflare, 2026 )	Qwen3.8-27B, post-trained	native schema head	32k †	130.9	1,990.5

Table 1: Systems by type, with readouts, input limits and median and 95th-percentile latency on the latency panel (§ 4 ; local systems on one H100). ∗ Estimated from sequential accuracy runs. † Our explicit budget (release defaults: 16k for Clef-27B and Eikos-27B, 8k for SimpleJev-27B, 256k for Torchcast-27B; CLM-8B’s serving example uses 2k). ‡ Applied by the author’s runtime: Decider-chat-31B cuts states beyond 4,096 tokens; Quyet-1.0-Large cuts states to at most 6,000 tokens and refuses more than ten options. § The server fails on the longest states on one 80 GB H100. ¶ Synchronous API, one question per request, an item’s questions sent in parallel.

Figure 2: Paired decision accuracy against (a) cost per 1,000 decision states and (b) median latency. The step line is the efficient frontier. Hosted costs are measured billing; self-hosted costs assume $2.50 per H100-hour on the latency panel, an upper bound for batched serving (§ 4 ). Markers give the system category; Laya, Quyet-1.0-Large and CLM-8B lie below the axis break.

System One models, hosted API
	Accuracy		Joint outcome (% of states)				$P(D{=}1\mid U)$		Errors with	Coverage
System	$D$	$U$	U1D1	U1D0	U0D1	U0D0	$U{=}1$	$U{=}0$	$U{=}1$ (%)	(%)
JEV	0.763	0.833	66.0	17.3	10.4	6.4	0.792	0.619	73.1	100.0
OpenAI Decisions	0.706	0.789	58.2	20.6	12.4	8.7	0.738	0.588	70.3	99.6
System One models, open weights
Eikos-27B	0.795	0.850	69.1	15.9	10.4	4.6	0.813	0.695	77.7	100.0

Table 2: Paired outcomes on the 19,663 paired test states (pooled counts).

D

: decision accuracy;

U

: share of states with every measured fact correct. The last two columns give the share of errors made with

U{=}1

and coverage (states with every answer supported). Bold: best fast system. ∗ Reasoning reference. † Coverage below 95%.

Figure 3: Paired outcomes on the 19,663 paired test states. (a) Share of states in each outcome; the white number is the share with every fact right but the wrong action. (b) Decision accuracy against the share of errors made with every fact right; marker shape gives the system category. The more accurate the system, the larger this share.

Figure 4: (a) Decision accuracy and (b) share of errors made with every fact right, overall and by domain (number of states under each label; “Visible fields”: visible-field probes, § C.2 ). Hatched: fewer than ten errors.

Figure 5: Cost of every wrong decision in the three simulators, by exact lookahead (extra steps added to the shortest remaining plan). Left of zero: at most one extra step; right: two or more, or unrecoverable. Upper bar: errors with every fact right; lower bar: with a wrong fact;

n

: wrong decisions. Systems below 0.5 accuracy or failing the coverage rule are omitted.

Figure 6: (a) Paired test states on which

k

of the 14 competent fast systems make a facts-right, action-wrong error, against a within-cell permutation null (ticks; labels give observed/null). (b) Accuracy of each competent system on states that

k

of the other thirteen fail (thin lines, by category) and of GPT-6.1 by

k

(dashed).

Figure 7: The diagnostic studies. Each sends the decision again with information added, as separate requests on the same states: B on generated states whose probes determine the action, C with the benchmark’s probes on real states, D with probes derived for the decision on ALFWorld. Green requests use the system’s own earlier answers.

Figure 8: States whose probes are sufficient by construction (1,200 test states). (a) Probe accuracy and decision accuracy under each input, and the gap from raw trace to facts only (points). (b) Raw-trace errors by type (

n

: errors). (c) Change vs. the raw trace for rewrites with the same facts; bold: 95% interval excludes zero. Header colours follow Figure 7 .

System One models, hosted API
		Change vs. alone (points)
System	Alone	+questions	+own	+correct
JEV	0.806	+1.8	+2.1	+4.0
		[+0.4, +3.2]	[+0.7, +3.3]	[+2.6, +5.5]
OpenAI Decisions	0.722	+1.7	+3.2	+5.0
		[+0.0, +3.5]	[+1.6, +5.0]	[+3.4, +6.7]

Table 3: Probe-then-decide on 2,783 paired test states: accuracy alone and change (points) after the probe questions, the system’s own answers or the correct answers, with 95% intervals below; gray: interval includes zero. Refused requests count as wrong (Perplexity Decider-27B: 268 states, 8k-token limit per question). † Quyet-1.0-Large: the 547 states it answers.

Figure 9: Adding information to real states: decision accuracy alone (gray) and the change when information is added (points; blue gain, orange loss; bold: 95% interval excludes zero). (a) Benchmark probes on 2,783 paired states. (b) Decision-oriented blocks on 644 ALFWorld states; outlined: the largest gain in each row. Header colours follow Figure 7 .

System One models, hosted API
		Change vs. decision alone (points)
		Status block with correct values					With own answers
System	Decision alone	Benchmark probes	Decision facts	State digest	Next sub-goal	Facts + sub-goal	Facts	Facts + sub-goal	Own sub-goal accuracy
JEV	0.716	+5.7	+5.3	+4.7	+16.3	+21.3	+4.8	+6.4	0.668
OpenAI Decisions	0.767	+4.0	+0.9	+1.4	+11.0	+7.9	+1.1	−2.6	0.775
System One models, open weights

Table 4: Decision-oriented blocks on 644 ALFWorld states: accuracy alone, change (points) with each block (correct values unless marked own) and accuracy on the sub-goal question. Gray: 95% interval includes zero. Quyet-1.0-Large is omitted (answers 2 of 644).

Figure 10: Controlled variations. (a) Accuracy with the full trace (ring) and with the history removed (dot). (b) ALFWorld accuracy by trace length, for states whose needed object was already seen and states that require exploration. (c) Matched edit triples: share of relevant edits that change the decision (filled) and of irrelevant edits that keep it (ring). Systems failing the coverage rule are marked or omitted.

Figure 11: Calibration of native decision distributions on test decisions. (a) Reliability diagrams; marker area is the share of decisions in the bin. (b) Expected calibration error (15 bins; hatched: coverage below 95%) and accuracy against mean confidence. GPT-6.1 is excluded (verbalized probabilities).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$U$	$D$	Directly observed	Not licensed
1	1	Measured facts and action both correct	That all relevant facts were understood
0	0	Some probe wrong; action wrong	That the probe error caused the wrong action
1	0	All measured facts correct; action wrong	That the failure lies purely in deciding
0	1	Some probe wrong; action correct	That the model used a shortcut, or that the wrong fact mattered

Appendix

Table 5: The four observable outcomes on a paired state and the conclusions each does and does not support.

		Records
Domain	Sources	Test	Paired test	Dev	Decision / probes
Agent decision states
Household (sim.)	ALFWorld unseen and seen games	9,097	9,097	840	admissible commands; location, holding, last outcome, progress
Grid world (sim.)	BabyAI-Text (BALROG)	2,181	2,181	236	6 actions; orientation, holding, target position, last outcome
Text game (sim.)	TextWorld cooking (TALES levels)	2,400	2,400	300	admissible commands; room, holding, ingredient state, item location
Customer service	$\tau^{2}$ -bench airline, retail	3,739	3,739	330	enumerated tool calls; record fields, policy constraints

Appendix

Table 6: Composition of the frozen benchmark (65,228 records). “Paired test” counts test states that carry factual probes. Dev records are reported separately and never pooled with test results. Household includes the targeted-coverage cohort; grid-world and text-game rows include newly generated confirmation cohorts and, for TextWorld, native rule changes.

		Label disagrees with the text
Group	Records	Location	Seen-at	Extra gold
A1	3,000	140	23	232
A2	300	11	0	47
A3	300	22	0	19
A4	1,500	54	20	0
A5	997	57	70	58

Appendix

Table 7: ALFWorld test records whose labels disagree with the trace text because receptacles share a simulator location. Location: the probe names a co-located receptacle. Seen-at: the sighting differs. Extra gold: the gold also contains an already searched receptacle (lenient).

System	All	Clean	Flagged
Gemma-4-31B	0.810	0.885	0.251
Qwen3.8-27B	0.792	0.864	0.295
Cygnet	0.776	0.834	0.334
Clef-27B	0.682	0.751	0.212
JEV	0.663	0.707	0.364

Appendix

Table 8: Share of ALFWorld decision errors made with all measured facts correct, on all 8,197 paired ALFWorld test records, on the 7,799 records without a location or seen-at disagreement, and on the 398 flagged records.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

System Switch: When Should a Fast Decision Model Stop and Think?
Oct 7, 2026Gian Luca BailoCascaded Inference LLM Decision-Making
Independent Researcher, Recco (Genoa), Italy
System1-Arena: Which Decisions Can Be Left to System 1?
Oct 15, 2026Tianyu Chen, Xinyang Liu, Ruyi An +2LLM Decision-Making LLM Evaluation
The University of Texas at Austin
The belief-action gap in language model agents
Oct 12, 2026Ayse Sila Okcu, Weisheng Wang, Tongmeng Xie +4LLM Decision-Making
University of Cambridge · Utrecht University · University of York +3

Oct 7, 2026J/K move · Enter open · S save

System Switch: When Should a Fast Decision Model Stop and Think?

Gian Luca Bailo

Independent Researcher, Recco (Genoa), Italy

Cascaded Inference LLM Decision-Making Selective Prediction VLM Reasoning Language Model Calibration

Open paper AskPDF arXiv

Dual-process agents pair a fast policy with a slow deliberative model. In real-time settings the slow model usually runs continuously; in turn-based agents and robot planners it is invoked on events such as uncertainty or a detected failure. We study a fast learned actor that takes every decision and hands control to a reasoning vision-language model only when a gate opens, while the game keeps running. We use closed-loop Doom and the new open "System One" typed-decision models, served through a common llama.cpp interface. On 900 held-out questions, (i) zero-shot decision models from 0.15B to 9B parameters choose to collect items 1.6-1.8 times more often than chance among their errors, in any option order, although the order changes some models' accuracy; (ii) accuracy, calibration and sensitivity (how well confidence separates right from wrong answers) are distinct: models of similar accuracy differ widely in AUROC, and the confidence of the most sensitive one tracks which kinds of situation it fails, not which answers are wrong; (iii) offline, deferring the least confident 30% of decisions to a reasoning model gains over random deferral in proportion to the actor's AUROC (rank correlation 0.87); with actor and rate chosen on held-out games the gain is +0.13 [0.08, 0.18] with doomLaya's option order and +0.08 [0.02, 0.14] with shuffled options, and reasoning carries about half of it; (iv) in closed loop (33 games, three seeds) no variant reaches the exit. Committing to plans, the reasoner's or a fixed explore rule's, opens more doors and makes an actor that stands still play; with the rule the agent dies more often. Told that some doors need keys, the reasoner takes ordinary doors for locked ones, which the state cannot tell apart; without that knowledge it goes back to collecting. We release code, prompts, data and logs.

Abstract

LLM Decision-Making

Explore similar work

System Switch: When Should a Fast Decision Model Stop and Think?

System1-Arena: Which Decisions Can Be Left to System 1?

The belief-action gap in language model agents