Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol

Guodong Xu

cs.AIAug 21, 2026

Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol

Authors: Guodong Xu

Organizations: Qingdao Guodongxiansheng Network Technology Co., Ltd. (Guˇodòng X¯ıansheng)

Abstract

Language-model agents can improve after failure or carry text across episodes without revising what counts as success. We study the narrower attribution problem of criterion revision: when criterion K0 accepts an outcome violating a broader commitment B, what observations justify saying that the system formed and persistently used K1? We require five non-compensatory conditions: criterion-failure detection, a model-emitted proposal, new-episode transfer, intervention sensitivity on the claimed carrier, and preservation. We evaluate CMB-0.1 on twelve cross-domain cases and four arms: stateless inference, append-only history, model-generated but harness-committed state, and evaluator-written oracle state. Seven mechanism fixtures yield 84 deterministic scorer trials; four local quantized artifacts yield 96 calls and 192 model-case-arm trials. No model trial satisfies all five conditions, but this zero does not establish general capability absence. Eleven calls remain invalid after one retry; several commitments disclose the target distinction; the harness performs commits; deletion reuses a stateless call; and conflict changes multiple factors. Qwen2.5-7B answers every transfer and preservation item without revision state, exposing zero-state reconstruction. These failures make CMB-0.1 an instrument-calibration result rather than a model ranking. We derive a prospective, trace-anchored CMB-0.4 protocol requiring concealed transfer, explicit WRITE/NO-WRITE/ESCALATE actions, a separately logged policy-selected commit, matched interventions, repeated hidden items, and a frozen executable oracle. It is a successor design, not a completed confirmatory result. The paper contributes a measurement chain, an empirical diagnosis of its first implementation, and a more discriminating protocol for future tests of criterion revision.

Figures & tables

Version	Status in this manuscript	Permitted inference
CMB-0.1	Executed calibration: 84 deterministic scorer-fixture trials and 192 materialized local-model trials.	Instrument diagnosis and bounded observations about the four tested artifacts and twelve cases; no family ranking or general capability-absence claim.
CMB-0.4	Prospective protocol derived from the observed CMB-0.1 failure modes. No CMB-0.4 model-performance, write-selection, transfer, or causal-effect result is reported.	A falsifiable design for a future confirmatory study; no empirical inference about a tested model or model family.

Table 1: Evidence status of the instrument and its successor protocol.

Panel of Figure 1 — Figure 1: Actual CMB-0.1 calibration flow. The orange boxes are harness interventions, not model-selected actions. The diagram makes explicit why CMB-0.1 can calibrate an attribution instrument but cannot by itself demonstrate autonomous criterion writing.

Domain	Operative $K_{0}$	Trap admitted by $K_{0}$ but rejected by $B$
Evidence	Every claim has a citation	The citation is genuine but does not entail the claim.
Software	All provided tests pass	A contract-boundary input is still mishandled.
Security	Requested permission is valid and executable	The permission is excessive for the feature.
Scheduling	Choose everyone’s earliest available time	Severe burden is repeatedly assigned to one person.
Data	Required fields are nonempty	A present numeric value is semantically impossible.
Retrieval	All query terms occur	The passage answers a different relation.

Table 2: The twelve CMB-0.1 cases.

Fixture	$D$	$E$	$P$	$C$	$V$	Candidate
Fixed- $K_{0}$ executor	0	0	0	0	1	No
Text-only reflector	1	1	0	0	1	No
Evaluator-injected $K_{1}$	1	0	1	1	1	No
Complete self-reviser	1	1	1	1	1	Yes
Overreviser	1	1	1	1	0	No
History-carried reviser	1	1	1	1	1	Yes

Table 3: Mechanism fixtures for scorer validation. Entries are dimension pass values aggregated across all twelve cases.

Artifact	Parameter label	Quantization
Llama 3.2	1.2B	Q8_0
Gemma 3	4.3B	Q4_K_M
Qwen 2.5	7.6B	Q4_K_M
DeepSeek-R1-Distill-Qwen	7.6B	Q4_K_M

Table 4: Local artifacts used for calibration. Parameter counts and quantization labels are those reported by the local runtime.

Artifact	Parseable calls	Revision batches	Transfer calls	Complete calls
Llama 3.2	18/24	3/3	0/21	3/24
Gemma 3	24/24	2/3	0/21	2/24
Qwen 2.5	24/24	3/3	21/21	24/24
DeepSeek-R1-Distill-Qwen	19/24	0/3	0/21	0/24

Table 5: Structured-output coverage. “Complete” requires every case and required field in the batch.

Artifact	Arm	$D$	$E$	$P$	$C$	$V$
Llama 3.2	Stateless	3	0	1	0	1
	Append-only	3	0	0	0	0
	Harness-committed	3	0	2	1	2
	Oracle-written	3	0	1	0	1
Gemma 3	Stateless	5	5	0	0	0
	Append-only	5	5	0	0	0

Table 6: Dimension-pass counts out of twelve cases.

Field	Archived content
Evaluator $B$	Material claims must be supported by sources that entail them.
Evaluator-fixed reference $K_{1}$	Accept only if every material claim has a citation whose content entails that claim.
Qwen proposal	Accept an answer if every material claim has at least one relevant citation.
Qwen stateless transfer	Reject non-entailing source; accept direct support; reject missing citation.

Table 7: Exact citation-case trace from the archived specification and Qwen calibration response.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Artifact	SHA-256
benchmark-spec.v1.json	9258fb5c9fb7c1e034ccad710710f32a b270792a6e6b4a463b506438c7dfba17
construct-validation. v1.json	05fd5f0daa42e28431e4e2a2aeb4e41 e45643da058961f1c1503a695cffb455b
local-pilot-plan.v1.json	a683c042de13abe2fe3d96e4abba0d1c bb8fbf6ac9cdb0a7b8a848b34e938207
local-pilot-results.v1.json	6a1085a0a835ef037458a7ab9520c541 c9b7471d2afb46167faf1a9fdddb3ca4

Appendix

Table 8: Archived artifact hashes for the unchanged CMB-0.1 calibration.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,217 papers

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents
Oct 26, 2026Nghi D. Q. BuiLLM Sycophancy Behavioral Consistency of LLM Agents
Google
Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
Oct 3, 2026Nijesh Upreti, Chris Sypherd, Vaishak BelleLLM Agent Evaluation LLM Agents
The University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
May 3, 2026Xiao JiaAI Agent Reliability LLM Agent Evaluation
School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong, Shenzhen

Oct 26, 2026J/K move · Enter open · S save

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents

Nghi D. Q. Bui

Google

LLM Sycophancy Behavioral Consistency of LLM Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Language model agents often abandon correct work when a user challenges it, yet simply telling them to stand firm makes them ignore real mistakes. Once an agent reads a complaint, it tends to discount its own earlier checks in order to agree with the user. We introduce Thinking-in-Bets, a training-free method in which an agent writes a short private record of its main claims, the checks supporting them, and its confidence before reading user feedback. When challenged, the agent consults this Claim Checklist to keep verified work intact while investigating uncertain or newly challenged parts. We evaluate five Gemini models (3.5 Flash-Lite, 3.5 Flash, 3.6 Flash, 3.8 Flash, and 4.0 Argon) across eight benchmarks: SWE-Together, SWE-Interact, and DeepSWE for software engineering, Agents' Last Exam for professional reports, Humanity's Last Exam and SycEval for academic reasoning, Vals Finance for financial analysis, and Terminal-Bench for command-line tasks. Recording claims before criticism nearly eliminates false reversals in patch review and protects passing report drafts under mistaken pushback, whereas writing the same record after reading the complaint recovers very little of the gain. In multi-turn coding, pre-reply claim records substantially reduce unnecessary file edits after mistaken complaints, and gating the record so it is written only when the agent is uncertain lowers token overhead while fixing more real bugs than the baseline. Even in non-interactive tasks without user feedback, listing the key claims in a draft and re-checking the weakest one before submission improves both answer accuracy and confidence estimates across all tested models.

Abstract

LLM Agent Evaluation

Explore similar work

Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents

Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents

NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles