When History Fails to Become Experience: Action Calibration in Language Agents

Jingyu Liu; Zhiwen Wang; Yuxin Jing; Huanyu Zhou; Yong Liu

cs.CLOct 2, 2026

When History Fails to Become Experience: Action Calibration in Language Agents

Authors: Jingyu Liu, Zhiwen Wang, Yuxin Jing, Huanyu Zhou, Yong Liu

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · ByteDance · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE

Abstract

Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we examine how agents use history. We find that history improves task completion overall, yet much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past actions and selectively records experience to guide subsequent decisions, improving task success beyond outcome labeling alone.

Figures & tables

Figure 1: Action calibration within an ongoing task. (A) Ordinary history simply retains the observation returned by the environment. (B) Outcome annotates the observation as the outcome of the previous action. (C) A calibrator selectively retains experience to guide later decisions.

Figure 2: Recovery from the same restored state. For a failed trajectory of length

L

, Panel A contrasts Raw (

L/3

history) with Reference (adding the recorded

L/3

–

2L/3

segment, not executed in the restored environment). Both conditions resume at

L/3

with a budget of 30 new actions, using the same 1,000 source trajectories. Panels B–C pool recovery by environment and model, respectively.

Figure 3: Prompt lengths at the first request after resumption, for 1,000 source trajectories. Panel A compares Raw with Reference; Panel B shows Reference medians (diamonds) and 10th–90th percentiles (lines) by environment. Both horizontal axes are logarithmic. The dashed line at 16,384 tokens is a visual reference, not a model context limit.

Current observation $O_{7}$ is retained in all five conditions.
Condition	Earlier history	Recent history
Raw	$(O_{1},A_{1}),(O_{2},A_{2}),(O_{3},A_{3})$	$(O_{4},A_{4}),(O_{5},A_{5}),(O_{6},A_{6})$
Recent-3-Only	Entire entries omitted	$(O_{4},A_{4}),(O_{5},A_{5}),(O_{6},A_{6})$
Observation-Only	$O_{1},O_{2},O_{3}$	$O_{4},O_{5},O_{6}$
Shuffle-All	$(O_{1},{\color[rgb]{0.85,0.425,0}A_{4}}),(O_{2},{\color[rgb]{0.85,0.425,0}A_{5}}),(O_{3},{\color[rgb]{0.85,0.425,0}A_{6}})$	$(O_{4},{\color[rgb]{0.85,0.425,0}A_{1}}),(O_{5},{\color[rgb]{0.85,0.425,0}A_{2}}),(O_{6},{\color[rgb]{0.85,0.425,0}A_{3}})$
Shuffle-Old	$(O_{1},{\color[rgb]{0.85,0.425,0}A_{2}}),(O_{2},{\color[rgb]{0.85,0.425,0}A_{3}}),(O_{3},{\color[rgb]{0.85,0.425,0}A_{1}})$	$(O_{4},A_{4}),(O_{5},A_{5}),(O_{6},A_{6})$

Table 1: Five history views at the same point in an interaction.

O_{t}

is an observation; action

A_{t}

responds to it and produces

O_{t+1}

. Orange denotes reassigned actions; the permutations are illustrative.

By environment
Group	Raw	Recent-3- Only	Observation- Only	Shuffle-All	Shuffle-Old
ALFWorld	67.08	52.44	61.88	64.64	68.08
ScienceWorld	39.12	31.88	35.88	34.84	38.56
WebShop	27.72	27.28	20.68	25.44	27.88
AppWorld	51.64	32.36	45.28	50.44	52.48
By model

Table 2: Task success (%) under history interventions. The two blocks summarize the same evaluation grid: 2,500 tasks per environment, 2,000 per model, and 10,000 overall, per condition. Table 4 gives the full breakdown.

Figure 4: Task-success changes from the matching model–environment Raw baseline, in percentage points (500 tasks per cell). Both panels use the same color scale. The final row and column give environment and model means; the lower-right cell gives the pooled change over 10,000 episodes per condition.

Figure 5: Probability of repeating the same action a third time. Each model receives the same

N

histories in all three conditions, ending in two identical actions and screened for feedback that does not support another repetition. Lower is better.

Figure 6: An outcome restatement and an unsupported inference in actor-generated experience. Highlighting marks the repeated fact or unsupported rule. Quotations come from real ALFWorld trajectories; the second experience is excerpted.

Figure 7: Full-rollout success (%) for Raw, Causal Transition, and Calibrator. Panel A pools 2,000 tasks per actor; Panel B pools 2,500 per environment (500 per model–environment cell). Labels show Calibrator gains over Raw in percentage points; Overall pools all 10,000 tasks per condition.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$N$	Raw	Causal	Experience
GPT-5.6-Sol	46	4 (8.70%)	3 (6.52%)	2 (4.35%)
DeepSeek-V4-Pro	131	33 (25.19%)	30 (22.90%)	25 (19.08%)
Qwen3.5-Plus	188	48 (25.53%)	41 (21.81%)	36 (19.15%)
Qwen3-32B	492	93 (18.90%)	77 (15.65%)	84 (17.07%)
Qwen3-8B	958	492 (51.36%)	434 (45.30%)	241 (25.16%)
Pooled	1,815	670 (36.91%)	585 (32.23%)	388 (21.38%)

Appendix

Table 3: Next-action repetition on fixed histories. Entries show repeated-action counts with rates in parentheses.

N

is the shared number of histories for each condition. Figure 5 plots these rates.

Model	Environment	Raw	Recent-3- Only	Obs.- Only	Shuffle- All	Shuffle- Old	Causal	Exp.
GPT-5.6-Sol	ALFWorld	448	421	456	456	453	449	441
	ScienceWorld	403	338	384	364	400	410	394
	WebShop	188	171	143	169	186	190	176
	AppWorld	427	274	394	426	430	437	421
DeepSeek-V4-Pro	ALFWorld	350	280	318	327	349	357	346
	ScienceWorld	293	235	267	255	285	297	282

Appendix

Table 4: Complete diagnostic success counts, out of 500 tasks per model–environment cell. Totals aggregate 10,000 episodes per condition. “Obs.-Only” denotes Observation-Only; “Exp.” denotes Experience.

Model	Environment	Raw	Causal	Check
GPT-5.6-Sol	ALFWorld	448	449	467
	ScienceWorld	403	410	410
	WebShop	188	190	195
	AppWorld	427	437	417
DeepSeek-V4-Pro	ALFWorld	350	357	359
	ScienceWorld	293	297	305

Appendix

Table 5: Task success with an explicit history-check reminder. Entries are successful-task counts out of 500 per model–environment cell. Overall totals pool 10,000 tasks per condition; the final row gives the corresponding success rates. Causal denotes Causal Transition.

Model	Environment	Valid reviews	Experience	NO_UPDATE
GPT-5.6-Sol	ALFWorld	5,848	4,182 (71.51)	1,666 (28.49)
	ScienceWorld	7,044	4,723 (67.05)	2,321 (32.95)
	WebShop	3,453	1,886 (54.62)	1,567 (45.38)
	AppWorld	5,636	5,482 (97.27)	154 (2.73)
DeepSeek-V4-Pro	ALFWorld	8,605	1,980 (23.01)	6,625 (76.99)
	ScienceWorld	8,757	2,471 (28.22)	6,286 (71.78)

Appendix

Table 6: Experience generation and NO_UPDATE decisions across five actors and four environments. Entries give counts with percentages of valid review responses in parentheses. Each cell covers 500 tasks; Overall pools responses across all 10,000 episodes.

Figure 10: LLM-judged changes in next-action quality with candidate experiences generated by GPT-5.6-Sol. Bars compare actions generated by each recipient actor with and without an experience on the same history. Rates use all completed comparisons, including ties; ties are omitted from the bars.

Case	Task evidence and retained experience	Subsequent behavior
Restatement ALFWorld Qwen3-8B	The task requires cleaning an egg and placing it in a fridge. put egg 2 in bowl 1 returns “Nothing happens.” Experience: “Putting the egg in a bowl does not successfully place it in the fridge.”	The action and identical experience recur at steps 11–13, with earlier judgments visible. Repeated writes restate an already recorded failure.
Unsupported claim WebShop Qwen3.5-Plus	A 3.4-oz, paraben-free shampoo below $40 is requested. The opened page shows a 3.5-oz product at$ 29.99 but does not verify the paraben-free attribute. Experience: “The selected 3.5 oz dry shampoo is paraben-free and priced under $40, confirming that product detail pages provide the necessary attribute verification to satisfy the user’s specific constraints.”	The actor buys the product next without inspecting features or description. The judgment overlooks the size mismatch and endorses an unverified attribute.
Verification WebShop Qwen3.5-Plus	A black, non-slip PU leather organizer below $80 is requested. The page verifies color, material, and a$ 45.99 price, but not non-slip functionality. Experience excerpt: “… though non-slip functionality requires further verification via features or description.”	The actor selects black, then inspects features at step 3 and description at step 5. The experience notes that the non-slip property still needs verification.

Appendix

Table 7: Examples of retained judgments and subsequent actions, with verbatim experience excerpts.

Model	Setting	ALFWorld	ScienceWorld	WebShop	AppWorld
GPT-5.6-Sol	Raw	89.60	80.60	37.60	85.40
	Causal Transition	89.80	82.00	38.00	87.40
	Calibrator	94.20	91.80	42.40	87.00
DeepSeek-V4-Pro	Raw	70.00	58.60	37.00	86.60
	Causal Transition	71.40	59.40	37.40	87.00
	Calibrator	75.00	67.40	43.40	84.20

Appendix

Table 8: Method-evaluation success rates (%), with 500 tasks per cell. The three conditions each contain 10,000 episodes.

Model	Training data	ALFWorld	ScienceWorld	WebShop	AppWorld
GPT-5.6-Sol	Content screening only	452	451	204	424
GPT-5.6-Sol	Full Calibrator	471	459	212	435
DeepSeek-V4-Pro	Content screening only	364	312	205	417
DeepSeek-V4-Pro	Full Calibrator	375	337	217	421
Qwen3.5-Plus	Content screening only	397	197	182	293
Qwen3.5-Plus	Full Calibrator	402	189	193	305

Appendix

Table 9: Training-data ablation of paired next-action comparisons. Entries are successful-task counts out of 500 per model–environment cell. Content screening only retains GPT’s review of the experience text but omits the subsequent action comparisons; Full Calibrator uses both stages. The final two rows pool 10,000 tasks per condition.

Model	Environment	Raw	Recent-3 Only	Obs. Only	Shuffle All	Shuffle Old	Causal	Exp.
GPT-5.6-Sol	ALFWorld	95	80	93	93	95	95	90
	ScienceWorld	83	63	72	79	79	85	80
	WebShop	45	41	31	41	39	45	45
	AppWorld	81	40	73	80	83	84	82
DeepSeek-V4-Pro	ALFWorld	67	59	66	68	61	65	67
	ScienceWorld	44	27	48	45	52	46	51

Appendix

Table 10: Native-chat success counts out of 100 per model–environment cell (2,000 episodes per condition). “Obs. Only” denotes Observation-Only; “Exp.” denotes Experience.

Model	History	ALFWorld	ScienceWorld	WebShop	AppWorld	Total
GPT-5.6-Sol	Raw	25	20	4	5	54
GPT-5.6-Sol	Reference	24	17	4	3	48
DeepSeek-V4-Pro	Raw	10	12	9	5	36
DeepSeek-V4-Pro	Reference	6	9	5	3	23
Qwen3.5-Plus	Raw	11	12	9	5	37
Qwen3.5-Plus	Reference	5	8	4	4	21

Appendix

Table 11: Recovery success counts with user–assistant history. Each model–environment cell contains 50 paired sources; the Total column is out of 200 per model. Overall counts are out of 250 per environment and 1,000 in total.

Memory-Augmented Language Model Agents

Memory-augmented language model agents use persistent facts, interaction histories, and skills to support personalization and long-horizon tasks.

See all 448 papers

SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
Jun 11, 2026Ziyi Wang, Yuxuan Lu, Yimeng Zhang +8AI Agent Reliability Reinforcement Learning
Northeastern University · Independent Researcher · Northwestern University
From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents
Oct 20, 2026Jiaxi Liu, Shixiong Kai, Zhentao Tang +4Reinforcement Learning RL for LLM Agents
Shanghai Jiao Tong University · Huawei Noah’s Ark Lab
Training Language Agents to Learn from Experience
May 19, 2026Yuval Shalev, Zifeng Ding, Mateja JamnikContinual Learning for LLM Agents Self-Improving Agents
University of Cambridge

Jun 11, 2026J/K move · Enter open · S save

SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Qun Liu+7

Northeastern University · Independent Researcher · Northwestern University

AI Agent Reliability Reinforcement Learning LLM Agent Training RL for Tool Use

Open paper AskPDF arXiv

Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.

Abstract

Memory-Augmented Language Model Agents

Explore similar work

SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

From Outcomes to Steps: Reliable Success-Aware Critic Learning for Long-Horizon Language Agents

Training Language Agents to Learn from Experience