Agent-Reflex: Teaching Language Agents to Act Reflectively

Guanting Dong; Xinyu Yang; Jiajie Jin; Yuyang Hu; Xiaoshuai Song; Yifei Chen; Xiaoxi Li; Jiejun Tan

cs.CLOct 13, 2026

Agent-Reflex: Teaching Language Agents to Act Reflectively

Authors: Guanting Dong, Xinyu Yang, Jiajie Jin, Yuyang Hu, Xiaoshuai Song, Yifei Chen, Xiaoxi Li, Jiejun Tan,

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China · National University of Singapore

Abstract

Large language models have evolved from single-turn chatbots into interactive agents capable of long-horizon reasoning. In real deployments, MCP tool services can return noisy feedback and users may intervene to correct the agent, but existing methods provide limited supervision for reflection learning and failure recovery. To bridge this gap, we propose Agent-Reflex, a general training framework for incentivizing proactive reflection in agents. Specifically, Agent-Reflex comprises three stages. First, Environment Reflection Skill Discovery (RSD) injects grounded tool-feedback perturbations into environment scaling, samples a capability-diverse LLM pool, mines matched success-failure pairs, and summarizes reusable reflection meta-skills. Building on this skill library, we then apply Reflective Supervised Fine-Tuning (ReSFT) that converts each skill into supervision for self-reflection, skill recovery, and user-centric correction, strengthening complementary reflective behaviors. Finally, we propose On-Policy Reflective Policy Optimization (ORPO) to combine agentic reinforcement learning under noisy interactions with reflection-guided on-policy distillation to further elicit proactive reflection. Built on Qwen3-8B, Agent-Reflex reaches 58.6 overall on five long-horizon tool-use benchmarks and 29.1 on AgentNoiseBench, surpassing strong environment-scaling and reflection baselines. Experiments on six challenging agent benchmark suites demonstrate the effectiveness of Agent-Reflex. Further analyses show that Agent-Reflex improves agents' robustness to noise while maintaining stable training dynamics, supporting scalable agent training. Our code is available at https://github.com/RUC-NLPIR/Agent-Reflex.

Figures & tables

Figure 1: From idealized interaction to reflective recovery. (a) Prior environments assume clean tool feedback, whereas (b) real ones return noisy observations and user corrections at arbitrary turns. (c) Success rate of GPT-5.2 and Claude-4.5-Sonnet under no, user, and tool noise. (d) Share of the eight tool-error families mined from 24,510 real error returns.

Figure 2: RSD and ReSFT . Stage 1 perturbs tool returns of 557 MCP-anchored executable environments with mined failure families, contrasting success/failure rollouts into an offline skill library. Stage 2 samples local-repair (SR), global-recovery (KR), and user-correction (UR) continuations from that library, keeping only verifier-validated ones with skill prompts stripped.

Figure 3: ORPO. A mixed rollout group of mostly clean and a few perturbed trajectories shares one group-normalized advantage. Re-scoring the same tokens with and without the skill card of their environment yields the detached, confidence-weighted OPD signal.

Method	VitaBench					$\tau^{2}$ -Bench				BFCL V4							ACEBench			Tool Sandbox	Overall
Method	Cro.	Del.	In.	OTA.	Avg.	Ret.	Air.	Tel.	Avg.	Web.	Mem.	Mul.	NoL.	Liv.	Irr.	Avg.	Ste.	Tur.	Avg.	Tool Sandbox	Overall
Foundation Models
GPT-5.2	24.3	52.5	45.8	28.3	37.7	81.6	62.5	95.8	80.0	75.5	45.8	48.5	81.9	70.4	88.7	62.9	95.0	50.0	68.0	73.3	64.4
Claude-4.5-Sonnet	23.5	52.0	48.3	28.3	38.0	86.2	70.1	98.0	84.7	81.0	65.0	61.4	88.7	81.1	86.6	73.2	100.0	70.0	82.0	85.0	72.6
Gemini 3 Pro	31.5	62.0	56.5	40.3	47.6	85.3	72.7	98.0	85.4	80.0	61.7	60.8	90.7	83.1	85.6	72.5	100.0	50.0	70.0	88.4	72.8
Qwen3-235B-A22B	14.5	45.0	32.0	15.8	26.8	71.9	45.6	58.0	58.5	54.0	23.9	45.4	37.4	68.9	81.7	48.0	100.0	63.3	78.0	77.1	57.7

Table 1: Main results on five tool-use benchmarks. Overall is the unweighted mean of the five benchmark averages; Avg. of BFCL V4 and ACEBench follows each benchmark’s official scoring rather than a simple mean. Best and second-best among 7B–8B methods are bolded and underlined (ties share a mark). ∗ Reproduced by us because the checkpoint is not open-sourced.

Method	AgentNoiseBench- $\tau^{2}$				AgentNoiseBench-Vita				Overall
Method	Clean	Tool Noise	User Noise	Avg.	Clean	Tool Noise	User Noise	Avg.	Overall
Foundation Models
GPT-5.2	81.0	61.0	72.0	66.5	53.0	45.0	45.0	45.0	55.8
Claude-4.5-Sonnet	73.0	57.0	65.0	61.0	51.0	41.0	41.0	41.0	51.0
Gemini-2.5-Pro	60.0	37.0	55.0	46.0	49.0	34.0	38.0	36.0	41.0
Qwen3-8B	26.2	14.1	26.2	20.2	11.4	3.3	5.5	4.4	12.3

Table 2: Results on AgentNoiseBench. We report performance under tool and user noise. Avg. is the mean of the tool-noise and user-noise scores, and Overall is the mean of the two benchmark averages. The Clean results of foundation models are taken from the original AgentNoiseBench report, so they may differ slightly from Table 1 . ∗ marks our reproduction.

Figure 4: Comparison of agent policy optimization algorithms. All runs start from Qwen3-8B and train for 100 RL steps on the same task pool under matched budgets. (a) Final accuracy on held-out benchmarks. (b) Mean training reward on the same tasks.

Variant	ANB- $\tau^{2}$	ANB-Vita
Agent-Reflex	45.9	12.3
w/o SR	41.3	8.9
w/o KR	42.4	10.2
w/o UR	43.0	10.9
w/o OPD	43.7	9.5
w/o weight	44.5	10.4

Table 3: Component ablation. AgentNoiseBench averages per variant; shading is the within-column normalized drop, so darker marks a more load-bearing module.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Failure family	Share	Typical signals	Recovery implication
Server/gateway unavailable	33.6%	500/502/503/504	Retry with bounded backoff; switch endpoint after the stop rule.
Authentication/permission	17.9%	401/403, permission denied	Refresh credentials, request authorization, or escalate; unchanged retry is insufficient.
Timeout	14.7%	timeout, read timeout	Retry or narrow the request after checking whether state changed.
Resource missing	10.5%	404, not found	Verify the identifier or discover the resource through search/list tools.
Argument/validation	10.2%	400/422, invalid argument	Repair schema, types, required fields, or enums before retrying.
Generic execution exception	8.3%	tool/operation failed	Inspect returned evidence and choose a typed recovery rather than blind repetition.

Appendix

Table 4: Empirical tool-failure taxonomy used for noise injection. Share is the fraction of non-exclusive pattern hits over 24,510 candidate error returns, matching Figure 1 (d). Families differ in whether an unchanged retry is admissible at all.

Provider	Model family	Checkpoints for trajectory sampling
Alibaba	Qwen3 [ Yang et al., 2025 ]	4B-Thinking, 8B, 14B, 32B, 235B-A22B
Meta	Llama [ Grattafiori et al., 2024 ]	3.1-8B-Instruct
Google	Gemma [ Team et al., 2025 ]	4-31B-it
OpenAI	GPT-OSS [ OpenAI et al., 2025 ]	120B
DeepSeek-AI	DeepSeek	V4-Flash (284B)

Appendix

Table 5: LLM agent pool for RSD trajectory sampling. The pool only supplies behavioral contrasts for skill mining; all training trajectories come from Qwen3-8B.

Figure 5: Per-axis Stability-Gated Success on AgentNoiseBench’s native taxonomy , normalized by the three-model mean on each axis (10 = that average). (a) Seven tool-side axes. (b) Six user-side. ∗ marks a lead clearing a 95% paired-bootstrap interval against both baselines, † the backbone alone.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation
Aug 12, 2026Zhixin Zhang, Xinke Jiang, Zhibang Yang +5LLM-Guided RL LLM Agents
School of Computer Science, Peking University, Beijing, China · National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China +2
Training Language Agents to Learn from Experience
May 19, 2026Yuval Shalev, Zifeng Ding, Mateja JamnikContinual Learning for LLM Agents Self-Improving Agents
University of Cambridge
Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL
Jun 12, 2026Yinglun ZhuLLM Agent Evaluation Language Model Calibration
University of California, Riverside

Aug 12, 2026J/K move · Enter open · S save

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

Zhixin Zhang, Xinke Jiang, Zhibang Yang, Weixuan Xu+4

School of Computer Science, Peking University, Beijing, China · National Engineering Research Center for Software Engineering, Peking University, Beijing, China · Key Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing, China+2

LLM-Guided RL LLM Agents Long-Horizon LLM Agents RL for LLM Agents Memory-Augmented Language Model Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Large language model agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.

Abstract

LLM Agent Training

Explore similar work

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

Training Language Agents to Learn from Experience

Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL