Reinforcement learning (RL) has become a de facto paradigm for building LLM-based agents that act, interact, and reason over extended task horizons. However, in active reasoning where agents must elicit new observations through interaction with the environment to solve the task, we find that outcome-based RL can induce a systematic failure mode which we call information self-locking (SeL): agents fail both to elicit informative feedback and to internalize obtained evidence. To understand the issue, we trace agentic behaviors into two coupled capabilities: Action Selection (AS), which determines observation streams, and Belief Tracking (BT), which updates the agent's internal task understanding. Theoretical and empirical analyses reveal a bidirectional bottleneck that leads to SeL: weak BT obscures the credit of informative actions, while weak AS deprives BT of useful evidence. This coupling weakens the learning signal for both capabilities and leads to SeL. To mitigate this issue, we propose AREW, a simple yet effective Advantage Reweighting method that uses easy-to-obtain directional critiques to reallocate credit within trajectories. Extensive experiments across 9 agentic tasks of varying complexity show that AREW significantly mitigates SeL, yielding up to 60-point gains in final performance. Code is available at https://github.com/unimpor/T3.
Figures & tables
Figure 1: Overall illustration of information self-locking (SeL) and its mitigation. (b,a,o) denote the agent’s internal belief, its chosen action, and the resulting feedback at each turn. Under vanilla outcome-based RL (top), the agent can become trapped in a self-locking regime: deficient belief tracking masks contributions of informative actions, leading to misaligned credit assignment. Our AReW (bottom) introduces advantage reweighting via directional critiques, correcting the learning signal and assisting to mitigate SeL in active reasoning.
Figure 2: (a)/(b): the training dynamics of outcome reward, per-turn AS, and per-turn BT proxies in PE-G S=2 and MediQ datasets (Qwen-2.5-7B-Instruct). (c)/(d): correlation between the reward and AS proxies in PE-G S=2 and MediQ (Qwen-2.5-7B-Instruct). In strong BT patterns (by human-defined rules or frontier LLMs), the same AS sequence exhibits stronger correlation with the final reward.
Figure 3: Training dynamics of rewards, evaluated under the PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW – as-only and PPO with AReW – as+bt .
Preference Estimation
Medical Diagnosis
Troubleshooting
PE-G S=2
PE-G S=3
PE-F D=8
PE-F D=6
MediQ
FloDial-Easy
FloDial-Hard
Direct Inference
o4-mini
17.11
21.15
8.42
12.47
74.67
35.00
26.33
Qwen-2.5-7B-Inst.
15.00
10.00
2.67
4.20
39.00
24.33
13.33
Llama-3.1-8B-Inst.
18.00
12.33
3.14
5.70
35.25
24.33
17.33
PPO-trained (Qwen-2.5-7B-Inst.)
Table 1: Main results (average outcome reward on test sets) across three domains and seven active reasoning tasks.
Figure 4: Evaluations of AS and BT capabilities under PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW .
Figure 5: Training dynamics wrt. training steps on (a) rewards, (b) tool execution errors, (c) consumed response tokens and (d) interaction turns over Tau2Bench-Telecom solo setting, comparing vanilla PPO and PPO with AReW on Qwen2.5-7B-Instruct.
Figure 6: (a): outcome-RL reduces sensitivity to interactive feedback while increasing belief consistency. (b)/(c): Evaluations of AS and BT capabilities under GRPO and GSPO (Qwen-2.5-7B-Instruct).
Figure 7: Training dynamics on Tau2Bench-Telecom under the standard dual-control setting, comparing vanilla PPO and PPO with AREW on Qwen2.5-14B-Instruct. We report (a) rewards, (b) message-turn ratio, and (c) assistant tool-turn ratio.
Vanilla
Perturbation ratio α
0
0.1
0.2
0.3
0.4
0.5
PE-G S=3
18.3
80.3
40.0
65.0
31.3
22.3
30.3
FloDial-Hard
21.3
36.0
30.3
29.0
27.6
30.6
23.3
Table 2: Final Performance under different strength of directional critique perturbation (controlled by α ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Robustness of AReW under structured critique perturbations on PE-G S=3 (PPO training with Qwen2.5-7B-Instruct). (a) Channel : using only AS critiques, only BT critiques, or both channels, with the removed channel set to 0 . (b) Position : keeping only the first 40% or last 40% of interaction-turn critiques, with the remaining labels set to 0 . (c) Sign : keeping only the last 40% of labels and filling the remaining labels with constant 0 , +1 , or −1 . Across all three perturbation families, AReW remains consistently above the vanilla baseline, indicating robustness not only to random noise but also to several forms of structured critique destruction.
Figure 9: (a) Training dynamics of rewards on PE-G S=3 (PPO training with Qwen2.5-7B-Instruct). Compared with the experiment in Fig. , this variant uses an alternative belief-elicitation prompt that encourages more aggressive confidence updates, with the rest of the training pipeline unchanged. (b) Training dynamics of rewards on under different strength of AReW .
vanilla RL
α -0
α -0.2
α -0.5
PE-G S=3
Intermediate reward
18.33
73.67
32.33
18.33
AReW
80.33 ↑ 6.66
65.00 ↑ 32.67
30.30 ↑ 11.97
FloDial-Hard
Intermediate reward
21.33
33.00
28.00
21.00
AReW
36.00 ↑ 3.00
29.00 ↑ 1.00
23.30 ↑ 2.30
Appendix
Table 3: Robustness comparing AReW with direct intermediate reward under random critique sign flipping (PPO training on PE-G S=3 and FloDial-Hard datasets, Qwen2.5-7B-Instruct), where each critique is independently flipped with probability α .
p -0.3
p -0.5
pos/neg
performance
pos/neg
performance
Intermediate reward
5.94
23.33
7.26
17.33
AReW
1.58
30.67 ↑ 7.34
4.22
26.00 ↑ 8.67
Appendix
Table 4: Robustness comparing AReW with direct intermediate reward under structured critique sign flipping, where each negative critique is independently flipped to positive with probability p (PPO training on PE-G S=3 dataset, Qwen2.5-7B-Instruct). We report the average final positive/negative token ratio (including flipped positives), pos/neg , and the resulting test performance under two corruption levels.
AS
BT
GPT-5 labeling
Used in the main text
GPT-5 labeling
Used in the main text
PE-G S=3
vanilla RL
0.177
0.049
0.238
-0.005
AReW – AS + BT
0.669 ↑ 0.49
0.834 ↑ 0.79
0.459 ↑ 0.22
0.014 ↑ 0.02
MediQ
vanilla RL
0.275
0.222
0.355
0.060
AReW – AS + BT
0.335 ↑ 0.06
0.539 ↑ 0.32
0.666 ↑ 0.31
0.103 ↑ 0.04
FloDial-Hard
vanilla RL
0.343
0.250
0.495
0.130
Appendix
Table 5: Evaluations of AS and BT capabilities over vanilla RL and AReW – AS + BT (PPO algorithm with Qwen-2.5-7B-Instruct). We compare two evaluation sources: (1) GPT-5 rubric-based labeling and (2) the critiques used in the main text. We annotate the improvement of AReW over vanilla RL in each cell. Across all three tasks, AREW consistently improves over vanilla RL under both evaluation sources.
Figure 10: Prompt Template for MovieRec Preference Estimation.
Reinforcement learning with verifiable rewards (RLVR) enhances reasoning in large language models (LLMs) through outcome-level feedback, yet recent approaches to finer-grained credit assignment often require auxiliary models, additional sampling, or privileged information. Although policy entropy provides a readily available signal, prioritizing uncertain positions under both reinforcement and penalization concentrates penalties where failed responses still retain alternatives for recovery, which can suppress opportunities for exploration. To address this, we introduce Entropic Advantage Policy Optimization (EAPO), an entropy-guided credit assignment method that treats success and failure asymmetrically. Specifically, motivated by the observation that success under uncertainty is less repeatable while confident failures tend to recur, EAPO couples normalized policy entropy with the sign of the response advantage to reinforce surprising success and correct repeated failure. It assigns stronger reinforcement to high-entropy decisions in successful responses and stronger penalties to low-entropy decisions in failed responses, while attenuating penalties at uncertain positions to preserve opportunities for recovery. By redistributing the response advantage across tokens, EAPO derives token-level credit directly from existing rollout signals without additional supervision. We validate EAPO on a range of reasoning tasks across both base and reasoning backbones, demonstrating that it achieves the best overall performance. We further show that EAPO promotes more effective exploration, broadening problem coverage and generating more diverse candidate answers.
Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.
Reinforcement Learning (RL) has traditionally focused on training specialized agents to optimize predefined reward functions within narrowly defined environments. However, the advent of powerful Large Language Models (LLMs) and increasingly complex, open-ended tasks has catalyzed a paradigm shift towards agentic paradigms within RL. This emerging framework extends beyond traditional RL by emphasizing the development of autonomous agents capable of goal-setting, long-term planning, dynamic strategy adaptation, and interactive reasoning in uncertain, real-world environments. Unlike conventional approaches that rely heavily on static objectives and episodic interactions, LLM-based Agentic RL incorporates cognitive-like capabilities such as meta-reasoning, self-reflection, and multi-step decision-making directly into the learning loop. In this paper, we provide a deep insight for looking the conceptual foundations, methodological innovations, and effective designs underlying this trend. Furthermore, we identify critical challenges and outline promising future directions for building LLM-based Agentic RL.