Reinforcement learning (RL) has become a de facto paradigm for building LLM-based agents that act, interact, and reason over extended task horizons. However, in active reasoning where agents must elicit new observations through interaction with the environment to solve the task, we find that outcome-based RL can induce a systematic failure mode which we call information self-locking (SeL): agents fail both to elicit informative feedback and to internalize obtained evidence. To understand the issue, we trace agentic behaviors into two coupled capabilities: Action Selection (AS), which determines observation streams, and Belief Tracking (BT), which updates the agent's internal task understanding. Theoretical and empirical analyses reveal a bidirectional bottleneck that leads to SeL: weak BT obscures the credit of informative actions, while weak AS deprives BT of useful evidence. This coupling weakens the learning signal for both capabilities and leads to SeL. To mitigate this issue, we propose AREW, a simple yet effective Advantage Reweighting method that uses easy-to-obtain directional critiques to reallocate credit within trajectories. Extensive experiments across 9 agentic tasks of varying complexity show that AREW significantly mitigates SeL, yielding up to 60-point gains in final performance. Code is available at https://github.com/unimpor/T3.
Figures & tables
Figure 1: Overall illustration of information self-locking (SeL) and its mitigation. (b,a,o) denote the agent’s internal belief, its chosen action, and the resulting feedback at each turn. Under vanilla outcome-based RL (top), the agent can become trapped in a self-locking regime: deficient belief tracking masks contributions of informative actions, leading to misaligned credit assignment. Our AReW (bottom) introduces advantage reweighting via directional critiques, correcting the learning signal and assisting to mitigate SeL in active reasoning.
Figure 2: (a)/(b): the training dynamics of outcome reward, per-turn AS, and per-turn BT proxies in PE-G S=2 and MediQ datasets (Qwen-2.5-7B-Instruct). (c)/(d): correlation between the reward and AS proxies in PE-G S=2 and MediQ (Qwen-2.5-7B-Instruct). In strong BT patterns (by human-defined rules or frontier LLMs), the same AS sequence exhibits stronger correlation with the final reward.
Figure 3: Training dynamics of rewards, evaluated under the PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW – as-only and PPO with AReW – as+bt .
Preference Estimation
Medical Diagnosis
Troubleshooting
PE-G S=2
PE-G S=3
PE-F D=8
PE-F D=6
MediQ
FloDial-Easy
FloDial-Hard
Direct Inference
o4-mini
17.11
21.15
8.42
12.47
74.67
35.00
26.33
Qwen-2.5-7B-Inst.
15.00
10.00
2.67
4.20
39.00
24.33
13.33
Llama-3.1-8B-Inst.
18.00
12.33
3.14
5.70
35.25
24.33
17.33
PPO-trained (Qwen-2.5-7B-Inst.)
Table 1: Main results (average outcome reward on test sets) across three domains and seven active reasoning tasks.
Figure 4: Evaluations of AS and BT capabilities under PPO algorithm with Qwen-2.5-7B-Instruct across vanilla PPO, PPO with AReW .
Figure 5: Training dynamics wrt. training steps on (a) rewards, (b) tool execution errors, (c) consumed response tokens and (d) interaction turns over Tau2Bench-Telecom solo setting, comparing vanilla PPO and PPO with AReW on Qwen2.5-7B-Instruct.
Figure 6: (a): outcome-RL reduces sensitivity to interactive feedback while increasing belief consistency. (b)/(c): Evaluations of AS and BT capabilities under GRPO and GSPO (Qwen-2.5-7B-Instruct).
Figure 7: Training dynamics on Tau2Bench-Telecom under the standard dual-control setting, comparing vanilla PPO and PPO with AREW on Qwen2.5-14B-Instruct. We report (a) rewards, (b) message-turn ratio, and (c) assistant tool-turn ratio.
Vanilla
Perturbation ratio α
0
0.1
0.2
0.3
0.4
0.5
PE-G S=3
18.3
80.3
40.0
65.0
31.3
22.3
30.3
FloDial-Hard
21.3
36.0
30.3
29.0
27.6
30.6
23.3
Table 2: Final Performance under different strength of directional critique perturbation (controlled by α ).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Robustness of AReW under structured critique perturbations on PE-G S=3 (PPO training with Qwen2.5-7B-Instruct). (a) Channel : using only AS critiques, only BT critiques, or both channels, with the removed channel set to 0 . (b) Position : keeping only the first 40% or last 40% of interaction-turn critiques, with the remaining labels set to 0 . (c) Sign : keeping only the last 40% of labels and filling the remaining labels with constant 0 , +1 , or −1 . Across all three perturbation families, AReW remains consistently above the vanilla baseline, indicating robustness not only to random noise but also to several forms of structured critique destruction.
Figure 9: (a) Training dynamics of rewards on PE-G S=3 (PPO training with Qwen2.5-7B-Instruct). Compared with the experiment in Fig. , this variant uses an alternative belief-elicitation prompt that encourages more aggressive confidence updates, with the rest of the training pipeline unchanged. (b) Training dynamics of rewards on under different strength of AReW .
vanilla RL
α -0
α -0.2
α -0.5
PE-G S=3
Intermediate reward
18.33
73.67
32.33
18.33
AReW
80.33 ↑ 6.66
65.00 ↑ 32.67
30.30 ↑ 11.97
FloDial-Hard
Intermediate reward
21.33
33.00
28.00
21.00
AReW
36.00 ↑ 3.00
29.00 ↑ 1.00
23.30 ↑ 2.30
Appendix
Table 3: Robustness comparing AReW with direct intermediate reward under random critique sign flipping (PPO training on PE-G S=3 and FloDial-Hard datasets, Qwen2.5-7B-Instruct), where each critique is independently flipped with probability α .
p -0.3
p -0.5
pos/neg
performance
pos/neg
performance
Intermediate reward
5.94
23.33
7.26
17.33
AReW
1.58
30.67 ↑ 7.34
4.22
26.00 ↑ 8.67
Appendix
Table 4: Robustness comparing AReW with direct intermediate reward under structured critique sign flipping, where each negative critique is independently flipped to positive with probability p (PPO training on PE-G S=3 dataset, Qwen2.5-7B-Instruct). We report the average final positive/negative token ratio (including flipped positives), pos/neg , and the resulting test performance under two corruption levels.
AS
BT
GPT-5 labeling
Used in the main text
GPT-5 labeling
Used in the main text
PE-G S=3
vanilla RL
0.177
0.049
0.238
-0.005
AReW – AS + BT
0.669 ↑ 0.49
0.834 ↑ 0.79
0.459 ↑ 0.22
0.014 ↑ 0.02
MediQ
vanilla RL
0.275
0.222
0.355
0.060
AReW – AS + BT
0.335 ↑ 0.06
0.539 ↑ 0.32
0.666 ↑ 0.31
0.103 ↑ 0.04
FloDial-Hard
vanilla RL
0.343
0.250
0.495
0.130
Appendix
Table 5: Evaluations of AS and BT capabilities over vanilla RL and AReW – AS + BT (PPO algorithm with Qwen-2.5-7B-Instruct). We compare two evaluation sources: (1) GPT-5 rubric-based labeling and (2) the critiques used in the main text. We annotate the improvement of AReW over vanilla RL in each cell. Across all three tasks, AREW consistently improves over vanilla RL under both evaluation sources.
Figure 10: Prompt Template for MovieRec Preference Estimation.