C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
Authors: Ai Jian, Haoran Pan, Eryu Guo, Haojin Yang, Jinghan He, Xiaoyun Zhang, Jiangbo Pei, Peiyu Liu
Organizations: Beijing University of Posts and Telecommunications · Peking University · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · University of International Business and Economics
Large language models are increasingly trained as multi-turn agents that interact with external tools. However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update. To address this issue, we propose Confidence-Guided Credit Correction (C3), a plug-in method for local credit correction. C3 identifies spans corresponding to undesirable behaviors and selectively corrects positive credit after advantage estimation, with the correction strength adaptively controlled by the behavior policy's confidence in the corresponding behavior, while leaving the task reward and original credit estimation unchanged. Across multiple multi-turn benchmarks, C3 improves performance under both {critic-based and critic-free reinforcement learning methods}. Injecting the same behavioral signals as turn-level or outcome rewards does not consistently improve performance. Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.
Figures & tables
Figure 1: Misguided reinforcement under outcome-only training. (a) Incorrect tool use and redundant search receive mixed token-level credit, with positive and negative advantages shown in rose and blue-grey . (b) Token-level credit construction in critic-based and critic-free optimization.
Figure 2: Overview of confidence-guided advantage correction for multi-turn agent reinforcement learning. A detector localizes an undesirable-behavior span, the actor-side correction is applied only within that span, and its strength follows the policy confidence.
Figure 3: Misguided reinforcement during RL training.
Method
Single-Hop QA
Multi-Hop QA
Avg.
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
Musique
Bamb.
SFT
35.2 ± 0.2
59.2 ± 0.2
37.8 ± 0.0
48.9 ± 0.2
64.1 ± 0.3
22.6 ± 0.2
46.2 ± 1.2
44.87 ± 0.17
Critic-Based Optimization
PPO
40.7 ± 0.5
62.1 ± 0.2
43.4 ± 0.1
55.2 ± 0.2
56.4 ± 0.3
28.8 ± 0.5
47.8 ± 2.8
47.79 ± 0.50
+ process reward
42.0 ± 0.4
62.3 ± 0.3
43.5 ± 0.2
55.0 ± 0.1
60.8 ± 0.3
23.9 ± 0.6
42.2 ± 2.4
47.10 ± 0.43
+ C 3
39.5 ± 0.3
63.5 ± 0.1
45.0 ± 0.3
55.6 ± 0.1
67.9 ± 0.3
29.7 ± 0.1
50.4 ± 0.0
50.20 ± 0.10
Table 1: Strict EM (%) on seven QA benchmarks with Qwen3-8B. All entries report mean ± standard deviation over four independent samples at temperature 0.8 with top- p=0.95 . Bold marks the best mean within each optimization family.
Figure 4: Panels (a,c) show smoothed training EM, and panels (b,d) show evaluation strict EM.
Figure 6
Figure 6: Greedy evaluation on 51,713 questions per arm.
Figure 8
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Dataset
Size
Evaluation
NQ
3,610
TriviaQA
11,313
PopQA
14,267
HotpotQA
7,405
2WikiMultiHopQA
12,576
MuSiQue
2,417
Appendix
Table 4: Summary of data scales used for SFT, RL training, and evaluation.
Undesirable behavior
Detection rule
Severity
Invalid action
Malformed or impermissible action
1.0
Query loop
Exact repetition of an earlier normalized query
1.0
Near repetition: trigram Jaccard ≥0.9 , length ≥8
0.5
Appendix
Table 5: Rule-based detector for undesirable behaviors.
Method
Success (%)
Iter.
Task score (%)
Iter.
T2PO
81.2
70
83.9
70
T2PO+C 3
82.2
120
86.0
130
Appendix
Table 6: Best validation performance.
Figure 8: Validation trajectories for official T2PO and T2PO+C 3 . The two panels show success rate and task score; gray diamonds mark the shared SFT initialization.
Figure 9: Periodic strict-EM evaluation curves for the seven Search QA benchmarks under the PPO family. The bottom-right panel shows the unweighted 7-BMK macro mean.
Figure 10: Periodic strict-EM evaluation curves for the seven Search QA benchmarks under the GDPO family. The bottom-right panel shows the unweighted 7-BMK macro mean.
Method
Success
Late-stage signature
Process PPO
69.0%→11.2%
Runaway reasoning, repetition, and severe truncation.
GDPO
68.8%→40.2%
Shortened trajectories that terminate before purchase.
PPO+C 3
66.2%→60.8%
No comparable collapse in the same window.
GDPO+C 3
71.4%→70.0%
Stable performance through late training.
Appendix
Table 7: Late-stage WebShop behavior on the 500-task validation benchmark.
Statistic
259
319
399
Change (259 → 399)
Success rate (%)
69.0
26.6
11.2
−57.8 pt
Truncated responses (%)
0.2
66.6
86.2
+86.0 pt
Trailing <think> median (chars)
328
7,962
8,432
≈26×
Generated tokens / trajectory
1,043
2,536
2,667
+155%
Completed turns / trajectory
18.9
11.2
7.7
−59%
Mean assistant chars / turn
951
1,245
1,354
+42%
Appendix
Table 8: Turn-level signature of the process-PPO collapse.
Figure 11: Mean searches across seven benchmarks. Shaded columns denotes single-hop tasks.
Figure 12: Complete Search QA case presentation. The outcome-only policy repeats the original relation, while the C 3 -corrected policy separates entity identification from attribution and returns the requested person.
Figure 13: Complete WebShop case presentation. The later policy returns to search after the first candidate fails an explicit requirement, refines the query, and verifies the replacement before purchase.
Figure 14: Complete WebShop failure/recovery case. The outcome-only trajectory keeps the first candidate and repeats an invalid option; the C 3 trajectory returns to search, changes candidate, and completes the requested purchase.