Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory
Reinforcement learning from internal feedback can improve language models without ground-truth answers or external verifiers. However, confidence and correctness can diverge during training, and continued optimization of average self-certainty can increase confidence while degrading reasoning accuracy. We introduce SIF, a reinforcement learning method that combines comparative reward construction with selective policy updates. SIF compares ranked token-level self-certainty profiles across multiple windows and peers, assigning positive rewards only when both levels of voting provide sufficient support. Sample and group level filtering determines which responses contribute to policy optimization. Experiments on three backbones across 12 mathematical and scientific reasoning benchmarks show that SIF's final checkpoints achieve higher average accuracy than the best-validation checkpoints of Intuitor and Vigor, with a gain of 4.71 percentage points over Intuitor on Qwen2.5-Math-1.5B. Further analyses show better retention of accuracy during continued optimization, while test-time training experiments demonstrate sustained improvements with as few as four rollouts per prompt.
Figures & tables
Figure 1: Performance overview of SIF algorithm.
Figure 2: Training dynamics and diagnostic experiments for Intuitor. (a) Training set accuracy across three backbones. (b) Variance of token-level self-certainty during training. (c) Validation accuracy of training variants using mean self-certainty within different rank intervals as rewards, with token scores sorted in descending order.
Figure 3: Overview of SIF. Token-level self-certainty scores are ranked and summarized across multiple windows. Window-level and peer-level voting jointly construct binary rewards, while response eligibility and group-level filtering determine participation in group-relative policy updates.
Figure 4: Training dynamics of SIF, Intuitor, and Vigor on Qwen3-1.7B: policy entropy (left) and training-batch accuracy (right).
Method
Mathematical Reasoning
Scientific Reasoning
Avg.
AIME
AMC23
Math500
Minerva
Olympiad
GSM8K
Omni
College
JEEBench
Theorem
Science
SciBench
Llama-3.2-3B-Instruct
3.12
17.39
41.20
15.26
13.67
72.86
12.15
33.06
13.77
23.39
42.95
7.31
24.68
+ TTRL
2.08
19.84
47.55
17.78
16.34
78.91
13.72
33.91
14.60
26.41
44.01
9.64
27.07
+ SCOPE
1.67
20.38
44.33
16.82
14.61
74.85
13.02
34.69
13.30
23.91
43.18
8.19
25.75
+ Intuitor
2.71
17.93
41.48
16.18
14.73
72.34
12.34
33.92
13.76
22.57
43.13
7.83
24.91
+ Vigor
1.25
11.14
39.30
10.06
12.22
67.67
9.74
26.32
9.94
20.05
44.38
7.16
21.60
Table 1: Accuracy (%) on 12 reasoning benchmarks across three backbones. All methods are trained and evaluated under identical settings within each backbone. Baseline results use checkpoint achieving the highest validation accuracy during training, while SIF reports final (last) and best-validation (best) checkpoints. Avg. denotes the unweighted mean across benchmarks. The best and second-best distinct scores within each backbone, excluding GRPO, are bolded and underlined , with ties highlighted equally. GRPO uses ground-truth (GT) answers.
Figure 5: Sustained improvements during Test-Time Training across four datasets with five rollout counts.
Method
Step
Math500
Minerva
Olympiad
College
JEEBench
mid
80.50
37.27
37.35
58.33
24.81
SIF
last
80.30
36.53
37.24
57.68
24.47
SIF w/o Peer
mid
79.45
36.17
34.11
57.13
23.11
last
79.20
35.85
33.17
55.86
23.11
SIF w/o Window
mid
79.40
36.26
38.58
57.42
23.54
last
79.40
35.98
37.87
56.95
23.57
Table 2: Ablation results (%) on five benchmarks using Qwen3-1.7B test time trained for 40 steps. Mid and Last denote the intermediate and final training checkpoints.
Figure 6: Additional ablation on Qwen3-1.7B.
Figure 7: Most frequent word proportion for SIF (left) and Intuitor (right). Gray dots represent individual responses; horizontal lines indicate means.
Figure 8: Variance dynamics of Intuitor and SIF on Qwen3-1.7B and Llama3.2-3B-Instruct. Arrows indicate the maximum gap between each model.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Checkpiont
MATH
MATH500
Minerva
Olympiad
CollegeMath
JEEBench
TheoremQA
Pass@1
Pass@8
Pass@1
Pass@8
Pass@1
Pass@8
Pass@1
Pass@8
Pass@1
Pass@8
Pass@1
Pass@8
Pass@1
Pass@8
Backbone Model: Qwen2.5-Math-1.5B
Vanilla
55.35
85.88
50.00
85.20
14.84
39.34
25.43
52.08
31.96
63.17
12.11
46.60
13.91
37.88
Global Step 10
57.09
86.12
52.20
86.60
15.95
40.44
26.63
52.37
34.92
64.30
12.74
47.96
15.62
40.75
Global Step 20
60.50
86.36
58.70
88.20
17.92
43.38
28.23
53.41
40.60
66.54
16.12
51.65
20.97
44.75
Global Step 30
64.22
87.36
62.62
89.60
20.36
42.65
29.45
55.04
47.40
67.00
17.26
51.65
24.39
46.75
Appendix
Table 5: Pass@1 and Pass@8 performance across training steps. Δ denotes the improvement from Base to Step 80 in percentage points.
Parameter
Value
Training budget
80 steps
Prompt batch size
64
Responses per prompt
8
PPO mini-batch size
64
Micro-batch per GPU
4
PPO epochs per batch
1
Appendix
Table 6: Optimization settings for test-time training. The PPO mini-batch size is expressed in prompts; each prompt contributes eight sampled responses before masking. Run-specific exceptions are described in the text.
Parameter
Value
Internal voting
Number of windows
10
Tokens per window
100
Minimum-length stride
50 tokens
Window wins required
8 of 10
Peer-voting threshold τp
3/4
Appendix
Table 7: Voting and generation settings. † The DeepSeek-R1-Distill-Qwen-1.5B uses a maximum prompt length of 4,096 tokens.
Parameter
Value
Initial model
-
Training split
MATH train
Online validation split
MATH test
Training epochs
1
Training steps
115
Prompt batch size
64
Appendix
Table 8: Optimization settings for all models MATH training run. The PPO mini-batch size is expressed in prompts; each prompt contributes eight sampled responses.
Parameter
Value
Internal voting
Number of windows
10
Tokens per window
100
Minimum-length stride
50 tokens
Window wins required
8 of 10
Peer-voting threshold τp
3/4
Appendix
Table 9: Internal voting and training generation settings for all models MATH run.
Backbone
Step
Acc. (%)
SC
Entropy
Qwen3-1.7B
30
73.43
55.03
0.0957
80
0.00
61.54
0.0280
Llama-3.2-3B
20
41.86
33.29
0.0721
(Instruct)
40
0.00
35.98
0.0161
Appendix
Table 10: Diagnostics for Intuitor . Acc. is MATH validation accuracy; SC and entropy are the logged training-batch means. These snapshots illustrate proxy–accuracy divergence.
Metric / rollout step
Filter responses Random 40%
Filter responses Random 20%
Filter responses Random 10%
Top-2 no filter
Validation accuracy (%)
Global Step 10
62.20
63.04
59.68
60.18
Global Step 20
67.65
72.80
73.11
75.89
Global Step 30
72.74
69.33
63.11
68.98
Global Step 40
52.55
54.87
45.78
65.54
Global Step 50
43.46
41.66
30.07
65.78
Appendix
Table 11: Validation accuracy of Intuitor with random response filtering and a top-2 reward baseline on Qwen3-1.7B trained on MATH, with one run per configuration. Random filtering excludes the indicated percentage of sampled responses from each training batch. The top-2 baseline assigns binary rewards according to mean self-certainty rankings without response filtering. Peak denotes the highest accuracy among the reported checkpoints; Drop is Peak minus Final in percentage points, calculated from the displayed values.
Figure 9: Word clouds of generated responses from the best and last checkpoints of Intuitor and our method on the same ten MATH500 problems. Larger words indicate higher average within-response frequencies within each panel; common stopwords are omitted.
Parameter
Value
Training budget
1 epoch
Prompt batch size
64
Responses per prompt
8
PPO mini-batch size
64
Micro-batch per GPU
4
PPO epochs per batch
1
Appendix
Table 12: Common baseline training settings. The configured PPO mini-batch size is expressed in prompts and expands to 512 sampled responses before distribution across GPUs.