Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

Minju Gwak; Alan Ritter

cs.AIOct 30, 2026

Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

Authors: Minju Gwak, Alan Ritter

Organizations: Georgia Institute of Technology

Abstract

Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.

Figures & tables

Figure 1: Correct answers leave room for preference. We investigate whether these trajectory preferences improve learning and downstream transfer.

Figure 2: Self-improvement with auxiliary rewards. SC or UID-Conj distinguishes reasoning traces with equal answer-level rewards, guiding online TTRL reward augmentation, offline DPO preferences, or weighted SFT. The updated policy is evaluated on verifiable reasoning and open-ended transfer tasks.

	Direct selectors				Borda-added selectors
Model	Maj.	SC	UID-Conj.	$\Delta$ best-worst (pp)	Maj.	SC + Borda	UID-Conj. + Borda	$\Delta$ best-worst (pp)
Qwen3-1.7B	25.9	25.4	25.7	0.78	25.9	26.3	27.0	1.11
Qwen3-4B	33.9	33.8	34.2	0.44	33.9	34.7	34.2	0.78

Table 1: Accuracy (%) averaged over rollout samples

n\in\{2,4,8\}

, over 300 pooled questions per model.

\Delta

best-worst is the within-setup selector range in percentage points, including Majority Voting (Maj.)

Figure 3: Mean accuracy of Auxiliary reward + Borda Voting vs. Majority Voting for Qwen3-1.7B (Top) and Qwen3-4B (Bottom), across rollout

n\in\{2,4,8\}

Figure 4: Trace quality for same answers. (Left) SC, UID-Conj. vs random correct selection, showing wins, losses, identical-response ties, and equal-score ties. Net denotes wins – losses in pp. (Right) Mean differences on four reasoning criteria. Results average eligible rollout budgets within each question, then questions equally.

Model	Quest.	Pairs	Net $\Delta$ (pp)	Total gap	Large gap	Same resp. (%)
Qwen3-1.7B	91	226	-21.98	-0.58	-0.46	28.21
Qwen3-4B	107	295	-15.42	-0.34	-0.27	30.69

Table 2: UID-Conj. vs SC on correct-only cohorts. Net

\Delta

computed as UID-Conj – SC; negative values favor SC (See Figure 4 ). Results are averaged across budgets within each question, then equally across questions.

Qwen3-1.7B
	In-Domain	Out-of-Domain
Method	AIME2026	HMMT2026	MinervaMath	GPQA-D	Macro-avg.	$\Delta$
Base model	30.00	9.70	25.15	38.79	25.91	0.00
TTRL	30.67	7.27	25.51	38.59	25.51	-0.40
Random-TTRL	32.67	7.88	25.88	37.58	26.00	+0.09
SC-TTRL	34.00	10.91	25.74	39.49	27.54	+1.63

Table 3: Online effectiveness on competition-level math benchmarks with known answers. Mean accuracy of 5 is reported in percent. Macro-avg. is the unweighted mean across benchmarks;

\Delta

is its percentage-point difference from the base model. Notably, TTRL adapts to AIME2026 using majority-voting rewards rather than ground-truth outcome rewards, explaining its stronger in-domain gains on Qwen3-4B. In contrast, UID-Conj TTRL improves all three out-of-domain benchmarks, suggesting better generalization beyond the adaptation distribution.

DPO
	In-Domain	Out-of-Domain
Method	AIME2026	HMMT2026	MinervaMath	GPQA-D	Macro-avg.	$\Delta$
Base	30.00	9.70	25.15	38.79	25.91	0.00
Random-DPO	38.00	8.48	26.18	38.48	27.78	+1.87
SC-DPO	32.00	11.52	26.18	37.58	26.82	+0.91
UID-Conj DPO	37.33	12.73	26.47	39.60	29.03	+3.12

Table 4: Offline effectiveness on competition-level math benchmarks with known answers. Mean acc. of 5 is reported in percent;

\Delta

is the macro-average difference from the base model in percentage points.

	Out-of-Domain Reasoning					Instruction-Following
	CriticBench		LabBench2			IHEval			IFBench
Method	(F1)	(Acc)	(Acc)	Macro-Avg.	$\Delta$	Ref.	Align.	Conflict	Loose	Strict	Macro-Avg.	$\Delta$
Base model	62.76	67.76	41.44	57.32	0.00	72.01	69.33	11.54	21.67	17.67	38.44	0.00
TTRL	61.65	66.76	41.24	56.55	-0.77	72.98	69.61	12.31	21.67	17.67	38.85	+0.40
Random-TTRL	63.11	68.00	40.92	57.34	+0.02	72.23	68.40	10.99	22.00	18.33	38.39	-0.05
SC-TTRL	62.10	67.28	41.80	57.06	-0.26	73.39	70.03	11.73	22.33	18.33	39.16	+0.72

Table 5: Transfer to critique, scientific reasoning, and instruction following. IFBench uses prompt-level loose and strict accuracy, and IHEval uses task-averaged composite scores. Macro-Avg. is the unweighted mean.

\Delta

denotes the percentage-point change in the corresponding macro-average relative to the base model.

Comparison	$\Delta$	95% CI
Online
UID-Conj vs. TTRL	$+1.22$	$[-0.25,\ 2.85]$
UID-Conj vs. Rand.-TTRL	$+1.23$	$[-0.06,\ 2.54]$
Offline
UID-Conj DPO vs. Rand.-DPO	$+1.25$	$[-0.90,\ 3.47]$
UID-Conj SFT vs. Uni.-SFT	$+1.93$	$[0.44,\ 3.44]$

Table 6: Paired differences for Qwen3-1.7B in percentage points (positive favors UID-Conj), with 95% paired prompt-cluster bootstrap CIs from 100,000 replicates.

	Verifiable				Transfer
Method	AIME	HMMT	MM	GPQA-D	CriticBench	LabBench2
Base Model	30.00	9.70	25.15	38.79	62.76	41.44
TTRL	35.33	13.33	25.07	40.71	62.46	41.20
Random-TTRL	38.67	10.91	25.66	38.48	63.86	41.52
SC-TTRL	41.33	13.94	26.25	39.80	63.02	41.08
UID-Conj. TTRL	37.33	13.94	26.10	41.21	63.19	42.00

Table 7: Inference on models trained on S1K datasets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Questions	Coverage	Experimental role
In-Domain
AIME2026	30	Full snapshot	Post-training evaluation; also used for AIME test-time adaptation for TTRL experiments.
Out-of-Domain
HMMT2026	33	Full snapshot	Post-training evaluation.
MinervaMath	272	Full snapshot	Post-training evaluation.
GPQA-D	198	Full Diamond subset	Post-training evaluation of science reasoning.

Appendix

Table 8: Composition of the verifiable evaluation suite.

Out-of-Domain
Benchmark	Cases	Selection and composition	Reported metrics
CriticBench	500	250 correct and 250 incorrect candidate answers; deterministic selection with coverage across source datasets.	Verdict accuracy and incorrect-class F1, averaged across five sample indices.
LabBench2	500	Deterministic selection of 500 unique question families with complete frozen text evidence, covering seven task types; five sampled responses per case.	Task-specific semantic, numeric, or recall judge scores in $[0,1]$ , averaged across response samples.
Instruction Following
IFBench	300	All official single-turn test examples; one greedy response per example.	Prompt-level loose accuracy (primary), plus strict/loose prompt and instruction accuracies.
IHEval	500	Stratified selection across nine tasks and reference, aligned, and conflict settings; one greedy response per case.	Official task-specific metrics, aggregated separately for reference, aligned, and conflict settings.

Appendix

Table 9: Composition of the transfer evaluation suite.

Method	AIME2026	HMMT2026	MM	Macro-avg.	$\Delta$ vs. Base
Base model	30.00	9.70	25.15	21.61	0.00
Warm-start-SFT	30.67	8.48	26.40	21.85	+0.23

Appendix

Table 10: Base model vs. Warm-start SFT on Qwen3-1.7B. Accuracy (%);

\Delta

in percentage points.

	(a) Veri			(b) Non-Veri		Overall
				CriticBench
Method	AIME	HMMT	GPQA-D	(F1)	(Acc)	Avg.
Online settings
UID-Mix	62.67	23.03	56.97	76.66	73.96	54.50
UID-Conj (default)	55.33	13.94	57.68	76.33	73.48	50.46
Offline settings (DPO)

Appendix

Table 11: UID-Mix vs. UID-Conj (default). Overall Avg. equally weights AIME, HMMT, GPQA-D, and CriticBench, with the CriticBench score defined as the mean of F1 and accuracy.

	Verifiable Reasoning			Transfer Reasoning
				CriticBench
Method	AIME2026	MinervaMath	GPQA-D	F1	Acc.
Base model	30.00	25.15	38.79	62.76	67.76
TTRL	30.67	25.51	38.59	61.65	66.76
Random-TTRL	32.67	25.88	37.58	63.11	68.00
Permuted-UID-TTRL	34.00	26.40	40.10	62.50	67.52

Appendix

Table 12: Trajectory-permutation controls on Qwen3-1.7B. All scores are percentages, reporting mean accuracy of 5. Permuted-UID-TTRL shuffles the realized auxiliary reward terms among majority-agreeing responses within each prompt group.

Comparison	$\Delta$	95% CI
Online: prompt-weighted held-out average
UID-Conj vs. TTRL	$+0.95$	$[-0.32,\ 2.27]$
UID-Conj vs. Rand.-TTRL	$+1.11$	$[-0.20,\ 2.43]$
Offline: prompt-weighted four-benchmark average
UID-Conj DPO vs. Rand.-DPO	$+0.79$	$[-0.60,\ 2.18]$
UID-Conj SFT vs. Rand.-SFT	$+0.98$	$[-0.30,\ 2.29]$

Appendix

Table 13: Secondary prompt-weighted paired differences for Qwen3-1.7B in percentage points (positive favors UID-Conj).

Language Model Self-Improvement

Language model self-improvement studies how models use their own outputs, feedback, and experience to improve with less external supervision.

See all 185 papers

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning
Oct 27, 2026Jixuan Leng, Si Si, Fei Wang +4Reinforcement Learning RL for Language Models
CMU · Google DeepMind · Google +1
A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language Models Mathematical Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai Jiao Tong University · The Chinese University of Hong Kong
H $^2$ SD: Hybrid Hindsight Self-Distillation
Jul 21, 2026Qiye Cai, Yichuan Ma, Weitao Ma +6RL for Language Model Reasoning Language Model Distillation
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +1

Oct 27, 2026J/K move · Enter open · S save

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh+3

CMU · Google DeepMind · Google+1

Reinforcement Learning RL for Language Models Multi-Objective Reinforcement Learning Multi-Task RL

Open paper AskPDF arXiv

When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.

Abstract

Language Model Self-Improvement

Explore similar work

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

H2^22SD: Hybrid Hindsight Self-Distillation

H $^2$ SD: Hybrid Hindsight Self-Distillation