Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

Qiqian Fu

cs.AIAug 22, 2026

Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

Authors: Qiqian Fu

Abstract

Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p < 0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.

Figures & tables

Arm	Prior	Mechanism
grpo	none	plain GRPO, groups of $n{=}4$ (the baseline)
hintonly	text	reference-solution prefix, cosine-annealed $95\%{\to}5\%$ , then off
uft001	text	hintonly + NLL on hint tokens, coef. 0.001 (full UFT)
uft002	text	hintonly + NLL on hint tokens, coef. 0.002
opd	distribution	pure on-policy distillation: advantage $=-$ reverse KL, no reward
reopd	distribution	opd + teacher-trajectory prefix replay

Table 1: The eleven arms. All train on the same 20,830-problem sparse pool for 500 steps with identical batch size, learning rate, and seed.

Domain	Set	$n$	Nature	Base (%)
in-domain	MathV (pooled)	1119	sparse-800 $+$ old-319, size-weighted 71.5/28.5
	sparse-800	800	training-distribution hard tail, never trained on	$\approx 0$
	old-319	319	earlier MathV360K slice; 69% Geometry3K 4-option MC	—
cross-domain	DynaMath	477	programmatically rendered; zero overlap with MathV360K	25.8
	easy	373		30.3
	hard	104		11.5

Table 2: Evaluation datasets, grouped by domain. The combined DynaMath base figure is reported as measured; the size-weighted average of the two split figures is 26.2, a 0.4 pp bookkeeping difference we do not resolve here.

Arm	Prior	sparse-800	old-319	MathV (pooled)	DynaMath
uft002	text	52.4	19.1	42.9	32.9
vapohl	value (HL-Gauss)	53.6	19.7	44.0	31.9
hintonly	text	51.4	21.3	42.8	31.2
hintsaf	text + distribution	49.4	26.0	42.7	30.8
uft001	text	52.0	15.7	41.6	30.8
saf	distribution (annealed)	49.6	30.4	44.1	30.6

Table 3: Blind-judged accuracy (%) of all eleven arms, sorted by the cross-domain metric (DynaMath, n=477, pooling the easy and hard splits). MathV (pooled, n=1119) is the in-domain metric, combining the sparse-800 and old-319 slices at their size weights (71.5/28.5); the two slice columns give its breakdown. The horizontal rule separates the six delivered-prior arms from the five undelivered-prior arms: the two groups do not overlap on either metric. Within-group orderings should not be read as rankings (single training seed; Section 7 ).

Figure 1: All eleven arms ranked by DynaMath accuracy. Every delivered-prior arm outperforms every undelivered-prior arm: the shaded band (29.1–30.6) separates the two groups and contains no arm. Differences within each group are inside judging noise and should not be ranked (Section 7 ). Dashed line: untrained base (25.8).

X	Y	$\rho$
sparse-800	DynaMath	$+0.89$
old-319	DynaMath	$-0.74$
MathV pooled (71.5/28.5)	DynaMath	$+0.76$
MathV pooled (50/50)	DynaMath	$+0.40$
sparse-800	old-319	$-0.66$
sparse-800	DynaMath-easy	$+0.87$

Table 4: Spearman rank correlation between evaluation sets over the eleven trained arms (n = 11). The five correlations computable from Table 3 were independently recomputed from the published numbers and agree with the values below to within 0.004; the two easy-split rows use per-arm accuracies on DynaMath-easy alone.

Figure 2: Which held-out set predicts cross-domain transfer? Each point is one trained arm; the

y

axis is accuracy on DynaMath (

n{=}477

), the only evaluation pool with no overlap with the training distribution. Left: the hardest in-domain held-out set (sparse-800) ranks arms nearly as DynaMath does (Spearman

\rho=+0.89

n=11

). Right: the set used as a “general-distribution” check since round 1 (old-319) anti-predicts transfer (

\rho=-0.74

): vapo tops old-319 while placing last among trained arms on DynaMath, and vapohl nearly inverts that. Dashed line: untrained base model (25.8).

Figure 3: Spearman rank correlations between evaluation sets over the eleven trained arms, computed from the blind-judged accuracies in Table 3 . sparse-800 is the strongest predictor of cross-domain accuracy ( $+0.89$ ); old-319 is strongly anti-correlated with it ( $-0.74$ ) and with sparse-800 itself ( $-0.66$ ). The pooled in-domain metric inherits its predictive validity mechanically from its sparse-800 component (the pool contains both sets; weights 71.5/28.5).

Earlier reading	Corrected
All arms lie on one specialization-versus-generalization trade-off curve; none advances both ends.	The apparent curve is capacity reallocation within MathV360K. Across domains there is no trade-off: stronger in-domain predicts stronger out-of-domain.
vapohl is the specialization extreme, paying a generalization tax.	vapohl is second-best cross-domain (31.9). Its low old-319 score reflects only that it stopped guessing option letters.
vapo is the robustness champion—best on the “general-distribution” set.	vapo is the weakest trained arm cross-domain (27.0, below plain grpo ). old-319 had the ranking inverted.

Table 5: Earlier readings and their corrections after the evaluation-validity analysis.

Figure 4: One change: replacing vapo ’s clipped-MSE value loss with HL-Gauss cross-entropy ( vapohl ), everything else identical. In-domain accuracy on sparse-800 moves

+14.4

pp and cross-domain DynaMath from 27.0 to 31.9, while old-319 drops—the arm stopped guessing option letters (Section 4.2 ).

Hypothesis	Verdict
Answer-space guessing	Rejected: 48.9% accuracy against an 8.7% aggregate guessing ceiling, with a diverse answer distribution.
Format-reward hacking	Rejected: correct rows carry genuine multi-step reasoning (median 270 characters).
Template leakage from training	Partial: roughly 3 pp of attributable inflation; rows without a template duplicate in the training pool still score 45.9% against a 3.5% base.
Composition artifacts on old-319	Confirmed—and followed to the evaluation-validity finding of Section 4 .

Table 6: Four reward-hacking hypotheses and their verdicts.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

May 28, 2026

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains -- and why it does so -- remain under-explored. We study cross-domain transfer in a 7B model whose SFT and RL post-training stages use only constraint-satisfaction puzzles, with no mathematics problems in the post-training data. To analyze how transfer emerges, we introduce a reasoning primitive-level framework that combines a 9-class span classifier with motif extraction, allowing us to segment chain-of-thought traces into primitive motifs and track their evolution across training stages and domains. We find that puzzle SFT induces a reasoning-primitive vocabulary, yielding a

+7

pp pass@32 gain on OlymMATH-Hard. Vanilla GSPO then composes these primitives into longer compute-verify chains, adding a further

+6

pp. However, this RL stage also suppresses exploratory primitives such as hypothesize and backtrack. To address this, we introduce a novelty bonus that rewards diverse correct rollouts, using perplexity under the reference model as a signal. This restores recovery primitives during RL and adds a further

+7

pp pass@32 relative to vanilla GSPO. Finally, the end-to-end recipe raises the hard-math capability ceiling from

16.0\%

at the OLMo3-7B-Instruct-SFT base to

36.0\%

, without adding any mathematics problems during the SFT or RL stages.

Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski +1

Fin AI Research · Salk Institute for Biological Studies

RL for Language Models RL for Language Model Reasoning Cross-Domain Transfer Learning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Ask PDF

Apr 9, 2026

Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing

In reinforcement learning with verifiable rewards (RLVR), challenging math questions whose sampled groups remain uniformly incorrect yield no informative update, letting easier questions dominate actor updates. This imbalance can contribute to distribution sharpening: low-

k

accuracy improves while problem coverage contracts. Hint scaffolding can restore informative updates by revealing a prefix of a verified teacher solution. However, existing hint-based approaches still face two issues: teacher--student distribution mismatch and the need to reduce hint exposure to match no-hint evaluation. We address these issues through two components. Distribution-Aligned Hint Synthesis (DAHS) constructs verified teacher hints conditioned on student-style responses. Backward Hint Annealing (BHA) anneals hint exposure across length buckets as bucket accuracy improves. BHA also uses per-question hint dropout to keep no-hint rollouts throughout RL training. Under the DAPO training framework, our method improves both pass@1 and pass@2048 over DAPO on AIME24/25/26 for Qwen3-1.7B-Base and Qwen3-4B-Base, and improves pass@1 on five secondary math benchmarks (Olympiad, MATH500, Minerva Math, AMC23, and GSM8K). On Llama-3.2-1B-Instruct, the gains concentrate in the high-

k

regime without recovering the base model's coverage. Overall, when verified teacher prefixes are available, DAHS+BHA mitigates DAPO-induced contraction of high-

k

problem coverage in the evaluated challenging-math RLVR settings.

Pei-Xi Xie, Che-Yu Lin, Cheng-Lin Yang

CyCraft AI Lab, Taiwan

Reinforcement Learning LLM Mathematical Reasoning Reinforcement Learning with Verifiable Rewards Curriculum RL

Ask PDF

Jun 2, 2026

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning

Reinforcement Learning from Verifiable Rewards (RLVR) has become the dominant approach for improving mathematical reasoning in large language models, yet current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states. Investigating this structure, we find that at the anchor token (the position immediately before the answer marker), correct rollouts converge naturally because they must produce the same answer (cosine similarity ~0.84), yet each retains residual variance from its unique reasoning path. Encouraging full alignment at this point pushes the model to extract a unified "correct decision" representation, reducing sensitivity to which reasoning path was taken. Based on this observation, we propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference. On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales, supported by ablations on loss type, anchor position, layer depth, and loss weight.

Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10

1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2

LLM Alignment RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Ask PDF

Abstract

Reinforcement Learning

Explore similar work

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning