Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

Qiqian Fu

cs.AIAug 22, 2026

Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

Authors: Qiqian Fu

Abstract

Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p < 0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.

Figures & tables

Arm	Prior	Mechanism
grpo	none	plain GRPO, groups of $n{=}4$ (the baseline)
hintonly	text	reference-solution prefix, cosine-annealed $95\%{\to}5\%$ , then off
uft001	text	hintonly + NLL on hint tokens, coef. 0.001 (full UFT)
uft002	text	hintonly + NLL on hint tokens, coef. 0.002
opd	distribution	pure on-policy distillation: advantage $=-$ reverse KL, no reward
reopd	distribution	opd + teacher-trajectory prefix replay

Table 1: The eleven arms. All train on the same 20,830-problem sparse pool for 500 steps with identical batch size, learning rate, and seed.

Domain	Set	$n$	Nature	Base (%)
in-domain	MathV (pooled)	1119	sparse-800 $+$ old-319, size-weighted 71.5/28.5
	sparse-800	800	training-distribution hard tail, never trained on	$\approx 0$
	old-319	319	earlier MathV360K slice; 69% Geometry3K 4-option MC	—
cross-domain	DynaMath	477	programmatically rendered; zero overlap with MathV360K	25.8
	easy	373		30.3
	hard	104		11.5

Table 2: Evaluation datasets, grouped by domain. The combined DynaMath base figure is reported as measured; the size-weighted average of the two split figures is 26.2, a 0.4 pp bookkeeping difference we do not resolve here.

Arm	Prior	sparse-800	old-319	MathV (pooled)	DynaMath
uft002	text	52.4	19.1	42.9	32.9
vapohl	value (HL-Gauss)	53.6	19.7	44.0	31.9
hintonly	text	51.4	21.3	42.8	31.2
hintsaf	text + distribution	49.4	26.0	42.7	30.8
uft001	text	52.0	15.7	41.6	30.8
saf	distribution (annealed)	49.6	30.4	44.1	30.6

Table 3: Blind-judged accuracy (%) of all eleven arms, sorted by the cross-domain metric (DynaMath, n=477, pooling the easy and hard splits). MathV (pooled, n=1119) is the in-domain metric, combining the sparse-800 and old-319 slices at their size weights (71.5/28.5); the two slice columns give its breakdown. The horizontal rule separates the six delivered-prior arms from the five undelivered-prior arms: the two groups do not overlap on either metric. Within-group orderings should not be read as rankings (single training seed; Section 7 ).

Figure 1: All eleven arms ranked by DynaMath accuracy. Every delivered-prior arm outperforms every undelivered-prior arm: the shaded band (29.1–30.6) separates the two groups and contains no arm. Differences within each group are inside judging noise and should not be ranked (Section 7 ). Dashed line: untrained base (25.8).

X	Y	$\rho$
sparse-800	DynaMath	$+0.89$
old-319	DynaMath	$-0.74$
MathV pooled (71.5/28.5)	DynaMath	$+0.76$
MathV pooled (50/50)	DynaMath	$+0.40$
sparse-800	old-319	$-0.66$
sparse-800	DynaMath-easy	$+0.87$

Table 4: Spearman rank correlation between evaluation sets over the eleven trained arms (n = 11). The five correlations computable from Table 3 were independently recomputed from the published numbers and agree with the values below to within 0.004; the two easy-split rows use per-arm accuracies on DynaMath-easy alone.

Figure 2: Which held-out set predicts cross-domain transfer? Each point is one trained arm; the

y

axis is accuracy on DynaMath (

n{=}477

), the only evaluation pool with no overlap with the training distribution. Left: the hardest in-domain held-out set (sparse-800) ranks arms nearly as DynaMath does (Spearman

\rho=+0.89

n=11

). Right: the set used as a “general-distribution” check since round 1 (old-319) anti-predicts transfer (

\rho=-0.74

): vapo tops old-319 while placing last among trained arms on DynaMath, and vapohl nearly inverts that. Dashed line: untrained base model (25.8).

Figure 3: Spearman rank correlations between evaluation sets over the eleven trained arms, computed from the blind-judged accuracies in Table 3 . sparse-800 is the strongest predictor of cross-domain accuracy ( $+0.89$ ); old-319 is strongly anti-correlated with it ( $-0.74$ ) and with sparse-800 itself ( $-0.66$ ). The pooled in-domain metric inherits its predictive validity mechanically from its sparse-800 component (the pool contains both sets; weights 71.5/28.5).

Earlier reading	Corrected
All arms lie on one specialization-versus-generalization trade-off curve; none advances both ends.	The apparent curve is capacity reallocation within MathV360K. Across domains there is no trade-off: stronger in-domain predicts stronger out-of-domain.
vapohl is the specialization extreme, paying a generalization tax.	vapohl is second-best cross-domain (31.9). Its low old-319 score reflects only that it stopped guessing option letters.
vapo is the robustness champion—best on the “general-distribution” set.	vapo is the weakest trained arm cross-domain (27.0, below plain grpo ). old-319 had the ranking inverted.

Table 5: Earlier readings and their corrections after the evaluation-validity analysis.

Figure 4: One change: replacing vapo ’s clipped-MSE value loss with HL-Gauss cross-entropy ( vapohl ), everything else identical. In-domain accuracy on sparse-800 moves

+14.4

pp and cross-domain DynaMath from 27.0 to 31.9, while old-319 drops—the arm stopped guessing option letters (Section 4.2 ).

Hypothesis	Verdict
Answer-space guessing	Rejected: 48.9% accuracy against an 8.7% aggregate guessing ceiling, with a diverse answer distribution.
Format-reward hacking	Rejected: correct rows carry genuine multi-step reasoning (median 270 characters).
Template leakage from training	Partial: roughly 3 pp of attributable inflation; rows without a template duplicate in the training pool still score 45.9% against a 3.5% base.
Composition artifacts on old-319	Confirmed—and followed to the evaluation-validity finding of Section 4 .

Table 6: Four reward-hacking hypotheses and their verdicts.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer
May 28, 2026Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski +1RL for Language Models RL for Language Model Reasoning
Fin AI Research · Salk Institute for Biological Studies
Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing
Apr 9, 2026Pei-Xi Xie, Che-Yu Lin, Cheng-Lin YangReinforcement Learning LLM Mathematical Reasoning
CyCraft AI Lab, Taiwan
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Jun 2, 2026Ziyue Wang, Aomufei Yuan, Yongfu Zhu +10LLM Alignment RL for Language Model Reasoning
1Peking University · 2JINGDONG · 3Shanghai Innovation Institute +2

May 28, 2026J/K move · Enter open · S save

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski, Fergal Reid

Fin AI Research · Salk Institute for Biological Studies

RL for Language Models RL for Language Model Reasoning Cross-Domain Transfer Learning Reinforcement Learning with Verifiable Rewards LLM Mathematical Reasoning

Open paper AskPDF arXiv

Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains -- and why it does so -- remain under-explored. We study cross-domain transfer in a 7B model whose SFT and RL post-training stages use only constraint-satisfaction puzzles, with no mathematics problems in the post-training data. To analyze how transfer emerges, we introduce a reasoning primitive-level framework that combines a 9-class span classifier with motif extraction, allowing us to segment chain-of-thought traces into primitive motifs and track their evolution across training stages and domains. We find that puzzle SFT induces a reasoning-primitive vocabulary, yielding a

+7

pp pass@32 gain on OlymMATH-Hard. Vanilla GSPO then composes these primitives into longer compute-verify chains, adding a further

+6

pp. However, this RL stage also suppresses exploratory primitives such as hypothesize and backtrack. To address this, we introduce a novelty bonus that rewards diverse correct rollouts, using perplexity under the reference model as a signal. This restores recovery primitives during RL and adds a further

+7

pp pass@32 relative to vanilla GSPO. Finally, the end-to-end recipe raises the hard-math capability ceiling from

16.0\%

at the OLMo3-7B-Instruct-SFT base to

36.0\%

, without adding any mathematics problems during the SFT or RL stages.

Abstract

Reinforcement Learning

Explore similar work

When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

Mitigating Distribution Sharpening in Math RLVR via Distribution-Aligned Hint Synthesis and Backward Hint Annealing

Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning