How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

Johnny Yu; Cyril Gorlla

cs.LGOct 26, 2026

How Much Does the Hint Matter? Scope Conditions for Hindsight Self-Distillation in Financial Reasoning

Authors: Johnny Yu, Cyril Gorlla

Organizations: University of California, Berkeley Berkeley, California, USA · CTGT San Francisco, California, USA

Abstract

Financial institutions operating under data-residency constraints are restricted to open-weight models small enough to locally host, which trail frontier systems on multi-step quantitative reasoning. On-policy distillation narrows such gaps by supervising a student on its own rollouts, but traditional OPD requires a strong external teacher, and on-policy self-distillation (OPSD) conditions on a full reference solution. We ask how much of that signal is necessary. We present Fin-SD, a localized hindsight self-distillation recipe for multi-step reasoning. Given a problem the student fails, the student itself identifies the first incorrect reasoning step and writes a short critique of that step. We truncate the student's trajectory immediately before the error (aiming to obtain a correct prefix), condition a frozen copy of the student on the same prefix plus the hint, and distill that hint-conditioned teacher into the student over a short rollout under reverse KL. No external model contributes a gradient. We apply Fin-SD to GPT-OSS-20B on a custom multi-step financial reasoning benchmark that requires deriving formula inputs from realistic scenarios via accounting conventions. We find: (i) replacing the critique with a generic-feedback control matches Fin-SD's entire 3.03 pp accuracy gain over the base model on the FinanceReasoning Hard subset, suggesting the gain comes from where supervision is applied, not hint content; (ii) the model writing the hint does not need to be stronger than the model receiving it: hints from same-family GPT-OSS-120B, at roughly 6x the parameters, yield no materially greater benefits than self-written hints; and (iii) Fin-SD matches on-policy self-distillation from full reference solutions on accuracy while generating 9.5% fewer tokens. With Fin-SD, general reasoning on MMLU-Pro and a fourteen-metric compliance audit on FinTrust show no material regression.

Figures & tables

Figure 1. Overview of Fin-SD. The student is sampled on a training problem and judged against the ground truth. On failures, a frozen copy of the student receives the query, the failing trajectory, and the answer, and returns the first faulty step $k$ , a short critique $h$ , and an error type; pure arithmetic slips are dropped without a gradient. The trajectory is truncated immediately before k, so both policies condition on the on-policy prefix before the localized error, with the frozen teacher also receiving $h$ . The student minimizes the per-token KL between the two next-token distributions over an L = 100 rollout. At each epoch boundary, the failure set is re-mined and the teacher re-frozen.

Panel of Figure 2 — Figure 2. One item from the training corpus, carried through the three generation phases of Section 4.1 and then through the localization and truncation of Section 3.2 .

Model and optimization
Setting	Value
Student model	GPT-OSS-20B ( OpenAI, 2025 )
Student model effort	Medium
LoRA attention $r$	$64$
LoRA $\alpha$	$128$
LoRA expert $r$	$8$

Table 1. Student fine-tuning hyperparameters.

Configuration	$n$	Accuracy (%)	95% CI	Completion rate (%)	95% CI	Tokens (k)	95% CI
Fin-SD	5	$\mathbf{72.19\pm 1.79}$	$[70.0,\,74.4]$	$\mathbf{92.19\pm 1.96}$	$[89.8,\,94.6]$	$\mathbf{508.5\pm 26.5}$	$[476,\,541]$
OPSD (full solution)	5	$70.42\pm 1.64$	$[68.4,\,72.5]$	$89.75\pm 1.35$	$[88.1,\,91.4]$	$562.1\pm 17.0$	$[541,\,583]$
Base	5	$69.16\pm 1.85$	$[66.9,\,71.5]$	$91.18\pm 1.95$	$[88.8,\,93.6]$	$525.8\pm 13.6$	$[509,\,543]$
SFT	3	$62.46\pm 3.50$	$[53.8,\,71.2]$	$99.02\pm 0.24$	$[98.4,\,99.6]$	$228.9\pm 3.9$	$[219,\,239]$

Table 2. Fin-SD against the three baselines. Mean

\pm

SD over seeds, with 95% confidence intervals.

Panel of Figure 3 — Figure 3. Accuracy relative to the base model on the FinanceReasoning Hard subset, including baselines and ablations. Points are mean differences in percentage points; bars are $t$ -based 95% confidence intervals. All configurations are $n=5$ seeds except SFT ( $n=3$ ); base is $n=5$ independent runs. The SFT interval extends to $-16.10$ and is clipped at the axis limit (arrow).

	Base	Fin-SD	$\Delta$
Overall accuracy (%)	$68.91$	$68.34$	$-0.57$ pp
Execution rate (%)	$98.56$	$98.77$	$+0.21$ pp
Mean tokens	1,742	1,692	$-2.87\%$

Table 3. MMLU-Pro.

Dimension / metric	Base	Fin-SD	Change
Trustfulness — hallucination (%) $\downarrow$	$30.0$	$30.0$	flat
Trustfulness — refuse-to-answer (%) $\uparrow$	$69.0$	$70.0$	$+$
Trustfulness — correctness (%) $\uparrow$	$56.0$	$61.0$	$+$
Trustfulness — informativeness (1–5) $\uparrow$	$3.91$	$3.79$	$-$
Robustness — answerable (1–6) $\uparrow$	$4.56$	$4.54$	$-$
Robustness — unanswerable (1–6) $\uparrow$	$3.75$	$3.92$	$+$

Table 4. FinTrust compliance evaluation.

$\pi_{H}$	$d$	Acc. (%)	Compl. (%)	Tok. (k)
Fin-SD ( self , RKL)		$72.19\pm 1.79$	$92.19\pm 1.96$	$508.5\pm 26.5$
self	FKL	$70.93\pm 2.17$	$91.43\pm 1.64$	$529.1\pm 13.1$
120b	FKL	$71.26\pm 0.92$	$91.77\pm 1.05$	$509.1\pm 19.5$
120b	RKL	$72.19\pm 1.13$	$92.69\pm 0.70$	$496.1\pm 11.4$
Rollout length ( $L=8192$ )		$71.09\pm 1.13$	$91.77\pm 0.87$	$513.5\pm 15.9$

Table 5. Ablation grid: Mean

\pm

SD over five seeds.

Self-Distillation

Self-distillation trains a model from its own predictions, representations, or experience to improve capability, efficiency, and robustness.

See all 143 papers

Sep 9, 2026

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning

Financial text, textbooks, and question-answer pairs are abundant, but only a small fraction is directly usable for reasoning-focused post-training. Existing QA pairs often lack explicit reasoning, sufficient context, or reliably verifiable answers, while textbooks must first be transformed into synthetic training examples. We present a data-centric pipeline that constructs complementary corpora by mining open-source reasoning traces, distilling financial instruction data, and generating knowledge-graph-guided question-answer pairs from financial educational material. After semantic deduplication, three lightweight sequence classifiers select finance-relevant examples, reject under-specified questions, and identify tasks suitable for reinforcement learning with compact rule-based verifiers. For model adaptation, we study supervised fine-tuning and reinforcement learning, while self-distilled fine-tuning and post-training model merging are used to prevent the loss of financial capabilities already present in the starting model. We evaluate the adapted language models using FINESSE-Bench, reporting aggregate performance and changes relative to their starting checkpoints. Across the selected comparisons, ordinary SFT reduces FINESSE-Bench accuracy by 3.2-4.0 percentage points, whereas self-distilled SFT improves over the corresponding starting models by 1.0-2.8 points. Equal-weight merging recovers 3.0 points over its SFT parent and finishes 0.9 points above the original model; GRPO on hard tasks adds 0.4 points after self-distilled SFT or 3.0 points when applied directly to verifiable tasks. These results show that retention-aware adaptation can improve financial reasoning without the regressions observed after ordinary SFT.

Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii +2

Financial Reasoning in Language Models RL for Language Model Reasoning Language Model Distillation LLM Post-Training Language Model Post-Training

Ask PDF

Jun 14, 2026

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal. But the common choice of a ground-truth answer is only an endpoint cue: on terse-answer tasks, the teacher falls silent at the intermediate positions where path-level guidance matters most. We propose Hindsight Self-Distillation (HSD), which conditions the teacher on a successful peer rollout drawn from the current training group. Such a peer is an exact sample from the success-conditioned policy, requiring no additional sampled rollouts. By providing a full successful continuation rather than only the final answer, the resulting credit signal concentrates at the divergence position between a failed rollout and a successful peer. Across Qwen3-8B and Qwen3-32B on math and code benchmarks, HSD obtains the best result against GRPO variants and on-policy distillation baselines, with the largest gains on terse-answer tasks such as AIME.

Yu Li, Shu Hong, Tian Lan

Department of Electrical and Computer Engineering, George Washington University

RL for Language Model Reasoning Credit Assignment in RL Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards On-Policy Distillation Teacher-Student Learning+1

Ask PDF

Jul 6, 2026

Rethinking On-Policy Self-Distillation for Thinking Models

Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.

Simran Kaur, Narutatsu Ri, Yinghui He +2

Princeton Language and Intelligence, Princeton University

LLM Self-Correction On-Policy Self-Distillation On-Policy Distillation Context Distillation Learning Using Privileged Information Large Reasoning Models

Ask PDF

Abstract

Self-Distillation

Explore similar work

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

Rethinking On-Policy Self-Distillation for Thinking Models