Cross-entropy (CE) training provides dense and scalable supervision for language models, but it optimizes next-token prediction under teacher forcing rather than sequence-level behavior under model rollouts. We introduce a feature-matching objective for language-model fine-tuning that targets sequence-level statistics of the completion distribution, providing dense semantic feedback without requiring a task-specific verifier or preference model. To optimize this objective efficiently, we propose energy-based fine-tuning (EBFT), which uses strided block-parallel sampling to generate multiple rollouts from nested prefixes concurrently, batches feature extraction over these rollouts, and uses the resulting embeddings to perform an on-policy policy-gradient update. We present a theoretical perspective connecting EBFT to KL-regularized feature-matching and energy-based modeling. Empirically, across Q&A coding, unstructured coding, and translation, EBFT matches RLVR and outperforms SFT on downstream accuracy while achieving a lower validation cross-entropy than both methods.
Figures & tables
Figure 1: Feature-matching loss grows with completion length. Conditional feature-matching loss (lower is better) as a function of completion length for Qwen2.5-1.5B fine-tuned with SFT on OpenCodeInstruct ( Ahmad et al., 2025 ) . Although this increase is expected even under a perfect model due to growing feature variance, part of the degradation reflects SFT’s inability to calibrate the model’s rollout distribution over long horizons.
Figure 2: EBFT achieves the lowest feature-matching loss across all completion lengths. Despite training with a rollout horizon of only 8 tokens, EBFT’s gains persist and grow at longer completions. RLVR worsens this loss relative to the base model.
Figure 3: EBFT improves downstream performance without sacrificing distributional calibration. From left to right, we plot HumanEval accuracy (greedy and pass@16), validation cross-entropy (CE), and conditional feature-matching (CFM) loss over training for Qwen2.5-1.5B fine-tuned on OpenCodeInstruct ( Ahmad et al., 2025 ) . SFT improves cross-entropy and CFM loss but lags on downstream accuracy. RLVR improves downstream accuracy but substantially degrades both calibration metrics relative to the base model (dashed line). EBFT achieves the best results across all four metrics, avoiding this tradeoff. CE and CFM losses are computed on a 1k-samples held-out subset of OpenCodeInstruct.
Figure 4: Overview of Energy-Based Fine-Tuning (EBFT). For each context c , the generator pθ samples n completions. A frozen feature network ϕ embeds each prompt–completion pair, producing features ϕ(c:y^j) for the sampled completions and ϕ(c:y) for the ground truth. Each completion receives a feature-matching reward measuring alignment with the ground-truth feature moment, and the generator is updated via REINFORCE with an RLOO baseline.
Table 1: EBFT outperforms SFT and matches or exceeds RLVR on downstream metrics, while achieving the best distributional calibration across all tasks. Best results per method on Q&A coding, unstructured coding, and translation. CE: validation cross-entropy; FM: feature-matching loss (both lower is better). “ws.”: warm-started from an SFT checkpoint. RLVR is inapplicable to unstructured coding where no verifier exists; EBFT still yields substantial gains over SFT in this setting. See Table 6 for per-benchmark results and Section 3 for full experimental details.
Figure 5: On translation, EBFT outperforms both SFT and RLVR on downstream accuracy, cross-entropy, and feature-matching loss. From left to right, we plot COMET scores on WMT22 and MTNT, validation cross-entropy, and CFM loss over training for Llama-3.2-1B fine-tuned on ALMA ( Xu et al., 2023 ) . EBFT achieves the lowest CE and CFM losses and matches SFT on WMT22 while clearly outperforming it on MTNT. RLVR underperforms SFT on all four metrics, with cross-entropy rising well above the base model (dashed line).
Figure 6: The CE regularization weight γ controls cross-entropy reduction without affecting downstream performance or feature-matching loss. We ablate EBFT with γ∈{0,0.03,0.1} on Qwen2.5-1.5B. Dashed and dotted lines indicate the base model and 2-epoch SFT, respectively. Larger γ accelerates CE reduction while downstream accuracy and CFM loss remain nearly identical across settings. Even pure feature matching ( γ=0 ) surpasses SFT on cross-entropy, confirming that the two objectives are aligned rather than in tension.
Figure 7: Feature network ablations: whitening and last-token pooling matter most; scaling the feature network does not help. HumanEval accuracy, validation cross-entropy, and CFM loss over training for EBFT ( γ=0 ) on Qwen2.5-1.5B with different feature network configurations. The default (last-token features with whitening from a frozen 1.5B copy) achieves the best downstream accuracy and CFM loss. Removing whitening and mean pooling cause the largest degradations. Random weights hurt only modestly, and replacing the 1.5B feature network with a frozen Qwen2.5-7B yields similar results, suggesting that pre-trained representations help but that naively scaling the feature network does not.
Figure 8: EBFT improvements are consistent across model scales. HumanEval accuracy, validation cross-entropy, and CFM loss over training for EBFT ( γ=0 ) applied to Qwen2.5-1.5B, 3B, and 7B. Each model uses a frozen copy of itself as the feature network. Dashed lines indicate base model performance. All three scales show substantial and qualitatively similar improvements across all four metrics, with no sign of diminishing returns.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Strided parallel rollouts for a sequence of length L=12 , stride S=4 and completion length G=4 .
Figure 10: Custom attention matrix A in for a sequence of length L=12 , stride S=4 and completion length G=4 . When the entry Aij is 0, the token in position i attends to the token in position j , and when it is −∞ it does not.
Parameter
Value
Q&A Code
Unstructured Code
Translation
Rollout Batch Size
16
Sequence Length
1024
Completion Length
8
8
4
Stride
8
8
2
Actor Learning Rate
1×10−6
Appendix
Table 2: Hyperparameter Configuration for EBFT runs.
Parameter
Value
Training Batch Size
64
Epochs
1
Max Length
2048
Learning Rate
1×10−5
Scheduler
Warmup + Cosine Decay to 0.1 × lr
Warmup
0.03
Appendix
Table 3: Hyperparameter Configuration for SFT warmstart runs.
Table 4: Hyperparameter Configuration for SFT baseline runs.
Parameter
Value
Q&A Code
Translation
Rollout Batch Size
{32, 64}
{128, 256}
Prompt Max Length
1024
Generate Max Length
1024
Actor Learning Rate
1×10−6
Temperature
1.0
Appendix
Table 5: Hyperparameter Configuration for RLVR baseline runs.
Figure 11: On Q&A Coding, increasing the cross-entropy weight α consistently lowers both validation cross-entropy and feature-matching losses, while SFT on full sequences yields faster initial downstream gains that quickly degrade. We sweep α∈{0,0.5,1.0} and γ∈{0,0.03,0.1} for EBFT initialized from base Qwen2.5-1.5B. As a baseline, we compare against SFT trained on full sequences (solid red), whereas the main text reports SFT trained only on the answer. SFT on full sequences improves downstream performance faster early on but quickly deteriorates, and answer-level cross-entropy rises. Setting α=0 and γ=0 (blue dotted) causes both cross-entropy and moment-matching losses to increase and leads to degraded pass@1 and pass@ k scores, indicating that both terms are necessary for stable training. We also report feature-matching loss with and without whitening; the non-whitened variant tracks more closely with cross-entropy, which is why we use it for comparison. Overall, increasing α has limited effect on downstream metrics but helps decrease both the cross-entropy and moment-matching objectives. The validation set is a 1k-sample held-out subset of OpenCodeInstruct.
Figure 12: On unstructured coding, EBFT achieves better downstream performance than SFT while reaching comparable levels on cross-entropy and feature-matching losses, even as downstream accuracy plateaus before these losses converge. We sweep α∈{0,0.5,1.0} and γ∈{0,0.03,0.1} for EBFT initialized from base Qwen2.5-1.5B. Cross-entropy and feature-matching losses continue to decrease throughout training even after downstream accuracy has plateaued, which we attribute to distribution shift between the training data and the downstream evaluation benchmarks. Setting α=0 (blue dotted) again leads to diverging cross-entropy, confirming the importance of the cross-entropy term. SFT (solid red) achieves the lowest cross-entropy and feature-matching losses but at the cost of weaker downstream performance compared to EBFT configurations with γ>0 . The validation set is a 1k-sample held-out subset of the unstructured coding data.
Figure 13: On translation, EBFT matches or exceeds RLVR on downstream COMET scores—outperforming it on MTNT—while consistently surpassing SFT across all benchmarks and evaluation settings. We sweep α∈{0,0.5,1.0} and γ∈{0,0.03,0.1} for EBFT initialized from base Llama-3.2-1B. On WMT’22, EBFT performs comparably to or slightly below RLVR, while on MTNT it clearly outperforms RLVR, and on OpenSubtitles the two are similar. EBFT surpasses SFT across all benchmarks and decoding strategies (greedy, best-of-1/4/16). Unlike in the coding setting, using α=0 and γ=0 (blue dotted) leads to degraded downstream performance, indicating that both terms are important for translation. SFT is trained only on answers, as training on full sequences led to worse downstream results. On cross-entropy and feature-matching losses, EBFT achieves lower values than SFT, with larger γ configurations (green) reaching the best feature-matching levels. The legend is shared with the previous figures. The validation set is a 1k-sample held-out subset of ALMA.
Figure 14: On translation BLEU scores, EBFT consistently outperforms both SFT and RLVR, while RLVR degrades at best-of-16 later in training—consistent with the distribution sharpening induced by RLVR. We report BLEU scores for the same α and γ sweep as the previous figure, initialized from base Llama-3.2-1B. Although RLVR is more competitive on BLEU than on COMET, EBFT still achieves higher scores across most benchmarks and decoding strategies. Notably, at best-of-16 (temp. 0.6), RLVR performance drops in the second half of training, suggesting that at large k the model suffers from reduced sample diversity, consistent with the distribution sharpening induced by RLVR ( Zhao et al., 2025 ) . EBFT configurations with γ>0 do not exhibit this degradation and maintain stable or improving BLEU throughout training. The legend is shared with the previous figures.
Figure 15: HumanEval/18: counting overlapping substring occurrences. EBFT correctly advances by one character after each match and therefore counts overlapping occurrences. By contrast, the base model uses Python’s non-overlapping string.count , while both SFT and RLVR advance by len(substring) , which misses overlap-sensitive cases and causes test failures.
Figure 16: HumanEval/94: largest prime digit sum. EBFT produces a self-contained executable solution by defining is_prime and then computing the digit sum of the largest prime in the list. The base model and RLVR instead rely on an undefined helper function, while SFT truncates mid-function and yields syntactically invalid Python, illustrating a recurring executability gap between EBFT and the competing methods.
Figure 17: HumanEval/105: filtering, ordering, and lexical mapping. EBFT correctly implements the full prompt: it filters to digits 1 to 9, sorts the valid entries, reverses the order, and maps each digit to its English name. The other outputs each satisfy only part of the specification: the base model returns the input unchanged, SFT incorrectly returns the original array whenever any element is out of range instead of filtering, and RLVR outputs numeric strings rather than digit names.
Figure 18: HumanEval/69: selecting the greatest frequency-qualified integer. EBFT returns the maximum positive integer whose frequency is at least its value, matching the full prompt semantics. The base model fails to provide a usable implementation, SFT returns the first qualifying key rather than the greatest one, and RLVR follows the right high-level logic but is not executable under strict evaluation because the output includes formatting artifacts outside valid code.
Figure 19: MTNT EN → FR: non-translation and multilingual runaway. EBFT remains on task and produces a direct French translation, whereas the base model and RLVR drift into multilingual template continuations that repeat the source sentence and append language-tagged variants. SFT avoids the runaway list structure but still fails to translate, remaining almost entirely in English.
Figure 20: MTNT EN → FR: truncation after template drift. EBFT produces a complete on-task translation, while the base model drifts into unrelated multilingual continuation and SFT paraphrases too loosely away from the original meaning. RLVR begins a language-tagged template expansion and terminates mid-token, yielding an unfinished output that is incompatible with strict evaluation.
Figure 21: MTNT EN → FR: semantic failure from dropped negation. Not all errors are formatting failures: this example shows that negation can be overlooked by the fine-tuned models. EBFT preserves the negative meaning of the source sentence, whereas RLVR drops the negation and changes “no more available rooms” into “more rooms.” The base model again exhibits multilingual repetition after an initially plausible translation, and SFT produces a weaker paraphrase that underspecifies the original statement.
Supervised fine-tuning (SFT) followed by reinforcement learning (RL) has become a standard post-training paradigm for large language models. This paradigm provides a cold-start for RL exploration, avoiding the inefficiency of pure RL where on-policy sampling yields insufficient positive samples. However, in practice, existing approaches often use a small amount of data for SFT initialization compared to the RL phase, which can cause the model to fit the limited samples and shift away from its pre-trained distribution. This distribution shift impedes the model's ability to effectively explore during subsequent RL training. To address this challenge, we propose that in low-data regimes, SFT should prioritize activating task-relevant capabilities rather than memorizing specific content. Along this line, we propose EKSFT (Entropy-KL Selective Fine-Tuning), which selectively masks tokens that exhibit either high entropy or high KL divergence from a reference model. By excluding these high-uncertainty, distribution-shifting tokens from imitation, EKSFT injects task-specific knowledge while preserving the integrity of the model's pre-trained distribution. Empirical evaluations on mathematical reasoning benchmarks demonstrate that EKSFT consistently outperforms standard SFT. Further RL fine-tuning from the EKSFT model yields consistently better post-RL performance, indicating improved exploration for the RL stage. Our codes and datasets are available at https://github.com/MINE-USTC/EKSFT.
Qi Liu, Mingdi Sun, Yongyi He +5
University of Science and Technology of China · Huawei Cloud
Fine-tuning a language model on contexts whose correct completion has a near-synonym competitor often fails silently. The cross-entropy loss decreases monotonically while the correct token never overtakes the competitor in rank. We study this regime across five transformer architectures spanning two families and a fivefold parameter range, on ten hand-selected near-synonym contexts. We instrument these failures with an order parameter combining the predicted distribution and pairwise embedding overlaps. It decomposes additively into a signal, tracking the model's commitment to the correct token over its nearest competitor, and a background drag, set by how the embedding bulk leaks probability into the score. This isolates two failure modes. In kinematic failure the signal stays small. In structural failure the drag actively worsens as fine-tuning proceeds. We observe sharp catapult-like jumps in the order parameter that resemble a phase transition. A central negative result organises the paper. The transitions are phantoms. The spontaneous-symmetry-breaking interpretation is ruled out by direct measurement. Catapult-like jumps still appear under LoRA fine-tuning with the token embedding matrix exactly unchanged during training, where no geometric phase transition is possible. The discontinuity lives entirely in the softmax readout. A small number of dimensionless quantities organise the trajectory across architectures. One is consistent across all five under full fine-tuning. A second sorts architectures into two classes by bulk embedding distribution and predicts LoRA sufficiency. As a blind test, the framework predicts the critical learning rate of a held-out architecture, not used to fit any parameter, to within 2.1% of a subsequent learning-rate sweep. Findings concern the near-synonym mechanism only and should not be extrapolated without recalibration.
Vaibhav Prakash, Jayasri Dontabhaktuni
Department of Physics, Mahindra University, Hyderabad, India
Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining. We first analyze next-token predictions using Shannon and Renyi entropies, revealing that pretrained models exhibit a regular multimodal entropy structure. These entropy peaks correspond to varying numbers of plausible alternatives, indicating that the base model intrinsically encodes rich distributional knowledge beyond the single supervised token. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure. At each step, LP-SFT constructs an adaptive support of alternative tokens and applies a locally normalized preservation loss to maintain the base model's relative structure among them, while standard cross-entropy independently optimizes the supervised token. Across mixed-domain and single-domain fine-tuning experiments, LP-SFT improves overall performance over vanilla SFT and recent SFT-enhancement baselines, achieving the best balance between pass@1 accuracy and pass@k performance. These results suggest that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.
Yueyang Wang, Baolong Bi, Shuo Lu +1
School of Mathematical Sciences, Peking University · Institute of Computing Technology, Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences +1