Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations

Mihir Dhanakshirur; Adam Ousherovitch; Ambuj Tewari

cs.LGSep 24, 2026

Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations

Authors: Mihir Dhanakshirur, Adam Ousherovitch, Ambuj Tewari

Organizations: Department of Statistics, University of Michigan

Abstract

Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Entropy Penalty ( $\lambda$ )	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean	Std. Dev.
	Individual Seeds (Pass@1)					Aggregate
$0.0$ (Baseline)	0.5080	0.4620	0.4960	0.4940	0.4960	0.4912	$\pm 0.0154$
$0.3$	0.4840	0.5140	0.5040	0.5100	0.5160	0.5056	$\pm 0.0116$
$0.5$	0.5020	0.5100	0.5100	0.5000	0.5180	0.5080	$\pm 0.0064$
$1.0$	0.5220	0.5180	0.5320	0.5340	0.5320	0.5276	$\pm 0.0064$
$2.0$	0.5260	0.5500	0.5440	0.5420	0.5320	0.5388	$\pm 0.0086$

Appendix

Table 4: Detailed per-seed Pass@1 Accuracy on GSM8K across varying entropy penalty (

\lambda

) strengths using the Qwen2.5-1.5B-Instruct model.

Entropy Penalty ( $\lambda$ )	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean	Std. Dev.
	Individual Seeds					Aggregate
$0.0$ (Baseline)	0.5450	0.5300	0.5150	0.5200	0.5250	0.5270	$\pm 0.0103$
$0.3$	0.5300	0.5550	0.5350	0.5200	0.5500	0.5380	$\pm 0.0129$
$0.7$	0.5700	0.5700	0.5600	0.5800	0.5500	0.5660	$\pm 0.0102$
$1.0$	0.5750	0.5700	0.5750	0.5600	0.5750	0.5710	$\pm 0.0058$
$1.5$	0.5600	0.5600	0.5550	0.5400	0.5550	0.5540	$\pm 0.0073$

Appendix

Table 5: Detailed per-seed Greedy Pass@1 Accuracy on MBPP across varying entropy penalty (

\lambda

) strengths using the Qwen2.5-1.5B-Instruct model.

Figure 4: GSM8K Pass@1 under greedy decoding and under temperature sampling (

T=0.8

k=8

) versus entropy penalty

\lambda

. Dashed lines mark each curve’s cross-entropy baseline; stars mark each curve’s best

\lambda

. Error bars show

\pm 1

standard deviation over seeds.

	Greedy Pass@1	Sampled Pass@1 ( $T=0.8$ )
0.0 (Baseline)	0.4912 $\pm$ 0.0154	0.3825 $\pm$ 0.0023
0.3	0.5056 $\pm$ 0.0116	–
0.5	0.5080 $\pm$ 0.0064	–
1	0.5276 $\pm$ 0.0064	0.4942 $\pm$ 0.0144
2	0.5388 $\pm$ 0.0086	–
3	0.5504 $\pm$ 0.0083	0.5246 $\pm$ 0.0060

Appendix

Table 6: GSM8K Pass@1 accuracy under varying entropy penalties, under greedy decoding and under temperature sampling (

T=0.8

k=8

samples per problem). Model: Qwen2.5-1.5B-Instruct . The sampled column is evaluated on a coarser

\lambda

grid and over 3 seeds; dashes mark configurations not run under sampling.

Figure 5: MBPP Pass@1 under greedy decoding and under temperature sampling (

T=0.8

k=4

) versus entropy penalty

\lambda

. Dashed lines mark each curve’s cross-entropy baseline; stars mark each curve’s best

\lambda

. Error bars show

\pm 1

standard deviation over 5 seeds.

	Greedy Pass@1	Sampled Pass@1 ( $T=0.8$ )
0.0 (Baseline)	0.5270 $\pm$ 0.0103	0.4223 $\pm$ 0.0064
0.3	0.5380 $\pm$ 0.0129	0.4878 $\pm$ 0.0117
0.7	0.5660 $\pm$ 0.0102	0.5137 $\pm$ 0.0153
1	0.5710 $\pm$ 0.0058	0.5235 $\pm$ 0.0093
1.5	0.5540 $\pm$ 0.0073	0.5215 $\pm$ 0.0106
2	0.5490 $\pm$ 0.0058	0.5170 $\pm$ 0.0041

Appendix

Table 7: MBPP Pass@1 accuracy under varying entropy penalties, under greedy decoding and under temperature sampling (

T=0.8

k=4

samples per problem). Model: Qwen2.5-1.5B-Instruct .

Entropy Penalty ( $\lambda$ )	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4	Mean	Std. Dev.
	Individual Seeds					Aggregate
0.0 (Baseline)	0.4113	0.4300	0.4238	0.4200	0.4263	0.4223	$\pm$ 0.0064
0.3	0.5038	0.4688	0.4950	0.4863	0.4850	0.4878	$\pm$ 0.0117
0.7	0.5413	0.5175	0.5100	0.4975	0.5025	0.5138	$\pm$ 0.0153
1	0.5312	0.5250	0.5100	0.5350	0.5162	0.5235	$\pm$ 0.0093
1.5	0.5125	0.5175	0.5100	0.5300	0.5375	0.5215	$\pm$ 0.0106

Appendix

Table 8: Detailed per-seed sampled Pass@1 accuracy on MBPP at

T=0.8

with

k=4

samples per problem, using Qwen2.5-1.5B-Instruct .

Figure 6: MATH Pass@1 under greedy decoding and under temperature sampling (

T=0.8

k=4

) versus entropy penalty

\lambda

for Qwen2.5-7B-Instruct . Dashed lines mark each curve’s cross-entropy baseline; stars mark each curve’s best

\lambda

. Error bars show

\pm 1

standard deviation over 3 seeds.

	Greedy Pass@1	Sampled Pass@1 ( $T=0.8$ )
0.0 (Baseline)	0.4200 $\pm$ 0.0071	0.3696 $\pm$ 0.0121
1	0.4583 $\pm$ 0.0062	0.4179 $\pm$ 0.0097
2	0.4400 $\pm$ 0.0163	0.4350 $\pm$ 0.0041
4	0.4717 $\pm$ 0.0118	0.4496 $\pm$ 0.0048
8	0.4667 $\pm$ 0.0143	0.4675 $\pm$ 0.0145

Appendix

Table 9: MATH Pass@1 accuracy under varying entropy penalties, under greedy decoding and under temperature sampling (

T=0.8

k=4

samples per problem). Model: Qwen2.5-7B-Instruct .

Entropy Penalty ( $\lambda$ )	Seed 0	Seed 1	Seed 2	Mean	Std. Dev.
	Individual Seeds			Aggregate
0.0 (Baseline)	0.3525	0.3775	0.3787	0.3696	$\pm$ 0.0121
1	0.4175	0.4300	0.4062	0.4179	$\pm$ 0.0097
2	0.4300	0.4400	0.4350	0.4350	$\pm$ 0.0041
4	0.4450	0.4475	0.4562	0.4496	$\pm$ 0.0048
8	0.4475	0.4813	0.4738	0.4675	$\pm$ 0.0145

Appendix

Table 10: Detailed per-seed sampled Pass@1 accuracy on MATH at

T=0.8

with

k=4

samples per problem, using Qwen2.5-7B-Instruct .

Figure 7: Greedy Pass@1 on GSM8K for five Renyi orders across penalty strengths. The dashed line is the cross-entropy baseline, which all curves share at

\lambda=0

. Error bars show

\pm 1

standard deviation over 3 seeds. Orders

\alpha\in\{0.5,1,2\}

are mutually indistinguishable;

\alpha=0.1

never separates from the baseline and

\alpha=0.25

collapses at

\lambda=8

	$\lambda=1$	$\lambda=3$	$\lambda=8$
$\alpha=0.1$	$0.4700\pm 0.0082$	$0.4883\pm 0.0225$	$0.4483\pm 0.0062$
$\alpha=0.25$	$0.4733\pm 0.0232$	$0.5133\pm 0.0366$	$0.1883\pm 0.0085$
$\alpha=0.5$	$0.5450\pm 0.0071$	$0.5717\pm 0.0143$	$0.5717\pm 0.0103$
$\alpha=1$ (Shannon)	$0.5517\pm 0.0287$	$0.5600\pm 0.0147$	$\mathbf{0.5800\pm 0.0212}$
$\alpha=2$	$0.5283\pm 0.0155$	$0.5733\pm 0.0170$	$0.5750\pm 0.0122$

Appendix

Table 11: Greedy Pass@1 on GSM8K by Renyi order

\alpha

and penalty strength

\lambda

, aggregated over 3 seeds. The

\lambda=0

baseline (

0.4717\pm 0.0062

) is shared by all orders.

Figure 8: GSM8K with and without a chain of thought. Left: answer-only greedy Pass@1 against the entropy penalty, with its cross-entropy baseline dashed. Right: change from each regime’s own baseline, so the two can be compared despite very different absolute accuracies. The chain-of-thought curve rises monotonically to

+9.04

points; the answer-only curve is flat through

\lambda=4

and then falls to

-5.60

. Error bars show

\pm 1

standard deviation over 5 seeds, and on the right the standard error of the difference from baseline.

$\lambda$	Pass@1	Change (95% CI)
0.0 (Baseline)	$0.1120\pm 0.0097$	—
0.3	$0.1164\pm 0.0067$	$+0.44\ [-0.72,+1.60]$
0.5	$0.1204\pm 0.0067$	$+0.84\ [-0.32,+2.00]$
1.0	$0.1244\pm 0.0066$	$+1.24\ [+0.09,+2.39]$
2.0	$0.1288\pm 0.0074$	$+1.68\ [+0.48,+2.88]$
3.0	$0.1212\pm 0.0047$	$+0.92\ [-0.14,+1.98]$

Appendix

Table 12: GSM8K answer-only greedy Pass@1 under varying entropy penalties, aggregated over 5 seeds, with the change from the cross-entropy baseline in percentage points (95% Welch interval).

Explore similar work

Label-Free Reinforcement Learning via Cross-Model Entropy
May 27, 2026Matt Gorbett, Hossein ShiraziReinforcement Learning Instruction Following
Independent Researcher · San Diego State University
Cross-Model Disagreement as a Label-Free Correctness Signal
Mar 26, 2026Matt Gorbett, Suman JanaLanguage Model Error Detection Confidence Estimation in Language Models
Independent Researcher · Department of Computer Science, Columbia University, New York, NY 10027, USA
A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards
Sep 27, 2026Andreas Plesner, Curtis Northcutt, Francisco Guzmán +1Reward Modeling LLM Reliability
ETH Zurich · Handshake AI

May 27, 2026J/K move · Enter open · S save

Label-Free Reinforcement Learning via Cross-Model Entropy

Matt Gorbett, Hossein Shirazi

Independent Researcher · San Diego State University

Reinforcement Learning Instruction Following RL Post-Training Reward-Free RL Reward Design for RL

Open paper AskPDF arXiv

Post-training large language models with reinforcement learning is bottlenecked by the reward signal. Existing approaches require either ground-truth verifiable rewards, restricting training to domains with automatic correctness checks (e.g., mathematics, code execution), or human preference labels, which are expensive to collect and prone to reward hacking. Recent label-free methods replace ground-truth verifiers with self-referential signals like majority voting or token entropy over a model's own outputs, but risk reinforcing a model's own errors. In this work we propose Cross-Model Entropy (CME), the mean log-likelihood of a generator's response under a separate verifier model, as a label-free reward signal for RL post-training. CME is continuous, training-free, and grounded in the principle that responses a verifier finds unsurprising are likely correct or high quality. Because the verifier is independent of the generator, the signal cannot be gamed through self-consistency. We integrate CME into GRPO with no other changes to the training loop, extending label-free RL to open-ended instruction following -- a regime where self-referential signals are inapplicable or poorly suited. On open-ended instruction following (UltraFeedback prompts, evaluated on AlpacaEval 2.0), CME rewards beat the untrained base in head-to-head LLM-as-Judge comparisons across four model families (Qwen, Llama, Gemma, OLMo) and three training regimes (pretrained, SFT, and instruction-tuned), with tie-adjusted win rates ranging from 52.5% to 71.4%. Code will be released upon publication.

Abstract

Explore similar work

Label-Free Reinforcement Learning via Cross-Model Entropy

Cross-Model Disagreement as a Label-Free Correctness Signal

A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards