One Reference or Many? Rethinking Multi-Reference Training for Intermediate-Entropy Text Generation

Maitreya Prafulla Chitale; Ketaki Mangesh Shetye; Yash More; Harshit Gupta; Manav Chaudhary; Manish Shrivastava; Vasudeva Varma

cs.CLSep 19, 2026

One Reference or Many? Rethinking Multi-Reference Training for Intermediate-Entropy Text Generation

Authors: Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Yash More, Harshit Gupta, Manav Chaudhary, Manish Shrivastava, Vasudeva Varma

Organizations: IIIT Hyderabad · Microsoft, India · Sentisum

Abstract

Natural language generation (NLG) ranges from highly constrained tasks, such as machine translation, to open-ended tasks like dialogue generation. Between them lie peer-review generation and image captioning, where one input admits several valid, overlapping outputs. We measure this intermediary region with Von Neumann entropy ( $\hat{H}\approx0.6$ vs. 0.15 translation, 0.92 dialogue). Single-reference training discards valid perspectives, while multi-reference training introduces high-variance, unaligned targets. To tackle this, we propose target-side consolidation, that is, merging all references for an input into a single target which preserves every distinct information unit. To this end, we introduce MERC-67K, pairing 36,537 papers and 31,014 images with their original and consolidated references. In human evaluation, models trained using consolidated reviews obtain an average rank of 1.70 (out of 4) for coverage against 2.51-2.98 for baselines, and win 63.5-74.0% of pairwise comparisons. Models trained using consolidated captions obtain an average rank of 1.87 for overall caption quality (accuracy and naturalness) against 2.64-2.82 for baselines and obtain the highest reference-free LongCLIPScore (0.839 vs best baseline's 0.816). For reviews, multi-reference training often under-performs zero-shot inference, and consolidation's gains over multi-reference training hold across Llama3.1, Mistralv0.3, and Qwen2.5. Consolidation uses $\sim$ 4 $\times$ (reviews) and 5 $\times$ (captions) fewer individual references during optimization. Dataset and code will be released upon acceptance.

Figures & tables

Figure 1: The NLG spectrum, classifying tasks by entropy from constrained to open-ended. The dashed box marks the intermediary region, which holds structured yet subjective tasks such as peer-review generation and image captioning.

Task	Entropy ( $\hat{H}$ )
Translation	0.15
Paraphrase Generation	0.35
Peer-Review Generation	0.61
Image Captioning	0.62
Story Generation	0.90
Dialogue Generation	0.92

Table 1: Average empirically normalized Von Neumann entropy (

\hat{H}=S/S_{\text{random}}

) across different NLG tasks. Higher values indicate more diverse references.

Peer-Review Generation (Papers)
Venue	Train	Val	Test
COLM 24	234	29	30
COLM 25	324	40	41
NeurIPS 23	2,686	336	336
NeurIPS 24	3,319	415	415
ICLR 24	4,507	564	563

Table 2: Papers and images per split in MERC-67K. Reviews use a venue-stratified 80-10-10 split and captions use the Karpathy split.

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	BS-F1
Single-Reference Systems
Llama-ZeroShot	Original	41.17	10.43	19.85	46.90	11.88	22.35	40.69	10.37	19.94	83.62
Llama-HighConf	Original	39.13	9.92	18.69	45.39	11.59	21.65	40.20	10.20	19.51	83.90
Multi-Reference Systems
Reviewer2	Original	29.77	7.55	15.80	58.62	15.43	32.30	22.33	5.60	11.76	83.71

Table 3: Automatic evaluation metrics comparing model outputs against the average of original unconsolidated target reviews. The highest score in each metric is marked in bold , while the second-highest is underlined . BS-F1 denotes BERTScore F1.

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	BS-F1
Single-Reference Systems
Llama-ZeroShot	Consolidated	44.98	14.85	21.26	68.62	22.74	32.54	33.87	11.16	15.99	85.48
Llama-HighConf	Consolidated	41.33	12.91	19.41	63.61	20.22	30.37	32.62	10.07	15.31	85.29
Multi-Reference Systems
Reviewer2	Consolidated	23.96	7.57	13.23	71.46	23.24	41.22	14.99	4.70	8.20	84.52

Table 4: Automatic evaluation metrics comparing model outputs against the consolidated target reviews. The highest score in each metric is marked in bold , while the second-highest is underlined . BS-F1 denotes BERTScore F1.

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	LCS
Single-Reference Systems
Molmo-SingleRef	Original	0.403	0.156	0.366	0.450	0.176	0.408	0.401	0.156	0.365	0.812
Multi-Reference Systems
BLIP-2 OPT-6.7B	Original	0.398	0.150	0.356	0.413	0.155	0.367	0.414	0.157	0.372	0.816
PaliGemma-3B	Original	0.412	0.162	0.376	0.469	0.185	0.425	0.393	0.155	0.360	0.801

Table 5: Automatic evaluation metrics comparing generated captions against the five original human written captions, averaged over references. LCS denotes LongCLIPScore, which is reference-free. The highest score in each metric is marked in bold , while the second-highest is underlined .

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	LCS
Single-Reference Systems
Molmo-SingleRef	Consolidated	0.392	0.159	0.347	0.635	0.260	0.561	0.303	0.124	0.269	0.812
Multi-Reference Systems
GIT-base	Consolidated	0.334	0.115	0.293	0.591	0.207	0.517	0.245	0.084	0.216	0.793
PaliGemma-3B	Consolidated	0.393	0.152	0.345	0.663	0.261	0.581	0.292	0.113	0.257	0.801

Table 6: Automatic evaluation metrics comparing generated captions against the single consolidated target caption. LCS denotes LongCLIPScore, which is reference-free. The highest score in each metric is marked in bold , while the second-highest is underlined .

Section	LC	LZ	LH	LM
Summary	2.08	2.50	2.57	2.85
Strengths	1.53	2.33	2.98	3.16
Weaknesses	1.71	2.81	2.78	2.70
Questions	1.48	2.40	2.92	3.20
Overall	1.70	2.51	2.81	2.98

Table 7: Average human-assigned rank per section (1-4, lower is better) for reviews generated by Llama-Consolidated (LC), Llama-ZeroShot (LZ), Llama-HighConf (LH), Llama-MultiRef (LM). The best-performing result is highlighted in bold , and the second-best result is underlined .

Model	Average Rank
Molmo-SingleRef	2.68
Molmo-MultiRef	2.64
Molmo-Consolidated	1.87
Molmo-ZeroShot	2.82

Table 8: Average human-assigned rank (1-4, lower is better) for captions generated by Molmo-Consolidated , Molmo-ZeroShot , Molmo-SingleRef , Molmo-MultiRef . The best-performing result is highlighted in bold and the second-best result is underlined .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Test Samples	Median Refs / Sample
Translation	3,003	2
Paraphrase Generation	3,246	3
Peer-Review Generation	3,659	4
Image Captioning	1,000	5
Story Generation	2,761	3
Dialogue Generation	1,142	5

Appendix

Table 9: Dataset statistics and median reference counts across the six evaluated NLG tasks used to compute empirical von Neumann entropy.

Hyperparameter	Value
Batch Size	1
Gradient Accumulation Steps	4
16-bit Floating Point Precision	True
Optimizer	adamw_torch
Learning Rate	$1\times 10^{-4}$
Max Gradient Norm	0.3

Appendix

Table 10: Hyperparameters Used During Finetuning

Set	n	Mean Tok	Median Tok
Raw Reviews	145,530	578.07	505
CR	36,537	1327.18	1272
Consolidated	3,659	704.67	702
Highconf	3,659	593.57	465
Multiref	3,659	508.48	430
ZeroShot	3,659	435.47	421

Appendix

Table 11: Output length statistics for peer-review generation (measured in Llama-3.1-8B-Instruct tokens).

Set	n	Mean Tok	Median Tok
Raw Captions	155,070	13.97	13
CR	31,014	27.88	27
Consolidated	1,000	25.20	24
Singleref	1,000	12.07	11
Multiref	1,000	12.36	12
Zeroshot	1,000	127.68	128

Appendix

Table 12: Output length statistics for image captioning (measured in Molmo-7B-O tokens).

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	BS-F1
Mistral-7B
Mistral-Consolidated	Original	41.09	8.32	16.14	36.97	7.45	14.34	52.39	10.71	21.07	83.86
Mistral-MultiRef	Original	37.56	7.59	15.58	40.72	8.33	17.08	41.95	8.42	17.37	83.57
Llama-3.1-8B
Llama-Consolidated-3k	Original	39.41	7.19	15.07	35.29	6.42	13.36	50.76	9.35	19.82	83.57

Appendix

Table 13: Ablation study across different model architectures evaluated against original unconsolidated target reviews. Bold values indicate the superior performance between the consolidation-driven and multi-reference paradigm for each respective model family. BS-F1 denotes BERTScore F1.

		F1			Precision			Recall
Model	Ground Truth	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L	BS-F1
Mistral-7B
Mistral-Consolidated	Consolidated	51.36	14.72	19.58	57.49	16.51	21.94	46.87	13.42	17.86	86.62
Mistral-MultiRef	Consolidated	38.62	9.14	15.68	54.62	13.28	22.90	32.42	7.53	12.88	84.70
Llama-3.1-8B
Llama-Consolidated-3k	Consolidated	47.70	11.94	17.83	53.20	13.34	19.91	43.79	10.94	16.34	86.10

Appendix

Table 14: Ablation study across different model architectures evaluated against the consolidated target reviews. Bold values indicate the superior performance between the consolidation-driven and multi-reference paradigm for each respective model family. BS-F1 denotes BERTScore F1.

Figure 2: Qualitative captioning example for image 3418504074.jpg . The single unified box compares the five human references against outputs from short-form trained variants and off-the-shelf long-form captioners.

		F1			Precision			Recall
Baseline	BS-F1	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Llama-ZeroShot	+2.17**	+12.05**	+6.06**	+3.42**	-3.81**	+1.01**	-4.50**	+17.45**	+7.65**	+6.22**
Llama-HighConf	+2.36**	+15.70**	+7.99**	+5.27**	+1.20**	+3.55**	-2.34**	+18.69**	+8.73**	+6.90**
Reviewer2	+3.14**	+33.07**	+13.33**	+11.44**	-6.63**	+0.51**	-13.19**	+36.32**	+14.09**	+13.99**
OpenReviewer	+2.07**	+14.40**	+7.34**	+4.83**	-2.03**	+2.33**	-3.52**	+18.87**	+8.53**	+7.19**
Llama-MultiRef	+2.31**	+16.71**	+8.37**	+5.63**	-0.51**	+3.12**	-3.44**	+20.61**	+9.35**	+7.79**

Appendix

Table 15: Paired significance tests of Llama-Consolidated against each baseline on the consolidated reviews. Entries are score differences (consolidated - baseline). significant differences (

p<0.01

) are denoted with **.

		F1			Precision			Recall
Baseline	BS-F1	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Llama-ZeroShot	+0.53**	-0.78**	-0.04	-1.25**	-9.95**	-2.44**	-5.59**	+9.63**	+2.74**	+3.82**
Llama-HighConf	+0.26**	+1.25**	+0.47**	-0.09	-8.44**	-2.15**	-4.89**	+10.12**	+2.91**	+4.25**
Reviewer2	+0.46**	+10.65**	+2.85**	+2.81**	-21.62**	-5.98**	-15.52**	+27.98**	+7.51**	+11.98**
OpenReviewer	+0.06**	-0.54**	-0.18**	-0.74**	-10.96**	-2.96**	-5.81**	+9.74**	+2.58**	+4.36**
Llama-MultiRef	+0.11**	+0.70**	+0.26**	-0.30**	-10.83**	-2.86**	-6.04**	+11.17**	+3.12**	+4.95**

Appendix

Table 16: Paired significance tests of Llama-Consolidated against each baseline on the original unconsolidated reviews. Entries are score differences (consolidated - baseline). significant differences (

p<0.01

) are denoted with **

		F1			Precision			Recall
Baseline (MultiRef)	BS-F1	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Llama-3.1-8B	+2.58**	+16.42**	+5.36**	+4.39**	+0.44**	+1.96**	-3.39**	+20.47**	+6.10**	+6.44**
Mistral-7B-v0.3	+1.92**	+12.74**	+5.58**	+3.91**	+2.87**	+3.22**	-0.96**	+14.46**	+5.89**	+4.98**
Qwen-2.5-7B	+2.15**	+13.59**	+4.38**	+4.14**	+2.51**	+2.34**	-1.39**	+15.40**	+4.56**	+5.21**

Appendix

Table 17: Paired significance tests of each consolidated model against its MultiRef counterpart on the consolidated reviews. significant differences (

p<0.01

) are denoted with **

		F1			Precision			Recall
Baseline (MultiRef)	BS-F1	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
Llama-3.1-8B	+0.70**	+5.57**	+1.10**	+0.58**	-6.31**	-1.23**	-4.70**	+17.88**	+3.50**	+5.81**
Mistral-7B-v0.3	+0.29**	+3.53**	+0.73**	+0.56**	-3.75**	-0.88**	-2.74**	+10.44**	+2.29**	+3.70**
Qwen-2.5-7B	+0.62**	+6.15**	+1.17**	+1.16**	-2.78**	-0.45**	-2.91**	+14.49**	+2.69**	+4.77**

Appendix

Table 18: Paired significance tests of each consolidated model against its MultiRef counterpart on the original unconsolidated reviews. Significant differences (

p<0.01

) are denoted with **.

		F1			Precision			Recall
Baseline	LCS	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
BLIP-2-6.7B	+0.02**	+0.07**	+0.06**	+0.04**	-0.10**	-0.00	-0.10**	+0.16**	+0.09**	+0.12**
GIT-Base	+0.05**	+0.14**	+0.09**	+0.10**	-0.08**	+0.02**	-0.09**	+0.23**	+0.13**	+0.18**
LLaVA-1.5-7B	+0.07**	+0.23**	+0.13**	+0.21**	+0.35**	+0.17**	+0.30**	-0.09**	+0.02**	-0.03**
MultiRef	+0.02**	+0.07**	+0.04**	+0.04**	-0.13**	-0.04**	-0.14**	+0.16**	+0.08**	+0.12**
PaliGemma-3B	+0.04**	+0.08**	+0.06**	+0.05**	-0.15**	-0.04**	-0.16**	+0.19**	+0.10**	+0.14**

Appendix

Table 19: Paired significance tests of Molmo-Consolidated against each baseline on the consolidated captions. Entries are score differences (Consolidated - baseline). Significant differences (

p<0.01

) are denoted with **.

		F1			Precision			Recall
Baseline	LCS	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
BLIP-2 (6.7B)	+0.02**	-0.03**	-0.01**	-0.03**	-0.11**	-0.04**	-0.10**	+0.12**	+0.04**	+0.09**
GIT-Base	+0.05**	+0.02**	+0.01**	-0.00	-0.12**	-0.04**	-0.12**	+0.21**	+0.08**	+0.16**
LLaVA-1.5 (7B)	+0.07**	+0.22**	+0.09**	+0.19**	+0.22**	+0.09**	+0.19**	-0.08**	-0.00	-0.05**
MultiRef	+0.02**	-0.04**	-0.03**	-0.05**	-0.14**	-0.07**	-0.14**	+0.12**	+0.03**	+0.09**
PaliGemma (3B)	+0.04**	-0.04**	-0.02**	-0.05**	-0.16**	-0.07**	-0.16**	+0.15**	+0.05**	+0.11**

Appendix

Table 20: Paired significance tests of Molmo-Consolidated against each baseline on the original human written captions. Entries are score differences (Consolidated - baseline). Significant differences (

p<0.01

) are denoted with **.

Figure 3: Pairwise win matrix indicating the percentage of instances in which the row model achieved a higher rank than the column model. Llama-Consolidated is ranked above the baselines in more head-to-head comparisons.

Figure 4: Heatmap of the average model rank across different review sections. Lower average ranks (indicating better performance) are highlighted in green, while higher ranks are shown in red.

Figure 5: Average annotator rank per review section where lower rank is better. Llama-Consolidated ranks first in all four sections, with the largest margin in Strengths and Questions .

Figure 6: Pairwise win matrix indicating the percentage of instances in which the row model achieved a higher rank than the column model. Molmo-Consolidated is ranked above the baselines in more head-to-head comparisons.

Figure 7: Inter-annotator agreement denoted by Kendall’s $W$ per review section.

Figure 8: Average ranks stratified by conference (ICLR 2024, ICLR 2025, ICLR 2026, NeurIPS 2023, NeurIPS 2024, COLM 2024, COLM 2025).

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 423 papers

Consolidating Rewarded Perturbations for LLM Post-Training
May 29, 2026Zheyu Zhang, Shuo Yang, Gjergji KasneciRL Post-Training Language Model Post-Training
Technical University of Munich
EGTR-Review: Efficient Evidence-Grounded Scientific Peer Review Generation via Multi-Agent Teacher Distillation
Jun 4, 2026Xinpeng Qiu, Wang Yihu, Zhifeng Liu +2Multi-Agent LLM Systems Evidence-Grounded Generation
Department of Information Management, Peking University · PKU-WUHAN Institute for Artificial Intelligence, Peking University
Unifying Conformal Language Tasks with In-Context Ensembles
Sep 2, 2026Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara +2Conformal Prediction Language Model Ensembles
Signal 1 AI · Layer 6 AI

May 29, 2026J/K move · Enter open · S save

Consolidating Rewarded Perturbations for LLM Post-Training

Zheyu Zhang, Shuo Yang, Gjergji Kasneci

Technical University of Munich

RL Post-Training Language Model Post-Training

Open paper AskPDF arXiv

Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top-K rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs K forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP (Consolidating Rewarded Perturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the language model. Across five language models from 0.5B to 8B and five tasks covering math, code, and creative writing, CoRP improves the base model by 8.1 points on average. Using one tenth of RandOpt's perturbation budget, CoRP exceeds single-inference RandOpt by 6.5 points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example.

Abstract

Text Generation

Explore similar work

Consolidating Rewarded Perturbations for LLM Post-Training

EGTR-Review: Efficient Evidence-Grounded Scientific Peer Review Generation via Multi-Agent Teacher Distillation

Unifying Conformal Language Tasks with In-Context Ensembles