Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment

Yasuaki Sumita; Tomoharu Iwata; Toshiyuki Tanaka

cs.CLOct 16, 2026

Controlling the Diversity-Quality Trade-Off of LLMs by Merging Models Before and After Alignment

Authors: Yasuaki Sumita, Tomoharu Iwata, Toshiyuki Tanaka

Organizations: Graduate School of Informatics, Kyoto University, Kyoto, Japan · NTT Communication Science Laboratories, NTT, Inc., Kyoto, Japan

Abstract

Alignment is known to change the output characteristics of large language models (LLMs). Base models tend to generate more diverse outputs, whereas aligned models tend to produce higher-quality outputs. The appropriate balance between the two depends on the task. Existing methods either require costly fine-tuning or, like temperature sampling, mainly improve lexical diversity rather than semantic diversity. We introduce a depth-aware FrankenMerging approach for controlling the diversity--quality trade-off between an aligned model and its base model without additional training. This approach combines the lower layers of an aligned model with the higher layers of its base model based on the hypothesis that lexical diversity is more closely associated with lower layers and semantic diversity with higher layers. Experiments showed that varying the number of layers taken from each model adjusts both diversity and quality. In most cases, the merged models exhibited higher semantic diversity than the aligned model, and at some split positions, they showed higher lexical diversity than both original models. They also demonstrated a better diversity--quality trade-off than temperature sampling in most settings. A supporting ablation, in which we reversed the merging order, yielded results largely consistent with the hypothesis.

Figures & tables

Figure 1: Overview of our approach: We sequentially combine layers from the 1st to the $i$ -th layer of an aligned model A with layers from the $(i+1)$ -th to the $L$ -th layer of its base model B, then output the resulting model as a new one.

Panel of Figure 2 — Figure 2: Lexical and semantic diversity of the base models (blue squares), the aligned models (red diamonds), and the merged models (purple circles) for LLaMA 2 (top row) and Qwen 2.5 (bottom row). The circles are connected in the order of $i$ , and the labels indicate some values of $i$ . The $x$ -axis and $y$ -axis denote $\text{Div}_{\text{lex}}$ and $\text{Div}_{\text{sem}}$ , respectively. Each point is the average over the 500 inputs, and the error bars indicate 95% confidence intervals obtained by 10,000 bootstrap resamples of the inputs.

Panel of Figure 3 — Figure 3: Output quality of the base models (blue dashed lines), the aligned models (red dotted lines), and the merged models (purple circles) for LLaMA 2 (top) and Qwen 2.5 (bottom). The $x$ -axis denotes $i$ , and the $y$ -axis denotes COMET for translation (left) and BERTScore for summarization (right). Each point is the average over the 500 inputs, and the error bars and shaded bands indicate 95% confidence intervals obtained by 10,000 bootstrap resamples of the inputs.

Panel of Figure 4 — Figure 4: Trade-off between diversity (horizontal) and quality (vertical) for temperature sampling with the base models (blue squares) and the aligned models (red diamonds) and for the merged models (purple circles) for LLaMA 2 (top row) and Qwen 2.5 (bottom row). From left to right, the panels in each row show translation and summarization with $\text{Div}_{\text{lex}}$ and translation and summarization with $\text{Div}_{\text{sem}}$ . The labels indicate some temperatures $T$ and values of $i$ . Each point is the average over the 500 inputs, and the error bars indicate 95% confidence intervals obtained by 10,000 bootstrap resamples of the inputs.

Model	Task	Merged	Base	Aligned	Merged $-$ Base	Merged $-$ Aligned
Lexical diversity
LLaMA 2	Translation	$43.80$	$11.89$	$15.98$	$+31.91$ [ $+29.80$ , $+34.02$ ] ∗	$+27.82$ [ $+26.16$ , $+29.46$ ] ∗
LLaMA 2	Summarization	$25.99$	$2.16$	$9.64$	$+23.83$ [ $+23.18$ , $+24.47$ ] ∗	$+16.35$ [ $+15.89$ , $+16.83$ ] ∗
Qwen 2.5	Translation	$5.03$	$0.00$	$0.00$	$+5.03$ [ $+4.41$ , $+5.71$ ] ∗	$+5.03$ [ $+4.52$ , $+5.71$ ] ∗
Qwen 2.5	Summarization	$1.27$	$0.00$	$0.00$	$+1.27$ [ $+1.19$ , $+1.37$ ] ∗	$+1.27$ [ $+1.19$ , $+1.37$ ] ∗
Semantic diversity

Table 1: Hypervolume (

\times 10^{-3}

) of the diversity–quality points of the merged models (all values of

i

) and of temperature sampling with the base and aligned models (all temperatures). The last two columns show the differences with 95% confidence intervals obtained by a paired bootstrap over the 500 inputs (10,000 resamples). An asterisk ( ∗ ) indicates

p<0.05

after Holm’s correction.

Panel of Figure 5 — Figure 5: Diversity of the base models (blue squares), the aligned models (red diamonds), the merged models of our approach (purple circles), and the swapped models (orange triangles, labeled “Inverse model” in the legend) for LLaMA 2 (top row) and Qwen 2.5 (bottom row). The points of each merging order are connected in the order of $i$ , and the labels indicate some values of $i$ . The $x$ -axis and $y$ -axis denote $\text{Div}_{\text{lex}}$ and $\text{Div}_{\text{sem}}$ , respectively. Each point is the average over the 500 inputs, and the error bars indicate 95% confidence intervals obtained by 10,000 bootstrap resamples of the inputs.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Prompt template
Translation	Translate the following text into English: {input} Translation:
Summarization	Summarize the following text: {input} Summary:
Story generation	Write a story based on the following writing prompt: {input} Story:
Dialogue generation	Continue the following dialogue: {input}

Appendix

Table 2: Prompt templates. {input} is replaced with the input text, and the line breaks are kept as shown.

Setting	Value
Library	vLLM 0.28.0
Top- $p$	1.0
Top- $k$	Not applied
Min- $p$	0.0
Repetition penalty	1.0
Presence penalty	0.0

Appendix

Table 3: Generation settings. The EOS token IDs are the union of those of the base model and the aligned model, and the maximum context length is the smaller of their values.

Software	Version
Python	3.12
PyTorch	2.13.0 (CUDA 13.0)
Transformers	5.16.1 (generation); 4.57.6 (merging and evaluation)
Tokenizers	0.23.1
Datasets	5.0.1
vLLM	0.28.0

Appendix

Table 4: Software versions.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints
Dec 18, 2025Aniruddha Roy, Jyoti Patel, Aman Chadha +2Language Model Merging LLM Alignment
HCL · Evalueserve · Apple (USA) +2
Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning
Sep 14, 2026Jiayi Yuan, Hangoo Kang, James Jihao Liu +4RL for Language Models Multi-Objective Language Model Alignment
University of Washington · 2Stanford University
The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation
Mar 25, 2026Mingyi LiuHomogenization in Language Models LLM Alignment
Independent Researcher

Dec 18, 2025J/K move · Enter open · S save

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

Aniruddha Roy, Jyoti Patel, Aman Chadha, Vinija Jain+1

HCL · Evalueserve · Apple (USA)+2

Language Model Merging LLM Alignment LLM Safety Alignment Latent Space Alignment

Open paper AskPDF arXiv

Merging large language models (LLMs) is a practical way to compose capabilities from multiple fine-tuned checkpoints without retraining. Yet standard schemes (linear weight soups, task vectors, and Fisher-weighted averaging) can preserve loss while quietly destroying alignment. We argue that merging is not a numerical trick but a geometry-constrained operation around an already-aligned anchor: fusion must be steered to respect safety geometry, not validated post hoc. We introduce AlignMerge, a geometry-aware merging framework that makes alignment an explicit invariant. In a local Fisher chart around an instruction-tuned base, we estimate an alignment subspace with projector P_A and optimize: L_AlignMerge = L_geo + lambda_align * L_align + lambda_bud * L_bud, where L_geo keeps the merge close to its experts in Fisher-Rao geometry, L_align penalizes motion along alignment-sensitive directions, and L_bud enforces a soft alignment budget. As the alignment functional we use the decoding-invariant Alignment Quality Index (AQI), a latent-space criterion that captures how cleanly aligned and misaligned behaviors separate in representation space. Across five model families (LLaMA-3 8B, Mistral 7B, Qwen 2, Phi-3.5, Gemma 2), merging safety anchors with task experts, AlignMerge improves alignment metrics (AQI, toxicity, LLM-judge alignment) while matching or exceeding the best expert on instruction-following, reasoning, and helpfulness. It also exhibits smaller alignment-subspace drift and fewer budget violations than Fisher soups, TIES, SafeMerge, and MergeAlign. These results make alignment-preserving merging a first-class design goal and suggest a path to geometry-aware composition of future foundation models.

Abstract

Language Model Merging

Explore similar work

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation