InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Yanggan Gu; Yuanyi Wang; Zhaoyi Yan; Yiming Zhang; Qi Zhou; Fei Wu; Hongxia Yang

cs.LGMay 20, 2025

InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models

Authors: Yanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang, Qi Zhou, Fei Wu, Hongxia Yang

Organizations: The Hong Kong Polytechnic University (PolyU) · Zhejiang University · PolyU-Daya Bay Technology and Innovation Research Institute

Abstract

Model fusion combines multiple Large Language Models (LLMs) with different strengths into a more powerful, integrated model through lightweight training methods. Existing works on model fusion focus primarily on supervised fine-tuning (SFT), leaving preference alignment (PA) --a critical phase for enhancing LLM performance--largely unexplored. The current few fusion methods on PA phase, like WRPO, simplify the process by utilizing only response outputs from source models while discarding their probability information. To address this limitation, we propose InfiFPO, a preference optimization method for implicit model fusion. InfiFPO replaces the reference model in Direct Preference Optimization (DPO) with a fused source model that synthesizes multi-source probabilities at the sequence level, circumventing complex vocabulary alignment challenges in previous works and meanwhile maintaining the probability information. By introducing probability clipping and max-margin fusion strategies, InfiFPO enables the pivot model to align with human preferences while effectively distilling knowledge from source models. Comprehensive experiments on 11 widely-used benchmarks demonstrate that InfiFPO consistently outperforms existing model fusion and preference optimization methods. When using Phi-4 as the pivot model, InfiFPO improve its average performance from 79.95 to 83.33 on 11 benchmarks, significantly improving its capabilities in mathematics, coding, and reasoning tasks.

Figures & tables

Figure 1: Overview of InfiFPO for implicit model fusion. We compute probabilities for preferred ( $\boldsymbol{y}_{w}$ ) and dispreferred ( $\boldsymbol{y}_{l}$ ) responses using both pivot and source models. Following length normalization and probability clipping, we identify the source model with the maximum normalized probability difference from the pivot model for fusion and preference alignment.

Types	General Data	Math Data	Code Data
Dataset	Infinity-Instruct	NuminaMath-1.5	KodCode-V1-SFT
Original Size	1.4M	1.4M	268k
Sample Size	60K	45K	45K

Table 1: Dataset Statistics.

Models	Math			Code		General Reasoning			InstFol	Text Reasoning		Avg	Model Size	GPU Hours
Models	GSM8K	MATH	ThmQA	MBPP	HEval	BBH	ARC	MMLU	IFEval	DROP	HS
Pivot Model
Phi-4	87.41	80.04	51.12	75.40	83.54	68.84	93.90	85.62	77.34	88.67	87.62		79.95	14B	$\sim$ 1.0M
Source Models
Qwen2.5-Instruct	91.13	78.16	47.25	81.70	83.54	77.59	92.20	80.22	85.01	85.56	88.28	80.97	14B	$\sim$ 1.8M
Mistral-Small	92.42	69.84	48.50	68.80	84.15	81.59	91.86	81.69	82.25	86.52	91.84	79.95	24B	$\sim$ 1.6M

Table 2: Main Results. * indicates that this method only uses Qwen2.5-Instruct, Qwen2.5-Coder, and Mistral-Small as source models. Abbreviations: InstFol (Instruction Following), ThmQA (TheoremQA), HEval (HumanEval), HS (HellaSwag). We use the packing technique for SFT, where multiple short examples are packed into a single input sequence to increase training efficiency.

Method	Math	Code	All
WRPO	75.94 $\downarrow$ 0.0	84.84 $\downarrow$ 0.0	82.80 $\downarrow$ 0.0
InfiFPO ${}_{\text{WRPO}}$	76.31 $\uparrow$ 0.4	85.20 $\uparrow$ 0.2	83.32 $\uparrow$ 0.5
IPO	75.29 $\downarrow$ 0.0	83.93 $\downarrow$ 0.0	82.38 $\downarrow$ 0.0
InfiFPO ${}_{\text{IPO}}$	76.37 $\uparrow$ 1.1	84.54 $\uparrow$ 0.5	83.15 $\uparrow$ 0.8

Table 3: Results of InfiFPO combined with different preference optimization objectives.

Method	LN	PC	Math	Code	All
Phi4			72.85 $\downarrow$ 0.0	79.47 $\downarrow$ 0.0	79.95 $\downarrow$ 0.0
InfiFPO			72.72 $\downarrow$ 0.1	78.42 $\downarrow$ 1.0	79.51 $\downarrow$ 0.4
	✓		75.06 $\uparrow$ 2.4	83.99 $\uparrow$ 4.5	82.86 $\uparrow$ 2.9
		✓	74.39 $\uparrow$ 1.3	81.32 $\uparrow$ 1.9	81.41 $\uparrow$ 1.5
	✓	✓	75.80 $\uparrow$ 3.1	85.15 $\uparrow$ 5.6	83.33 $\uparrow$ 3.3

Table 4: Albation Study on Length Normalization (LN) and Probability Clipping (PC).

Num	Math	Code	All
1	74.91 $\downarrow$ 0.0	81.91 $\downarrow$ 0.0	81.54 $\downarrow$ 0.0
2	75.53 $\uparrow$ 0.6	82.49 $\uparrow$ 0.5	82.20 $\uparrow$ 0.6
3	75.60 $\uparrow$ 0.6	83.99 $\uparrow$ 2.0	82.74 $\uparrow$ 1.2
4	75.25 $\uparrow$ 0.3	85.21 $\uparrow$ 3.3	83.14 $\uparrow$ 1.6
5	75.80 $\uparrow$ 0.8	85.15 $\uparrow$ 3.2	83.33 $\uparrow$ 1.7

Table 5: Albation Study on Number of Source Models.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Prompt Format
IFEval	{prompt}\nPlease directly give the correct answer:
ARC-C	Question: {question}\nA . {textA}\nB . {textB}\nC . {textC}\nD . {textD}\nDirectly give me the correct answer option, and then explain:
Hellaswag	{ctx}\nQuestion : Which ending makes the most sense?\nDirectly give me the correct choice, you can further explain it or not.\nA . {A}\nB . {B}\nC . {C}\nD . {D}\nYou may choose from ’A’, ’B’, ’C’, ’D’.\nAnswer :
BBH	Follow the given examples and answer the question.\n {_hint}\nQ : {{input}}\nA : Let’s think step by step.
DROP	You will be asked to read a passage and answer a question. Some examples of passages and Q&A are provided below.\n {drop_examples}\n \n ## Your Task\n ---\n {prompt}\nThink step by step, then write a line of the form "Answer: $ANSWER" at the end of your response.
MMLU	{_hint}\nQuestion : {{input}}\nA . {{A}}\nB . {{B}}\nC . {{C}}\nD . {{D}}\n \nFor simple problems:\nDirectly provide the answer with minimal explanation.\n \nFor complex problems:\nUse this step-by-step format:\n #### Step 1: [Concise description]\n [Brief explanation]\n #### Step 2: [Concise description]\n [Brief explanation]\n \nRegardless of the approach, always conclude with:\nThe answer is [the_answer_letter].\nwhere the [the_answer_letter] is one of A, B, C or D.\n \nLet ’s think step by step.

Appendix

Table 6: Prompt format used for different datasets.

Model/Method	Math	Code	All
Phi-4	72.86	79.47	79.95
InfiFusion	73.23	82.43	81.44
SFT-WRPO	73.15	82.21	81.58
InfiFPO	73.59	83.15	82.19

Appendix

Table 7: Experimental results on UltraFeedback

Model / Method	Math	Code	All
Qwen-2.5-Instruct	72.18	82.62	80.97
FuseChat	75.96	83.50	83.22
InfiFusion	76.09	83.38	83.16
SFT-WRPO	76.42	84.29	83.46
InfiFPO	76.46	85.41	84.01

Appendix

Table 8: Experimental results using Qwen2.5-14B-Instruct as target model

Model / Method	Math	Code	All
Skywork-Reward-Llama-3.1-8B-v0.2
Phi-4	72.86	79.47	79.95
InfiFusion	74.32	82.47	81.96
SFT-WRPO	74.28	83.44	81.93
InfiFPO	74.53	84.88	82.67
ArmoRM-Llama3-8B-v0.1

Appendix

Table 9: Results using different reward models

Model / Method	Math	Code	All
Phi-4	72.86	79.47	79.95
FuseChat	73.54	82.97	82.12
- without sequence alignment	72.30	79.16	79.43
InfiFusion	75.54	82.67	82.38
- without sequence alignment	73.47	79.98	80.11
InfiFPO	75.60	83.99	82.74

Appendix

Table 10: Impact of sequence alignment on model fusion methods

Model / Method	25k	50k	75k	100k
Phi-4	79.95	79.95	79.95	79.95
SFT-WRPO	81.58	82.14	82.80	82.89
InfiFPO	82.55	82.99	83.33	83.56

Appendix

Table 11: Performance scaling with data volume

Model / Method	Math	Code	All
Qwen2.5-1.5B-Instruct
Base Model	46.02	47.16	50.31
InfiFusion	49.59	50.49	53.11
SFT-WRPO	50.16	51.21	53.70
InfiFPO	50.70	52.75	54.61
Qwen2.5-3B-Instruct

Appendix

Table 12: Results with different small-scale pivot models

Model type	Logps (preferred)	Logps (dispreferred)
Pivot model	-0.4	-0.7
Source model 1	-0.6	-0.5
Source model 2	-0.3	-0.8

Appendix

Table 13: Example of model probabilities before PC

Model type	Logps (preferred)	Logps (dispreferred)	Selected
Pivot model	-0.4	-0.7	-
Source model 1	-0.4	-0.7	-
Source model 2	-0.3	-0.8	✓

Appendix

Table 14: Example of model probabilities after PC

OpenAi HumanEval - Task Type: Code Generation
InfiFPO Behavior	Baseline Behavior
"prompt_1": "from typing import List\n\n\ndef remove_duplicates(numbers: List[int]) ->List[int]:\n """ From a list of integers, remove all elements that occur more than once.\n Keep order of elements left the same as in the input.\n >>>remove_duplicates([1, 2, 3, 2, 4])\n [1, 3, 4]\n """\n"
"completion": "def remove_duplicates(numbers: List[int]) ->List[int]:\n """ From a list of integers, remove all elements that occur more than once.\n Keep order of elements left the same as in the input.\n >>>remove_duplicates([1, 2, 3, 2, 4])\n [1, 3, 4]\n """\n counts = {}\n for number in numbers:\n counts[number] = counts.get(number, 0) + 1\n \n result = []\n for number in numbers:\n if counts[number] == 1:\n result.append(number)\n \n return result\n","result": "passed","passed": true	"completion": "from typing import List\n\ndef remove_duplicates(numbers: List[int]) ->List[int]:\n """ From a list of integers, remove all elements that occur more than once.\n Keep order of elements left the same as in the input.\n >>>remove_duplicates([1, 2, 3, 2, 4])\n [1, 3, 4]\n """\n seen = set()\n duplicates = set()\n result = []\n\n for number in numbers:\n if number in duplicates:\n continue\n if number in seen:\n duplicates.add(number)\n else:\n seen.add(number)\n result.append(number)\n\n return result\n","result": "failed: ","passed": false
"prompt_2": "\nFIX = """\nAdd more test cases.\n"""\n\ndef vowels_count(s):\n """Write a function vowels_count which takes a string representing\n a word as input and returns the number of vowels in the string.\n Vowels in this case are ’a’, ’e’, ’i’, ’o’, ’u’. Here, ’y’ is also a\n vowel, but only when it is at the end of the given word.\n\n Example:\n >>>vowels_count("abcde")\n 2\n >>>vowels_count("ACEDY")\n 3\n """\n"
"completion": "def vowels_count(s):\n """Write a function vowels_count which takes a string representing\n a word as input and returns the number of vowels in the string.\n Vowels in this case are ’a’, ’e’, ’i’, ’o’, ’u’. Here, ’y’ is also a\n vowel, but only when it is at the end of the given word.\n\n Example:\n >>>vowels_count("abcde")\n 2\n >>>vowels_count("ACEDY")\n 3\n """\n vowels = "aeiouAEIOU"\n count = 0\n for char in s:\n if char in vowels:\n count += 1\n if s[-1].lower() == ’y’:\n count += 1\n return count\n", "result": "passed", "passed": true	"completion": "def vowels_count(s):\n """Write a function vowels_count which takes a string representing\n a word as input and returns the number of vowels in the string.\n Vowels in this case are ’a’, ’e’, ’i’, ’o’, ’u’. Here, ’y’ is also a\n vowel, but only when it is at the end of the given word.\n\n Example:\n >>>vowels_count("abcde")\n 2\n >>>vowels_count("ACEDY")\n 3\n """\n vowels = "aeiouAEIOU"\n count = 0\n for i, char in enumerate(s):\n if char in vowels:\n count += 1\n elif char == ’y’ and i == len(s) - 1:\n count += 1\n return count\n\n# Additional test cases\nprint(vowels_count("hello")) # 2\nprint(vowels_count("sky")) # 1\nprint(vowels_count("rhythm")) # 0\nprint(vowels_count("fly")) # 1\nprint(vowels_count("boy")) # 2\nprint(vowels_count("try")) # 1\nprint(vowels_count("my")) # 1\nprint(vowels_count("y")) # 1\nprint(vowels_count("a")) # 1\nprint(vowels_count("e")) # 1\nprint(vowels_count("i")) # 1\nprint(vowels_count("o")) # 1\nprint(vowels_count("u")) # 1\n", "result": "failed: Test 5","passed": false,

Appendix

Table 15: Case study on code generation.

Case	Problem Prompt	SFT Output	FPO Output	Gold	Comment
0	Problem: How many ways are there to divide a set of 8 elements into 5 non-empty ordered subsets? Please analyze and write your final answer after the phrase "The answer is":	SFT predicts 126000, by mistakenly recalling $S(8,5)$ and applying $5!$ permutations, leading to a significant overestimate.	InfiFPO correctly computes $S(8,5)=105$ , applies $5!=120$ , yielding $105\times 120=12600$ .	11760	InfiFPO demonstrates recursive computation of Stirling numbers, while SFT simply recalls a wrong value, lacking procedural reasoning.
1	Problem: What is the value of $\int_{-\infty}^{+\infty}\sin(3t)\cdot\sin(t/\pi)/t^{2}\,dt$ ? Please analyze and write your final answer after the phrase "The answer is":	SFT predicts 0, oversimplifying based on oscillation intuition, neglecting the structure of the integral.	InfiFPO expands product of sines, applies Fourier integral properties, identifies non-zero singular contribution, yielding the correct result.	1.0	InfiFPO’s success stems from correct application of orthogonality and singularity analysis, while SFT lacks symbolic manipulation depth.
2	Problem: Given x = 0.157, what is the value of $x\times\frac{\prod_{n=1}^{\infty}(1-\frac{x^{2}}{n^{2}\pi^{2}})}{\sin(x)}$ ? Please analyze and write your final answer after the phrase "The answer is":	Both SFT and InfiFPO predict 0.157, recognizing Euler’s reflection formula, correctly simplifying numerator and denominator.	Both predict 0.157. FPO’s derivation explicitly links product form to sine function identity.	0.157	Both models give correct answer, but InfiFPO provides a more explicit and pedagogical derivation path.

Appendix

Table 16: Comparison between SFT and InfiFPO on TheoremQA benchmark. InfiFPO shows stronger symbolic reasoning, procedural consistency, and structural understanding, leading to better robustness and interpretability.

Explore similar work

Apr 27, 2026cs.LG

Intrinsic Mutual Information as a Modulator for Preference Optimization

Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15%. Our code is available at https://github.com/liavonpenn/rmipo.

Peng Liao, Peijia Zheng, Lingbo Li +2

1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University

LLM Alignment Offline Preference Optimization LLM Training

Ask PDF

Jul 14, 2025cs.LG

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data

The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data. This naturally leads to the question of whether such logs can be fully leveraged to fuse LLMs' complementary capabilities. Although prior work has explored various strategies for integrating multiple LLMs, we argue that practical fusion must meet two essential requirements: (1) compatibility with real-world serving scenarios (e.g., local and API-based serving), and (2) flexibility to operate at different stages of the LLM pipeline to meet varied user needs (e.g., fine-tuning and inference stages). To this end, we introduce LLMFusionBench, a large-scale benchmark for LLM fusion that spans 14 tasks across five domains, with responses from 20 open-source LLMs (8B--671B) totaling 103M tokens. Building on LLMFusionBench, we propose FusionFactory, a systematic framework with three elaborated levels: (1) query-level fusion via tailored LLM routers, (2) thought-level fusion leveraging retrieved abstract reasoning templates, and (3) model-level fusion via distillation from top-ranked responses. Experiments show that FusionFactory consistently outperforms the best individual LLM across all 14 benchmarks, with the optimal fusion configuration varying across benchmarks, highlighting the promise of multi-LLM log data as a practical foundation for fusing diverse LLM capabilities.

Tao Feng, Haozhen Zhang, Zijie Lei +5

University of Illinois Urbana-Champaign · Nanyang Technological University · Meta Monetization AI +2

LLM Evaluation LLM Routing Language Model Ensembles Multi-LLM Collaboration

Ask PDF

Apr 22, 2026cs.AI

HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs

Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over dispreferred responses in their entirety and lacks the granularity to provide feedback on subsections of many-step solutions typical of reasoning tasks. Existing methods excel at either stable preference learning (e.g., DPO variants like KTO and RSO) or structured reasoning (e.g., ReMA's multi-agent RL framework, Tree of Thoughts), but fail to merge these complementary strengths. We propose HiPO (Hierarchical Preference Optimization), an extension of DPO that separates responses into reasoning segments (query clarification and context, reasoning steps, and answer) and computes loss as a weighted sum of the DPO loss for each segment. Our approach enables segment-specific training while maintaining DPO's computational efficiency and training stability. We demonstrate that for multiple 7B LLMs fine-tuned using HiPO and DPO on the Math Stack Exchange preference dataset, the models trained with HiPO outperform the others on a variety of common math benchmarks and achieve greater organization, logical flow, and consistency as measured by GPT-4.1.

Darsh Kachroo, Arjun Prasaath Anbazhagan, Adriana Caraeni +2

Vellore Institute of Technology, Chennai · Northwestern University · University of Massachusetts Amherst +2

LLM Alignment Direct Preference Optimization Mathematical Reasoning Preference Optimization LLM Reasoning

Ask PDF