Fusion Anything: A Generalized Multimodal Foundation Model

Huizi Cui; Zongbo Han; Chenggong Ding; Naichuan Xiao; Jialong Yang; Jingdong Chen; Guangyu Wang; Qinghua Hu

cs.LGAug 16, 2026

Fusion Anything: A Generalized Multimodal Foundation Model

Authors: Huizi Cui, Zongbo Han, Chenggong Ding, Naichuan Xiao, Jialong Yang, Jingdong Chen, Guangyu Wang, Qinghua Hu, +1 more

Organizations: Tianjin University · Beijing University of Posts and Telecommunications

Abstract

Making prediction with multimodal data is widely used in diverse scenarios. Existing multimodal fusion models, once deployed, can only handle predefined modalities (e.g., vision, text and audio) and single task, making it difficult to quickly adapt to new downstream applications. Therefore, a natural yet aggressive question arises - whether there exists a general multimodal fusion model that can be applied to arbitrary modality combinations and arbitrary prediction tasks. We argue that a unified multimodal fusion model should not depend on specific modalities and should instead encode transferable patterns of multimodal correlation. To this end, we propose a simple and effective learning paradigm based on training on large-scale synthetic multimodal datasets generated with Structural Multimodal Causal Models (SMCMs), which formally characterizes the generative processes of real-world multimodal data. Building on this framework, we propose the Fusion Anything Model (FAM), a foundation model for generalized multimodal data fusion. By constructing large-scale synthetic multimodal data with diverse correlation patterns, our model encodes transferable multimodal correlations during training and activates appropriate associations through in-context examples during inference. Extensive experiments on 18 real-world datasets spanning 12 modalities and 11 prediction tasks demonstrate that our model achieves competitive performance with specialized models without task-specific adaptation.

Figures & tables

Figure 1 : Fusion Anything Model for generalized multimodal prediction. A unified foundation model accommodates arbitrary modality combinations and diverse prediction tasks, enabling direct prediction without task-specific parameter adaptation.

Figure 2 : Synthetic multimodal dataset generation. (a) Real-world multimodal observations are produced by different sensors perceiving the same underlying object. (b) Motivated by this process, we generate synthetic modalities from a common latent representation through distinct generation mechanisms. (c) An example SMCM with three modalities $M_{1}$ , $M_{2}$ , and $M_{3}$ . The root representation comprises label-related and label-independent components, and the CCM controls the allocation of latent information through shared and private nodes. Shared nodes provide common information to multiple modalities, while each private node supplies modality-specific variation to its corresponding modality. Varying these node configurations and connections induces diverse multimodal correlation patterns.

Figure 3 : Overview of FAM. (a) Training: synthetic datasets generated by SMCMs are organized to in-context prediction tasks, with query-label guiding the optimization of the proposed FAM across tasks. (b) Inference: given labeled context examples from a real-world multimodal task, FAM predicts labels for unlabeled query samples by leveraging intra-modality, cross-modality, and sample-level attentions.

Figure 4 : Comparison of accuracy ranking between FAM and representative specialized fusion methods across diverse multimodal datasets. Our model achieves performance comparable to these specialized methods without parameter adaptation.

Figure 5 : Comparison between FAM and specialized fusion models across diverse multimodal datasets.

Method	GAMMA		RAVDESS		NYUDv2		MET-Meme(E)		TCGA-SKCM		MultiOFF		TCGA-KIRC		ROSMAP
Method	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$
TabPFN	65.00	80.00	57.92	89.12	57.80	87.99	33.65	64.40	55.10	56.32	61.74	62.81	72.73	79.34	73.58	86.35
Ours	85.00	92.00	70.00	94.25	62.84	90.85	37.91	67.02	59.18	60.53	65.77	68.44	75.76	72.31	76.42	84.56

Table 1 : Comparison of the original TabPFN and the proposed FAM.

Figure 6 : Performance scaling with different fractions of contextual examples.

Dataset	Modalities	Combination	Concat	EFT	MTO	CAF	TMC	Avg.	Ours
MultiOFF	Image+Text	DINOv3+Qwen3	58.39	59.96	61.30	61.30	61.07	60.40	65.77
		ResNet18+Qwen3	58.61	57.27	59.96	59.06	57.49	58.48	63.09
		DINOv3+BERT	62.42	60.40	56.82	59.73	65.10	60.89	66.44
RAVDESS	Audio+Visual	WavLM+DINOv3	56.74	26.60	60.83	56.39	56.87	51.49	70.00
		Whisper+DINOv3	77.71	32.08	68.82	66.94	77.99	64.71	76.04
		WavLM+ResNet18	37.92	18.68	47.08	37.64	35.90	35.44	58.54

Table 2 : Comparison under different pre-trained encoder configurations. For each configuration, the best and second-best results among all methods are highlighted in bold and underlined , respectively.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Concat		EFT		MTO		OTM		CAF		LS		TMC		Ours
Dataset	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	ACC $\uparrow$	AUC $\uparrow$
GAMMA	65.00 $(\pm\,5.00)$	85.59 $(\pm\,1.58)$	63.33 $(\pm\,16.07)$	74.22 $(\pm\,15.70)$	63.33 $(\pm\,12.58)$	83.11 $(\pm\,8.20)$	63.33 $(\pm\,10.41)$	76.44 $(\pm\,15.28)$	58.33 $(\pm\,5.77)$	77.48 $(\pm\,8.23)$	63.33 $(\pm\,7.64)$	77.07 $(\pm\,17.49)$	60.00 $(\pm\,5.00)$	83.56 $(\pm\,3.11)$	85.00 $(\blacktriangle\,22.62)$	92.00 $(\blacktriangle\,12.36)$
TCGA-KIRC	71.72 $(\pm\,4.63)$	70.11 $(\pm\,8.51)$	70.71 $(\pm\,4.63)$	72.59 $(\pm\,9.62)$	71.72 $(\pm\,4.63)$	66.94 $(\pm\,5.74)$	75.76 $(\pm\,9.09)$	65.98 $(\pm\,4.22)$	69.70 $(\pm\,6.06)$	73.55 $(\pm\,5.28)$	61.62 $(\pm\,9.74)$	62.53 $(\pm\,5.17)$	63.64 $(\pm\,3.03)$	67.91 $(\pm\,3.56)$	75.76 $(\blacktriangle\,6.49)$	72.31 $(\blacktriangle\,3.79)$
NYUDv2	55.45 $(\pm\,0.64)$	84.03 $(\pm\,0.37)$	30.73 $(\pm\,2.31)$	62.85 $(\pm\,3.27)$	42.86 $(\pm\,1.79)$	67.11 $(\pm\,7.45)$	52.34 $(\pm\,3.24)$	79.67 $(\pm\,4.07)$	54.18 $(\pm\,1.39)$	83.93 $(\pm\,1.15)$	55.81 $(\pm\,1.86)$	83.39 $(\pm\,1.65)$	53.57 $(\pm\,0.64)$	83.42 $(\pm\,0.97)$	62.84 $(\blacktriangle\,13.56)$	90.85 $(\blacktriangle\,13.08)$
CMU-MOSI	57.43 $(\pm\,1.77)$	75.55 $(\pm\,0.59)$	53.60 $(\pm\,2.62)$	62.00 $(\pm\,11.40)$	67.06 $(\pm\,1.27)$	72.71 $(\pm\,2.45)$	69.83 $(\pm\,1.77)$	72.89 $(\pm\,2.87)$	67.69 $(\pm\,1.53)$	75.10 $(\pm\,1.08)$	61.61 $(\pm\,3.08)$	71.92 $(\pm\,4.93)$	65.45 $(\pm\,1.91)$	72.42 $(\pm\,4.90)$	72.30 $(\blacktriangle\,9.06)$	79.58 $(\blacktriangle\,7.78)$
Enrico	52.97 $(\pm\,0.71)$	85.65 $(\pm\,0.57)$	26.71 $(\pm\,2.99)$	63.37 $(\pm\,1.57)$	47.15 $(\pm\,4.85)$	76.15 $(\pm\,5.96)$	45.66 $(\pm\,1.20)$	78.63 $(\pm\,2.82)$	49.77 $(\pm\,3.33)$	83.64 $(\pm\,1.46)$	50.91 $(\pm\,4.23)$	85.38 $(\pm\,2.32)$	51.94 $(\pm\,1.62)$	85.31 $(\pm\,1.15)$	52.74 $(\blacktriangle\,6.30)$	83.89 $(\blacktriangle\,4.16)$

Appendix

Table 3 : Comparison results across different multimodal datasets. Results of the specialized fusion methods are reported as mean

\pm

standard deviation, with the standard deviation shown below the mean in the same secondary-line font size as the

\blacktriangle

\blacktriangledown

marker. The marker below each Ours ACC or AUC reports its increase/decrease relative to the mean of the corresponding metric over the specialized fusion methods.

Dataset	TabPFN			Ours
Dataset	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
GAMMA	65.00	80.00	48.48	85.00 $(\blacktriangle\,20.00)$	92.00 $(\blacktriangle\,12.00)$	78.02 $(\blacktriangle\,29.54)$
TCGA-KIRC	72.73	79.34	61.18	75.76 $(\blacktriangle\,3.03)$	72.31 $(\blacktriangledown\,7.03)$	69.44 $(\blacktriangle\,8.26)$
NYUDv2	57.80	87.99	50.51	62.84 $(\blacktriangle\,5.04)$	90.85 $(\blacktriangle\,2.86)$	57.66 $(\blacktriangle\,7.15)$
CMU-MOSI	71.28	79.03	70.57	72.30 $(\blacktriangle\,1.02)$	79.58 $(\blacktriangle\,0.55)$	71.84 $(\blacktriangle\,1.27)$
Enrico	51.37	83.16	37.29	52.74 $(\blacktriangle\,1.37)$	83.89 $(\blacktriangle\,0.73)$	41.25 $(\blacktriangle\,3.96)$

Appendix

Table 4 : Comparison of TabPFN and our model across different multimodal datasets. The

\blacktriangle

\blacktriangledown

marker indicates the increase/decrease of Fusion Anything relative to the corresponding TabPFN result in percentage points. Ours results that match or exceed TabPFN are shown in bold.

Explore similar work

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities
Jul 28, 2026Mingqiao Ye, Zhaochong An, Zhitong Gao +11Modality-Specific Encoders Complementary Modalities
1EPFL · University of Copenhagen · 4CUHK +3
Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis
Jul 14, 2026Alexios Filippakopoulos, Elias Kallioras, Nikolaos Xiros +2Multimodal Fusion Cross-Modal
National Technical University of Athens, Greece · Athena Research Center, Greece · University of Bern, Switzerland +2
SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning
May 13, 2026Truong Pham, Anay Majee, Rishabh IyerMultimodal Alignment Modality Alignment
The University of Texas at Dallas

Jul 28, 2026, cs.CVJ/K move · Enter open · S save

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu+10

1EPFL · University of Copenhagen · 4CUHK+3

Modality-Specific Encoders Complementary Modalities Modalities Reference-Guided Multimodal In-Context Verification

Open paper AskPDF arXiv

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

Abstract

Explore similar work

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

Segregate, Refine, Integrate: Decomposing Multimodal Fusion for Sentiment Analysis

SMA: Submodular Modality Aligner For Data Efficient Multimodal Learning