CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion

Sijie Mai; Shiqin Han

cs.CVFeb 22, 2026

CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion

Authors: Sijie Mai, Shiqin Han

Organizations: South China Normal University

Abstract

Modality gap significantly restricts the effectiveness of multimodal fusion. Previous methods often use techniques such as diffusion models and adversarial learning to reduce the modality gap, but they typically focus on one-to-one alignment without exposing the data points of the source modality to the global distribution information of the target modality. To this end, leveraging the characteristic of rectified flow that can map one distribution to another via a straight trajectory, we extend rectified flow for modality distribution mapping. Specifically, we leverage the one-to-many mapping' strategy in rectified flow that allows each data point of the source modality to observe the overall target distribution. This also alleviates the issue of insufficient paired data within each sample, enabling a more robust distribution transformation. Moreover, to achieve more accurate distribution mapping and address the ambiguous flow directions in one-to-many mapping, we design adaptive relaxed alignment', enforcing stricter alignment for modality pairs belonging to the same sample, while applying relaxed mapping for pairs not belonging to the same sample or category. Additionally, to prevent information loss during distribution mapping, we introduce `cyclic rectified flow' to ensure the transferred features can be translated back to the original features, allowing multimodal representations to learn sufficient modality-specific information. After distribution alignment, our approach achieves very competitive results on multiple tasks of multimodal affective computing even with a simple fusion method, and visualizations verify that it can effectively reduce the modality gap.

Figures & tables

Panel of Figure 1 — Figure 1: (a) shows that there are significant differences in the distribution of different modalities in the feature space, and (b) show that vanilla multimodal model (using feature concatenation and MLP networks for fusion) even underperforms language-only model. After applying our CaReFlow to reduce modality gap, multimodal model significantly outperforms language-only model.

Figure 2: The comparison between tradition methods, rectified flow [ 21 ] , and our CaReFlow. CaReFlow implements one-to-many mapping, adaptive relaxed alignment, and cyclic information flow to improve the effect of modality alignment.

Figure 3: (a) The diagram of the model pipeline and (b) the forward and backward information flows in CaReFlow.


		CMU-MOSI					CMU-MOSEI
Model	Venue	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Self-MM [ 54 ]	AAAI 2021	45.8	84.9	84.8	0.731	0.785	53.0	85.2	85.2	0.540	0.763
C-MIB [ 31 ]	TMM 2023	47.7	87.8	87.8	0.662	0.835	52.7	86.9	86.8	0.542	0.784
DMD [ 20 ]	CVPR 2023	44.9	84.3	84.3	0.726	0.788	52.8	84.6	84.6	0.538	0.768
DEVA † [ 46 ]	AAAI 2025	46.3	86.3	86.3	0.730	0.787	52.3	86.1	86.2	0.541	0.769

Table 1: The results on CMU-MOSI and CMU-MOSEI. The results labeled with † are obtained from original papers, and other results are obtained from our experiments. The best results are highlighted in bold and the second best results are marked with underlines.


	CH-SIMS v2
Model	Acc5↑	Acc3↑	Acc2↑	F1↑	MAE↓	Corr↑
MISA [ 14 ]	47.5	68.9	78.2	78.3	0.342	0.671
MAG-BERT [ 36 ]	49.2	70.6	77.1	77.1	0.346	0.641
Self-MM [ 54 ]	53.5	72.7	78.7	78.6	0.315	0.691
MMIM [ 12 ]	50.5	70.4	77.8	77.8	0.339	0.641

Table 2: The comparison with baselines on CH-SIMS-v2.

Panel of Figure 4 — Figure 4: Results on (a) UR-FUNNY and (b) MUStARD datasets.


	CH-SIMS-v2			CMU-MOSI
Model	Acc5 $\uparrow$	Acc2 $\uparrow$	MAE $\downarrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	MAE $\downarrow$
W/O Distribution Alignment	55.1	78.4	0.312	45.9	86.7	0.661
W/O Cyclic Information Flow	56.3	81.7	0.296	47.2	87.9	0.643
W/O Adaptive Relaxed Alignment	56.8	82.4	0.283	47.9	88.5	0.639
W/O One-to-Many Mapping	57.4	79.8	0.287	47.2	87.0	0.643

Table 3: Ablation experiments on CMU-MOSI and CH-SIMS-v2.


	CMU-MOSI						CMU-MOSEI					CH-SIMS-v2
Model	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Paramaters $\downarrow$	Acc7 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$	Acc5 $\uparrow$	Acc3 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
ARGF [ 27 ]	50.5	87.4	87.3	0.623	0.846	184.43M	53.2	87.6	87.5	0.524	0.798	51.8	71.0	78.4	78.7	0.314	0.677
MulT [ 41 ]	40.7	88.1	88.1	0.755	0.821	185.51M	51.7	86.5	86.6	0.583	0.794	54.6	74.2	80.8	80.7	0.300	0.738
Deep CCA [ 1 ]	43.8	87.2	87.2	0.722	0.815	184.59M	51.0	84.6	84.7	0.574	0.775	51.3	71.7	77.9	77.9	0.334	0.668
CLGSI [ 53 ]	45.8	89.0	88.9	0.642	0.851	186.31M	55.7	87.2	87.1	0.504	0.797	52.5	71.0	78.2	78.3	0.313	0.683

Table 4: Discussion on different distribution mapping methods on the CMU-MOSI, CMU-MOSEI, and CH-SIMS-v2 datasets.

Panel of Figure 5 — Figure 5: The visualizations for distributions of unimodal features using t-SNE [ 42 ] . CL denotes contrastive learning.

Panel of Figure 6 — Figure 6: The Acc2 w.r.t the change of sample ratio $\beta$ , number of Euler steps $\frac{1}{dt}$ , forward loss weight $\alpha_{f}$ , and backward loss weight $\alpha_{b}$ .


Fusion Methods	Acc5 $\uparrow$	Acc3 $\uparrow$	Acc2 $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	Corr $\uparrow$
Tensor Fusion [ 55 ]	58.6	77.9	83.6	83.6	0.280	0.754
Graph Fusion [ 27 ]	58.1	76.0	83.1	83.1	0.283	0.749
Low-rank Modality Fusion [ 23 ]	58.0	75.9	82.6	82.7	0.286	0.747
CubeMLP [ 38 ]	58.1	75.6	83.2	83.3	0.277	0.748
Simple Fusion Network (Default)	57.9	75.9	82.9	82.9	0.277	0.745

Table 5: Comparison of fusion methods on CH-SIMS-v2.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 771 papers

Oct 6, 2026

Dynamic Alignment and Calibration for Multimodal Learning

Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existing methods still suffer from two limitations: (i) static cross-modal alignment strategies usually impose uniform constraints on all samples while overlooking sample-wise variations, potentially leading to unreasonable over-alignment; and (ii) confidence- or uncertainty-aware fusion methods often fail to adequately account for feature magnitude and confidence differences across modalities. For modality pairs with significant feature magnitude differences or small confidence gaps, it might be unreliable to strictly align fusion weights according to confidence. To address these issues, we propose an Alignment- and Calibration-driven Multimodal Learning framework (ACML). Specifically, ACML incorporates a dynamic cross-modal triplet alignment module, which enforces strong semantic consistency for high-confidence positive pairs while encouraging diverse representation learning between high- and low-confidence positive pairs according to their confidence gaps. Additionally, ACML introduces a difference-aware attention calibration strategy that adaptively adjusts attention regularization based on feature magnitude and confidence differences across modalities, thereby mitigating biases caused by unreasonable fusion constraints. Extensive experiments on multiple multimodal benchmark datasets demonstrate that ACML consistently achieves superior performance and robustness over recent state-of-the-art methods.

Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu +1

School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu,China · Tianyijiaotong Technology Ltd.,China · School of Computer Science and Technology, Hainan University, Haikou, China

Multimodal Robustness Cross-Modal Alignment Uncertainty-Aware Multimodal Fusion Multimodal Contrastive Learning

Ask PDF

Jun 8, 2026

Explicit Representation Alignment for Multimodal Sentiment Analysis

Multimodal affective analysis aims to understand human sentiment and emotion by jointly modeling heterogeneous modalities such as text and images. However, multimodal models often fail to consistently outperform strong text-only baselines, with performance varying significantly across fusion strategies. In this work, we identify representation misalignment between independently pretrained modality encoders as a key bottleneck for effective multimodal learning, and show through controlled experiments that alignment prior to fusion is often more important than fusion complexity. To address this issue, we propose a unified multimodal affective analysis framework that leverages vision-language models (VLMs) to convert visual content into structured textual descriptions, projecting heterogeneous modalities into a shared linguistic space and enabling interpretable text-centric reasoning. To further improve robustness, we introduce a hybrid learning strategy that combines semantic token selection with a batch-level uniformity regularization objective, encouraging a more dispersed and stable global feature space while mitigating noise introduced by VLM-generated descriptions. Experiments on multiple multimodal sentiment and emotion benchmarks show that our method consistently outperforms strong unimodal and multimodal baselines, achieving state-of-the-art performance. Our analysis further highlights the critical role of representation alignment in multimodal affective learning.

Baode Wang, Ziming Wang, Huacan Wang +2

AgentAlpha

Cross-Modal Alignment Sentiment Analysis Multimodal Emotion Recognition Multimodal Sentiment Analysis Multimodal Fusion Multimodal Understanding

Ask PDF

Apr 20, 2026

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective

Multimodal affective computing aims to predict humans' sentiment, emotion, intention, and opinion using language, acoustic, and visual modalities. However, current models often learn spurious correlations that harm generalization under distribution shifts or noisy modalities. To address this, we propose a causal modality-invariant representation (CmIR) learning framework for robust multimodal learning. At its core, we introduce a theoretically grounded disentanglement method that separates each modality into causal invariant representation' and environment-specific spurious representation' from a causal inference perspective. CmIR ensures that the learned invariant representations retain stable predictive relationships with labels across different environments while preserving sufficient information from the raw inputs via invariance constraint, mutual information constraint, and reconstruction constraint. Experiments across multiple multimodal benchmarks demonstrate that CmIR achieves state-of-the-art performance. CmIR particularly excels on out-of-distribution data and noisy data, confirming its robustness and generalizability.

Sijie Mai, Shiqin Han

School of Computer Science, South China Normal University

Disentangled Representation Learning Causal Representation Learning Multimodal Robustness Multimodal Disentangled Representation Learning Sentiment Analysis Multimodal Emotion Recognition+2

Ask PDF

Abstract

Multimodal Fusion

Explore similar work

Dynamic Alignment and Calibration for Multimodal Learning

Explicit Representation Alignment for Multimodal Sentiment Analysis

Learning Invariant Modality Representation for Robust Multimodal Learning from a Causal Inference Perspective