Harmonized Tabular-Image Fusion via Gradient-Aligned Alternating Learning

Longfei Huang; Yang Yang

cs.CVApr 2, 2026

Harmonized Tabular-Image Fusion via Gradient-Aligned Alternating Learning

Authors: Longfei Huang, Yang Yang

Organizations: Nanjing University of Science and Technology

Abstract

Multimodal tabular-image fusion is an emerging task that has received increasing attention in various domains. However, existing methods may be hindered by gradient conflicts between modalities, misleading the optimization of the unimodal learner. In this paper, we propose a novel Gradient-Aligned Alternating Learning (GAAL) paradigm to address this issue by aligning modality gradients. Specifically, GAAL adopts an alternating unimodal learning and shared classifier to decouple the multimodal gradient and facilitate interaction. Furthermore, we design uncertainty-based cross-modal gradient surgery to selectively align cross-modal gradients, thereby steering the shared parameters to benefit all modalities. As a result, GAAL can provide effective unimodal assistance and help boost the overall fusion performance. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SoTA) tabular-image fusion baselines and test-time tabular missing baselines. The source code is available at https://github.com/njustkmg/ICME26-GAAL.

Figures & tables

Fig. 1: We visualize the gradient conflicts and evaluate the performance of existing multimodal solutions on DVM dataset. (a) Multimodal and image gradients often show negative cosine similarity in naive joint learning, indicating severe gradient conflicts. (b) Existing multimodal solutions underperform in tabular-image fusion, failing to fully exploit unimodal image potential.

Fig. 2: The framework of the GAAL method. (a) Gradient-Aligned Alternating Learning that only one modality’s learner is updated at each step. Uncertainty-based cross-modal gradient surgery utilizes gradients from cross-modal hard samples to guide the optimization of the shared classifier for the current modality. (b) Uncertainty-based Gradient Guidance that samples hard examples from another modality to provide cross-modal gradient guidance.

Unimodal Learning
Method	DVM			SUNAttribute			CelebA
Method	Multi	Image	Tabular	Multi	Image	Tabular	Multi	Image	Tabular
Resnet50	-	0.8743	-	-	0.8361	-	-	0.8146	-
MLP	-	-	0.8742	-	-	0.8082	-	-	0.7775
Tabular-image Fusion
CF	0.8793	0.0058	0.8792	0.8278	0.5544	0.8298	0.7740	0.5654	0.7723

TABLE I: The accuracy results on DVM, SUNAttribute, and CelebA datasets. The best performance is bolded, and the second-best is underlined. The image performance of test-time tabular missing methods represents their multimodal performance. The “*” indicates the version that follows the same pretrained weight setting as test-time tabular missing methods.

Panel of Figure 4 — Fig. 3: Sensitivity to hyper-parameter $\lambda^{I}$ and $\lambda^{T}$ on DVM and SUNAttribute datasets.

Panel of Table 5 — TABLE II: Results of ablation studies on DVM and SUNAttribute datasets.

Panel of Figure 6 — Fig. 4: Convergence results of GAAL on DVM and SUNAttribute datasets.

Fig. 5: Impact of constraint margin

\epsilon

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 757 papers

Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model
Sep 26, 2026Longfei Huang, Shangdong Yang, Yang YangCross-Modal Representation Transfer Multimodal Learning
Nanjing University of Science and Technology · Nanjing University of Posts and Telecommunications
Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification
Jun 3, 2026Feixiang Zhou, Jianyang Xie, Zhuangzhi Gao +10Missing-Modality Learning Cross-Modal Alignment
School of Eye and Vision Sciences, University of Liverpool, U.K. · School of Computer Science, University of Exeter, U.K. · School of Computer Science and Engineering, South China University of Technology, China +4
Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning
Jun 10, 2026Jiaqi LuoFine-Tuning Adapter Tuning
School of Mathematical Sciences, Soochow University, No.1 Shizi Street, Suzhou, 215006, Jiangsu Province, China

Sep 26, 2026J/K move · Enter open · S save

Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model

Longfei Huang, Shangdong Yang, Yang Yang

Nanjing University of Science and Technology · Nanjing University of Posts and Telecommunications

Cross-Modal Representation Transfer Multimodal Learning Transfer Learning Missing-Modality Learning

Open paper AskPDF arXiv

Multimodal tabular-image learning is gaining growing attention, yet it faces challenges due to tabular data unavailable at test time. A practical solution involves transferring tabular knowledge to images during training to enhance the performance of image models at inference. However, the overlooked yet important challenges lie in the modality imbalance between images and tables, as well as their asymmetric modality relationship in cross-modal transfer, which limits the auxiliary role of tabular data. To address these issues, we propose Skewed Knowledge Transfer (SKT), which asymmetrically transfers tabular knowledge to improve the image model by adaptive integration of modality gradients in a shared parameter space. Specifically, we first introduce a multimodal shared head, which allows the model to benefit from cross-modal structure without adding additional parameters. We then design a two-step Nash Bargaining strategy to effectively leverage tabular gradients. In the first step, SKT seeks a point of modality balance and uses preference awareness in the second step to steer combined gradients toward image-beneficial directions. Furthermore, we theoretically analyze the Pareto improvement and convergence of SKT. To this end, tabular knowledge is explicitly transferred to enhance image models. Empirical experiments on widely used tabular-image datasets reveal that SKT consistently improves image unimodal performance by using tabular data as auxiliary information.

Abstract

Multimodal Fusion

Explore similar work

Learning Through Game: Skewed Transfer of Tabular Knowledge to Strengthen Image Model

Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification

Parameter-Efficient Adapter Tuning for Tabular-Image Multimodal Learning