Reasoning-Driven Multimodal LLM for Domain Generalization
Authors: Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang
Organizations: Xidian University · Microsoft Research Asia · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application
This paper addresses the domain generalization (DG) problem in deep learning. While most DG methods focus on enforcing visual feature invariance, we leverage the reasoning capability of multimodal large language models (MLLMs) and explore the potential of constructing reasoning chains that derives image categories to achieve more robust predictions under domain shift. To this end, we systematically study the role of reasoning in DG using DomainBed-Reasoning, a newly constructed extension of DomainBed dataset, in which each sample is paired with class-relevant reasoning chains. Our analysis reveals two key challenges: (i) fine-tuning MLLMs with reasoning chains for classification is more challenging than direct label supervision, since the model must optimize complex reasoning sequences before label prediction; and (ii) mismatches in reasoning patterns between supervision signals and fine-tuned MLLMs lead to a trade-off between semantic richness (informative but harder to optimize) and optimization efficiency (easier to optimize but less informative). To address these issues, we propose RD-MLDG (Reasoning-Driven Multimodal LLM for Domain Generalization), a framework with two components: (i) MTCT (Multi-Task Cross-Training), which introduces an additional direct classification pathway to guide reasoning supervision; and (ii) SARR (Self-Aligned Reasoning Regularization), which preserves the semantic richness of reasoning chains while mitigating reasoning-pattern mismatches via iterative self-labeling. Experiments on standard DomainBed datasets (PACS, VLCS, OfficeHome, TerraInc) demonstrate that RD-MLDG achieves state-of-the-art performances, highlighting reasoning as a promising complementary signal for robust out-of-domain generalization.
Figures & tables
Figure 1: Illustrative examples of the printer in OfficeHome (Art Painting, Clipart, Product, Real World). Although the visual appearance differs substantially, the highlighted green segments of the reasoning chains remain highly consistent, capturing class-relevant cues that are likely to generalize well across domains.
Figure 2: Overview of the DomainBed-Reasoning construction pipeline. GPT-4o generates multi-stage reasoning chains ( <SUMMARY> , <CAPTION> , <REASONING> , <REFLECTION> , <CONCLUSION> ) without access to ground-truth labels. Multiple candidates are sampled and filtered through rejection sampling to obtain coherent reasoning chains, which form the foundation for analyzing reasoning challenges in DG.
Figure 3: Illustration of Challenge 1 (see Sec. 4.1 ). (A) Classification token probabilities on target-domain test data under zero-shot and SFT, with and without reasoning; (B) Probability distributions of all tokens on source-domain training data under zero-shot and SFT; (C) Training dynamics: (I–II) classification token probabilities on source-domain training data after SFT; (III) loss curves comparing reasoning-based and no-thinking SFT.
Figure 4: Illustration of Challenge 2 (see Sec. 4.2 ). (A) Token probability distributions on source-domain training data under zero-shot and fine-tuned settings: GPT-4o reasoning (A.I, A.II) vs InternVL-8B reasoning (A.III, A.IV). (B) Top-15 tokens with the largest entropy reduction during optimization, highlighting differences between GPT-4o (B.I) and InternVL-8B (B.II). (C) Qualitative examples corresponding to (B.I) and (B.II), with tokens receiving the strongest optimization gains highlighted in red .
Method
Venue
PACS
VLCS
OfficeHome
TerraInc
Avg.
ResNet-50 Based Method.
CORAL ( Sun and Saenko, 2016 )
ICCV’16
86.20 ± 0.30
78.80 ± 0.60
68.70 ± 0.30
47.60 ± 1.00
70.33
MLDG ( Li et al., 2018a )
AAAI’18
84.90 ± 1.00
77.20 ± 0.40
66.80 ± 0.60
47.70 ± 0.90
69.15
Mixstyle ( Zhou et al., 2021 )
ICLR’21
85.20 ± 0.30
77.90 ± 0.50
60.40 ± 0.20
44.00 ± 0.70
66.88
DomainDrop ( Guo et al., 2023a )
ICCV’23
87.90 ± 0.30
79.80 ± 0.30
68.70 ± 0.10
51.50 ± 0.40
71.98
SMOS ( Luo et al., 2024 )
CVPR’24
89.40 ± 0.30
79.80 ± 0.10
71.60 ± 0.10
55.40 ± 0.40
74.05
Table 1: Multi-source DG results (%) on DomainBed benchmark datasets. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
OfficeHome
TerraIncognita
Idx
Method
Art
Clipart
Product
Real
Avg.
L100
L38
L43
L46
Avg.
1
GPT-4o
87.10
85.36
95.20
92.82
90.12
67.28
59.71
64.47
50.49
60.49
2
InternVL3-2B
3.79
3.64
2.70
3.49
3.41
0.16
0.25
0.15
0.33
0.22
3
+ CLS only
88.59
81.49
94.46
92.10
89.16
75.26
64.66
69.98
55.99
66.47
4
+ Reasoning only (Baseline)
87.03
82.13
93.77
91.45
88.60
74.79
61.26
69.40
58.56
66.00
5
+ MTCT
89.00
83.71
95.49
93.53
90.43 (↑1.83%p)
77.82
65.03
72.71
59.41
68.74 (↑2.74%p)
Table 2: Ablation study on each component on OfficeHome and TerraIncognita dataset.
Figure 5: Analysis of MTCT (Sec. 5.1 ). Probability distributions on TerraInc source domain training data under direct label prediction, reasoning-only SFT, and MTCT SFT: (A) all tokens and (B) class tokens.
Figure 6: Analysis of SARR (see Sec. 5.2 ) from token entropy on TerraInc (A) and OfficeHome (B). We track the entropy of selected class-relevant (A.I & B.I) and domain-specific tokens (A.II & B.II) across fine-tuning iterations, with vertical bars denoting variation across samples. Blue curves correspond to fitting GPT-4o reasoning chains, and orange curves to InternVL3-8B self-generated reasoning chains.
Figure 7: Impact of the number of self-labeling rounds N in SARR (Sec. 5.2 ) on TerraInc using InternVL3-8B. Top: classification accuracy on four domains (L100, L38, L43, L46) and their average, with paired t -test p -values for adjacent N shown on the right axis (reference line at p=0.05 ). Bottom: zero-shot token probability distributions (log-scale) on L100, using supervision data from each self-labeling round.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
bird
bobcat
cat
coyote
dog
empty
opossum
rabbit
raccoon
squirrel
Avg.
Domain divergence (Visual)
0.209
0.251
0.314
0.387
0.213
0.144
0.266
0.167
0.258
0.176
0.239
Domain divergence (Text)
0.054
0.048
0.103
0.114
0.093
0.126
0.091
0.103
0.142
0.115
0.099
Appendix
Table 3: Cross-domain divergence of visual embeddings and reasoning-chain embeddings for each TerraInc class.
L100
L38
L43
L38
Avg.
N=0
41.78
39.25
40.84
36.18
39.51
N=1
21.66
22.14
17.56
15.57
19.23
N=2
18.49
17.98
15.07
13.64
16.30
N=3
17.07
15.74
14.43
12.01
14.81
Appendix
Table 4: Rejection rate across SARR iterations (N) on the four TerraInc domains.
TerraIncognita
Idx
Method
L100
L38
L43
L46
Avg.
1
InternVL3-2B + Reasoning only (Baseline)
74.79
61.26
69.40
58.26
66.00
2
InternVL3-2B + MTCT
77.82
65.03
72.71
59.41
68.74 (↑2.74%p)
3
InternVL3-2B + MTCT + SARR
82.43
66.54
73.62
61.15
70.94 (↑4.94%p)
4
InternVL3-8B + Reasoning only
74.42
60.39
68.21
55.21
64.56
5
InternVL3-8B + MTCT
76.78
60.55
71.94
59.47
67.19 (↑2.63%p)
Appendix
Table 5: MTCT and SARR Performance on LLaVA-1.5 for TerraInc (%).
Method
Venue
A
C
P
R
Avg.
DANN ( Ganin et al., 2016 )
IJCAI’16
55.20
49.30
48.40
58.40
52.80
CORAL ( Sun and Saenko, 2016 )
ICCV’16
55.60
52.80
50.30
59.40
54.50
IRM ( Arjovsky et al., 2019 )
-
54.90
53.20
48.60
59.20
54.00
MMD ( Li et al., 2018b )
ICCV’18
55.10
52.00
50.30
59.30
54.20
OrgMixup ( Zhang et al., 2018 )
ICLR’18
56.00
54.40
50.40
61.00
55.50
Mixup ( Yan et al., 2020 )
-
55.50
54.10
49.40
59.40
54.60
Appendix
Table 6: SDG results (%) on OfficeHome. One domain is used as the source domain and the others are used as the target domain. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
FGVC-Aircraft
Idx
Method
Base
New
H
1
InternVL3-8B
20.35
17.83
19.01
2
+ Reasoning only (Baseline)
39.20
15.44
22.11
3
+ MTCT
56.42
18.60
27.95 (↑5.84%p)
4
+ MTCT + SARR
60.11
24.33
34.63 (↑12.52%p)
Appendix
Table 7: Base-to-new generalization results on FGVC-Aircraft.
Method
VQA
VE
Real
Cartoon
Pencil
Oil
Avg.
Real
Cartoon
Pencil
Oil
Avg.
ViT
55.03
48.52
47.76
48.65
49.99
72.15
52.51
57.72
58.78
60.29
CLIP ( Radford et al., 2021a )
58.23
49.11
50.41
49.41
51.79
73.10
55.85
61.61
60.93
62.87
BLIP ( Li et al., 2023a )
59.19
50.29
51.32
50.88
52.92
66.73
48.26
52.93
53.62
55.39
Open-Source Models.
BLIP2-FlanT5-XL ( Li et al., 2023a )
65.29
64.41
61.18
62.92
63.45
63.82
73.13
72.24
72.00
70.30
Appendix
Table 8: VQA and VE results (%) on VOLDOGER benchmark dataset. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
Figure 8: Input prompt template used for reasoning-chain generation. The prompt consists of two parts: (i) a question instruction paired with an image and candidate label options, and (ii) explicit guidelines requiring the model (GPT-4o) to output reasoning in a structured five-section format (SUMMARY, CAPTION, REASONING, REFLECTION, CONCLUSION). These generated reasoning chains are then paired with ground-truth labels and used to fine-tune downstream multimodal LLMs.
Multimodal models ideally should generalize to unseen domains while remaining data-efficient to reduce annotation costs. To this end, we introduce and study a new problem, Semi-Supervised Multimodal Domain Generalization (SSMDG), which aims to learn robust multimodal models from multi-source data with few labeled samples. We observe that existing approaches fail to address this setting effectively: multimodal domain generalization methods cannot exploit unlabeled data, semi-supervised multimodal learning methods ignore domain shifts, and semi-supervised domain generalization methods are confined to single-modality inputs. To overcome these limitations, we propose a unified framework featuring three key components: Consensus-Driven Consistency Regularization, which obtains reliable pseudo-labels through confident fused-unimodal consensus; Disagreement-Aware Regularization, which effectively utilizes ambiguous non-consensus samples; and Cross-Modal Prototype Alignment, which enforces domain- and modality-invariant representations while promoting robustness under missing modalities via cross-modal translation. We further establish the first SSMDG benchmarks, on which our method consistently outperforms strong baselines in both standard and missing-modality scenarios. Our benchmarks and code are available at https://github.com/lihongzhao99/SSMDG.
Generalizing image classification across domains remains challenging in critical tasks such as fundus image-based diabetic retinopathy (DR) grading and resting-state fMRI seizure onset zone (SOZ) detection. When domains differ in unknown causal factors, achieving cross-domain generalization is difficult, and there is no established methodology to objectively assess such differences without direct metadata or protocol-level information from data collectors, which is typically inaccessible. We first introduce domain conformal bounds (DCB), a theoretical framework to evaluate whether domains diverge in unknown causal factors. Building on this, we propose GenEval, a multimodal Vision Language Models (VLM) approach that combines foundational models (e.g., MedGemma-4B) with human knowledge via Low-Rank Adaptation (LoRA) to bridge causal gaps and enhance single-source domain generalization (SDG). Across eight DR and two SOZ datasets, GenEval achieves superior SDG performance, with average accuracy of 69.2% (DR) and 81% (SOZ), outperforming the strongest baselines by 9.4% and 1.8%, respectively.
Post-training via Reinforcement Learning (RL) has enabled Large Reasoning Models (LRMs) to achieve strong performance in individual domain. However, real-world applications increasingly require general-purpose reasoners rendering strong performance across diverse domains. Mixed-domain post-training aims to achieve this goal by jointly training with mixed domain data, but this often induces capability compromise and degradation among different domains. Existing methods attribute this performance degradation to harmful cross-domain interactions and propose various strategies to mitigate them, but these strategies may also impede beneficial knowledge sharing across domains and in turn fail to match or surpass single-domain performance. To address this problem, we propose Multi-domain Contrastive Policy Optimization (MCPO), which uses contrastive learning to utilize both positive and negative cross-domain interactions for knowledge sharing and competition. Specifically, we partition each rollout generated by LRMs according to its underlying reasoning structures and use reasoning segments to capture these structures. We thus formulate positive and negative pairs of reasoning segments as mutually augmented examples, which provide supportive and competing signals for knowledge sharing. Subsequently, we design complementary contrastive objectives for cross-domain knowledge sharing and intra-domain knowledge consolidation, targeting compatibility across domains and discriminability within each domain to form a harmonious reasoning space. Experimental results across a broad range of domains show that MCPO alleviates performance degradation caused by mixed-domain training and outperforms single-domain training in most cases.
Zongji Yu, Wenshui Luo, Yiliu Sun +5
Shanghai Jiao Tong University · Shanghai AI Laboratory · Shandong University