Reasoning-Driven Multimodal LLM for Domain Generalization
Authors: Zhipeng Xu, Zilong Wang, Xinyang Jiang, Dongsheng Li, De Cheng, Nannan Wang
Organizations: Xidian University · Microsoft Research Asia · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application
This paper addresses the domain generalization (DG) problem in deep learning. While most DG methods focus on enforcing visual feature invariance, we leverage the reasoning capability of multimodal large language models (MLLMs) and explore the potential of constructing reasoning chains that derives image categories to achieve more robust predictions under domain shift. To this end, we systematically study the role of reasoning in DG using DomainBed-Reasoning, a newly constructed extension of DomainBed dataset, in which each sample is paired with class-relevant reasoning chains. Our analysis reveals two key challenges: (i) fine-tuning MLLMs with reasoning chains for classification is more challenging than direct label supervision, since the model must optimize complex reasoning sequences before label prediction; and (ii) mismatches in reasoning patterns between supervision signals and fine-tuned MLLMs lead to a trade-off between semantic richness (informative but harder to optimize) and optimization efficiency (easier to optimize but less informative). To address these issues, we propose RD-MLDG (Reasoning-Driven Multimodal LLM for Domain Generalization), a framework with two components: (i) MTCT (Multi-Task Cross-Training), which introduces an additional direct classification pathway to guide reasoning supervision; and (ii) SARR (Self-Aligned Reasoning Regularization), which preserves the semantic richness of reasoning chains while mitigating reasoning-pattern mismatches via iterative self-labeling. Experiments on standard DomainBed datasets (PACS, VLCS, OfficeHome, TerraInc) demonstrate that RD-MLDG achieves state-of-the-art performances, highlighting reasoning as a promising complementary signal for robust out-of-domain generalization.
Figures & tables
Figure 1: Illustrative examples of the printer in OfficeHome (Art Painting, Clipart, Product, Real World). Although the visual appearance differs substantially, the highlighted green segments of the reasoning chains remain highly consistent, capturing class-relevant cues that are likely to generalize well across domains.
Figure 2: Overview of the DomainBed-Reasoning construction pipeline. GPT-4o generates multi-stage reasoning chains ( <SUMMARY> , <CAPTION> , <REASONING> , <REFLECTION> , <CONCLUSION> ) without access to ground-truth labels. Multiple candidates are sampled and filtered through rejection sampling to obtain coherent reasoning chains, which form the foundation for analyzing reasoning challenges in DG.
Figure 3: Illustration of Challenge 1 (see Sec. 4.1 ). (A) Classification token probabilities on target-domain test data under zero-shot and SFT, with and without reasoning; (B) Probability distributions of all tokens on source-domain training data under zero-shot and SFT; (C) Training dynamics: (I–II) classification token probabilities on source-domain training data after SFT; (III) loss curves comparing reasoning-based and no-thinking SFT.
Figure 4: Illustration of Challenge 2 (see Sec. 4.2 ). (A) Token probability distributions on source-domain training data under zero-shot and fine-tuned settings: GPT-4o reasoning (A.I, A.II) vs InternVL-8B reasoning (A.III, A.IV). (B) Top-15 tokens with the largest entropy reduction during optimization, highlighting differences between GPT-4o (B.I) and InternVL-8B (B.II). (C) Qualitative examples corresponding to (B.I) and (B.II), with tokens receiving the strongest optimization gains highlighted in red .
Method
Venue
PACS
VLCS
OfficeHome
TerraInc
Avg.
ResNet-50 Based Method.
CORAL ( Sun and Saenko, 2016 )
ICCV’16
86.20 ± 0.30
78.80 ± 0.60
68.70 ± 0.30
47.60 ± 1.00
70.33
MLDG ( Li et al., 2018a )
AAAI’18
84.90 ± 1.00
77.20 ± 0.40
66.80 ± 0.60
47.70 ± 0.90
69.15
Mixstyle ( Zhou et al., 2021 )
ICLR’21
85.20 ± 0.30
77.90 ± 0.50
60.40 ± 0.20
44.00 ± 0.70
66.88
DomainDrop ( Guo et al., 2023a )
ICCV’23
87.90 ± 0.30
79.80 ± 0.30
68.70 ± 0.10
51.50 ± 0.40
71.98
SMOS ( Luo et al., 2024 )
CVPR’24
89.40 ± 0.30
79.80 ± 0.10
71.60 ± 0.10
55.40 ± 0.40
74.05
Table 1: Multi-source DG results (%) on DomainBed benchmark datasets. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
OfficeHome
TerraIncognita
Idx
Method
Art
Clipart
Product
Real
Avg.
L100
L38
L43
L46
Avg.
1
GPT-4o
87.10
85.36
95.20
92.82
90.12
67.28
59.71
64.47
50.49
60.49
2
InternVL3-2B
3.79
3.64
2.70
3.49
3.41
0.16
0.25
0.15
0.33
0.22
3
+ CLS only
88.59
81.49
94.46
92.10
89.16
75.26
64.66
69.98
55.99
66.47
4
+ Reasoning only (Baseline)
87.03
82.13
93.77
91.45
88.60
74.79
61.26
69.40
58.56
66.00
5
+ MTCT
89.00
83.71
95.49
93.53
90.43 (↑1.83%p)
77.82
65.03
72.71
59.41
68.74 (↑2.74%p)
Table 2: Ablation study on each component on OfficeHome and TerraIncognita dataset.
Figure 5: Analysis of MTCT (Sec. 5.1 ). Probability distributions on TerraInc source domain training data under direct label prediction, reasoning-only SFT, and MTCT SFT: (A) all tokens and (B) class tokens.
Figure 6: Analysis of SARR (see Sec. 5.2 ) from token entropy on TerraInc (A) and OfficeHome (B). We track the entropy of selected class-relevant (A.I & B.I) and domain-specific tokens (A.II & B.II) across fine-tuning iterations, with vertical bars denoting variation across samples. Blue curves correspond to fitting GPT-4o reasoning chains, and orange curves to InternVL3-8B self-generated reasoning chains.
Figure 7: Impact of the number of self-labeling rounds N in SARR (Sec. 5.2 ) on TerraInc using InternVL3-8B. Top: classification accuracy on four domains (L100, L38, L43, L46) and their average, with paired t -test p -values for adjacent N shown on the right axis (reference line at p=0.05 ). Bottom: zero-shot token probability distributions (log-scale) on L100, using supervision data from each self-labeling round.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
bird
bobcat
cat
coyote
dog
empty
opossum
rabbit
raccoon
squirrel
Avg.
Domain divergence (Visual)
0.209
0.251
0.314
0.387
0.213
0.144
0.266
0.167
0.258
0.176
0.239
Domain divergence (Text)
0.054
0.048
0.103
0.114
0.093
0.126
0.091
0.103
0.142
0.115
0.099
Appendix
Table 3: Cross-domain divergence of visual embeddings and reasoning-chain embeddings for each TerraInc class.
L100
L38
L43
L38
Avg.
N=0
41.78
39.25
40.84
36.18
39.51
N=1
21.66
22.14
17.56
15.57
19.23
N=2
18.49
17.98
15.07
13.64
16.30
N=3
17.07
15.74
14.43
12.01
14.81
Appendix
Table 4: Rejection rate across SARR iterations (N) on the four TerraInc domains.
TerraIncognita
Idx
Method
L100
L38
L43
L46
Avg.
1
InternVL3-2B + Reasoning only (Baseline)
74.79
61.26
69.40
58.26
66.00
2
InternVL3-2B + MTCT
77.82
65.03
72.71
59.41
68.74 (↑2.74%p)
3
InternVL3-2B + MTCT + SARR
82.43
66.54
73.62
61.15
70.94 (↑4.94%p)
4
InternVL3-8B + Reasoning only
74.42
60.39
68.21
55.21
64.56
5
InternVL3-8B + MTCT
76.78
60.55
71.94
59.47
67.19 (↑2.63%p)
Appendix
Table 5: MTCT and SARR Performance on LLaVA-1.5 for TerraInc (%).
Method
Venue
A
C
P
R
Avg.
DANN ( Ganin et al., 2016 )
IJCAI’16
55.20
49.30
48.40
58.40
52.80
CORAL ( Sun and Saenko, 2016 )
ICCV’16
55.60
52.80
50.30
59.40
54.50
IRM ( Arjovsky et al., 2019 )
-
54.90
53.20
48.60
59.20
54.00
MMD ( Li et al., 2018b )
ICCV’18
55.10
52.00
50.30
59.30
54.20
OrgMixup ( Zhang et al., 2018 )
ICLR’18
56.00
54.40
50.40
61.00
55.50
Mixup ( Yan et al., 2020 )
-
55.50
54.10
49.40
59.40
54.60
Appendix
Table 6: SDG results (%) on OfficeHome. One domain is used as the source domain and the others are used as the target domain. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
FGVC-Aircraft
Idx
Method
Base
New
H
1
InternVL3-8B
20.35
17.83
19.01
2
+ Reasoning only (Baseline)
39.20
15.44
22.11
3
+ MTCT
56.42
18.60
27.95 (↑5.84%p)
4
+ MTCT + SARR
60.11
24.33
34.63 (↑12.52%p)
Appendix
Table 7: Base-to-new generalization results on FGVC-Aircraft.
Method
VQA
VE
Real
Cartoon
Pencil
Oil
Avg.
Real
Cartoon
Pencil
Oil
Avg.
ViT
55.03
48.52
47.76
48.65
49.99
72.15
52.51
57.72
58.78
60.29
CLIP ( Radford et al., 2021a )
58.23
49.11
50.41
49.41
51.79
73.10
55.85
61.61
60.93
62.87
BLIP ( Li et al., 2023a )
59.19
50.29
51.32
50.88
52.92
66.73
48.26
52.93
53.62
55.39
Open-Source Models.
BLIP2-FlanT5-XL ( Li et al., 2023a )
65.29
64.41
61.18
62.92
63.45
63.82
73.13
72.24
72.00
70.30
Appendix
Table 8: VQA and VE results (%) on VOLDOGER benchmark dataset. The best performances in comparisons are highlighted in bold and the second-best ones are marked with underlines .
Figure 8: Input prompt template used for reasoning-chain generation. The prompt consists of two parts: (i) a question instruction paired with an image and candidate label options, and (ii) explicit guidelines requiring the model (GPT-4o) to output reasoning in a structured five-section format (SUMMARY, CAPTION, REASONING, REFLECTION, CONCLUSION). These generated reasoning chains are then paired with ground-truth labels and used to fine-tune downstream multimodal LLMs.