Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization

Ayan Banerjee; Kuntal Thakur; Sandeep Gupta

cs.CVMar 12, 2026

Human Knowledge Integrated Multi-modal Learning for Single Source Domain Generalization

Authors: Ayan Banerjee, Kuntal Thakur, Sandeep Gupta

Organizations: Impact Lab, Arizona State University

Abstract

Generalizing image classification across domains remains challenging in critical tasks such as fundus image-based diabetic retinopathy (DR) grading and resting-state fMRI seizure onset zone (SOZ) detection. When domains differ in unknown causal factors, achieving cross-domain generalization is difficult, and there is no established methodology to objectively assess such differences without direct metadata or protocol-level information from data collectors, which is typically inaccessible. We first introduce domain conformal bounds (DCB), a theoretical framework to evaluate whether domains diverge in unknown causal factors. Building on this, we propose GenEval, a multimodal Vision Language Models (VLM) approach that combines foundational models (e.g., MedGemma-4B) with human knowledge via Low-Rank Adaptation (LoRA) to bridge causal gaps and enhance single-source domain generalization (SDG). Across eight DR and two SOZ datasets, GenEval achieves superior SDG performance, with average accuracy of 69.2% (DR) and 81% (SOZ), outperforming the strongest baselines by 9.4% and 1.8%, respectively.

Figures & tables

Figure 1: SDG attempts for DR has traditionally failed to consistently outperform ERM primarily due to existing gap in causal factors among data sets. Knowledge from human experts in the field can fill the causal gap but are usually qualitative and ambiguous. This paper demonstrates GenEval that quantifies human expert knowledge, refines them, and integrates specialized foundational model such as MedGemma-4B through LoRA based fine tuning to bridge the causal gap between domains while removing ambiguity of knowledge.

Method	Aptos	EyePACS	Messidor	Messidor 2	p value
MMD [ 51 ]	49.3 $\pm$ 1.0	69.3 $\pm$ 1.1	64.1 $\pm$ 4.8	69.6 $\pm$ 0.6	0.3
CDANN [ 52 ]	48.1 $\pm$ 0.7	73.1 $\pm$ 0.3	55.8 $\pm$ 1.8	61.2 $\pm$ 1.3	0.9
SD-ViT [ 73 ]	46.5 $\pm$ 0.8	71.1 $\pm$ 0.7	63.9 $\pm$ 0.9	71.4 $\pm$ 0.2	0.2
SPSD-ViT [ 27 ]	51.6 $\pm$ 1.1	73.3 $\pm$ 0.4	64.0 $\pm$ 1.4	72.9 $\pm$ 0.1	0.09
ERM-ViT [ 75 ]	48.5 $\pm$ 0.9	70.7 $\pm$ 1.7	62.7 $\pm$ 1.6	69.5 $\pm$ 2.5	NA
GenEval (Ours)	73.46	83.18	67.70	79.64

Table 1: Comparison of SOTA DG improvement techniques across DR datasets. Leave-one-domain-out evaluation: each column shows accuracy when that domain is held out as the target and the remaining three are used for training. Bold italic is highest accuracy, underlined is second highest. p value denotes statistical significance of performance gain over ERM-ViT, p

>

0.05 implies insignificant result.

Figure 2: Contributions: a) Algorithms 1 and 2 to obtain domain conformal bounds and source domain conformance degree. Utilizing SDCD, ablation studies are performed to refine the knowledge set into an optimal set that maximizes SDCD. b) GenEval, a specialized foundational vision language model based method that integrates image and expert knowledge in textual form for a multi-modal prompt driven DR grading and SOZ detection schemes.

Key LoRA hyperparameters

Adapter rank (

r

) = 16; scaling factor (

\alpha

) = 16; LoRA dropout = 0.05; target modules = { q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj }; modules to save = { lm_head, embed_tokens }; trainable params =

\approx

95M (2.4% of 4B); task type = CAUSAL_LM .

Training time (hh:mm:ss)

Single domain: A = 01:03:35; E = 10:14:20; M1 = 00:20:41; M2 = 00:34:11.

Multi domain: A+E+M1 = 12:01:30; A+E+M2 = 12:23:19; A+M1+M2 = 01:58:43; E+M1+M2 = 11:18:07.

Inference time

Single domain: 423.96 ± 33.52 ms [383.14, 458.72]; Multi domain: 385.46 ± 29.24 ms [359.71, 427.35].

Table 2: LoRA setup and fine-tuning time. A = APTOS 2019 , E = EyePACS , M1 = Messidor-1 , M2 = Messidor-2 .

Sou rce	Target	Best Baseline	Baseline Accuracy	K SDCD	D SDCD	GenEval	K + D SDCD
Mess	Aptos	SPSD-ViT [ 27 ]	48.3 %	92.7%	16%	56.0%	98.03%
idor	Messidor2	SPSD-ViT [ 27 ]	62.2%	99.1%	87.1%	65.18%	99.14%
	EyePACS	SPSD-ViT [ 27 ]	57.4%	17.02%	36.4%	80.04%	94.94%
Mess	Aptos	SPSD-ViT [ 27 ]	52.8%	32.1 %	17.7%	69.69%	76.3%
idor2	Messidor	SD-ViT [ 73 ]	61.0%	99.9%	98.2%	67.67%	100.00%
	EyePACS	SPSD-ViT [ 27 ]	72.5%	34.6%	41.31%	77.79%	96.34%

Table 3: SDG results on diabetic retinopathy datasets. Knowledge is K and Data is D We evaluate our approach across different source-target domain pairs using state-of-the-art methods. All results are reported as classification accuracy (%).

Ablation	SDCD (%)	Accuracy (%)
no ablation	59.000	65.01
remove microaneurysms	68.000	70.02
remove hemorrhages, or exudates	71.683	71.05
remove venous beading	82.765	73.21
remove Neovascularization	82.81	73.23

Table 4: Knowledge refinement through ablations guided by SDCD and corresponding accuracy.

Figure 3: Variation of SDCD metric with pixel signal to noise ratio due to instability in Mahalanobis distance, and effect on correlation of SDCD with Accuracy given by Lemma 1 .

Method	APTOS	Messidor	IDRiD	DeepDR	FGADR	RLDL	Avg
GDRNet	52.8	65.7	70.0	40.0	7.5	44.3	46.7
DECO	59.7	70.1	74.8	40.3	9.9	49.3	50.68
GenEval	73.2	69.5	70.6	59.2	56.9	67.6	66.2

Table 5: Extended single-source domain generalization on diabetic retinopathy. Train on EyePACS, test on six external targets. Metric: accuracy (%). Best scores in bold .

VLM Method	APTOS	Messidor	Average
CLIP [ 64 ]	44.3	39.6	41.9
OrdinalCLIP [ 45 ]	45.7	41.8	43.8
CLIP-DR [ 87 ]	46.3	47.3	46.8
GenEval- (Ours)	72.0	78.2	75.1

Table 6: VLM SDG F1 score (%) for DR Classification.

Source	Target	CuPKL [ 37 ]	CuPKL Avg	GenEval	GenEval Avg
PCH	UNC	82.3	88.1	91.0	90
UNC	PCH	93.8		89.0

Table 7: Single-source domain generalization on SOZ detection. Results are reported as F1-score (%). Best scores in bold .

Figure 4: MedGemma-4B LoRA fine-tuning overview. 2 2 2 Rank 16, $\alpha=16$ , dropout 0.05, targets: q,k,v,o, up,down,gate; LR $2\times 10^{-4}$ ; batch 1 (accum 32); 1 epoch; AdamW fused; bf16/fp16.

Dataset	Accuracy (%)	F1-Score (%)	Recall (%)
APTOS	61.80	60.25	62.15
EyePACS	79.66	77.61	78.95
Messidor	67.92	67.58	69.44
Messidor-2	77.52	75.78	76.71
Average	71.73	70.31	71.31

Table 8: Zero-shot performance of MedGemma-4B (no DR-specific fine-tuning).

Symptom	Key Observations and Diagnostic Relevance
Microaneurysms	Tiny red capillary dilations in the retina; earliest sign of mild NPDR. Their progression correlates with disease severity [ 9 ] .
Hemorrhages	Includes dark red dot/blot and flame-shaped types indicating microvascular leakage. Severe NPDR is marked by more than 20 hemorrhages in all quadrants; risk of PDR rises to 50% within a year [ 83 ] .
Hard Exudates	Sharp yellow lipid-rich deposits from chronic leakage, often in/near the macula. Indicative of risk for diabetic macular edema (DME), a major cause of vision loss [ 30 ] .
Cotton Wool Spots	Fluffy white retinal lesions caused by nerve-fiber-layer infarctions. Signify retinal ischemia in moderate to severe NPDR [ 72 ] .
Subhyaloid Hemorrhages	Boat- or D-shaped hemorrhages between the retina and the hyaloid face, typically from ruptured neovascular vessels. Hallmark of proliferative DR [ 12 ] .
Neovascularization	Fragile new vessel growth on the optic disc (NVD) or elsewhere on the retina (NVE). Defining trait of PDR; untreated cases face 60% vision loss within five years [ 22 ] .

Table 9: Clinical Signs of DR and Their Diagnostic Significance

Dataset	Micro.	Hemorr.	Exud.	Composite
APTOS	42.3 $\pm$ 5.1	48.6 $\pm$ 4.7	61.2 $\pm$ 5.4	51.0 $\pm$ 4.2
EyePACS	46.1 $\pm$ 4.3	54.0 $\pm$ 4.9	65.8 $\pm$ 5.0	55.2 $\pm$ 4.1
Messidor	44.7 $\pm$ 5.6	50.2 $\pm$ 5.1	63.1 $\pm$ 5.5	53.0 $\pm$ 4.6
Messidor-2	49.0 $\pm$ 4.8	56.3 $\pm$ 4.5	68.4 $\pm$ 4.7	57.0 $\pm$ 3.9
Mean	45.5	52.3	64.6	54.1

Table 10: Sample per–dataset IoU (%)

\pm

std. dev. on a held-out validation subset (illustrative). Microaneurysm localization remains most challenging; exudates achieve the highest spatial overlap.

Method	EyePACS	Messidor	Messidor-2	Average
DRGen [ 6 ]	67.5 $\pm$ 1.8	46.7 $\pm$ 0.1	61.0 $\pm$ 0.1	58.4
ERM-ViT [ 75 ]	67.8 $\pm$ 1.4	45.5 $\pm$ 0.2	58.8 $\pm$ 0.4	57.3
SD-ViT [ 73 ]	72.0 $\pm$ 0.8	45.4 $\pm$ 0.1	58.5 $\pm$ 0.2	58.6
SPSD-ViT [ 27 ]	71.4 $\pm$ 0.8	45.6 $\pm$ 0.1	58.8 $\pm$ 0.2	58.6
GenEval (Ours)	77.8 $\pm$ 0.8	49.0 $\pm$ 0.2	64.0 $\pm$ 0.2	63.6
p value	$<$ 0.01	0.04	0.02	$<$ 0.01

Table 12: SDG: Training on Aptos. Accuracy %

Method	APTOS	EyePACS	Messidor-2	Average
DRGen [ 6 ]	41.7 $\pm$ 4.3	43.1 $\pm$ 7.9	44.8 $\pm$ 0.9	43.2
ERM-ViT [ 75 ]	45.3 $\pm$ 1.3	52.4 $\pm$ 3.2	58.2 $\pm$ 3.2	51.9
SD-ViT [ 73 ]	44.3 $\pm$ 0.9	53.2 $\pm$ 1.6	57.8 $\pm$ 2.4	51.7
SPSD-ViT [ 27 ]	48.3 $\pm$ 1.1	57.4 $\pm$ 2.1	62.2 $\pm$ 1.6	55.9
GenEval (Ours)	56.0 $\pm$ 0.8	80.0 $\pm$ 2.1	65.2 $\pm$ 2.4	67.1
p value	$<$ 0.01	$<$ 0.01	0.03	$<$ 0.01

Table 13: SDG: Training on Messidor. Accuracy %

Method	APTOS	EyePACS	Messidor	Average
DRGen [ 6 ]	40.9 $\pm$ 3.9	69.3 $\pm$ 1.0	61.3 $\pm$ 0.8	57.7
ERM-ViT [ 75 ]	47.9 $\pm$ 2.1	67.4 $\pm$ 0.9	59.6 $\pm$ 3.9	58.3
SD-ViT [ 73 ]	51.8 $\pm$ 0.9	68.7 $\pm$ 0.6	62.0 $\pm$ 1.7	60.8
SPSD-ViT [ 27 ]	52.8 $\pm$ 2.0	72.5 $\pm$ 0.3	61.0 $\pm$ 0.8	62.1
GenEval (Ours)	69.7 $\pm$ 1.8	77.8 $\pm$ 0.3	67.7 $\pm$ 0.8	71.7
p value	$<$ 0.01	0.03	0.02	$<$ 0.01

Table 14: SDG: Training on Messidor-2. Accuracy %

Symptom	Key Observations and Diagnostic Relevance
Single Activation	Large, isolated activation region; primary indicator of epileptogenic focus [ 36 ] .
Gray Matter Activation	Activation primarily in gray matter with minimal white matter overlap; characteristic neuronal seizure activity [ 36 ] .
Hemispheric Asymmetry	Unilateral or asymmetric activation patterns; seizure foci show lateralized activity [ 36 ] .
Temporal Consistency	Persistent activation patterns across multiple time points; epileptogenic regions maintain consistent activity [ 36 ] .
Network Isolation	Activation distinct from resting-state networks (RSN); independence indicates pathological activity [ 36 ] .
Spatial Localization	Focal activation confined to specific anatomical regions; spatially constrained patterns [ 36 ] .

Table 15: Clinical Signs of SOZ and Their Diagnostic Significance

PCH
Method	Acc (%)	Prec (%)	Rec (%)	F1 (%)
CuPKL GPT-4o	88.4	93.8	93.8	93.8
GenEval	81.0	93.0	86.0	89.0
UNC
CuPKL GPT-4o	70.0	90.3	75.0	82.3
GenEval	83.0	93.0	88.0	91.0

Table 16: Single Domain Generalization SOZ detection using rs-fMRI. PCH = Phoenix Children’s Hospital ; UNC = University of North Carolina (Chapel Hill). Best scores in bold .

Source	Target	K SDCD (%)	D SDCD (%)	GenEval Acc. (%)
Messidor	APTOS	93.10	16.00	56.20
Messidor	Messidor2	98.70	87.10	65.10
Messidor	EyePACS	22.00	36.40	80.20
Messidor2	APTOS	30.50	17.70	69.60
Messidor2	Messidor	99.60	98.20	67.70
Messidor2	EyePACS	38.20	41.31	77.90

Table 17: SDG (YOLOv11 knowledge model): K SDCD vs. D SDCD and GenEval accuracy. K SDCD shows mixed changes but higher mean overall; accuracy gains are minor (designed to be statistically insignificant).

Explore similar work

Bridge: Basis-Driven Causal Inference Marries VFMs for Domain Generalization
Apr 29, 2026Mingbo Hong, Feng Liu, Caroline Gevaert +2Domain Generalization Cross-Domain Generalization
University of Twente · 2Drexel University
Reasoning-Driven Multimodal LLM for Domain Generalization
Feb 27, 2026Zhipeng Xu, Zilong Wang, Xinyang Jiang +3Domain Generalization Cross-Domain Generalization
Xidian University · Microsoft Research Asia · National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application
Bi-Level Optimization for Single Domain Generalization
Apr 7, 2026Marzi Heidari, Hanping Zhang, Hao Yan +1Cross-Domain Generalization
School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada

Apr 29, 2026J/K move · Enter open · S save

Bridge: Basis-Driven Causal Inference Marries VFMs for Domain Generalization

Mingbo Hong, Feng Liu, Caroline Gevaert, George Vosselman+1

University of Twente · 2Drexel University

Domain Generalization Cross-Domain Generalization Causal Inference

Open paper AskPDF arXiv

Detectors often suffer from degraded performance, primarily due to the distributional gap between the source and target domains. This issue is especially evident in single-source domains with limited data, as models tend to rely on confounders (e.g., illumination, co-occurrence, and style) from the source domain, leading to spurious correlations that hinder generalization. To this end, this paper proposes a novel Basis-driven framework for domain generalization, namely Bridge, that incorporates causal inference into object detection. By learning the low-rank bases for front-door adjustment, Bridge blocks confounders' effects to mitigate spurious correlations, while simultaneously refining representations by filtering redundant and task-irrelevant components. Bridge can be seamlessly integrated with both discriminative (e.g., DINOv2/3, SAM) and generative (e.g., Stable Diffusion) Vision Foundation Models (VFMs). Extensive experiments across multiple domain generalization object detection datasets, i.e., Cross-Camera, Adverse Weather, Real-to-Artistic, Diverse Weather Datasets, and Diverse Weather DroneVehicle (our newly augmented real-world UAV-based benchmark), underscore the superiority of our proposed method over previous state-of-the-art approaches. The project page is available at: https://mingbohong.github.io/Bridge/.

Abstract

Explore similar work

Bridge: Basis-Driven Causal Inference Marries VFMs for Domain Generalization

Reasoning-Driven Multimodal LLM for Domain Generalization

Bi-Level Optimization for Single Domain Generalization