Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

Mohammed Rahman Sherif Khan Mohammad; Ardhendu Behera; Sandip Pradhan; Swagat Kumar; Amr Ahmed

Figure 1: The core trade-off of VLM adaptation and our proposed solution. (a) Lightweight adapters (e.g., Tip-Adapter [ 43 ] ) are fast but limited, as they only reason over the single, global image feature. (b) Heavyweight adapters (e.g., GraphAdapter [ 22 ] ) are more powerful, reasoning over visual patches, but their complex GNN adds a permanent, significant cost at inference time. (c) Our proposed method , TOGA (the best of both), uses a powerful auxiliary teacher. This teacher, our modality-aware graph transformer (MGT), performs deep bi-modal reasoning on a unified graph of patches and text. It supervises its knowledge into the lightweight “student” adapter, which is the only component used at inference, achieving high performance with the efficiency of (a).

Figure 2: Our Training-Only Graph Adapter ( TOGA ) is an asymmetric supervision pipeline. At train time (red dotted region and

\rightarrow

denote training only), we use a three-branch ensemble: (1) a frozen

L_{\text{ZS}}

branch, (2) a lightweight student (Cache Model)

L_{\text{Cache}}

, and (3) our powerful, auxiliary graph teacher

L_{\text{Graph}}

. The teacher enriches multi-scale patches and text embeddings via unimodal Transformers, then constructs a modality-aware graph transformer to perform cross-modal reasoning. A dual-loss objective regularizes the teacher’s knowledge into the student’s adapter

\mathcal{A}

. At test time , the entire teacher branch is discarded, resulting in zero additional inference cost.

Figure 3

Method	ImageNet	-R	-A	-V2	-Sketch	Avg
CLIP [ 27 ]	66.7	74.0	47.8	60.8	46.2	59.1
CoOp [ 47 ]	71.5	75.2	49.7	64.2	48.0	61.1
CoCoOp [ 46 ]	71.0	76.2	50.6	64.1	48.8	62.1
MaPLe [ 19 ]	70.7	77.0	50.9	64.1	49.2	62.4
ProGrad [ 48 ]	72.2	74.6	49.4	64.7	47.6	61.7
KgCoOp [ 37 ]	71.2	76.7	50.7	64.1	49.0	62.3

Table 2: Out-of-distribution (OOD) robustness accuracy (%) on ImageNet and its shifted variants; Avg: mean across five datasets.

	Loss Objective	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
Euro	$\mathcal{L}_{CE}$	65.1	73.1	77.9	83.2	88.1
	$\mathcal{L}_{CE}+\mathcal{L}_{CE}^{Graph}$	67.1	74.5	79.3	82.2	87.5
	$\mathcal{L}_{CE}+\mathcal{L}_{Focal}^{Graph}$	67.4	74.9	80.3	84.1	89.4
Cal	$\mathcal{L}_{CE}$	94.0	94.3	95.4	95.5	95.8
	$\mathcal{L}_{CE}+\mathcal{L}_{CE}^{Graph}$	94.0	94.6	95.4	95.5	96.1
	$\mathcal{L}_{CE}+\mathcal{L}_{Focal}^{Graph}$	94.3	95.0	95.7	95.8	96.3

Table 3: Ablation of the dual-objective co-training strategy.

\mathcal{L}_{CE}

denotes the cross-entropy loss on the final logits.

\mathcal{L}_{CE}^{Graph}

and

\mathcal{L}_{Focal}^{Graph}

are the auxiliary losses applied to the teacher. Dataset abbreviation follows the same as table 1 .

	Pooling Strategy	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
EuroSAT	$\sum_{\text{All}}$ (Global)	63.4	72.2	79.5	83.4	88.7
	$\sum_{\text{Top-}\mathbb{N}=25}$	65.3	73.3	79.9	83.7	88.9
	$\sum_{\text{Top-}\mathbb{N}=50}$	67.4	74.9	80.3	84.1	89.4
	$\sum_{\text{Top-}\mathbb{N}=75}$	64.9	73.0	79.6	83.4	88.8
Caltech	$\sum_{\text{All}}$ (Global)	94.1	94.6	95.5	95.6	96.2
	$\sum_{\text{Top-}\mathbb{N}=25}$	94.2	94.8	95.6	95.8	96.1

Table 4: Global pooling (

\sum_{\text{All}}

) vs Top-

\mathbb{N}

filtering (

\sum_{\text{Top-}\mathbb{N}}

). The Top-

\mathbb{N}

strategy explicitly filters out non-discriminative patches.

Figure 4: Qualitative visualization of Discriminative Node Filtering. Our Top-

\mathbb{N}

filtering learns to retain high-scoring, discriminative foreground patches (Green) while suppressing non-informative background nodes (Blue). This resolves feature dilution inherent in global pooling, yielding a cleaner signal by focusing on key object parts (e.g., ant’s head, flower pistil, cat’s eyes). Samples from Caltech-101; more visualizations in supplementary.

	Patch Strategy	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
Euro	Grid $2\times 2$	65.0	72.9	80.0	83.7	88.8
	Grid $3\times 3$	61.7	72.5	78.8	83.6	88.8
	Grid $4\times 4$	66.6	74.5	77.9	83.7	88.9
	MultiScale	67.4	74.9	80.3	84.1	89.4
Cal	Grid $2\times 2$	93.7	94.5	95.5	95.6	96.2
	Grid $3\times 3$	93.8	94.6	95.6	95.6	96.0

Table 5: Ablation of patch grid granularity. Comparison of fixed-scale grids vs MultiScale strategy (18 patches).

	Configuration	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
Euro	w/o MGT	61.9	71.5	77.2	80.3	85.7
Euro	w/ MGT	67.4	74.9	80.3	84.1	89.4
Cal	w/o MGT	94.0	94.6	95.4	95.2	95.8
Cal	w/ MGT	94.3	95.0	95.7	95.8	96.3
Food	w/o MGT	86.0	86.3	86.5	86.4	87.2
Food	w/ MGT	86.2	86.5	86.7	86.8	87.5

Table 6: Impact of Modality-aware Graph Transformer (MGT).

	T	M	F	P	1-shot	2-shot	4-shot	8-shot	16-shot
EuroSAT	✓	✓	✓	✓	67.4	74.9	80.3	84.1	89.4
	✓	✓	✓	✗	64.1	71.3	76.8	81.0	86.7
	✓	✓	✗	✓	63.4	72.2	79.5	83.4	86.5
	✓	✗	✓	✓	61.9	71.5	77.2	80.3	85.7
	✗	✓	✓	✓	66.2	72.6	78.9	81.8	87.4
Aircraft	✓	✓	✓	✓	31.0	34.8	38.3	44.2	48.4

Table 7: Internal teacher ablation. T: unimodal Transformer, M: MGT, F: TopK filtering, P: patch–text edges. Cross-modal graph reasoning is the primary contributor ( M + P ), while T and F offer smaller complementary gains.

Figure S1 : Performance trajectories of different adapter-based architectures. The curves show the few-shot accuracy of the model on several additional datasets, complementing the main results presented in the paper.

Shots	Method	Venue	INet	SUN	Air	Euro	Cars	Food	Pets	Flow	Cal	DTD	UCF	Avg
0	CLIP [ 27 ]	ICML’22	66.7	62.6	24.7	47.5	65.3	86.1	89.1	71.4	92.9	43.6	66.7	65.1
1	CoOp [ 47 ]	IJCV’22	68.0	67.3	26.2	50.9	67.1	82.6	90.3	72.7	93.2	50.1	70.7	67.2
	CoOp [ 47 ]	IJCV’22	65.7	67.0	20.8	56.4	67.5	84.3	90.2	78.3	92.5	50.1	71.2	67.6
	CoCoOp [ 46 ]	CVPR’22	69.4	68.7	28.1	55.4	67.6	84.9	91.9	73.4	94.1	52.6	70.4	68.8
	PLOT++ [ 3 ]	ICLR’23	66.4	66.7	28.6	65.4	68.8	86.1	91.8	80.4	94.3	54.5	74.3	70.7
	KgCoOp [ 37 ]	CVPR’23	68.9	68.4	26.8	61.9	66.7	86.4	92.1	74.7	94.2	52.7	72.8	69.6

Table S1 : Comparison of few-shot classification accuracy (%) on 11 benchmark datasets. We evaluate our method against several SOTA prompt-based approaches. The best performance in each shot-group is marked in bold . Our results are highlighted in gray. Dataset abbreviations: INet (ImageNet), SUN (SUN397), Air (Aircraft), Euro (EuroSAT), Cars (Stanford Cars), Food (Food101), Pets (OxfordPets), Flow (Flowers102), Cal (Caltech101), DTD (Describable Textures), UCF (UCF101).

Figure S2 : t-SNE visualization of logits on the Caltech101 dataset (

16

-shot). The plots depict the separation of

8

object categories, ranging from distinct objects like stop sign to complex shapes like windsor chair. (a) Baseline CLIP provides a strong initial separation (

0.618

). (b) Tip-Adapter-F introduces minor variance, slightly reducing the score to

0.592

. (c) Ours further refines the manifold, tightening the clusters for difficult classes (e.g., crocodile and brontosaurus) and achieving the highest distinctiveness with a Silhouette score of

0.693

. This highlights the benefit of our discriminative node filtering in suppressing background noise.

Figure S3 : This visualization highlights the model’s ability to distinguish between fine-grained culinary categories. (a) Baseline CLIP and (b) Tip-Adapter-F show comparable performance (

\approx 0.58

), with noticeable overlap between visually similar classes (e.g., macarons and cup cakes). (c) Ours (ViT-B/16) significantly enhances class purity, leading to superior clustering of features and raising the Silhouette score to

0.732

. This confirms that our MGT-guided distillation effectively transfers local ingredient-level details into the student cache.

Figure S4 : t-SNE visualization of logits on the EuroSAT dataset (

16

-shot). We visualize the embedding space for

8

randomly selected categories. (a) Baseline CLIP fails to form distinct clusters for texture-heavy classes, resulting in a negative Silhouette score of

-0.043

. (b) Tip-Adapter-F improves separability (

0.211

) but retains ambiguity between vegetation classes. (c) Ours demonstrates that the asymmetric graph supervision effectively disentangles complex spectral signatures, achieving a Silhouette score of

0.522

and showing clear separation between semantically adjacent classes like Pasture Land and Herbaceous Vegetation Land.

Figure S5 : Extended Qualitative Visualization of Discriminative Node Filtering. This figure provides additional samples demonstrating the efficacy of our

\text{Top-}\mathbb{N}

filtering across various Caltech-101 classes. The learned filter successfully identifies and retains high-scoring, discriminative foreground patches (Green) covering key object features (e.g., the segmented body of the scorpion, the silhouette of the scissors, and the spike pattern of the stegosaurus). Concurrently, it suppresses non-informative background regions (Blue). This mechanism ensures that the GNN Teacher’s final visual feature is robust against feature dilution, yielding a high-fidelity relational signal for cache supervision.

Shots	Backbone	Method	Datasets
Shots	Backbone	Method	SUN	Air	Euro	Cars	Food	Pets	Flowers	Calt	DTD	UCF
1	ResNet-101	Base	60.9	19.9	38.7	64.2	80.6	87.1	64.8	91.0	43.4	63.4
	ResNet-101	Ours	63.5	22.7	62.1	66.1	81.0	87.8	81.1	91.9	50.5	68.0
	ViT-B/32	Base	63.8	20.8	49.9	61.1	80.3	87.2	67.8	91.5	48.2	65.9
	ViT-B/32	Ours	66.2	24.3	60.8	62.9	81.0	88.5	84.5	92.6	52.5	70.2
	ViT-B/16	Base	65.0	25.9	53.4	67.1	85.0	89.2	69.7	93.6	47.1	70.9

Table S2 : Comparison of few-shot classification accuracy (%). The table compares the baseline Tip-Adapter-F (denoted as Base ) against the Proposed method (denoted as Ours ) across various CLIP backbones. Distinct blocks represent different shot settings (

K=\{1,2,4,8\}

). Best results are highlighted in bold .

Figure S6 : Accuracy vs. Efficiency Trade-off (8-Shot Average). We compare the total inference cost (GFLOPs) against average few-shot accuracy across 10 benchmark datasets. Our method demonstrates strong scaling efficiency : while the transition from ViT-B/16 to ViT-L/14 incurs higher computational cost (due to the larger backbone resolution and depth), it yields a massive +5.1% performance gain , reaching a state-of-the-art average accuracy of 85.9%. This highlights that our GNN-based adaptation module effectively leverages the capacity of stronger backbones without hitting a saturation point.

Backbone	Res.	Dim	Backbone		GNN (Ours)		Cache
Backbone	Res.	Dim	GFLOPs	Param	GFLOPs	Param	GFLOPs	Param
RN101	224	512	19.6	56.3	4.5	13.5	0.02	8.2
ViT-B/32	224	512	8.7	87.9	4.5	13.5	0.02	8.2
ViT-B/16	224	512	33.7	86.2	4.5	13.5	0.02	8.2
ViT-L/14	224	768	155.7	304.0	10.2	30.4	0.03	12.3

Table S3 : Efficiency Analysis. We report GFLOPs (billions) and Parameters (millions) for the Backbone, our GNN module, and the Cache Adapter. Note that our GNN introduces marginal overhead compared to the backbone.

Figure S7 : Efficiency Profile: Inference vs. Training. Left: Breakdown of inference computational cost (GFLOPs). Our method (GNN + Cache) introduces only marginal overhead compared to the pre-trained backbone, scaling efficiently to larger models (e.g., only +6.1% overhead for ViT-L/14). Right: Analysis of trainable parameters. By keeping the massive backbone frozen (grey), our approach requires optimizing only a lightweight set of parameters (

\sim

21M–43M) for the GNN and Adapter, significantly reducing training memory and time costs compared to full fine-tuning.

Figure S8 : Hyperparameter Sensitivity Analysis. We visualize the impact of key hyperparameters on test accuracy, highlighting the necessity of our validation-based tuning protocol.

Hyperparameter	Symbol	Search Range	Scale
Learning Rate	$\eta$	$[10^{-5},5\times 10^{-2}]$	Log-Uniform
Contrastive Weight	$\lambda_{con}$	$[0.1,1.0]$	Uniform
Graph Logit Scale	$\gamma$	$[0.1,5.0]$	Uniform
Focal Loss Gamma	$\gamma_{focal}$	$[0.5,3.0]$	Uniform
Tip-Adapter Alpha	$\alpha$	$[1.0,10.0]$	Uniform

Table S4 : Hyperparameter Optimization Space. Optimization parameters are tuned on the validation set to adapt to domain granularity and shot availability.

Figure S9 : Quantitative comparison between the Baseline (Tip-Adapter) and Ours (Graph Refined) models on four test examples from the Caltech-101 dataset. Each panel displays the ground truth (GT) label, the input image, and the top-5 classifications with their associated probabilities. Green bars denote the correct class, while red bars indicate an incorrect top-1 prediction. The results demonstrate that the graph-based method corrects the baseline’s misclassifications (e.g., in the first and third panels) and significantly improves confidence in correct predictions (e.g., in the second panel).

Caltech101
Nodes	1-Shot	2-Shot	4-Shot	8-Shot	16-Shot
5	94.0	94.7	95.7	95.7	96.0
9	94.0	94.7	95.6	95.7	96.0
14	94.1	94.7	95.6	95.7	96.1
18	94.3	95.0	95.7	95.8	96.3
21	94.1	94.8	95.6	95.7	96.2

Table S5 : Few-shot classification accuracy (in %) for the model across varying numbers of Nodes (rows) and Shots per class (columns) on the Caltech and EuroSAT datasets.

EuroSAT
Configuration	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
w/o Text	64.6	70.5	75.5	79.8	84.2
w/ Text	67.4	74.9	80.3	84.1	89.4
Caltech101
w/o Text	93.6	93.6	95.1	95.2	95.7
w/ Text	94.3	95.0	95.7	95.8	96.3

Table S6 : Impact of Text Nodes in MGT.

MGT	Trans	1 Shot	2 Shot	4 Shot	8 Shot	16 Shot
EuroSAT
✗	✓	62.9	72.1	77.4	80.8	87.5
✓	✗	67.4	74.9	80.3	84.1	89.4
Aircraft
✗	✓	27.4	31.6	34.9	41.5	45.7
✓	✗	31.0	34.8	38.3	44.2	48.4

Table S7 : Comparison of MGT vs. Transformer architectures across different datasets and shot configurations.

Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters

Abstract

Figures & tables

Explore similar work

Hold-One-Shot-Out (HOSO) for Validation-Free Few-Shot CLIP Adapters

Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning

GP-Adapter: Gaussian Process CLIP-Adapter for Few-Shot Out-of-Distribution Detection