FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification

Ankita Das; Ambarish Parthasarathy; Sumohana S. Channappayya; C. Krishna Mohan

cs.CVOct 7, 2026

FedSSMCoOp: SSM Encoders for light-weight Federated Prompt Learning for Few-shot Classification

Authors: Ankita Das, Ambarish Parthasarathy, Sumohana S. Channappayya, C. Krishna Mohan

Organizations: Department of Computer Science and Engineering Indian Institute of Technology Hyderabad · Department of Artificial Intelligence Indian Institute of Technology Hyderabad · Department of Electrical Engineering Indian Institute of Technology Hyderabad

Abstract

Vision-Language Models (VLMs) have shown strong performance across a wide range of downstream vision tasks, thanks to the complementary information contained in the respective domains. Despite the performance gains, most of these approaches rely on aligning these domains using the cosine similarity metric, which fails to capture token-level structure and cross-modal interactions prior to the classification stage. This is especially critical in biomedical applications under federated constraints, where data sharing is restricted, labeled data is scarce at each site, and it differs widely across institutions, leading to substantial statistical heterogeneity. To overcome this issue, we propose FedSSMCoOp, a federated few-shot image classification framework that enables multimodal learning while preserving data privacy. With the help of the SSM-based Vision Mamba and Cross Mamba blocks, and by optimizing only the soft-prompt and communication-prompt updates in the federated setting, the framework prioritizes both computation and performance. Importantly, this eliminates the need to use an external Large Language Model (LLM) for feature alignment. The framework is further trained and evaluated on various biomedical image datasets, and its performance is assessed. The proposed framework delivers stable performance relative to the baselines and is, on average, 1.96 times lighter. The corresponding script will be made available soon.

Figures & tables

Method	FL	PL	Vim	MIB	Medical	Few-shot
Federated Learning
FedAvg McMahan et al. (2017)	✓	✗	✗	✗	$\sim$	✗
FedProx Li et al. (2020)	✓	✗	✗	✗	$\sim$	✗
SCAFFOLD Karimireddy et al. (2020)	✓	✗	✗	✗	✗	✗
Prompt Learning
CoOp Zhou et al. (2022b)	✗	✓	✗	✗	✗	✓

Table 1: Comparison of related methods with FedSSMCoOp. FL : Federated Learning, PL : Prompt Learning, Vim : Vision Mamba encoder, MIB : Mamba Interaction Block. ✓ : supported, ✗ : not supported,

\sim

: partially supported.

Figure 1: The architecture of the proposed FedSSMCoOp framework. On each client, a frozen Vison Mamba encoder produces image tokens $V$ while the text encoder produces text tokens $T$ . The COMO block fuses both the streams bidirectionally into text-aware and image-aware features, which are mean-pooled to form $\mathrm{Pr}_{student}$ which is optimized using the cross-entropy loss $\mathcal{L}_{CE}$ . Only the prompt parameters are sent to the server for aggregation.

Dataset	Modality	Organ	#Cls	Classes	Train / Test / Val
BTMRI 1	Brain MRI	Brain	4	Glioma, Meningioma, Normal, Pituitary	2854 / 1717 / 1141
BUSI Al-Dhabyani et al. (2020)	Breast US	Breast	3	Benign, Malignant, Normal	389 / 236 / 155
CHMNIST Kather et al. (2016)	Histopathology	Colon	8	Adipose, Complex Stroma, Debris, Empty, Immune, Normal Mucosa, Simple Stroma, Tumor Epithelium	2496 / 1504 / 1000
COVID Tahir et al. (2021)	Chest X-ray	Lungs	4	COVID-19, Lung Opacity, Normal, Viral Pneumonia	10582 / 6351 / 4232
LC25000 Borkowski et al. (1912)	Histopathology	Lung & Colon	5	Colon Adeno., Colon Benign, Lung Adeno., Lung Benign, Lung SCC	12500 / 7500 / 5000
OCTMNIST Kermany et al. (2018)	Retinal OCT	Retina	4	CNV, Drusen, DME, Normal	97477 / 1000 / 10832

Table 2: Overview of the eleven biomedical imaging datasets used in our evaluation.

Model/Modality	BioMedCLIP	CoCoOp	KgCoOp	PromptFL	PromptFL+FedProx	FedOTP	BioMedCoOp	FedSSMCoOp
Vision Modality	86M	86M	86M	38M	38M	38M	86M	26M
Text Modality	110M	38M	38M	38M	38M	38M	38M	38M
Others	-	Prompt (35k)	Prompt (659k)	Prompt (2k)	Prompt (2k)	Prompt (4k)	Prompt (86M)	Cross-Mamba (5M)
Total Parameters	196M	124M	125M	77M	77M	77M	210M	69M

Table 3: The corresponding modality-wise parameters of the models. Note that the PromptFL framework makes use of a ResNet-based model in the vision modality.

Dataset	FedBiomedCLIP	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedTPG	FedMVP	FedMPT	FedBiomedCoOp	FedSSMCoOp (Ours)
BTMRI	57.08 $\pm$ 0.00	48.86 $\pm$ 7.92	51.19 $\pm$ 4.62	41.60 $\pm$ 3.29	36.93 $\pm$ 5.44	31.43 $\pm$ 3.93	47.13 $\pm$ 10.10	44.321	23.12	50.67 $\pm$ 4.74	53.64 $\pm$ 1.78
BUSI	51.27 $\pm$ 0.00	43.22 $\pm$ 6.27	39.69 $\pm$ 4.73	43.30 $\pm$ 12.72	37.43 $\pm$ 12.65	33.33 $\pm$ 6.91	37.57 $\pm$ 7.95	40.678	55.93	46.61 $\pm$ 7.79	40.11 $\pm$ 6.75
CHMNIST	29.06 $\pm$ 0.00	49.53 $\pm$ 1.90	45.48 $\pm$ 2.53	30.57 $\pm$ 6.17	33.60 $\pm$ 3.78	34.06 $\pm$ 9.48	50.82 $\pm$ 5.69	50.931	12.50	52.87 $\pm$ 2.43	53.39 $\pm$ 1.67
COVID	45.84 $\pm$ 0.00	44.69 $\pm$ 1.03	43.52 $\pm$ 11.06	42.35 $\pm$ 8.13	42.30 $\pm$ 8.20	37.50 $\pm$ 2.70	46.31 $\pm$ 8.46	41.474	17.08	43.59 $\pm$ 16.49	46.71 $\pm$ 10.13
LC25000	51.20 $\pm$ 0.00	58.72 $\pm$ 2.63	55.99 $\pm$ 7.84	42.90 $\pm$ 5.52	55.50 $\pm$ 5.33	26.25 $\pm$ 8.40	64.97 $\pm$ 4.24	37.840	20.00	61.88 $\pm$ 2.99	68.61 $\pm$ 4.67
OCTMNIST	35.00 $\pm$ 0.00	43.30 $\pm$ 2.18	32.43 $\pm$ 2.00	41.80 $\pm$ 9.72	40.65 $\pm$ 9.55	26.25 $\pm$ 8.40	35.23 $\pm$ 4.45	22.100	25.00	32.57 $\pm$ 9.67	34.93 $\pm$ 5.76

Table 4: The quantitative few-shot classification accuracy (

k=1

) results on the respective baselines and the proposed framework on different datasets. For each dataset, best-performing and second-best performing models are reported.

Figure 2: The UMAP plots of the proposed model against the baselines on different dataset. Observe that the proposed FedSSMCoOp framework has compact and well-separated embeddings in the text domain and a relatively looser cluster in the vision domain. This explains its ability of capturing discriminative features.

Figure 3: Visualization of activation maps of different models. The visualizations of baselines; FedCoCoOp, FedKgCoOp and FedBiomedCoOp are reported on 16 shot and the proposed FedSSMCoOp on 1 shot.

Component / Objective	No-LLM (FedSSMCoOp)	LLM-Server	LLM-Client	LLM-Client+Server
LLM prototype bank $P_{g}$	✗	Server-side	Client-side	Both
Client text branch	Prompt only	Prompt only	Dual (prompt+ $P_{g}$ )	Dual (prompt+ $P_{g}$ )
Prompt–prototype fusion	✗	✗	Learnable $\alpha$	Learnable $\alpha$
Classification loss $\mathcal{L}_{CE}$	✓	✓	✓	✓
Alignment loss $\mathcal{L}_{FedSCCM}$	✗	✓	✗	✓
Distillation loss $\mathcal{L}_{FedKD}$	✗	✗	✗	✓

Table 5: Architectural and training-objective differences across the four ablation configurations. The LLM prototype bank

P_{g}

is precomputed offline and never used during federated training; only the soft prompt is communicated.

Dataset	No-LLM	LLM-Server	LLM-Client	LLM-Client+Server
BTMRI	53.64 $\pm$ 1.78	44.05 $\pm$ 7.55	50.55 $\pm$ 2.64	46.11 $\pm$ 6.81
BUSI	40.11 $\pm$ 6.75	39.55 $\pm$ 2.83	39.41 $\pm$ 9.32	37.71 $\pm$ 1.84
CHMNIST	53.39 $\pm$ 1.67	52.24 $\pm$ 1.25	54.01 $\pm$ 2.25	52.22 $\pm$ 1.47
COVID	46.71 $\pm$ 10.13	41.90 $\pm$ 8.94	45.59 $\pm$ 11.29	41.99 $\pm$ 11.27
LC25000	68.61 $\pm$ 4.67	64.38 $\pm$ 6.39	71.78 $\pm$ 4.15	64.67 $\pm$ 6.39
OCTMNIST	34.93 $\pm$ 5.76	31.13 $\pm$ 5.92	32.73 $\pm$ 6.22	31.60 $\pm$ 6.98

Table 6: Ablation study on the proposed framework with inclusion of LLMs at the server and client ends across all datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	ViT + no LLM	ViT + LLM	Vim + no LLM	Vim + LLM	Ours (w/o LLM)	Ours (with LLM)
BTMRI	58.33 $\pm$ 11.79	50.67 $\pm$ 4.74	49.79 $\pm$ 2.45	47.20 $\pm$ 5.77	53.64 $\pm$ 1.78	44.05 $\pm$ 7.55
BUSI	41.24 $\pm$ 2.04	46.61 $\pm$ 7.79	39.55 $\pm$ 10.51	37.15 $\pm$ 9.91	40.11 $\pm$ 6.75	39.55 $\pm$ 2.83
CHMNIST	42.75 $\pm$ 2.38	57.87 $\pm$ 2.43	47.72 $\pm$ 2.01	39.07 $\pm$ 1.85	53.39 $\pm$ 1.67	52.24 $\pm$ 1.25
COVID	45.37 $\pm$ 7.75	43.59 $\pm$ 16.49	41.11 $\pm$ 3.10	37.60 $\pm$ 6.21	46.71 $\pm$ 10.13	41.90 $\pm$ 8.94
LC25000	60.94 $\pm$ 5.61	61.88 $\pm$ 2.99	65.49 $\pm$ 2.25	51.48 $\pm$ 2.78	68.61 $\pm$ 4.67	64.38 $\pm$ 6.39
OCTMNIST	33.83 $\pm$ 5.88	31.57 $\pm$ 9.67	35.47 $\pm$ 3.51	28.90 $\pm$ 4.53	34.93 $\pm$ 5.76	31.13 $\pm$ 5.92

Table 7: Ablation study (% accuracy) across different model variants under 1-shot setting for all biomedical datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	4 Clients			8 Clients			16 Clients
	$\alpha$ =0.1	$\alpha$ =0.5	$\alpha$ =1.0	$\alpha$ =0.1	$\alpha$ =0.5	$\alpha$ =1.0	$\alpha$ =0.1	$\alpha$ =0.5	$\alpha$ =1.0
BTMRI	31.59 $\pm$ 12.96	32.17 $\pm$ 12.90	32.81 $\pm$ 15.43	33.72 $\pm$ 15.92	53.64 $\pm$ 1.78	42.46 $\pm$ 6.20	34.50 $\pm$ 15.50	52.21 $\pm$ 7.94	33.59 $\pm$ 16.13
BUSI	40.53 $\pm$ 10.81	40.96 $\pm$ 9.76	41.24 $\pm$ 10.81	40.82 $\pm$ 11.71	40.11 $\pm$ 6.75	40.11 $\pm$ 10.17	41.00 $\pm$ 11.20	42.30 $\pm$ 11.00	38.72 $\pm$ 5.63
CHMNIST	36.37 $\pm$ 3.80	36.30 $\pm$ 4.84	36.19 $\pm$ 4.37	33.38 $\pm$ 4.91	53.39 $\pm$ 1.67	36.39 $\pm$ 3.82	34.22 $\pm$ 4.82	35.18 $\pm$ 4.15	37.20 $\pm$ 5.94
COVID	34.84 $\pm$ 6.65	34.82 $\pm$ 6.49	34.56 $\pm$ 7.49	34.10 $\pm$ 7.20	46.71 $\pm$ 10.13	34.88 $\pm$ 6.62	33.68 $\pm$ 6.99	34.29 $\pm$ 6.58	34.84 $\pm$ 7.12
LC25000	48.56 $\pm$ 3.63	48.77 $\pm$ 3.72	49.17 $\pm$ 3.41	48.49 $\pm$ 2.79	68.61 $\pm$ 4.67	48.67 $\pm$ 3.41	48.54 $\pm$ 3.07	48.72 $\pm$ 3.31	48.92 $\pm$ 2.92

Table 8: Effect of client count and data heterogeneity on FedSSMCoOp accuracy (%) at 1-shot, averaged over 3 seeds. For each of the datasets, Best performing and second best performing models are reported.

Model	BTMRI	BUSI	CHMNIST	COVID	LC25000	OCTMNIST	DermaMNIST	RetinaMNIST	Kvasir	CTKidney	KneeXray
FedBiomedCLIP	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G	33.70G
FedCoCoOp	38.04G	34.17G	53.55G	38.04G	38.04G	49.67G	41.92G	53.55G	41.92G	38.04G	38.04G
FedKgCoOp	38.04G	34.17G	53.55G	38.04G	38.04G	49.67G	41.61G	53.55G	41.92G	38.04G	38.04G
PromptFL	26.33G	22.46G	41.84G	26.33G	30.21G	26.33G	37.96G	26.33G	41.84G	26.33G	30.21G
PromptFL + FedProx	26.33G	22.46G	41.84G	26.33G	30.21G	26.33G	37.96G	26.33G	41.84G	26.33G	30.21G
FedOTP	41.94G	34.13G	73.14G	41.94G	49.74G	41.94G	65.34G	49.74G	73.14G	41.94G	41.94G

Table 9: The corresponding computational complexity (GFLOPs) with respect to the baselines and the proposed framework across all biomedical datasets.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Federated setup
$K$	Number of clients (hospitals)
$k$	Client index, $k\in\{1,\dots,K\}$
$t$	Communication round index
$T$	Total number of communication rounds
$E$	Number of local epochs per round

Appendix

Table 10: Summary of notation used in FedSSMCoOp.

Dataset	FedBiomedCLIP	FedCoCoOp	FedKgCoOp	FedBiomedCoOp	FedSSMCoOp (Ours)
BTMRI	57.08	26.42 $\pm$ 4.40	23.30	31.00	23.74 $\pm$ 2.67
BUSI	51.27	37.85 $\pm$ 13.23	17.37	22.10	36.44 $\pm$ 13.15
CHMNIST	29.06	22.07 $\pm$ 1.95	23.14	22.85	17.87 $\pm$ 1.75
COVID	45.84	25.29 $\pm$ 7.30	31.29	29.40	30.23 $\pm$ 8.98
LungColon	51.20	22.97 $\pm$ 3.69	28.80	26.15	20.80 $\pm$ 6.52
OCTMNIST	35.00	25.40 $\pm$ 0.86	21.80	22.45	19.03 $\pm$ 5.72

Appendix

Table 11: Top-1 accuracy (%) across all datasets and methods for zero-shot evaluation. Best and second-best results are highlighted for each dataset.

Dataset	FedBiomedCLIP	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedBiomedCoOp	FedSSMCoOp (Ours)
BTMRI	57.08 $\pm$ 0.00	48.86 $\pm$ 7.92	51.19 $\pm$ 4.62	41.60 $\pm$ 3.29	36.93 $\pm$ 5.44	31.43 $\pm$ 3.93	50.67 $\pm$ 4.74	53.64 $\pm$ 1.78
BUSI Al-Dhabyani et al. (2020)	51.27 $\pm$ 0.00	43.22 $\pm$ 6.27	39.69 $\pm$ 4.73	43.30 $\pm$ 12.72	37.43 $\pm$ 12.65	33.33 $\pm$ 6.91	46.61 $\pm$ 7.79	40.11 $\pm$ 6.75
CHMNIST Kather et al. (2016)	29.06 $\pm$ 0.00	49.53 $\pm$ 1.90	45.48 $\pm$ 2.53	30.57 $\pm$ 6.17	33.60 $\pm$ 3.78	34.06 $\pm$ 9.48	52.87 $\pm$ 2.43	53.39 $\pm$ 1.67
COVID Tahir et al. (2021)	45.84 $\pm$ 0.00	44.69 $\pm$ 1.03	43.52 $\pm$ 11.06	42.35 $\pm$ 8.13	42.30 $\pm$ 8.20	37.50 $\pm$ 2.70	43.59 $\pm$ 16.49	46.71 $\pm$ 10.13
LC25000	51.20 $\pm$ 0.00	58.72 $\pm$ 2.63	55.99 $\pm$ 7.84	42.90 $\pm$ 5.52	55.50 $\pm$ 5.33	26.25 $\pm$ 8.40	61.88 $\pm$ 2.99	68.61 $\pm$ 4.67
OCTMNIST	35.00 $\pm$ 0.00	43.30 $\pm$ 2.18	32.43 $\pm$ 2.00	41.80 $\pm$ 9.72	40.65 $\pm$ 9.55	26.25 $\pm$ 8.40	32.57 $\pm$ 9.67	34.93 $\pm$ 5.76

Appendix

Table 12: The quantitative few-shot classification accuracy (

k=1

) results on the respective baselines and the proposed framework on different datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedSSMCoOp (Ours)
BTMRI	49.80 $\pm$ 3.56	53.37 $\pm$ 6.67	37.20 $\pm$ 6.41	34.20 $\pm$ 5.38	39.42 $\pm$ 11.99	57.40 $\pm$ 4.59
BUSI	48.31 $\pm$ 3.00	42.66 $\pm$ 7.50	32.43 $\pm$ 7.02	26.80	57.73 $\pm$ 0.24	45.48 $\pm$ 3.21
CHMNIST	53.08 $\pm$ 1.51	44.99 $\pm$ 3.10	26.27 $\pm$ 12.55	32.93 $\pm$ 5.60	23.40 $\pm$ 5.15	60.61 $\pm$ 2.68
COVID	41.78 $\pm$ 11.49	28.72 $\pm$ 6.34	20.90 $\pm$ 7.79	37.63 $\pm$ 18.39	26.32 $\pm$ 14.12	54.17 $\pm$ 7.22
LungColon	53.17 $\pm$ 12.04	46.44 $\pm$ 6.89	46.44 $\pm$ 6.89	45.90 $\pm$ 4.92	40.81 $\pm$ 16.30	79.09 $\pm$ 6.20
OCTMNIST	37.33 $\pm$ 5.00	33.10 $\pm$ 2.45	31.40 $\pm$ 6.10	31.57 $\pm$ 9.67	40.00 $\pm$ 4.81	32.50 $\pm$ 5.80

Appendix

Table 13: Few-shot classification accuracy (%) under the 2-shot setting across biomedical datasets.For each of the datasets, Best performing and second best performing models are reported.

Dataset	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedSSMCoOp (Ours)
BTMRI	51.20 $\pm$ 4.10	48.71 $\pm$ 0.32	46.80 $\pm$ 4.00	32.23 $\pm$ 5.86	46.67 $\pm$ 12.04	60.12 $\pm$ 2.07
BUSI	47.46 $\pm$ 15.16	46.33 $\pm$ 2.34	56.43 $\pm$ 0.71	55.55 $\pm$ 2.75	39.30 $\pm$ 12.10	55.08 $\pm$ 5.96
CHMNIST	46.65 $\pm$ 9.26	40.43 $\pm$ 6.76	36.33 $\pm$ 3.68	33.70 $\pm$ 0.86	37.33 $\pm$ 3.07	71.23 $\pm$ 2.84
COVID	37.14 $\pm$ 5.31	40.56 $\pm$ 13.44	48.40 $\pm$ 0.71	48.10 $\pm$ 1.66	27.10 $\pm$ 8.49	53.81 $\pm$ 8.96
LungColon	55.00 $\pm$ 2.88	44.65 $\pm$ 6.88	48.27 $\pm$ 0.88	46.50 $\pm$ 4.15	52.80 $\pm$ 2.49	88.24 $\pm$ 1.67
OCTMNIST	47.27 $\pm$ 7.45	41.35 $\pm$ 3.10	38.37 $\pm$ 5.68	31.23 $\pm$ 14.20	36.53 $\pm$ 14.48	48.03 $\pm$ 1.90

Appendix

Table 14: Few-shot classification accuracy (%) under the 4-shot setting across biomedical datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedSSMCoOp (Ours)
BTMRI	55.12 $\pm$ 4.20	52.28 $\pm$ 4.14	48.00 $\pm$ 3.77	30.03 $\pm$ 5.95	41.83 $\pm$ 4.10	73.62 $\pm$ 1.64
BUSI	49.43 $\pm$ 11.53	40.68 $\pm$ 14.56	46.50 $\pm$ 8.20	43.10 $\pm$ 6.45	36.47 $\pm$ 8.73	61.30 $\pm$ 8.86
CHMNIST	57.45 $\pm$ 5.08	41.29 $\pm$ 1.05	42.10 $\pm$ 4.90	39.25 $\pm$ 3.10	42.47 $\pm$ 7.77	78.01 $\pm$ 0.60
COVID	52.69 $\pm$ 6.43	42.59 $\pm$ 8.91	46.80 $\pm$ 5.15	44.20 $\pm$ 2.80	31.97 $\pm$ 12.54	64.03 $\pm$ 4.18
LungColon	65.01 $\pm$ 4.54	50.32 $\pm$ 4.11	51.15 $\pm$ 3.20	48.40 $\pm$ 4.10	44.77 $\pm$ 4.98	92.40 $\pm$ 1.10
OCTMNIST	47.47 $\pm$ 3.40	44.25 $\pm$ 2.90	40.10 $\pm$ 5.10	34.80 $\pm$ 9.15	32.80 $\pm$ 12.95	61.20 $\pm$ 1.70

Appendix

Table 15: Few-shot classification accuracy (%) under the 8-shot setting across biomedical datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	FedCoCoOp	FedKgCoOp	PromptFL	PromptFL+FedProx	FedOTP	FedSSMCoOp (Ours)
BTMRI	60.86 $\pm$ 5.00	53.21 $\pm$ 3.14	52.60 $\pm$ 5.11	37.00 $\pm$ 8.65	48.23 $\pm$ 6.49	76.06 $\pm$ 1.31
BUSI BUSI	60.87 $\pm$ 6.20	51.70 $\pm$ 5.34	50.15 $\pm$ 7.40	47.50 $\pm$ 5.10	40.30 $\pm$ 10.28	69.21 $\pm$ 5.04
CHMNIST	62.17 $\pm$ 3.35	48.23 $\pm$ 0.79	46.40 $\pm$ 4.30	43.10 $\pm$ 2.85	49.93 $\pm$ 8.58	81.51 $\pm$ 1.71
COVID	44.54 $\pm$ 2.50	29.59 $\pm$ 9.41	41.20 $\pm$ 5.90	39.80 $\pm$ 3.40	23.20 $\pm$ 11.15	72.52 $\pm$ 1.62
LungColon	73.71 $\pm$ 4.01	52.30 $\pm$ 3.90	54.80 $\pm$ 2.15	51.20 $\pm$ 3.50	55.67 $\pm$ 5.09	92.97 $\pm$ 1.32
OCTMNIST	57.13 $\pm$ 8.04	47.10 $\pm$ 3.15	43.40 $\pm$ 5.25	38.50 $\pm$ 8.60	39.87 $\pm$ 16.34	57.60 $\pm$ 3.95

Appendix

Table 16: Few-shot classification accuracy (%) under the 16-shot setting across biomedical datasets. For each of the datasets, Best performing and second best performing models are reported.

Dataset	Discriminative signal	Favoured backbone
CHMNIST	Global morphology	Vision Mamba
COVID	Global context	Vision Mamba
LC25000	Tissue context	Vision Mamba
RetinaMNIST	Global structures	Vision Mamba
CTKidney	Fine details	Shot-dependent
BUSI	Local lesions	Attention

Appendix

Table 17: Observed relationship between dataset characteristics and best-performing backbone family, based on the few-shot results across shot settings.

Explore similar work

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging
Jul 15, 2026Shengchao Chen, Ting ShuMultimodal Foundation Model Cross-Modality
School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia
Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification
Apr 27, 2026Xiaoliu Luo, Minxue Xiao, Ting Xie +5Vision-Language Model Adaptation Medical Image Classification
Chongqing University of Technology · Hebei University of Technology · Nanyang Technological University +3
FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models
May 27, 2026Xucong Wang, Pengkun Wang, Zhe Zhao +3Multi-Label Classification Fedavg
University of Science and Technology of China (USTC) · Suzhou Institute for Advanced Research, USTC · City University of Hong Kong

Jul 15, 2026, cs.CVJ/K move · Enter open · S save

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Shengchao Chen, Ting Shu

School of Artificial Intelligence, Shenzhen University Shenzhen, China · Australian AI Institute, University of Technology Sydney Sydney, Australia

Multimodal Foundation Model Cross-Modality Medical Imaging Wireless Foundation Models Medical Visual Question Answering

Open paper AskPDF arXiv

Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM

^2

, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM

^2

equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable

O(1/\sqrt{T})

convergence and generalization guarantees. FM

^2

further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.

Abstract

Explore similar work

FM2^22: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

FM $^2$ : Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging