FedVP: Federated Learning across Heterogeneous Models by Pooling Locally Trained Verifiers
Organizations: University of Bath · Artificial Intelligence Innovation and Incubation Institute, Fudan University
Abstract
Conventional federated learning relies on the shared model backbone to aggregate client updates, limiting collaboration among institutions that deploy large language models (LLMs) from different families and at different scales. We propose Federated Verifier Pool (FEDVP), a new federated learning paradigm that enables heterogeneous clients to collaborate through shared verification rather than parameter aggregation. Each client trains a verifier using local data and responses generated by its chosen LLM backbone. The server pools these independently trained verifiers and combines their judgments to select answers without merging model parameters. This allows both LLM backbones and verifiers to differ in architecture and size, while keeping the LLM backbones frozen and deriving collaborative gains from improved answer selection. We evaluate FEDVP across five public medical question-answering datasets treated as federated clients, using Qwen and LLaMA backbones ranging from 3B to 14B parameters. FEDVP consistently improves performance across all five clients and outperforms the strongest conventional federated baselines by 1.46-5.79 percentage points in mean accuracy.
Figures & tables
| Center | Backbone | Verifier |
|---|---|---|
| BioASQ | LLaMA-3.2-3B | GRM (LLaMA-3.1-8B) |
| MedMCQA | Qwen2.5-7B | Skywork-Llama v0.2 (LLaMA-3.1-8B) |
| MedQA | LLaMA-3-8B | OffsetBias (LLaMA-3-8B) |
| MMLU-Med | Qwen2.5-3B | ArmoRM (LLaMA-3-8B) |
| PubMedQA | Qwen2.5-14B | Skywork-Qwen3 V2 (Qwen3-8B) |
| Method | BioASQ | MedMCQA | MedQA | MMLU-Med | PubMedQA | Avg. |
|---|---|---|---|---|---|---|
| Target backbone: LLaMA-3-8B-Instruct | ||||||
| FedIT | 86.29 | 57.47 | 60.80 | 66.26 | 76.60 | 69.48 |
| FFA-LoRA | 91.13 | 58.59 | 61.59 | 65.64 | 76.40 | 70.67 |
| FLoRA | 87.90 | 57.69 | 61.35 | 63.19 | 75.60 | 69.15 |
| FedPrompt | 92.74 | 59.77 | 61.74 | 69.33 | 77.60 | 72.24 |
| FedProx | 83.87 | 46.74 | 54.52 | 60.12 | 72.20 | 63.49 |
| Method | BioASQ | MedMCQA | MedQA | MMLU-Med | PubMedQA | Avg. |
|---|---|---|---|---|---|---|
| Target backbone: LLaMA-3-8B-Instruct | ||||||
| FedMD ( ) | 95.97 | 57.06 | 61.90 | 65.64 | 76.20 | 71.36 |
| FedMD + SC ( ) | 93.55 | 57.52 | 61.98 | 66.26 | 75.40 | 70.94 |
| FedMKT ( ) | 95.97 | 55.49 | 61.59 | 65.64 | 73.80 | 70.50 |
| FedMKT + SC ( ) | 95.97 | 56.90 | 61.59 | 63.80 | 75.20 | 70.69 |
| FedVP (Heterogeneous) | 94.35 | 58.24 | 66.77 | 70.55 | 78.60 | 73.70 |
| Client A | Client B | Client C | Client D | Client E | Pool | |
|---|---|---|---|---|---|---|
| Target backbone | BioASQ GRM | MedMCQA Skywork-Llama | MedQA OffsetBias | MMLU-Med ArmoRM | PubMedQA Skywork-Qwen3 | FedVP (Heterogeneous) |
| Llama-3.2-3B | 69.69 | 70.95 | 67.51 | 68.18 | 71.52 | 72.85 |
| (65.74) | ||||||
| Qwen2.5-7B | 74.08 | 74.90 | 72.66 | 73.49 | 74.94 | 75.80 |
| (71.25) | ||||||
| Llama-3-8B | 72.09 | 71.81 | 70.56 | 68.42 | 72.73 | 73.70 |
| Selector | BioASQ | MedMCQA | MedQA | MMLU-Med | PubMedQA | Avg. | (pp) |
| Target backbone: LLaMA-3-8B-Instruct | |||||||
| FedVP (Teacher pool) | 94.35 | 58.24 | 66.77 | 70.55 | 78.60 | 73.70 | — |
| Distilled verifier (Public data) | 96.77 | 57.85 | 68.34 | 71.17 | 78.60 | 74.55 | |
| Distilled verifier (Local data) | 96.77 | 57.16 | 68.66 | 73.62 | 78.00 | 74.84 | |
| Target backbone: Qwen2.5-7B-Instruct | |||||||
| FedVP (Teacher pool) | 96.77 | 60.27 | 68.26 | 76.07 | 77.60 | 75.80 | — |
| Selector | BioASQ | MedMCQA | MedQA | MMLU-Med | PubMedQA | Avg. |
|---|---|---|---|---|---|---|
| Evaluation generator: LLaMA-3-8B-Instruct | ||||||
| CoT ( ) | 81.45 | 41.50 | 45.56 | 49.69 | 55.80 | 54.80 |
| Self-Consistency ( ) | 93.55 | 52.21 | 62.06 | 62.58 | 77.40 | 69.56 |
| Pretrained single verifier ( ) | 80.65 | 50.56 | 58.68 | 59.51 | 59.40 | 61.76 |
| Pretrained verifier pool ( ) | 80.65 | 50.97 | 57.66 | 61.96 | 57.00 | 61.65 |
| FedVP (Uniform) ( ) | 94.35 | 58.12 | 67.95 | 70.55 | 78.40 | 73.87 |
| Uniform | Heterogeneous | |||
|---|---|---|---|---|
| Aggregation rule | LLaMA | Qwen | LLaMA | Qwen |
| Mean probability | 73.87 | 75.43 | 73.70 | 75.80 |
| Mean logit | 74.21 | 75.42 | 74.25 | 75.77 |
| Max probability | 73.33 | 74.41 | 73.68 | 75.32 |
| Member majority vote | 73.82 | 74.96 | 73.40 | 75.27 |
| LLaMA-3-8B | Qwen2.5-7B | |||
|---|---|---|---|---|
| Pool configuration | Avg. | (pp) | Avg. | (pp) |
| Full pool | 73.70 | – | 75.80 | – |
| w/o Client A | 73.63 | -0.07 | 75.71 | -0.09 |
| w/o Client B | 73.32 | -0.38 | 75.43 | -0.37 |
| w/o Client C | 73.54 | -0.16 | 76.00 | +0.20 |
| w/o Client D | 73.65 | -0.05 | 75.37 | -0.43 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Center | Train | Val. | Test | Format |
|---|---|---|---|---|
| BioASQ | 494 | 0 | 124 | yes/no |
| MedMCQA | 2,000 | – | 4,183 | 4 options |
| MedQA | 10,178 | 1,272 | 1,273 | 4 options |
| MMLU-Med | 1,299 | 162 | 163 | 4 options |
| PubMedQA | 450 | 50 | 500 | yes/no/maybe |
| Total | 14,421 | 6,243 |
| FedIT | FFA-LoRA | FLoRA | FedPrompt | |
|---|---|---|---|---|
| Trainable | LoRA | LoRA | new /round | 20 soft tokens |
| Rank / | 32 / 16 | 32 / 16 | 32 / 32 | – |
| Targets | q | q, v | q, v | input emb. |
| Optimizer | AdamW | SGD | AdamW | Adam |
| LR | 3e-4 | 5e-3 | 3e-4 | 0.3 |
| Schedule | linear | const. | linear | const. |
Heterogeneous Federated Learning
Heterogeneous Federated Learning studies how to train collaborative models when clients differ in data, tasks, model architectures, or computing resources.
See all 319 papers