Decouple, Purify and Unite: Semantic-Structural Prototype Learning for Federated Medical Segmentation
Authors: Xingyue Zhao, Wenke Huang, Linghao Zhuang, Yanzhou Su, Zhifeng Wang, Haoyu Zhao, Mengfan Li, Junjun He, +6 more
Organizations: Department of Neurosurgery, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China · The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Ant Group, China · DAMO Academy, Alibaba Group, China · Department of Medical Engineering, Xijing 986 Hospital, Fourth Military Medical University, Xi’an, China · Shanghai Innovation Institute, Shanghai, China · Faculty of Applied Sciences, Macao Polytechnic University, Macao, China · School of Computer Science, Wuhan University, Wuhan, China · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China
Federated learning enables medical institutions to train a global model without sharing data, yet feature heterogeneity from diverse scanners or protocols remains challenging. Existing representation-based methods face two limitations: 1) Incomplete Contextual Representation Learning: single-layer or coupled representations overlook multi-level structural cues and entangle regional semantics with boundary details. 2) Layerwise Style and Aggregation Biases: domain-specific style discrepancies across intermediate layers degrade prototypes, while aggregation that overlooks client distribution shifts can further amplify bias. We propose FedBCS+, federated decoupled contextual alignment with style-purified aggregation. We employ Frequency-domain Style Recalibration (FSR) in prototype construction to decouple content-style representations and extract style-purified prototypes. Built upon these purified features, Decoupled Contextual Prototype Alignment (DCPA) explicitly decouples multi-level features into semantic and structural prototypes and aligns regional semantics and fine-grained anatomical structures separately. Style-purified Semantic Prototype Aggregation (S2PA) measures each client's purified prototype divergence from the global consensus and adaptively reweights aggregation toward under-represented clients to reduce consensus bias. On five heterogeneous medical segmentation benchmarks spanning histopathology, MRI, ultrasound, and colonoscopy, FedBCS+ achieves the highest mean Dice among the compared methods. A convergence analysis further characterizes how aggregation and alignment affect the optimization bound.
Figures & tables
Fig. 1 : Two client-side representation challenges in prototype-based federated medical segmentation. Challenge 1 : Single-layer alignment misses complementary multi-level context, while prototypes dominated by region-center semantics underrepresent sparse boundary structures. Coupling these cues in a single prototype further causes cross-client misalignment. Challenge 2 : Site-specific style discrepancies propagate across network layers, producing conflicting feature distributions and corrupting the resulting prototypes. FedBCS+ addresses these issues with DCPA for decoupled semantic–structural alignment and FSR for style-purified representation learning; the purified prototypes subsequently guide S2PA aggregation.
Fig. 2 : Overview of FedBCS+. On each client, the UNet backbone applies frequency-domain style recalibration to its deeper encoder stages and extracts multi-level encoder and decoder features. DCPA constructs semantic prototypes from background and foreground-center features and structural prototypes from foreground-boundary features in each fused stream. The foreground semantic and structural branches are aligned independently. On the server, S2PA clusters the uploaded prototypes using FINCH and uses their distances from the global consensus to determine the parameter aggregation weights.
Fig. 3 : Foreground ratio distributions across clients. The four 2D benchmarks and the 3D FeTS 2022 benchmark exhibit substantial variation in target size across clients. Each row represents a client, and each column represents a foreground ratio bin. The distributions are normalized separately for each client.
TABLE I : Datasets and client configurations. M denotes the number of federated clients used for each benchmark.
Fig. 4 : Qualitative segmentation results across the five benchmarks. Each row corresponds to a benchmark. The Input column illustrates appearance variations across imaging domains, and the GT column shows the ground-truth contours in green . In the segmentation results, green indicates true positives, red indicates false negatives, and blue indicates false positives. For FeTS 2022, the whole-tumor region is shown in an axial slice, with the same slice and crop used for all methods. FedBCS+ delineates the targets accurately with fewer missed regions and false positives.
Method
P (M)
Δ (%)
FLOPs (G)
Δ (%)
Nproto
Dice (%)
FedAvg
8.267
–
24.147
–
0
71.14
FedProto
8.267
–
24.147
–
2
70.56
FPL
8.267
–
24.147
–
2
72.25
FedContrast
8.268
+0.01
24.383
+0.98
9
56.90
FedPLVM
8.267
–
24.147
–
≈212
70.64
FedBCS †
8.268
+0.01
24.383
+0.98
4
73.64
TABLE II: Client-side cost and segmentation accuracy on Histology. P denotes the model parameter count, and Nproto is the number of prototypes uploaded per client per round. FLOPs are computed for a 256×256 input, including the FFT operations. The Δ columns report percentage increases relative to UNet. FSR adds 902 parameters and increases FLOPs by 0.98%. A dagger denotes our conference version.
Histology (4 clients)
Prostate (6 clients)
Method
TCIA
CRC
KIRC
TNBC
Avg.
RUNMC
BIDMC
BMC
HK
I2CVB
UCL
Avg.
General FL Methods
FedAvg
71.78 ± 3.20
75.38 ± 1.77
73.19 ± 0.55
64.22 ± 4.17
71.14 ± 1.10
82.29 ± 2.10
71.79 ± 3.84
82.50 ± 2.26
81.74 ± 3.06
82.99 ± 3.25
73.55 ± 2.70
79.14 ± 1.44
FedProx
70.45 ± 3.48
75.08 ± 1.09
72.33 ± 1.89
63.75 ± 5.26
70.40 ± 0.43
80.92 ± 2.01
71.37 ± 5.36
80.52 ± 2.45
81.31 ± 2.99
81.01 ± 3.60
71.55 ± 4.31
77.78 ± 1.71
MOON
70.87 ± 3.35
74.66 ± 0.98
72.21 ± 1.85
64.26 ± 4.24
70.50 ± 0.59
82.83 ± 2.17
72.09 ± 6.04
81.83 ± 1.75
81.78 ± 4.18
82.78 ± 3.53
74.74 ± 5.52
79.34 ± 1.29
HarmoFL
74.40 ± 2.62
70.94 ± 2.54
72.09 ± 0.76
62.57 ± 3.03
70.00 ± 1.65
84.45 ± 2.13
77.56 ± 5.77
83.74 ± 2.17
82.60 ± 3.48
84.47 ± 2.93
76.75 ± 2.64
81.59 ± 1.60
TABLE III : Comparison with state-of-the-art federated learning methods on four 2D medical segmentation benchmarks. Dice scores are expressed as percentages and reported as means and standard deviations over five folds. The Avg. columns report results averaged across clients. The best results are shown in bold and the second best are underlined . A dagger denotes our conference version [ 24 ] .
Method
TC
WT
ET
Avg. Dice
General FL Methods
FedAvg
80.10
91.24
81.47
84.27
FedProx
80.94
90.78
81.86
84.53
MOON
80.49
91.29
82.14
84.64
HarmoFL
80.31
91.48
81.28
84.36
FedRDN
80.42
91.03
81.79
84.42
TABLE IV: Brain tumor segmentation on FeTS 2022. The experiment uses nine clients and the first cross-validation split. TC, WT, and ET denote tumor core, whole tumor, and enhancing tumor, respectively. Dice scores are expressed as percentages, and the final column reports the mean across the three tumor regions. The best results are shown in bold and the second best are underlined . A dagger denotes our conference version.
Fig. 5 : Foreground and background feature separability. The upper panels show t-SNE visualizations of encoder features learned by FedAvg and FedBCS+ on the four 2D benchmarks and FeTS 2022. The lower panel compares their silhouette scores. FedBCS+ achieves higher scores across all five benchmarks, indicating clearer separation between foreground and background features.
Method
TCIA
CRC
KIRC
TNBC
Avg.
FedAvg
60.45
72.83
70.69
64.58
67.14
FedProx
60.22
72.85
69.65
64.90
66.91
MOON
60.38
72.89
70.68
64.59
67.14
FPL
59.25
73.55
69.22
66.30
67.08
FedPLVM
68.89
59.65
70.64
68.60
66.95
FedBCS †
70.63
59.54
71.58
69.93
67.92
TABLE V: Leave-one-site-out generalization on Histology. Each site is excluded from federated training in turn and evaluated without fine-tuning at the round with the highest mean Dice over the three training sites. The experiment uses the first cross-validation split. Dice scores are expressed as percentages, and Avg. denotes the mean across held-out sites. The best results are shown in bold and the second best are underlined . A dagger denotes our conference version.
Setting
FSR
DCPA
S2PA
Dice
FedAvg
71.14 ± 1.10
Individual contribution
+ FSR
✓
72.55 ± 1.13
+ DCPA
✓
72.21 ± 0.43
+ S2PA
✓
73.81 ± 1.09
Cumulative
TABLE VI: Component ablation on Histology. Dice scores are expressed as percentages and reported as means and standard deviations over five folds. The upper block evaluates each module added individually to FedAvg. The lower block progressively combines the modules to form FedBCS+. The best results are shown in bold and the second best are underlined .
Alignment target
Dice
Semantic prototype only
70.87
Structural prototype only
70.30
Coupled prototype (CDPA † )
71.50
Joint (DCPA)
72.23
TABLE VII: Fine-grained analysis of DCPA on Histology. The alignment module is evaluated on a single cross-validation split, with Dice scores expressed as percentages. We compare semantic-only, structural-only, and joint alignment with the coupled CDPA module from the conference version, marked by a dagger. The best result is shown in bold .
Fig. 6 : Hyperparameter sensitivity on Histology. Panels (a)–(c) show the effects of the DCPA contrastive temperature τ , the DCPA loss weight λDCPA , and the S2PA aggregation momentum βema , respectively. Each parameter is varied in a single cross-validation run while the remaining parameters are fixed at their default values.
Fig. 7 : Convergence and cross-client consistency on Prostate. (a) Dice scores across communication rounds in a single cross-validation run. Shaded bands indicate round-to-round fluctuations, and dashed lines extend runs that stopped early at their final value. FedBCS+ reaches a peak Dice score of 83.8% and remains the highest curve from round 142 onward. (b) Cross-client standard deviation of the per-client mean Dice scores in Table III for all compared methods. Lower values indicate more consistent performance. FedBCS+ reduces the standard deviation from 4.62 for FedAvg to 3.52; only PathFL and HarmoFL obtain lower values, both with lower mean Dice.
Fig. 8 : Frequency-domain style separation. (a) Exchanging amplitude spectra between images from two Prostate clients transfers appearance while preserving anatomy. Each column retains the phase of its original image after the exchange. (b) Across encoder stages on Prostate and Histology, domain classification accuracy reaches 86–99% using amplitude spectra but remains near chance using phase spectra. (c) On Histology, with a separate model trained for each domain, amplitude divergence consistently exceeds phase divergence throughout the encoder. The left axis shows spectral divergence, and the right axis shows divergence of batch normalization statistics. These results demonstrate persistent style discrepancies in intermediate features.
Fig. 9 : Prototype entanglement on Polyp. Prototypes are computed from fused decoder features. The upper panel shows that center pixels are approximately 14 times as numerous as boundary pixels, causing a shared foreground prototype to be dominated by center features. The lower panel compares mean cosine similarities between each prototype and boundary or center features. Structural prototypes align more closely with boundary features, while semantic prototypes align more closely with center features, motivating separate alignment.
Department of Computational and Data Sciences, Indian Institute of Science Bengaluru, Karnataka 560012, India · Department of IISc Mathematics Initiative, Indian Institute of Science Bengaluru, Karnataka 560012, India
Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway. · Central Staff, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway.