FedDyMem: Efficient Federated Learning with Dynamic Memory and Memory-Reduce for Unsupervised Image Anomaly Detection
Authors: Silin Chen, Andy Liu, Kangjian Di, Yichu Xu, Han-Jia Ye, Wenhan Luo, Ningmu Zou
Organizations: School of Integrated Circuits, Nanjing University, Suzhou, 215163, Jiangsu, China. · National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210023, Jiangsu, China. · School of Artificial Intelligence, Nanjing University, Nanjing, 210023, Jiangsu, China. · Hong Kong University of Science and Technology, Hong Kong. · Interdisciplinary Research Center for Future Intelligent Chips (Chip-X), Nanjing University, Suzhou, 215163, Jiangsu, China.
Unsupervised image anomaly detection (UAD) has become a critical process in industrial and medical applications, where models are trained on normal samples without requiring labeled anomalies. However, even normal images may contain sensitive product designs, manufacturing information, or institution-specific population characteristics, making centralized data collection difficult in practical deployments. Federated UAD provides a promising decentralized solution, but it remains challenging because clients use only normal training samples, their feature distributions remain highly heterogeneous within the same task, and communication should be efficient while limiting the exposure of sensitive client information. To address these challenges, we propose FedDyMem, an efficient federated learning method with dynamic memory and memory-reduce for unsupervised image anomaly detection. Instead of exchanging full model parameters, FedDyMem facilitates knowledge sharing between the clients and server through reduced dynamic memory banks. In the local clients, a memory generator and a metric loss are employed to improve the consistency of the feature distribution for normal samples, leveraging the local model to update the memory bank dynamically. For efficient communication and data privacy, a memory-reduce method based on weighted averages is proposed to significantly decrease the scale of memory banks. This representation reduces communication overhead and empirically lowers reconstruction fidelity under inversion attacks. On the server, global memory is constructed and distributed to individual clients through k-means aggregation. Experiments on six federated UAD tasks constructed from eleven public industrial and medical datasets show that FedDyMem consistently improves anomaly detection and localization while reducing communication cost.
Figures & tables
Figure 1: Illustration of federated learning in UAD. As shown on the left, for the same inspection or screening task, an intra-class distribution bias exists between local models, caused by clients possessing different and incomplete subsets of normal data within the same inspection or screening task. As shown in the top right, federated UAD aims to construct a shared normal distribution for one-class anomaly detection across clients. Examples of type differences in anomaly detection are shown in the bottom right.
Figure 2: Overview of our proposed FedDyMem framework.
Figure 3: Illustration of the memory feature generation process in each client model.
Figure 4: Illustration of Memory-reduce.
Task
Partitioning attribute
#Groups
N
α
Brain
Data source
3
5
0.1
X-ray
Data source
3
5
0.1
PCB
PCB product type
6
5
0.1
Texture
Texture class
5
5
0.1
MPDD
Metal-part type
6
5
0.1
Wafer
Wafer product type
14
5
0.1
Table 1: Default non-IID client partition used in the main experiments.
Figure 5: Local metric loss curves of the first client over 200 communication rounds on four federated UAD tasks.
Figure 6: Illustration of I-AUROC over communication rounds on six federated UAD tasks.
Figure 7: Comparison of feature distributions on the texture task after 200 communication rounds. (a) t-SNE ( Maaten and Hinton (2008) ) visualization of local and global memory banks generated by five clients using FedAvg (top row) and FedDyMem (bottom row). (b) Comparison of global memory banks aggregated from FedDyMem and FedAvg clients, together with the corresponding I-AUROC results.
Methods
Communication
Wafer
PCB
Texture
MPDD
Brain
X-ray
Local
-
57.35( ± 0.71)
51.89( ± 0.76)
75.94( ± 0.39)
54.39( ± 0.77)
50.66( ± 0.87)
61.02( ± 0.58)
FedAvg
Model Parameters: ≈ 10.66M, (SCAFFOLD) ≈ 10.66M × 2
63.91( ± 0.36)
54.18( ± 0.33)
80.32( ± 0.81)
61.87( ± 0.76)
50.16( ± 0.63)
61.48( ± 0.56)
FedProx
63.98( ± 0.51)
54.11( ± 0.48)
80.22( ± 0.78)
61.83( ± 0.61)
50.47( ± 0.34)
61.45( ± 0.66)
FedPHP
63.94( ± 0.56)
54.13( ± 0.36)
80.19( ± 0.62)
61.80( ± 0.63)
50.38( ± 0.39)
61.45( ± 0.37)
APPLE
65.92( ± 0.33)
56.79( ± 0.65)
82.43( ± 0.51)
61.03( ± 0.77)
50.82( ± 0.45)
68.47( ± 0.51)
FedBN
63.89( ± 0.35)
54.09( ± 0.79)
80.13( ± 0.77)
61.90( ± 0.61)
50.04( ± 0.54)
61.45( ± 0.72)
Table 2: Comparison of Image-AUROC (%) results on six federated UAD tasks under the default non-IID protocol.
Methods
Pixel-AUROC (%)
PRO (%)
Wafer
PCB
Texture
MPDD
Wafer
PCB
Texture
MPDD
Local
58.27( ± 1.20)
68.12( ± 1.11)
72.63( ± 0.98)
87.02( ± 1.05)
47.76( ± 0.52)
45.86( ± 1.31)
56.97( ± 0.48)
51.31( ± 0.73)
FedAvg
66.56( ± 0.70)
69.30( ± 0.34)
76.16( ± 0.73)
91.67( ± 0.94)
67.00( ± 0.30)
54.04( ± 0.57)
53.35( ± 1.49)
55.07( ± 1.25)
FedProx
66.41( ± 0.74)
69.10( ± 1.28)
76.08( ± 1.28)
91.59( ± 1.02)
67.25( ± 0.33)
54.04( ± 1.28)
53.62( ± 0.70)
54.96( ± 1.16)
FedPHP
66.44( ± 1.38)
69.11( ± 0.42)
76.03( ± 0.57)
91.56( ± 0.73)
67.16( ± 0.38)
54.23( ± 1.16)
53.37( ± 0.47)
54.86( ± 1.42)
APPLE
69.88( ± 0.95)
71.34( ± 0.85)
78.06( ± 1.08)
92.12( ± 1.19)
69.02( ± 0.42)
60.11( ± 1.45)
54.14( ± 1.16)
55.84( ± 0.76)
Table 3: Comparison of Pixel-AUROC (%) and PRO (%) results on four federated UAD tasks under the default non-IID protocol.
Methods
Setting
Wafer
PCB
Texture
MPDD
Brain
X-ray
All normal training data are jointly accessible.
PatchCore
Centralized
75.56/77.92/71.80
91.25/96.12/77.97
99.04/97.46/88.76
85.00/98.88/71.31
95.69/-/-
78.10/-/-
SimpleNet
Centralized
69.92/72.16/70.51
84.35/96.17/70.47
99.32/96.43/85.59
86.91/98.92/74.65
92.79/-/-
77.96/-/-
UniAD
Centralized
74.05/83.23/76.56
83.79/96.74/79.59
98.92/94.90/89.63
69.28/87.65/70.06
87.51/-/-
84.89 /-/-
MambaAD
Centralized
76.34/84.69/84.67
94.59/ 99.11 / 91.52
98.41/97.99/88.15
88.11/96.74/90.58
97.73/-/-
82.67/-/-
InpFormer
Centralized
81.85 / 85.02 / 85.05
94.74 /98.87/88.36
99.82 / 98.67 / 95.56
91.09 / 99.55 / 94.56
98.73 /-/-
83.44/-/-
Table 4: Comparison with representative UAD methods under centralized and federated settings. Results are reported as the mean I-AUROC/P-AUROC/PRO (%).
Methods
Setting
Wafer ( N=14 )
Brain ( N=3 )
UniAD
FedAvg
67.48/76.06/61.75
74.81/-/-
MambaAD
FedAvg
65.91/69.32/55.66
86.42/-/-
InpFormer
FedAvg
64.15/71.16/53.45
83.31/-/-
Ours
FedDyMem
70.20 / 77.62 / 73.37
88.24 /-/-
Table 5: Comparison under the group-disjoint federated setting. Results are reported as the mean I-AUROC/P-AUROC/PRO (%).
Figure 8: Impact of different local epochs on clients.
Ratio
Methods
Brain
MPDD
Texture
20%
FedAvg
50.06/-/-
58.65/90.72/54.22
77.18/72.64/50.53
Ours
85.05/-/-
79.60/97.97/75.68
98.34/96.92/81.82
40%
FedAvg
50.28/-/-
58.27/90.41/54.11
78.50/74.16/50.20
Ours
86.49/-/-
81.23/97.97/77.54
99.09/96.89/82.53
60%
FedAvg
50.08/-/-
60.62/91.01/54.20
78.44/74.14/51.75
Ours
87.59/-/-
82.96/97.94/78.91
99.09/96.86/83.47
Table 6: Impact of per-round local training sample ratio. Results are reported as the mean I-AUROC/P-AUROC/PRO (%).
rEMA
DW
Brain
MPDD
Texture
-
-
84.47/-/-
78.63/95.72/74.17
97.91/95.32/83.13
✓
-
86.75/-/-
80.48/98.88/74.85
98.66/96.88/85.67
-
✓
87.62/-/-
81.88/97.85/77.73
98.99/96.99/85.88
✓
✓
89.27 /-/-
83.93 / 98.97 / 79.43
99.24 / 97.04 / 86.81
Table 7: Ablation study of rEMA and Dynamic Weighting (DW) in the memory-reduce.
Figure 9: Impact of parameter initialization.
Factor
Setting
PCB
Brain
Hg×Wg
2×2
79.02/93.38/78.89
86.27/-/-
4×4
82.24/95.74/80.96
88.73/-/-
8×8
83.34/ 96.03 / 81.09
89.27 /-/-
16×16
83.36 /95.87/80.99
88.93/-/-
knn
1
78.84/93.99/78.31
86.15/-/-
3
83.34 / 96.03 / 81.09
89.27 /-/-
Table 8: Impact of hyperparameters and backbones. Results are reported as I-AUROC/P-AUROC/PRO (%).
Configuration
N
Tˉagg
Texture
Average Aggregation
5
0.0241
92.76/96.10/84.42
Kclu=Q,Iclu=5
5
0.2373
97.94/95.77/85.12
Kclu=Q,Iclu=10
5
0.2885
99.09/97.07/86.01
Kclu=Q,Iclu=15
5
0.3429
99.24/97.04/ 86.81
Kclu=Q,Iclu=20
5
0.3987
99.22/97.05/86.81
Kclu=Q,Iclu=40
5
0.5375
99.27/97.03/86.77
Table 9: Effectiveness and scalability of K -means. Results are reported as the mean per-round server aggregation time Tˉagg (s) measured over 200 communication rounds and I-AUROC/P-AUROC/PRO (%).
+Proj
+MG
+KM
I-AUROC (%)
P-AUROC (%)
Wafer
PCB
Texture
MPDD
Brain
X-ray
Wafer
PCB
Texture
MPDD
-
-
-
67.69
69.58
85.86
70.41
69.03
68.92
70.23
83.23
85.22
94.19
✓
-
-
68.93
74.10
87.27
78.28
74.95
72.71
74.29
89.26
90.84
95.77
✓
✓
-
70.96
81.26
92.76
81.05
85.94
78.12
77.01
94.97
96.10
97.96
✓
✓
✓
71.96
83.34
99.24
83.93
89.27
80.30
78.40
96.03
97.04
98.97
Table 10: Ablation study of structure for FedDyMem.
Ratio
Methods
Brain
MPDD
Texture
0%
FedAvg
50.16/-/-
61.87/91.67/55.07
80.32/76.16/53.35
Ours
89.27/-/-
83.93/98.97/79.43
99.24/97.04/86.81
1%
FedAvg
50.14/-/-
59.43/89.91/54.91
77.97/73.93/49.74
Ours
88.90/-/-
83.86/98.76/79.40
98.97/96.95/85.85
5%
FedAvg
50.12/-/-
58.49/88.07/53.24
72.96/68.34/47.59
Ours
88.50/-/-
83.52/98.47/79.55
98.35/96.87/85.44
Table 11: Robustness to contaminated local training data. Ratio denotes the proportion of contaminated samples included in each client’s local training set. Results are reported as the mean I-AUROC/P-AUROC/PRO (%).
Figure 10: Robustness to dynamic client participation.
Dataset
Local-only (w/o B(T) )
Local Adaptation (w/ B(T) )
Wafer
57.35/58.27/47.76
71.66/75.96/73.11
PCB
51.89/68.12/45.86
82.88/95.67/79.96
Texture
75.94/72.63/56.97
99.11/96.80/85.91
MPDD
54.39/87.02/51.31
81.83/98.92/78.36
Brain
50.66/-/-
87.77/-/-
X-ray
61.02/-/-
79.71/-/-
Table 12: Local adaptation for a newly arriving client after federated training. Results are reported as the mean I-AUROC/P-AUROC/PRO (%).
Method
N
αDir
Texture
MPDD
FedAvg
5
0.01
61.22/56.09/52.94
55.36/86.95/50.45
Ours
5
0.01
99.12/96.85/85.30
83.00/98.90/79.15
FedAvg
5
0.5
89.42/79.24/63.50
63.07/91.31/67.34
Ours
5
0.5
99.14/96.85/85.48
84.80/98.99/79.59
FedAvg
5
1.0
97.26/86.67/77.35
63.81/92.37/70.89
Ours
5
1.0
99.13/96.87/87.07
84.58/98.96/79.81
Table 13: Performance under different client counts N and Dirichlet partition parameters αDir .
Injected carpet ratio ρ (# samples)
Carpet
Average
5% (14)
96.11/93.50/80.61
97.86/94.75/81.42
10% (28)
96.91/93.76/80.80
97.87/94.99/81.23
20% (56)
97.47/96.15/81.54
98.24/95.10/83.01
50% (140)
98.79/97.42/86.01
98.52/95.28/84.53
100% (279)
99.10 / 98.42 / 86.71
99.03 / 96.96 / 86.73
Table 14: Robustness to a rare normal mode. Results are reported as I-AUROC/P-AUROC/PRO (%).
Figure 11: Qualitative comparison of FedAvg and our proposed method. The results were obtained from a randomly selected client using a global memory bank.
Figure 12: Qualitative reconstruction results before and after applying the proposed memory-reduce operation on Texture, PCB, and Brain.
Attack
Dataset
+Memory-
SSIM ↓
PSNR ↓
FID ↑
Methods
Reduce
Deconvolution Inverting Method
Texture
-
0.6350
25.9003
246.9916
✓
0.2088
17.4787
415.9250
PCB
-
0.5213
25.8625
281.5855
✓
0.3379
12.1827
358.7053
Brain
-
0.2273
11.0985
300.5543
Table 15: Quantitative reconstruction attack results before and after memory-reduce.
Method
Server-visible Message
ROC-AUC ↓
TPR@1% FPR ↓
Advantage ↓
FedAvg
Δθn(T)
0.6024±0.0206
0.0619±0.0104
0.1687±0.0420
Raw-memory Sharing
{Mn,i(T)}i
0.7537±0.0403
0.3936±0.0125
0.6583±0.0428
FedDyMem
Rn(T)
0.5310±0.0223
0.0288±0.0122
0.0960±0.0390
Table 16: Membership inference on the texture task under the passive honest-but-curious server threat model. Values are reported as mean ±95% normal-approximation confidence intervals over five held-out federations.
Method
Server-visible Message
Client Domain Inference
Client Property Inference
Balanced Accuracy(%) ↓
Macro F1(%) ↓
AUROC ↓
Balanced Accuracy(%) ↓
F1 Score (%) ↓
FedAvg
Δθn(T)
66.8±5.9
64.5±6.2
0.840±0.050
76.1±5.1
75.0±5.5
Raw-memory Sharing
{Mn,i(T)}i
94.7±2.7
94.1±2.9
0.985±0.010
95.2±2.5
95.0±2.7
FedDyMem
Rn(T)
54.9±6.4
51.7±7.0
0.720±0.060
65.7±6.2
64.9±6.5
Table 17: Client domain and property inference on the texture task under the passive honest but curious server threat model. Values are reported as mean ± standard deviation across five partition level attacker train and test splits constructed from 20 independently generated federation partitions.
Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad, Serbia · Department of Informatics, Modeling, Electronics, and Systems, University of Calabria, Italy
Digital Factory Vorarlberg GmbH, Dornbirn, Austria · University of Salzburg, Salzburg, Austria · Vorarlberg University of Applied Sciences, Dornbirn, Austria