FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains

Jolle Verhoog; Ali Burak Ünal; Holger Caesar

cs.CVOct 1, 2026

FedCKA: Representation-Guided Layer Personalization for Federated 3D Perception Across Driving Domains

Authors: Jolle Verhoog, Ali Burak Ünal, Holger Caesar

Organizations: faculty of Cognitive Robotics, Mechanical Engineering, TU Delft, Delft, Netherlands

Abstract

Robust perception in intelligent vehicles demands 3D object detectors that remain dependable under domain shifts, such as changes in time of day, location, or weather. However, due to costly annotation and rare shifts, some environments lack sufficient data to train a standalone detector. Federated learning offers a privacy-preserving framework for collaborative model training, enabling clients to benefit from shared learning across diverse environments. Yet, this framework traditionally relies on a single global consensus model, which struggles to perform across heterogeneous local data distributions. Local conditions are better captured by adapting a subset of the model, but many personalization approaches rely on predefined layer partitions or fixed personalization ratios, thereby limiting adaptation to client-specific divergence. To reduce this rigidity, we propose FedCKA, a Centered Kernel Alignment (CKA)-based strategy that dynamically handles the personalization-globalization trade-off. Specifically, FedCKA computes layer-wise feature similarities between local client models and the global consensus model during training. By converting layer-wise similarity scores into client-specific aggregation masks, FedCKA selectively shares representation-consistent layers. Evaluation on a unified multi-domain benchmark based on nuScenes shows that FedCKA outperforms established federated baselines, including FedBN, FedRep, and FedSelect, improving average NDS by 7 percentage points over the strongest baseline. The findings offer both a comparative benchmark and a promising direction for robust federated 3D perception across shifts in location, weather, and illumination. Code is available at https://github.com/j-verhoog/FedCKA.

Figures & tables

Panel of Figure 1 — Fig. 1 : nuScenes [ 8 ] samples showing clear versus rainy weather (columns) across Boston daytime and Singapore nighttime conditions (rows).

Fig. 2 : Proposed client-specific aggregation via CKA-based layer masking. At round $t$ , each client compares its locally updated model $\widetilde{C}_{i,t}$ with the previous global model $G_{t-1}$ using layer-wise CKA to generate a binary mask. Only unmasked layers are transmitted and aggregated by the server to form $G_{t}$ . $G_{t}$ is returned to the clients, where each client overwrites its unmasked layers while masked layers retain their local parameters to obtain $C_{i,t}$ .

Domain	A	B	C	D	E
Location	Boston	Boston	Sing.	Sing.	Sing.
Time	Day	Day	Day	Night	Night
Weather	Clear	Rain	Clear	Clear	Rain
Train scenes	265	125	226	71	13
Val scenes	53	24	58	12	3

TABLE I : Federated domain definitions and corresponding train/validation scene counts. “Sing.” denotes Singapore.

Method	Domain A	Domain B	Domain C	Domain D	Domain E	Avg.
Centralized	0.61 / 0.66 / 0.84	0.61 / 0.68 / 0.85	0.72 / 0.67 / 0.83	0.60 / 0.64 / 0.85	0.58 / 0.58 / 0.84	0.65 / 0.66 / 0.84
Own Domain Only	0.58 / 0.64 / 0.81	0.47 / 0.51 / 0.75	0.59 / 0.62 / 0.76	0.14 / 0.24 / 0.47	0.01 / 0.08 / 0.04	0.52 / 0.57 / 0.74
Globalization
FedAvg [ 2 ]	0.28 / 0.37 / 0.54	0.27 / 0.36 / 0.51	0.34 / 0.38 / 0.53	0.28 / 0.35 / 0.56	0.20 / 0.33 / 0.51	0.30 / 0.37 / 0.53
FedDyn [ 17 ]	0.38 / 0.48 / 0.60	0.34 / 0.46 / 0.59	0.42 / 0.48 / 0.60	0.32 / 0.43 / 0.63	0.25 / 0.35 / 0.63	0.38 / 0.47 / 0.60
PCGrad [ 19 , 20 ]	0.34 / 0.41 / 0.56	0.32 / 0.41 / 0.56	0.40 / 0.44 / 0.57	0.33 / 0.39 / 0.59	0.21 / 0.34 / 0.55	0.36 / 0.42 / 0.57

TABLE II: Comparison of FL methods across domains. Non-federated runs are in italics. Performance is reported as mAP / NDS / CAP (Car AP); Avg. denotes the validation-scene-weighted average. Within each family, the best-performing methods are bolded, and the second-best are underlined. For FedCKA, threshold-based masking uses

\tau=0.25

Fig. 3 : Qualitative comparison of 3D object detection performance under severe domain shift (Singapore night rain) in BEV view. On the left are the Ground-Truth bounding boxes. In the middle are the predictions from the baseline FedAvg model, illustrating missed detections. On the right, the predictions from the proposed FedCKA framework show successfully recovered detections.

TABLE III: Cross-domain generalization matrices for client models M A–E. Each matrix reports NDS performance across validation domains D A–E. The diagonal shows performance when the client is evaluated on its own local domain, while the off-diagonal entries show cross-domain performance.

Setting	NDS	Mask	Setting	NDS	Mask
$\rho=0.01$	0.551	1.2%	$\tau=0.05$	0.598	0.7%
$\rho=0.10$	0.489	10.7%	$\tau=0.25$	0.635	4.7%
$\rho=0.40$	0.516	41.3%	$\tau=0.50$	0.600	7.8%
			$\tau=0.75$	0.596	4.4%

TABLE IV : FedCKA ablations for percentage- and threshold-based masking in Eqs. 5 and 6 , reporting average NDS and final client personalization ratio. The two masking strategies are mutually exclusive in these experiments: percentage-based masking uses

\rho

, while threshold-based masking uses

\tau

Method	Avg. pers.	Mask metadata	Total upload payload
FedAvg	0.0%	–	25.00 GB
FedDyn	0.0%	–	25.00 GB
PCGrad	0.0%	–	25.00 GB
FedBN	0.2%	0.25 KB	24.95 GB
FedRep	6.0%	0.25 KB	23.50 GB
FedCKA	3.5%	0.25 KB	24.12 GB

TABLE V : Cumulative client-to-server upload over

40

federated rounds with

5

clients and a

125

MB base model. Mask metadata is reported per active masking round across all clients; total payload is cumulative over training. Avg. pers. is the client- and round-averaged fraction withheld from aggregation; thus, the

40\%

target used by FedSelect and FedMC corresponds to

20\%

on average. Raw-data centralization is included for reference.

Method	NDS	CAP
Centralized	$0.658\pm 0.014$	$0.830\pm 0.014$
Own Domain Only	$0.567\pm 0.004$	$0.739\pm 0.002$
FedMC	$0.559\pm 0.006$	$0.744\pm 0.003$
FedSelect	$0.545\pm 0.004$	$0.736\pm 0.008$
FedCKA	$0.635\pm 0.008$	$0.810\pm 0.014$

TABLE VI : Comparison of non-federated baselines and personalized federated learning methods. Results are reported as mean

\pm

standard deviation over three runs.

Explore similar work

Apr 17, 2026cs.CV

Fed3D: Federated 3D Object Detection

3D object detection models trained in one server plays an important role in autonomous driving, robotics manipulation, and augmented reality scenarios. However, most existing methods face severe privacy concern when deployed on a multi-robot perception network to explore large-scale 3D scene. Meanwhile, it is highly challenging to employ conventional federated learning methods on 3D object detection scenes, due to the 3D data heterogeneity and limited communication bandwidth. In this paper, we take the first attempt to propose a novel Federated 3D object detection framework (i.e., Fed3D), to enable distributed learning for 3D object detection with privacy preservation. Specifically, considering the irregular input 3D object in local robot and various category distribution between robots could cause local heterogeneity and global heterogeneity, respectively. We then propose a local-global class-aware loss for the 3D data heterogeneity issue, which could balance gradient back-propagation rate of different 3D categories from local and global aspects. To reduce communication cost on each round, we develop a federated 3D prompt module, which could only learn and communicate the prompts with few learnable parameters. To the end, several extensive experiments on federated 3D object detection show that our Fed3D model significantly outperforms state-of-the-art algorithms with lower communication cost when providing the limited local training data.

Suyan Dai, Chenxi Liu, Fazeng Li +1

School of Automation Science and Enginnering, South China University of Technology, Guangzhou 510641, P.R.China. · State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, 110169, China.

3D Object Detection Federated Learning 3D Generation Heterogeneity

Ask PDF

Jun 30, 2026cs.GR

Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

Deploying 3D point cloud analysis in privacy-sensitive, resource-constrained settings faces two barriers: data cannot be centralized, and models must run on limited edge hardware. We present a multi-seed benchmark jointly evaluating federated learning (FL) and knowledge distillation (KD) for 3D point cloud classification. It spans 13 FL algorithms and 10 KD objectives (a 130-pair cross-product) across 504 training runs, evaluated on ModelNet40 and a clinical craniosynostosis dataset. We report three findings. First, under extreme non-IID label skew, standalone FL degrades sharply: on ModelNet40, the strongest method reaches 76.32% against a 92.26% centralized reference; on clinical data, the best reaches 75.83% against 100%. Second, distillation successfully compresses the teacher into a student 74.51% smaller and roughly twice as fast at inference, often matching or surpassing the teacher. Third, the combined pipeline exposes an evaluation pitfall: when distillation keeps a hard-label cross-entropy term on a labeled proxy split, a collapsed federated teacher (8.50%) paired with Logit-MSE still yields a 92.94% student. This 84.4-point gap reflects the proxy labels rather than the federated model, reusing the very labels whose privacy motivated federation. Objectives without hard labels instead track teacher quality (

r \approx 0.99

) and collapse when the teacher does. We therefore recommend evaluating FL-KD pipelines with label-free distillation so reported accuracy reflects the federated teacher, not the proxy.

Aizierjiang Aiersilan

University of Macau

Federated Learning Point Clouds Knowledge Distillation

Ask PDF

Jun 10, 2026cs.CV

Understanding Cross-Sensor Feature Variations for Generalizable 3D Perception

Radar-camera BEV perception often suffers from degraded performance when evaluated across datasets, as changes in driving scenes, sensor configurations, and environmental conditions can alter both the input observations and the internal fused representations. This work studies this issue from the perspective of source-domain variation modeling, aiming to improve the robustness of BEV-based 3D detectors without relying on target-domain samples. We introduce a framework that characterizes visual scene variations in the frequency domain and uses them to synthesize diverse source-domain views. By comparing the resulting fused BEV representations, the framework further captures how image-level variations influence multi-modal BEV features. These variation patterns are then used to regularize the detector, encouraging the learned fusion space to remain stable under latent scene changes. The proposed method is applied only during training and leaves the inference pipeline unchanged. Experiments on cross-dataset radar-camera 3D detection between View-of-Delft and TJ4DRadSet demonstrate consistent improvements over multiple BEV fusion backbones, and the gains remain effective when a small amount of target-domain data is available.

Xin Qiu, Wenjie Liu, Fuyuan Ai +3

Zhejiang University

3D Object Detection Multi-Sensor Fusion Bird'S-Eye View Cross-View Fusion Autonomous Driving Perception Perception

Ask PDF