Organizations: Department of Computer Science and Business Systems Gargi Memorial Institute of Technology Affiliated to Maulana Abul Kalam Azad University of Technology Balarampur, Mouza Beralia, Baruipur, Kolkata 700144, West Bengal, India · Engineering Sciences, Homi Bhabha National Institute Anushaktinagar, Mumbai, Maharashtra 400094, India · Computer and Informatics Group / Radioactive Ion Beam Facilities Group Variable Energy Cyclotron Centre 1/AF, Bidhannagar, Kolkata 700064, West Bengal, India
Talking-head generation has advanced rapidly with diffusion-based generative models, but training usually depends on centralized face-video and speech datasets, raising major privacy concerns. The problem is more acute for personalized talking-head generation, where identity-specific data are highly sensitive and often cannot be pooled across users or devices. PrivFedTalk is presented as a privacy-aware federated framework for personalized talking-head generation that combines conditional latent diffusion with parameter-efficient identity adaptation. A shared diffusion backbone is trained across clients, while each client learns lightweight LoRA identity adapters from local private audio-visual data, avoiding raw data sharing and reducing communication cost. To address heterogeneous client distributions, Identity-Stable Federated Aggregation (ISFA) weights client updates using privacy-safe scalar reliability signals computed from on-device identity consistency and temporal stability estimates. Temporal-Denoising Consistency (TDC) regularization is introduced to reduce inter-frame drift, flicker, and identity drift during federated denoising. To limit update-side privacy risk, secure aggregation and client-level differential privacy are applied to adapter updates. The implementation supports both low-memory GPU execution and multi-GPU client-parallel training on heterogeneous shared hardware. Comparative experiments on the present setup across multiple training and aggregation conditions with PrivFedTalk, FedAvg, and FedProx show stable federated optimization and successful end-to-end training and evaluation under constrained resources. The results support the feasibility of privacy-aware personalized talking-head training in federated environments, while suggesting that stronger component-wise, privacy-utility, and qualitative claims need further standardized evaluation.
Figures & tables
Figure 1: System-level overview of PrivFedTalk. The server maintains the shared diffusion backbone θ and the current global adapter state ϕ , while each participating client performs private local adaptation using lightweight LoRA modules. Only protected adapter updates Δ~ϕ and privacy-safe scalar reliability signals (s,n) are returned to the server, which applies identity-stable federated aggregation (ISFA) to produce the next global adapter.
Figure 2: Practical execution pipeline of the PrivFedTalk implementation. Configuration files specify model, federation, privacy, and runtime settings. The launcher initialises the training job; the server handles client sampling, logging, and ISFA-based aggregation. Selected clients are dispatched to GPU workers for adapter-only local training, with additional clients handled in subsequent waves. Client-side processing covers private-data loading, local objective computation, gradient clipping, differential privacy, and secure aggregation. The aggregated state is fed back to the server and written to checkpoints for personalised inference via the shared backbone and a client-specific adapter.
Figure 3: Validation curves over 100 federated communication rounds for PrivFedTalk. The three plots show validation loss (left), identity similarity (centre), and temporal stability (right). The best checkpoint is observed at round 97, just before convergence, reflecting typical stochastic behaviour under heterogeneous client participation.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
FID ↓
ID Similarity ↑
Temporal Jitter ↓
PrivFedTalk
7.3395
0.009683
1.3042
391.3217
0.018287
0.199548
FedAvg [ 9 ]
7.3395
0.009683
1.3042
391.3345
0.018285
0.199548
FedProx [ 10 ]
7.3395
0.009683
1.3042
391.3342
0.018289
0.199548
Table 1: Comparison using the best validation-loss checkpoint from each finished 2-GPU 1000-round run. The FedProx row reflects the repository’s current 2-GPU FedProx-config path. The internal sync-proxy metric is omitted from the main table because it is an implementation proxy rather than a standard SyncNet/LSE-style evaluation metric.
Variant
LPIPS ↓
FID ↓
ID ↑
TempJit ↓
SyncProxy ↑
Full PrivFedTalk
1.304183
391.324
0.01828779
0.199548637
0.083392685
Adapters-only
1.304184
391.322
0.01828694
0.199548656
0.083392585
DP-only
1.304182
391.320
0.01828744
0.199548647
0.083392678
TDC-only
1.304185
391.335
0.01828647
0.199548647
0.083392686
ISFA-only
1.304185
391.328
0.01828745
0.199548660
0.083392598
Table 2: Evaluation-level ablation comparison across the currently available PrivFedTalk variants.
In cross-silo federated learning (FL), sensitive text datasets remain confined to local organizations due to privacy regulations, making repeated training for each downstream task both communication-intensive and privacy-demanding. A promising alternative is to generate differentially private (DP) synthetic datasets that approximate the global distribution and can be reused across tasks. However, pretrained large language models (LLMs) often fail under domain shift, and federated finetuning is hindered by computational heterogeneity: only resource-rich clients can update the model, while weaker clients are excluded, amplifying data skew and the adverse effects of DP noise. We propose a flexible participation framework that adapts to client capacities. Strong clients perform DP federated finetuning, while weak clients contribute through a lightweight DP voting mechanism that refines synthetic text. To ensure the synthetic data mirrors the global dataset, we apply control codes (e.g., labels, topics, metadata) that represent each client's data proportions and constrain voting to semantically coherent subsets. This two-phase approach requires only a single round of communication for weak clients and integrates contributions from all participants. Experiments show that our framework improves distribution alignment and downstream robustness under DP and heterogeneity.
With the rapid advancement of diffusion models, talking face generation has made remarkable progress. However, existing diffusion-based methods still require task-specific fine-tuning and large-scale audiovisual datasets, resulting in high computational costs that hinder scalability and accessibility of diffusion-based approaches across the research community. To address this, we propose a finetuning-free paradigm that directly performs talking face generation using the pretrained weights of Stable Diffusion and IP-Adapter. This backbone leverages the visual embedding capability of IP-Adapter to mine lip-related semantics from the pretrained Stable Diffusion. To address the challenges of identity drift, synchronization errors, and temporal instability, we also design three trainable-parameterfree components: (1) the Structurist, which explicitly disentangles and reassembles lip and appearance features to mitigate identity drift and appearance distortion; (2) the Structure Controller, which adaptively refines embeddings based on quasi-monotonic motion trends for precise lip synchronization; and (3) the Noise Sensor, which introduces Gaussian prior to detect and suppress flicker and jitter artifacts and enhance temporal consistency. Experimental results show that our method outperforms existing SOTA approaches in both lip-sync accuracy (at least 0.16 gain in PCLD) and visual fidelity (at least 0.7 improvement in FID), establishing a novel fine-tuning-free diffusion framework for talking face generation.
Hao Wu, Xiangyang Luo, Hao Wang +3
Information Engineering University · Huai’an University · Chongqing University of Post and Telecommunications +1
Record-level differential privacy exposes a structural misalignment in personalized federated learning when client-specific variation is low-dimensional while training repeatedly releases high-dimensional updates. In this paper, we address this misalignment by releasing a private client context once and confining repeated adaptation to a fixed coefficient space. Beyond dimensionality reduction, the factorized generator induces an adaptive optimization geometry that reshapes noisy updates, and controlled ablations show that most of its private-training gain is retained by radial evolution. To further reduce the communication cost, we realize the Gaussian mechanism for coefficient updates directly through variable-length quantization with finite expected code length, so that the quantization error itself serves as the required privacy perturbation rather than extra distortion. Across MNIST and CIFAR-10, our design matches or outperforms full-model private adaptation across privacy budgets and client heterogeneity, while reducing protected uplink by a factor of 2.67 at ε=16 on CIFAR-10 with comparable future-client accuracy.
Yilin Xu, Chun Hei Michael Shiu, Chih Wei Ling +1
Department of Computer Science, City University of Hong Kong · Department of Electrical and Computer Engineering, University of British Columbia · School of Computer Science and Engineering, Hebrew University of Jerusalem