Privacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication

Idris Zakariyya; Pai Chet Ng; Kaushik Bhargav Sivangi; S. Mohammad Sheikholeslami; Konstantinos N. Plataniotis; Fani Deligianni

cs.CVMar 22, 2026

Privacy-Preserving Federated Action Recognition via Differentially Private Selective Tuning and Efficient Communication

Authors: Idris Zakariyya, Pai Chet Ng, Kaushik Bhargav Sivangi, S. Mohammad Sheikholeslami, Konstantinos N. Plataniotis, Fani Deligianni

Organizations: School of Computing Science, University of Glasgow, G12 8QQ, Glasgow, UK · Infocomm Technology Cluster, Singapore Institute of Technology, Singapore · The Edward S. Rogers Sr. Dept. of Electrical and Computer Engineering, University of Toronto, Canada

Abstract

Federated video action recognition enables collaborative model training without sharing raw video data, yet remains vulnerable to two key challenges: model exposure and communication overhead. Gradients exchanged between clients and the server can leak private motion patterns, while full-model synchronization of high-dimensional video networks causes significant bandwidth and communication costs. To address these issues, we propose Federated Differential Privacy with Selective Tuning and Efficient Communication for Action Recognition, namely FedDP-STECAR. Our FedDP-STECAR framework selectively fine-tunes and perturbs only a small subset of task-relevant layers under Differential Privacy (DP), reducing the surface of information leakage while preserving temporal coherence in video features. By transmitting only the tuned layers during aggregation, communication traffic is reduced by over 99% compared to full-model updates. Experiments on the UCF-101 dataset using the MViT-B-16x4 transformer show that FedDP-STECAR achieves up to 70.2% higher accuracy under strict privacy ( $ε=0.65$ ) in centralized settings and 48% faster training with 73.1% accuracy in federated setups, enabling scalable and privacy-preserving video action recognition. Code available at https://github.com/izakariyya/mvit-federated-videodp

Figures & tables

Figure 1: Vulnerabilities in federated video learning. (a) Model exposure: Shared gradients may reveal sensitive motion motion or identity. (b) Communication overhead: Synchronizing large video models cause heavy traffic and latency.

\hat{g}\leftarrow\frac{1}{|\mathcal{B}_{m}|}\sum_{i}\frac{g_{i}}{\max(1,\|g_{i}\|_{2}/C)}+\mathcal{N}(0,\sigma^{2}C^{2}I)

Algorithm 1 Our FedDP-STECAR

Stage	Dim	Blk	Stride (t,h,w)
Patch Emb.	96	1	(2,4,4)
Stage1	192	2	(1,8,8) $\rightarrow$ (1,4,4)
Stage2	384	3	(1,4,4) $\rightarrow$ (1,2,2)
Stage3	384	8	(1,2,2)
Stage4	768	2	(1,2,2) $\rightarrow$ (1,1,1)
Norm	768	-	-

Table 1: MViT-B-16x4 Architecture summary. Patch embedding uses Conv3d

(3,96,3\times 7\times 7)

with positional encoding; selective tuning updates the final transformer stage and classification head.

Figure 2: Centralized DP video activity recognition performance across varying

\epsilon

(fixed

\delta=1e\text{-}4

and clipping norm

C=1.0

), using full fine-tuning, selective tuning, and integrated top-level sampling (without replacement). The baseline MViT-B-16x4 was pretrained on Kinetics-400 and fine-tuned on UCF101.

Model	Accuracy (%)	Runtime (min)	Traffic (MB)	Comm. Delay (s)
DP-Adam-FT	22.42	339.38	1456	174.72
DP-Adam-FTTLS	21.10	342.04	1456	174.72
DP-Adam-Sel	73.06	179.68	3.10	0.37
DP-Adam-SelTLS	72.58	179.64	3.10	0.37

Table 2: FedAvg on UCF101 at

\epsilon=1.33

(

\delta=10^{-4}

C=1.0

); MViT-B-16x4 pretrained on Kinetics-400.

Model	$\boldsymbol{\epsilon}$	Accuracy (%)	Runtime (min)
DP-Adam-FTFA	1.33	28.3	681.16
DP-Adam-FTFN	1.33	28.7	684.93
DP-Adam-SelFA	1.33	74.1	359.54
DP-Adam-SelFN	1.33	73.5	360.31

Table 3: FedAvg vs. FedNova at

\epsilon{=}1.33

(

\delta{=}10^{-4}

): selective tuning improves accuracy and halves runtime.

Clients	Rounds	$\boldsymbol{\epsilon}$	FT-FA (%)	Sel-FA (%)	FT-FN (%)	Sel-FN (%)
2	10	1.33	28.3	74.1	28.7	73.5
5	20	1.33	30.6	73.8	32.0	74.0
10	20	1.42	27.5	73.7	27.9	73.5
20	20	3.15	18.3	72.5	19.1	72.5

Table 4: Scalability across clients, rounds, and privacy budgets (

\delta{=}10^{-4}

C{=}1.0

Differentially Private Federated Learning

Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.

See all 60 papers

FedVideoMAE: Efficient Federated Video Moderation with Differential Privacy and Secure Aggregation
Dec 21, 2025Ziyuan Tao, Chuanzhi Xu, Sandaru Jayawardana +4Federated Learning Aggregation Video Content Moderation
School of Computing, Macquarie University, Sydney, Australia · School of Computer Science, The University of Sydney, Sydney, Australia · School of Electrical and Computer Engineering, The University of Sydney, Sydney, Australia
FedCVU: Federated Learning for Cross-View Video Understanding
Mar 23, 2026Shenghan Zhang, Run Ling, Ke Cao +2Federated Learning Video Understanding
Software College, Northeastern University, Shenyang, China · University of Science and Technology of China, Hefei, China · University of Chinese Academy of Sciences, Beijing, China +1
PrivEraserVerify: Efficient, Private, and Verifiable Federated Unlearning
Apr 14, 2026Parthaw Goswami, Md Khairul Islam, Ashfak YeafiFederated Unlearning Privacy-Preserving Federated Learning
Department of Electronics and Communication Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh · Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USA · Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna-9203, Bangladesh

Dec 21, 2025J/K move · Enter open · S save

FedVideoMAE: Efficient Federated Video Moderation with Differential Privacy and Secure Aggregation

Ziyuan Tao, Chuanzhi Xu, Sandaru Jayawardana, Adnan Mahmood+3

School of Computing, Macquarie University, Sydney, Australia · School of Computer Science, The University of Sydney, Sydney, Australia · School of Electrical and Computer Engineering, The University of Sydney, Sydney, Australia

Federated Learning Aggregation Video Content Moderation Video Representation Learning Privacy-Preserving ML Secure Aggregation Federated Learning

Open paper AskPDF arXiv

Short-form video moderation is increasingly pushed toward edge and privacy-sensitive settings, where users may intend videos for a limited audience, such as friends or private groups, but sending raw clips to a central server can broaden exposure, consume bandwidth, and add moderation latency. Federated learning can keep videos on device, but unprotected model updates may still leak information, and full-video backbones are expensive to communicate. We present FedVideoMAE, a privacy-preserving federated framework for violence detection that adapts a frozen VideoMAE backbone with lightweight LoRA and prompt parameters. Each training round combines self-supervised masked video reconstruction with client-side differential privacy and pairwise masked aggregation (SA) of adapter updates. Violence labels are held out from federation and used only for downstream evaluation, separating private representation learning from supervised assessment. On RWF-2000, exchanging 5,518,848 trainable parameters instead of the 156,371,328-parameter instantiated pretraining state gives a 28.3x model-state payload ratio. FedVideoMAE reaches 77.25% test accuracy without DP or SA, while accuracy under DP+SA remains in the 65.25-66.00% range. Transfer experiments on RLVS and binary UCF-Crime show similar behavior. These results characterize the privacy-utility trade-off for edge video moderation. Code is available at: https://github.com/zyt-599/FedVideoMAE

Abstract

Differentially Private Federated Learning

Explore similar work

FedVideoMAE: Efficient Federated Video Moderation with Differential Privacy and Secure Aggregation

FedCVU: Federated Learning for Cross-View Video Understanding

PrivEraserVerify: Efficient, Private, and Verifiable Federated Unlearning