RoMod: Temporal Routing Modulation via Mixture-of-Experts for Video Anomaly Detection

Chao Huang; Pengfei Wei; Benfeng Wang; Chengliang Liu; Wei Wang; Li Shen; Wenqi Ren; Xiaochun Cao

cs.CVOct 4, 2026

RoMod: Temporal Routing Modulation via Mixture-of-Experts for Video Anomaly Detection

Authors: Chao Huang, Pengfei Wei, Benfeng Wang, Chengliang Liu, Wei Wang, Li Shen, Wenqi Ren, Xiaochun Cao

Organizations: Sun Yat-sen University · University of Macau

Abstract

Intermediate-layer features from multimodal large language models have shown strong potential for video anomaly detection (VAD), yet the origin of their discriminative power remains unclear. We study this question using sparse mixture-of-experts (MoE) models, whose explicit expert structure and sparse activation make their internal computation easier to inspect. With a fully frozen backbone and no additional training, we find that anomaly-related evidence is concentrated in a small set of experts. These experts recur across layers, spontaneously specialize in different anomaly types, and together form a dynamic routing subnetwork. We further show that the output channels most strongly influenced by these experts are also the hidden dimensions that contain the most anomaly-relevant information. Routing statistics can therefore serve as an internal anomaly cue that complements semantic features.Based on these findings, we propose RoMod, an efficient VAD framework trained with only $5\%$ of weakly labeled videos. RoMod includes a Routing-Modulated Fusion module, RoMF, and a Routing-aware Temporal Network, RoTN. RoMF uses routing signals to adaptively recalibrate hidden semantic channels. Its design also prevents the routing branch from bypassing semantic features and making predictions on its own. RoTN captures the temporal evolution of anomalies from onset to persistence and termination. Experiments on three benchmarks show that RoMod achieves state-of-the-art performance while running substantially faster than dense backbones of comparable size.

Figures & tables

Figure 1: Observations on a frozen MoE backbone on XD-Violence test set. (a) Comparison of readout APs: native answer ( $71.37\%$ ), hidden probe ( $82.87\%$ ), and a single expert’s activation count ( $77.40\%$ ). (b) Layer-wise expert activation contrast between normal and abnormal videos (red: high, green: low frequency).

Figure 2: Sparse expert specialization and input-dependent routing. (a) Per-expert discriminability from activation counts alone, with no hidden features or trainable parameters. (b) Routing preference over L20–L39 relative to the global average, grouped by event type; cells are comparable horizontally across sub-panels.

Figure 3: Overall framework of RoMod. A single forward pass yields the semantic hidden state from the optimal layer $l^{\star}$ and holistic routing statistics across all $L$ layers; RoMF turns the routing vector into per-channel scales that recalibrate the centred hidden state, and RoTN emits clip scores incrementally under causal attention.

			UBnormal	UCF-Crime	XD-Violence
Method	Venue	Data	AUC (%)	AUC (%)	AUC (%)	AP (%)
UR-DMU [ Zhou et al., 2023 ]	AAAI’23	100%	–	86.97	94.02	81.66
VadCLIP [ Wu et al., 2024 ]	AAAI’24	100%	–	88.02	–	84.51
VERA [ Ye et al., 2025 ]	CVPR’25	100%	–	86.55	88.26	70.54
$\pi$ -VAD [ Majhi et al., 2025 ]	CVPR’25	100%	–	90.33	–	85.37
RefineVAD [ Lee et al., 2026 ]	AAAI’26	100%	–	88.92	–	88.66

Table 1: Comparison on three benchmarks. Data denotes the fraction of training videos used.

Table 5

Backbone	Type	Act. (B)	FLOPs (T)	RoMF (ms)	RoTN (ms)	Thpt. (FPS)	UCF-Crime AUC (%)	XD-Violence AP (%)
Qwen3.6-35B-A3B	Sparse	2.438	5.39	0.322	1.093	51.83	88.92	90.50
Qwen3VL-30B-A3B	Sparse	2.850	6.12	0.318	1.085	36.50	87.23	88.33
Gemma-4-26B-A4B	Sparse	3.684	10.37	0.325	0.839	39.56	86.79	87.24
Qwen3.8-27B	Dense	24.353	54.13	—	—	23.19	83.45	85.12

Table 4: Cost and streaming performance in online mode. Activated parameters (Act.) and FLOPs are measured per step; for the dense backbone Act. counts all non-embedding parameters.

Figure 4: Qualitative behaviour and layer-wise gains. (a) Frame-level scores of RoMod (blue) and Hidden-only (orange) on three videos; pink bands are ground-truth windows. (b, c) Layer-wise AP on XD-Violence and AUC on UCF-Crime.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Where	Meaning
Architecture and indices
$L$ , $l$	everywhere	number of MoE decoder layers; layer index. $L_{\mathrm{moe}}$ is used when comparing backbones with different depths.
$E$ , $e$	everywhere	number of routed experts per layer; expert index.
$K_{e}$	everywhere	number of experts activated per token (Top- $K_{e}$ gating), $K_{e}=8$ .
$D$ , $c$	everywhere	hidden width; hidden-channel index.
$T$ , $t$	everywhere	number of clips in a video; clip index.

Appendix

Table 5: Symbol conventions. Letters that carry more than one role are disambiguated by subscript or by decoration; the table records which role applies where.

Seed Index	UBnormal	UCF-Crime	XD-Violence
Seed Index	AUC (%)	AUC (%)	AUC (%)	AP (%)
Run 1 (Seed 42, default)	79.17	91.32	97.34	92.49
Run 2 (Seed 1024)	78.36	90.64	96.85	91.82
Run 3 (Seed 2024)	78.85	91.03	97.12	92.15
Run 4 (Seed 3407)	77.42	89.65	96.08	90.73
Run 5 (Seed 7777)	78.11	90.28	96.53	91.36

Appendix

Table 6: Multi-seed evaluation across

5

independent runs on three benchmarks using

5\%

training data. Seed 42 is the default split reported in the main paper; all other runs fall within a

\sim 1.8

-point range of it.

Figure 5: Statistical hypothesis testing of category-dependent routing specialization on XD-Violence. (a) Layer-averaged pairwise Jensen–Shannon divergence matrix across six anomaly types. Every pair yields $\hat{p}=0.0005$ under $2{,}000$ non-parametric permutation tests (marked ∗∗∗ ), which is the resolution bound of the test. (b) Directional Kullback–Leibler divergence matrix $\mathrm{KL}(\mathrm{Row}\parallel\mathrm{Col})$ , quantifying the asymmetric information loss when approximating one category’s routing preference by another.

Figure 6: Channel–expert alignment on the default backbone (Qwen3.6-35B-A3B, XD-Violence). (a) Channel-level regression scatter at Layer $40$ ( $D=2048$ ). Green rings mark the Top- $50$ high-discriminability channels, which form a wedge envelope concentrated at large expert driving contributions $\Phi_{c}$ ( $r=0.589$ , asymptotic $p<10^{-10}$ ). (b) Layer-wise alignment across all $40$ MoE layers: Pearson $r$ and Spearman $\rho$ (top), Top- $50$ channel overlap count and hypergeometric enrichment significance $-\log_{10}p_{\mathrm{enrich}}$ (bottom; the dash-dotted line marks $\alpha_{\mathrm{Bonf}}=1.25\times 10^{-3}$ ). Both metrics sit at chance level throughout L1–L18 and rise sharply from L19 onward.

Figure 7: Cross-architecture verification on Gemma-4-26B-A4B (XD-Violence). (a) Channel-level scatter at Layer $30$ ( $D=2816$ ). Despite fewer experts and a wider hidden state, the concentration of discriminative channels at large $\Phi_{c}$ replicates ( $r=0.259$ , asymptotic $p<10^{-10}$ ). (b) Layer-wise alignment across $30$ layers ( $\alpha_{\mathrm{Bonf}}=1.67\times 10^{-3}$ ), showing the same transition: near-zero alignment in shallow layers, monotonic escalation through middle-to-late depths, and $16/30$ layers reaching significance.

Backbone	$L_{\mathrm{moe}}$	$D$	$E$	$\bar{r}$ (all)	$\bar{r}$ (upper)	$r_{\max}$ (layer)	Sig. layers ( $p<\alpha_{\mathrm{Bonf}}$ )	Peak Top-50 overlap (vs. chance baseline)
Qwen3.6-35B-A3B	40	2048	256	0.230	0.410	0.589 (L40)	25 / 40	23 / 50 ( $19.0\times$ , $p\approx 10^{-26}$ )
Gemma-4-26B-A4B	30	2816	128	0.107	0.196	0.326 (L28)	16 / 30	12 / 50 ( $13.5\times$ , $p\approx 10^{-11}$ )

Appendix

Table 7: Cross-architecture summary of channel–expert alignment on XD-Violence.

\bar{r}

(all) is the mean Pearson correlation over all MoE layers;

\bar{r}

(upper) is the mean over the upper half of the network. Significant layers satisfy

p_{\mathrm{enrich}}<\alpha_{\mathrm{Bonf}}=0.05/L_{\mathrm{moe}}

with

r>0

. The chance overlap baseline is

\mathbb{E}[X]=K_{\mathrm{ch}}^{2}/D

with

K_{\mathrm{ch}}=50

Intervention	$K_{\mathrm{ko}}$ /layer	Masked	AP (%)	AUC (%)
Unperturbed (frozen LM-head)	0	0	71.37	85.62
Mask Top- $K_{\mathrm{ko}}$	1	20	69.85 ( $-1.52$ )	84.32 ( $-1.30$ )
	5	100	67.42 ( $-3.95$ )	81.65 ( $-3.97$ )
	10	200	65.58 ( $-5.79$ )	79.20 ( $-6.42$ )
	20	400	64.12 ( $-7.25$ )	76.85 ( $-8.77$ )
Mask control (rank $>20$ )	1	20	71.18 ( $-0.19$ )	85.50 ( $-0.12$ )

Appendix

Table 8: Causal intervention via layer-wise expert knockout on XD-Violence (layers 20–39,

97{,}396

clips). Scores are read from the frozen language head with no fitted parameters, so the reference row equals the native-answer readout of Table 14 . Masking the

K_{\mathrm{ko}}

most anomaly-preferring experts degrades the readout disproportionately to the removed capacity, whereas masking an identical number of control experts (rank

>20

) leaves it largely intact. Interventions are applied at all

20

middle-to-late layers simultaneously.

Figure 8: Complete 40-layer Fisher separability profiles on the frozen Qwen3.6-35B-A3B backbone.

J(l)

(Eq. 12 ) is computed from the terminal-token hidden states

\mathbf{h}_{N_{\mathrm{seq}}}^{(l)}

, with no training and no temporal context. Top: XD-Violence (

73{,}814

normal /

23{,}582

abnormal clips), peak

0.6842

at L28,

6.3\times

gain, plateau L28–L34. Middle: UCF-Crime (

42{,}746

3{,}664

), peak

0.5459

at L33,

2.8\times

gain, plateau L28–L37. Bottom: UBnormal (

1{,}711

2{,}201

), peak

0.1378

at L28,

9.9\times

gain, plateau L28–L31. Blue shading marks the shallow-collapse region (L1–L17); the coloured band marks the high-plateau zone. Absolute scales differ by an order of magnitude across datasets, yet the shallow floor, the L18–L27 rise and the plateau onset at L28 are shared.

Dataset	Clips (normal / abn.)	Shallow collapse	$\bar{J}$ (L1–L17)	High plateau	Peak layer / $J_{\text{peak}}$	Gain	Plateau width / $\bar{J}$
XD-Violence	73,814 / 23,582	L1–L17	0.109	L28–L34	L28 / 0.6842	$6.3\times$	7 / 0.661
UCF-Crime	42,746 / 3,664	L1–L17	0.195	L28–L37	L33 / 0.5459	$2.8\times$	10 / 0.521
UBnormal	1,711 / 2,201	L1–L17	0.014	L28–L31	L28 / 0.1378	$9.9\times$	4 / 0.128

Appendix

Table 9: Stage boundaries of the Fisher separability profiles in Figure 8 . The gain is

J_{\text{peak}}

divided by the mean

J

over the shallow-collapse region. Plateau width counts layers whose

J(l)

stays within

10\%

of the peak.

Figure 9: Layer-wise separability criterion

G(l)

on XD-Violence under different supervision budgets. Curves show

G(l)

estimated from clip-level pseudo-labels induced on

1\%

5\%

and

10\%

of weakly labelled training videos, against the

100\%

ground-truth reference. These are pseudo-label budgets, not zero-shot readouts. Pseudo-label noise attenuates the absolute value but preserves the ordering: all curves share the shallow floor at

G\approx 0.22

, the rise from L20, and the plateau from L28. Spearman rank correlation with the ground-truth curve over L20–L40 is

0.78

0.93

and

0.96

for the three budgets.

Selection signal	Test GT	Selected layer $l^{\star}$			UBnormal	UCF-Crime	XD-Viol.
Selection signal	Test GT	UBn	UCF	XD	AUC	AUC	AP
Oracle (downstream argmax)	yes	29	34	31	79.31	91.44	92.61
Fisher $J(l)$ (hidden space)	yes	28	33	28	79.17	91.32	92.49
$G(l)$ , $1\%$ pseudo-labels	no	28	31	27	78.52	90.95	92.10
$G(l)$ , $5\%$ pseudo-labels (default)	no	28	33	28	79.17	91.32	92.49
$G(l)$ , $10\%$ pseudo-labels	no	29	33	28	79.31	91.32	92.49

Appendix

Table 10: Adaptive versus oracle layer selection (

\lambda_{\mathrm{reg}}=0.1

, offline mode). Oracle selection maximizes the downstream test metric per dataset and is an upper bound conditional on the default

5\%

Stage-II budget ; it requires test ground truth and is therefore not deployable.

J(l)

ranks layers using test-set temporal boundaries in hidden space only.

G(l)

rows are fully deployable, using pseudo-labels induced from the stated fraction of weakly labelled training videos. In every row the Stage-II supervision budget is held at

5\%

, so the rows isolate the effect of layer selection and are not comparable to Table 3 , which varies the budget end-to-end. Dataset sub-columns follow the same left-to-right order on both sides of the table.

$\lambda_{\mathrm{reg}}$	$l^{\star}$	Region	Feature alignment	Thpt. (FPS)	XD-AP (%)	$\Delta$ AP vs. $\lambda_{\mathrm{reg}}\!=\!0$
0.00	33	Plateau	Fully developed	51.83	90.70	–
0.05	31	Plateau	Fully developed	51.83	90.63	$-0.07$
0.10	28	Plateau (onset)	Fully developed	51.83	90.50	$\mathbf{-0.20}$
0.20	26	Emergence	Partially established	51.83	89.14	$-1.56$
0.40	21	Emergence	Nascent / incomplete	51.83	85.42	$-5.28$
0.80	20	$l_{\min}$ bound	Negligible alignment	51.83	84.61	$-6.09$

Appendix

Table 11: Sensitivity of the depth regularization weight

\lambda_{\mathrm{reg}}

on XD-Violence (online streaming mode,

5\%

supervision budget). Throughput is unchanged across settings because all MoE layers contribute to the global routing fingerprint

\mathbf{R}_{\mathrm{clip}}

regardless of which layer supplies the semantic state.

\lambda_{\mathrm{reg}}=0.1

is the default operating point.

Dataset	$l^{\star}$	AUC (%)	AP (%)	Pos. base rate (%)
UBnormal	28	$74.31\pm 0.79$	$79.28\pm 0.63$	56.13
UCF-Crime	33	$87.16\pm 0.48$	$45.82\pm 0.76$	0 7.91
XD-Violence	28	$94.72\pm 0.31$	$88.14\pm 0.42$	24.24

Appendix

Table 12: Routing-only linear discriminant baseline. A linear head is fitted on

\mathbf{R}_{\mathrm{clip}}(t)

alone, without hidden states, temporal context or the backbone language head. Evaluation layers match the adaptive selection of Section 4.1 ; values are mean

\pm

std over

5

random seeds. AUC is frame-level ROC-AUC.

Single-branch references
Operator	Form	UBnormal	UCF-Crime	XD-Violence
Operator	Form	AUC (%)	AUC (%)	AP (%)
Hidden only	$\bar{\mathbf{H}}$	72.40	82.90	84.87
Routing only	$\mathbf{W}\mathbf{R}$	74.31	87.16	88.14
Fusion operators
Direct concatenation	$[\bar{\mathbf{H}};\mathbf{W}\mathbf{R}]$	74.26	87.32	88.63

Appendix

Table 13: Comparison of fusion operators. All rows use one fusion block without RoTN, at the adaptively selected layer (

l^{\star}=28/33/28

for UBnormal / UCF-Crime / XD-Violence). The two reference rows are the single-branch limits: hidden-only is the

\times\,/\,\times

row of Table 3 , routing-only is Table 12 . Metrics follow the main text: AUC for UBnormal and UCF-Crime, AP for XD-Violence. Best per column in bold.

Backbone model	Activated / total params	XD-Violence		UCF-Crime	UBnormal
Backbone model	Activated / total params	AP (%)	AUC (%)	AUC (%)	AUC (%)
Random guess	—	24.24	50.00	50.00	50.00
Qwen3.6-35B-A3B	$\sim$ 2.4B / 35B	71.37	85.62	74.95	58.27
Qwen3VL-30B-A3B	$\sim$ 2.8B / 30B	76.65	89.02	75.83	59.41
Gemma-4-26B-A4B	$\sim$ 3.7B / 26B	71.11	86.33	73.80	57.14

Appendix

Table 14: Zero-shot performance of frozen MoE backbones across three benchmarks without parameter tuning, read from native language output logits on visual clips.

$\lambda_{\mathrm{s}}$	AUC (%)	AP (%)	TV ( $\downarrow$ )	Mean $\bar{a}$
0.05	97.05	90.89	0.0156	0.349
0.10	97.34	92.49	0.0212	0.319
0.50	96.94	91.01	0.0198	0.372
1.00	97.21	92.01	0.0213	0.317
2.00	96.77	90.73	0.0128	0.361
5.00	96.38	89.66	0.0102	0.401

Appendix

Table 15: Hyperparameter sensitivity sweep over

\lambda_{\mathrm{s}}

and

\lambda_{\mathrm{p}}

on XD-Violence (

l^{\star}=28

, averaged over

3

random seeds). The controlled variable is fixed at its operating value (

\lambda_{\mathrm{p}}=0.05

in panel (a),

\lambda_{\mathrm{s}}=0.1

in panel (b)). Best AP per panel in bold. The peak matches the full model of Table 1 (

97.34\%

AUC /

92.49\%

AP).

Component	Hyperparameter	Setting / value
Data sampling	Weak supervision ratio	$5\%$ of weakly labelled training videos
	Frames per clip ( $T_{f}$ )	$24$ frames
	Sampled frames per clip ( $K$ )	$4$ uniform frames
	Input image dimensions	$336\times 336$ RGB (Lanczos interpolation)
RoMF module	Semantic hidden dimension ( $D$ )	$2048$ (inherited from MLLM backbone)
	Global routing dimension ( $L\cdot E$ )	$10{,}240$ ( $40$ layers $\times$ $256$ experts)

Appendix

Table 16: Hyperparameter specifications for Stage-II RoMod training and temporal modelling. All values correspond to the default configuration used for every reported result.

Resource attribute	Specification / environment version
Central processing unit (CPU)	Intel(R) Xeon(R) Platinum 8358 @ 2.60 GHz (128 vCPUs)
Host memory (RAM)	1007 GB
Graphics processing units (GPUs)	4 $\times$ NVIDIA A100 80GB PCIe (79 GB VRAM each)
CUDA & driver version	CUDA 12.6 / NVIDIA driver 580.126.09
Operating system	Ubuntu 22.04.2 LTS (Linux kernel 5.15.0-43-generic)
Core deep learning framework	PyTorch 2.7.0+cu126 / TorchVision 0.22.0

Appendix

Table 17: Hardware specifications and software deployment environment.

Panel of Figure 10 — Figure 10: Extended qualitative anomaly localization curves on XD-Violence and UCF-Crime (Part 1). Top: normal surveillance ( v=HksDvkX6Bjk ), long-range multi-segment riot ( v=gsz_P8t-KM4 ), and abrupt motion transitions in fighting ( Kill.Bill.Vol.1.2003 ). Bottom: robustness to illumination shifts in a normal video ( v=5AlAh8J3WqE ), extended high-density crowd riot ( v=OFxPTJxA5pg ), and recurring civil unrest ( v=pFamvR9CpYw ). Solid blue: RoMod; dashed orange: zero-shot hidden baseline; pink bands: ground truth; dashed red: $\tau_{\mathrm{dec}}=0.5$ .

Panel of Figure 11 — Figure 11: Extended qualitative anomaly localization curves on XD-Violence and UCF-Crime (Part 2). Top: high-noise normal surveillance ( v=tI7fktKY6OU ), sustained multi-agent street riot ( v=iegHZ_UWWsA ), and ultra-long monitoring ( v=Dh-3BlhAOnE , $>18{,}000$ frames). Bottom: ambient normal scene ( v=fkps18H3SXY ), transient vehicle explosion with rapid smoke dissipation ( v=_BgJEXQkjNQ ), and dense fighting interactions ( Ip.Man.3.2015 ).

Explore similar work

Representation--Behavior Alignment for Explainable Weakly-Supervised Video Anomaly Detection
Oct 4, 2026Chao Huang, Pengfei Wei, Kaige Li +4Training-Free Video Anomaly Detection Fine-Grained Video Understanding
Sun Yat-sen University · University of Macau
CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection
May 22, 2026Hyeongmuk Lim, Youngbum HurTraining-Free Video Anomaly Detection Fine-Grained Video Understanding
Department of Industrial Engineering, Inha University, Incheon, Republic of Korea
Linguistic Relative Policy Optimization for Video Anomaly Reasoning
Jul 1, 2026Jiaxu Leng, Jiankang Zheng, Mengjingcheng Mo +4Complementarity
School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China · Chongqing College of Artificial Intelligence, Chongqing, China

Oct 4, 2026, cs.CVJ/K move · Enter open · S save

Representation--Behavior Alignment for Explainable Weakly-Supervised Video Anomaly Detection

Chao Huang, Pengfei Wei, Kaige Li, Chengliang Liu+3

Sun Yat-sen University · University of Macau

Training-Free Video Anomaly Detection Fine-Grained Video Understanding

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) provide a natural way to make video anomaly detection more explainable. However, their final decisions do not always fully use the discriminative information contained in their hidden states, an issue we refer to as representation--behavior misalignment. We decompose this gap into a capacity component that measures discriminative information never aggregated into the readout position, and a directional component that measures the angular mismatch between the optimal and the native normal--abnormal axis at that position. Across multiple video anomaly detection benchmarks and MLLM backbones the directional component dominates, and residual-stream tracing shows that native-axis separability rises sharply in several mid-to-late attention layers. Because both components are governed by attention rather than MLP updates, we propose Representation--Behavior Alignment (RBA), a parameter-efficient method that adapts those layers using video-level labels alone while updating about 0.012% of the backbone parameters. Experiments on three benchmarks show that RBA improves native-readout performance and better aligns the model's decision direction with discriminative representations, and it produces anomaly decisions and explanations through a single generative process.

Abstract

Explore similar work

Representation--Behavior Alignment for Explainable Weakly-Supervised Video Anomaly Detection

CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection

Linguistic Relative Policy Optimization for Video Anomaly Reasoning