Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection

Zhaolin Cai; Fan Li; Huiyu Duan; Lijun He; Guangtao Zhai

cs.CVFeb 27, 2026

Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection

Authors: Zhaolin Cai, Fan Li, Huiyu Duan, Lijun He, Guangtao Zhai

Organizations: Xinjiang University · Xi’an Jiao Tong University · Shanghai Jiao Tong University · Shanghai AI Laboratory

Abstract

Video anomaly detection (VAD) aims to identify abnormal events in videos. Traditional VAD methods generally suffer from the high costs of labeled data and full training, thus some recent works have explored leveraging frozen multi-modal large language models (MLLMs) in a tuning-free manner to perform VAD. However, their performance is limited as they directly inherit pre-training biases and cannot adapt internal representations to specific video contexts, leading to difficulties in handling subtle or ambiguous anomalies. To address these limitations, we propose a novel intervention framework, termed SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations. Our approach first leverages the gradient-free representational separability analysis (RSA) to identify top attention heads as latent anomaly experts (LAEs) which are most discriminative for VAD. Then a hierarchical meta-controller (HMC) generates dynamic rectification signals by jointly conditioning on global context and these LAE outputs. The signals execute targeted, anisotropic scaling directly upon the LAE representation manifolds, amplifying anomaly-relevant dimensions while suppressing inherent biases. Extensive experiments on mainstream benchmarks demonstrate our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data, establishing it as a powerful new direction for video anomaly detection. The code will be released upon the publication.

Figures & tables

Figure 1: Comparison of traditional full-training methods, existing tuning-free methods and our proposed SteerVAD. Our method overcomes the issue of costly training resources and inherent biases with minimal data required from pre-trained foundation models compared to previous methods.

Panel of Figure 2 — Figure 2: 3D UMAP visualization of representation manifolds of normal (blue) and anomalous (red) events from InternVL, illustrating their geometric structure. Each manifold is rendered from two perspectives using (a) cubic interpolation and (b) triangulation.

Figure 3: Framework overview of SteerVAD. We first apply the Representational Separability Analysis to find top $K$ Latent Anomaly Experts inside frozen MLLM. During the single pass, The global context vector $\mathbf{c}$ and LAE features $\{\mathbf{h}_{i}\}$ are extracted. The Hierarchical Meta-Controller ingests these signals, using Global Scrutiny Gate and Local Gating Module to generate manipulation signals ( $s_{\text{global}}$ , $\{\mathbf{g}_{i}\}$ ). These signals perform Anisotropic Manifold Scaling to rectify LAE features. A lightweight Anomaly Scorer receives the rectified features and outputs the final anomaly curve. Detected anomalous frames can be passed to the full MLLM to produce a textual explanation.

Mode	Methods	Backbone	Params	UCF	XD
Mode	Methods	Backbone	Params	AUC (%)	AP (%)
Weakly Supervised	Wu et al. ( Wu et al., 2020 )	I3D	-	82.44	73.20
	MIST ( Feng et al., 2021 )	I3D	-	82.30	-
	RTFM ( Tian et al., 2021 )	I3D	-	84.30	77.81
	S3R ( Wu et al., 2022 )	I3D	-	85.99	80.26
	MSL ( Li et al., 2022b )	I3D	-	85.30	78.28

Table 1: Comparison with existing methods on the UCF-Crime and XD-Violence datasets.

Panel of Figure 4 — Figure 4: Visualization of manifold rectification via t-SNE on aggregated features from the LAEs for the XD-Violence and UCF-Crime datasets.

Configuration / Strategy	GSG	LGM	Steering	AUC (%)
Part A: HMC Component Ablation
Full Model	✓	✓	Anisotropic	87.15
– w/o Global Gate	✗	✓	Anisotropic	85.94
– w/ Additive Steering	✓	✓	Additive	85.02
– w/o LGM (Static)	✓	✗	Static Scaling	84.21
– Linear Probing	✗	✗	NA	81.33

Table 2: Ablation studies on the UCF-Crime dataset, analyzing the impact of key components and expert selection strategies.

Seed ID	7270	860	5390	5191	5734	6265	466	4426	5578	8322	Statistics
Selected LAEs	Identical (L18H4, L23H24, L21H21, L22H7)										100% Match
AUC (%)	87.05	87.12	87.18	87.14	87.09	87.21	87.16	87.10	87.13	87.19	87.15 $\pm$ 0.04

Table 3: Stability analysis of LAE selection and downstream performance across 10 independent runs with different random seeds on the 1% calibration subset.

Data Ratio	AUC (%)	Improvement	Training Time
1% (Ours)	87.15	-	$<$ 1 min
10%	87.29	+0.14	$\sim$ 5 min
25%	87.35	+0.20	$\sim$ 13 min
50%	87.39	+0.24	$\sim$ 25 min
100%	87.42	+0.27	$\sim$ 49 min

Table 4: Data efficiency analysis. The rapid performance saturation indicates that the geometric steering policy converges to an optimal solution with minimal supervision (1%).

Number of Experts ( $K$ )		Number of Frames ( $F$ )
Value	AUC (%)	Value	AUC (%)
2	85.91	4	82.54
4	87.15	8	85.03
8	87.09	16	87.15
16	86.88	24	87.23

Table 5: Hyperparameter analysis of selected LAEs (

K

) and input frames (

F

) on UCF-Crime.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	Input Shape	Output Shape	Activation / Notes
Input $\mathbf{c}$	$[B,3584]$	-	Global context vector
Linear-1	$[B,3584]$	$[B,128]$	-
BatchNorm1d	$[B,128]$	$[B,128]$	Stabilizes training
ReLU	$[B,128]$	$[B,128]$	Non-linearity
Linear-2	$[B,128]$	$[B,1]$	Output logit
Sigmoid	$[B,1]$	$[B,1]$	Produces $s_{\text{global}}$

Appendix

Table 6: Architecture of the global scrutiny gate (GSG). The input batch size is denoted by

B

Layer	Input Shape	Output Shape	Activation / Notes
Input $\mathbf{c}$	$[B,3584]$	-	Shared across $K$ adapters
Linear-down ( $r=4$ )	$[B,3584]$	$[B,4]$	bias=False
Linear-up	$[B,4]$	$[B,128]$	bias=False
Tanh	$[B,128]$	$[B,128]$	Produces $\mathbf{g}_{i}\in[-1,1]$

Appendix

Table 7: Architecture of a single low-rank adapter within the LGM.

Hyperparameter	Value	Description
Data Processing & Model Architecture
$L$	48 frames	The duration of each non-overlapping video segment.
$F$	16 frames	The number of frames sampled from each segment as input to the MLLM.
$K$	4	The number of top-scoring latent anomaly experts (LAEs) selected by RSA.
HMC Training
Epochs	1000	The total number of training epochs for the HMC and anomaly scorer.

Appendix

Table 8: Consolidated list of hyperparameters for the SteerVAD framework.

Component	Parameters	Size / Complexity
Parameter Count & Disk Size
Global Scrutiny Gate	459,265	—
Local Gating Module	59,392	—
HMC (Total)	518,657	0.99 MB
Anomaly Scorer	513	<0.01 MB
Total Trainable	519,170	~0.99 MB

Appendix

Table 9: Architectural size and computational complexity of SteerVAD’s trainable modules.

Metric	Value
Total Training Epochs	1000
Batch Size	64
Learning Rate	0.001
Total Training Time	27.83 seconds (0.46 minutes)
Average Time per Epoch	0.03 seconds
Peak GPU Memory Usage	203.88 MB

Appendix

Table 10: Training cost statistics for calibrating the HMC and Anomaly Scorer.

Sampled Frames ( $F$ )	Avg. Inference Time (s)	Peak VRAM (GB)
4	0.43	17.2
8	0.78	18.3
16	1.52	20.4
24	2.29	22.3

Appendix

Table 11: Inference cost as a function of the number of sampled frames (

F

) per segment.

Source (Calibration)	Target (Evaluation)	Metric	HiProbeVAD (Rep)	SteerVAD (Ours)	Gap
UCF-Crime (1%)	XD-Violence (Unseen)	AP (%)	70.66	71.31	+0.65
XD-Violence (1%)	UCF-Crime (Unseen)	AUC (%)	80.82	81.04	+0.22

Appendix

Table 12: Cross-dataset generalization comparison. To ensure fairness, we reproduce HiProbeVAD under identical settings with InternVL3-8B backbone with same prompt1 setting in App.C.2.

Backbone	Params	UCF-Crime (AUC %)	XD-Violence (AP %)
LLaVA-OV ( Li et al., 2024 )	$\sim$ 7B	81.52	74.65
Qwen2.5-VL ( Bai et al., 2025 )	$\sim$ 7B	84.11	79.82
InternVL3 ( Zhu et al., 2025 )	$\sim$ 8B	87.15	83.02

Appendix

Table 13: Cross-model generalization performance. The framework is applied to different most widely used and advanced frozen MLLM backbones.

Panel of Figure 5 — Figure 5: Evolution of anomaly expert activations across different calibration data scales. The consistent distribution of hotspots from 1% to 100% visually confirms that the identification of anomaly experts saturates rapidly.

Panel of Figure 6 — Figure 6: Evolution of anomaly expert activations across different calibration data scales. The consistent distribution of hotspots from 1% to 100% visually confirms that the identification of anomaly experts saturates rapidly.

Ratio (N:A)	1:9	2:8	3:7	4:6	5:5 (Default)	6:4	7:3	8:2	9:1
AUC (%)	83.67	85.69	85.75	86.34	87.15	87.00	85.70	85.62	83.72

Appendix

Table 14: Impact of normal/abnormal ratio in the calibration set on UCF-Crime performance. The model maintains robust detection capabilities across a wide range of class distributions.

Supervision Type	Granularity	AUC (%)
Segment-Level (Default)	Fine	87.15
Video-Level	Coarse	87.01

Appendix

Table 15: Impact of supervision granularity on UCF-Crime performance. SteerVAD maintains high performance even with coarse video-level labels.

Figure 7: Jaccard similarity analysis of the Top-20 identified anomaly experts across 10 random seeds. The high similarity scores confirm that the expert selection is structurally robust to sampling variations.

Class	AUC (%)	Class	AUC (%)
Abuse	68.84	Robbery	84.02
Arrest	82.58	RoadAccident	89.64
Arson	59.02	Shooting	85.74
Assault	95.17	Shoplifting	81.57
Burglary	75.47	Stealing	80.18
Explosion	79.72	Vandalism	88.16

Appendix

Table 16: Detailed per-class AUC (%) performance on UCF-Crime. The active steering mechanism demonstrates consistent robustness across diverse anomaly types, with particularly high performance in dynamic events.

Panel of Figure 8 — Figure 8: Visualization of top attention head selection frequency across 10 independent random seeds. The emergence of consistent high-frequency heads confirms that the model reliably identifies the same intrinsic functional circuits for anomaly detection, demonstrating robustness against random calibration splits.

Metric	Linearity	Expert Overlap	Calibration Time	Speedup
RSA (Ours)	Linear	100% (Reference)	0.14 s	49 $\times$
Silhouette	Non-Linear	100% Identical	0.86 s	8 $\times$
k-NN Purity	Non-Linear	100% Identical	6.87 s	1 $\times$

Appendix

Table 17: Comparison of RSA with non-linear selection metrics regarding expert overlap and computational efficiency. RSA identifies the identical set of experts as expensive non-linear metrics but with significantly higher speed.

Panel of Figure 9 — Figure 9: Validation of RSA efficiency and consistency. (Top) Average computation time per method, showing RSA’s superior speed advantage. (Bottom) Heatmaps of head importance scores generated by RSA, Silhouette, and k-NN. The nearly identical visual patterns confirm that RSA captures the same underlying geometric structure as computationally expensive non-linear metrics.

Design Variant	Input Signal	AUC (%)	Observation
Local-Aware	$[c,h_{i}]$ (Concat)	86.35	Overfits to local noise & Self-Bias
SteerVAD (Ours)	Global Context $c$ Only	87.15	Robust Top-Down Modulation

Appendix

Table 18: Design ablation for HMC input signals on UCF-Crime. Conditioning solely on global context prevents overfitting to local feature noise and ensures robust top-down modulation.

Panel of Figure 10 — Figure 10: Semantic impact of feature rectification on an example from XD-Violence.

Panel of Figure 11 — Figure 11: Qualitative results of SteerVAD on UCF-Crime and XD-Violence datasets. Each panel shows a representative video snippet and the anomaly curve with one normal and one abnormal video. The shaded regions denote the ground truth. Further generated descriptions are also provided.

Figure 12: Further qualitative results on UCF-Crime dataset. Covered all 13 abnormal category and normal examples.

Figure 13: Further qualitative results on XD-Violence dataset. Covered all 6 abnormal category and normal examples.

Panel of Figure 14 — Figure 14: Further visualizations of SteerVAD on XD-Violence datasets. Our steering method effectively rectify the biased or ambiguous focus on the original videos.

Panel of Figure 15 — Figure 15: Further visualizations of SteerVAD on UCF-Crime datasets. Our steering method effectively rectify the biased or ambiguous focus on the original videos.

Explore similar work

SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere
May 8, 2026Chao Huang, Pengfei Wei, Wei Wang +5Video Anomaly Detection Zero-Shot Anomaly Detection
Shenzhen Campus of Sun Yat-sen University · Harbin Institute of Technology, Shenzhen
Geometry-Aware Semantic Reasoning for Training Free Video Anomaly Detection
Mar 10, 2026Ali Zia, Usman Ali, Muhammad Umer Ramzan +3Video Anomaly Detection Multimodal Anomaly Detection
School of Computing, Engineering & Mathematical Sciences, La Trobe University, Melbourne, Australia · School of Engineering and Applied Sciences, GIFT University, Gujranwala 52250, Pakistan
CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection
May 22, 2026Hyeongmuk Lim, Youngbum HurInterpretable Anomaly Detection VLM Reasoning
Department of Industrial Engineering, Inha University, Incheon, Republic of Korea

May 8, 2026J/K move · Enter open · S save

SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere

Chao Huang, Pengfei Wei, Wei Wang, Jie Wen+4

Shenzhen Campus of Sun Yat-sen University · Harbin Institute of Technology, Shenzhen

Video Anomaly Detection Zero-Shot Anomaly Detection

Open paper AskPDF arXiv

Video anomaly detection (VAD) aims to automatically identify events that deviate from normal patterns in untrimmed surveillance videos. Existing methods universally depend on large-scale annotations or task-specific training procedures, severely limiting their rapid deployment to novel scenes. We observe that intermediate-layer features of pre-trained multimodal large language models (MLLMs) already encode rich anomaly semantics, yet existing approaches rely on the language output pathway and fail to exploit the geometric discriminability latent in these representations. Based on this finding, we propose SphereVAD, a fully training-free, zero-shot VAD framework that recasts anomaly discrimination as von Mises-Fisher (vMF) likelihood-ratio geodesic inference on the unit hypersphere, unleashing latent discriminability through principled geometric reasoning rather than learning new representations. Specifically, SphereVAD first applies Frechet mean centering to unfold feature distributions and eliminate domain biases, then employs Holistic Scene Attention (HSA) to reinforce feature consistency using cross-video priors, and finally performs vMF-guided Spherical Geodesic Pulling (SGP) to align ambiguous segments with directional prototypes on the spherical manifold. This training-free pipeline requires only minimal synthetic images for calibration. SphereVAD establishes new state-of-the-art results among training-free approaches on three major benchmarks and remains competitive with fully supervised baselines. Code will be available upon acceptance.

Abstract

Explore similar work

SphereVAD: Training-Free Video Anomaly Detection via Geodesic Inference on the Unit Hypersphere

Geometry-Aware Semantic Reasoning for Training Free Video Anomaly Detection

CoReVAD: A Contextual Reasoning Framework for Training-Free Video Anomaly Detection