Multimodal Training with Knowledge Traction for Video Anomaly Detection
Organizations: Xidian University, Xi’an, China
Abstract
The task of weakly supervised video anomaly detection is challenging. Existing approaches mainly model video frame features and determine whether anomalies are present based on their magnitudes. In this work, we propose a novel method that incorporates external knowledge into training to improve model performance, robustness, and generalization. We leverage textual prompts and a large language model (LLM) to establish prior information for the specific task, and employ dictionary learning to normalize and compress this information. Moreover, during experiments, we observed inconsistencies in the textual descriptions extracted from similar video frames by image caption models. To address this, we devise a method that leverages an LLM and auxiliary information to ensure consistency in the textual descriptions of similar frames within a video. Finally, we introduce a feature processing approach that incorporates prior knowledge and cross-modal attention mechanisms. This approach learns the relationships and characteristics among different modalities to construct more accurate and reliable feature representations, thereby improving the accuracy and stability of downstream tasks. Experimental results demonstrate that our method surpasses other state-of-the-art approaches.
Figures & tables
| Supervision | Method | Feature | AUC (%) |
| Semi-supervised | Mem-AE | - | 71.2 |
| DLAN-AC | - | 74.7 | |
| SVD-GAN | - | 78.42 | |
| Weakly-supervised | MIL-Rank | 86.3 | |
| GCN | 84.44 | ||
| CLAWS | 89.67 |
| Supervision | Method | Feature | AUC (%) |
| Semi-supervised | Conv-AE | - | 50.6 |
| S-SVDD | - | 58.5 | |
| GODS | 70.46 | ||
| Weakly-supervised | MIL-Rank | 75.41 | |
| GCN | 82.12 | ||
| MIST | 82.3 |
| Baseline | VTA | TVA | KT | ShanghaiTech | UCF-Crime |
|---|---|---|---|---|---|
| ✔ | ✘ | ✘ | ✘ | 92.36 | 83.31 |
| ✔ | ✘ | ✔ | ✘ | 97.71 | 86.33 |
| ✔ | ✘ | ✔ | ✔ | 93.04 | 82.69 |
| ✔ | ✔ | ✔ | ✘ | 97.66 | 86.74 |
| ✔ | ✔ | ✔ | ✔ | 98.17 | 87.11 |
| Normal Count | Anomaly Count | ShanghaiTech | UCF-Crime |
|---|---|---|---|
| 2500 | 2500 | 97.41 | 85.97 |
| 2500 | 5000 | 97.83 | 86.59 |
| 5000 | 2500 | 97.28 | 86.41 |
| 5000 | 5000 | 98.17 | 87.11 |
| 10000 | 10000 | 97.95 | 86.93 |
| Number of Convolutions | ShanghaiTech | UCF-Crime |
|---|---|---|
| 1 | 96.23 | 84.89 |
| 2 | 97.19 | 85.76 |
| 4 | 97.64 | 86.88 |
| 8 | 98.17 | 87.11 |
| 16 | 97.78 | 86.72 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Prompt Text | Normal Description | Anomalous Description |
|---|---|---|---|
| ShanghaiTech | I need to detect the abnormality of a certain street in the campus. On this street, only normal walking is normal. Other situations, such as passing by bicycle, passing by fast running, etc., are abnormal. Please help me to randomly generate text descriptions of normal and abnormal situations. Note that it is randomly generated. | "People leisurely walking down the street" | "A person riding a bicycle through the pedestrian area." |
| "Students chatting while strolling along the sidewalk." | "Someone roller-skating at a high speed down the sidewalk." | ||
| "A group of friends walking together at a moderate pace." | "A jogger sprinting through the street segment." | ||
| "Someone carrying a backpack and walking calmly." | "Individuals playing a game of frisbee in the middle of the street." | ||
| "Students walking with their hands in their pockets." | "Someone skateboarding recklessly down the sidewalk. " | ||
| "A student wearing headphones, walking at a steady pace." | "A student running with a backpack on their back." |
| Type | Item | Value |
| Visual processing | FPS | 16 |
| Resize | 224 | |
| Padding | 0 | |
| Augmentation | 10-crop | |
| Feature dimension | 2048 | |
| Text processing | Feature dimension | 2048 |
| Scenes | Prompt Text | Normal Description | Anomalous Description |
|---|---|---|---|
| Steal | I need to detect anomalies related to theft. The detection scenes are usually the interior scenes of shops or houses, which are rich and varied. Please help me randomly generate text descriptions of normal and abnormal scenarios. | "A well-lit retail store with shelves neatly arranged, customers browsing products, and employees assisting them." | "A dimly lit retail store with broken shelves, items scattered on the floor, and a person acting suspiciously and concealing items in their bag." |
| "A cozy living room with comfortable furniture, family members watching TV, and soft music playing in the background." | "A disheveled living room with overturned furniture, shattered glass, and signs of forced entry through a broken window." | ||
| "A quiet bookstore with customers peacefully reading, classical music playing, and a barista serving coffee at the in-store cafe." | "A chaotic bookstore with books strewn all over the floor, a fire alarm blaring, and an individual stuffing books into a bag hurriedly." | ||
| Explosion | I need to detect anomalies about explosions. The detection scenarios are rich and diverse. The cause of the explosion could be various reasons. Please help me randomly generate text descriptions of normal and abnormal scenarios. | "A bustling parking lot on a sunny day, cars coming and going as people run errands." | "A deafening explosion shatters the quietness of the parking lot, sending debris and smoke into the air." |
| "A quiet residential street with children playing, neighbors chatting, and birds chirping." | "Chaos erupts on the residential street as a building collapses due to a gas leak, causing panic and confusion." | ||
| "A serene gas station with vehicles filling up, attended by workers in their uniforms." | "An intense fire breaks out at the gas station, flames engulfing fuel pumps and sending plumes of black smoke into the sky." |
Video Anomaly Detection
Video anomaly detection identifies unusual events or behaviors in video, often when examples of the anomalies are scarce or unknown.
See all 117 papers