How to Explain your Unexplainable Video Anomaly Detector
Organizations: Eindhoven University of Technology, Eindhoven, The Netherlands
Abstract
Weakly-supervised Video Anomaly Detection (VAD) models assign anomaly scores from high-dimensional features, and their predictions remain unexplained. Saliency maps show where a model looks but not what it relies on, and the descriptions produced by Vision-Language Models (VLMs) are generated by branches that the anomaly score never reads. To address this faithfulness gap, we propose Explain The Unexplainable (ETU), a classifier-agnostic framework that audits frozen VAD models through counterfactual interventions in their feature space. ETU maps video features to a discrete dependency graph of agents, actions and outcomes (), regenerates the features by graph-conditioned diffusion with selected nodes of the graph masked, and measures the change in the score of a frozen detector; one set of interventions is applied unchanged to every detector. The generator is trained only to reconstruct the input features, so it cannot learn the contrast it is asked to measure, and we validate the audit with experiments that test every alternative explanation of its results, from the cost of regeneration to random deletions of matched size. On UCF-Crime features obtained from a CLIP encoder, we show that substituting the scene semantics results in the largest drop for most detectors, and that removing the outcome chain of the graph () changes the anomaly score of all six audited detectors far more than removing the same number of random nodes, even though the random deletion perturbs the features about as much. We show that this targeted behavior comes from the divergence objective of the graph embedding. Without it, the two interventions become indistinguishable, and with a divergence objective in the generator as well, the score change of the detector follows the displacement of the reconstruction rather than the semantics that were removed.
Figures & tables
| Detector | I3D | CLIP | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| UCF-Crime (AUC) | XD-Violence (AP) | UCF-Crime (AUC) | XD-Violence (AP) | |||||||||||||
| MGFN | ||||||||||||||||
| UR-DMU | ||||||||||||||||
| RTFM | ||||||||||||||||
| PEL4VAD | – | – | – | – | – | – | – | – | ||||||||
| at | at | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Detector | 100 | 300 | 600 | 100 | 300 | 600 | |||
| MGFN | 4.26 | 38.73 | |||||||
| UR-DMU | 0.43 | 6.43 | |||||||
| RTFM | 3.72 | 22.64 | |||||||
| DSANet | 4.92 | 37.99 | |||||||
| VadCLIP | 3.77 | 25.84 | |||||||
| ETU | Divergence-trained generator | Divergence-free ETU | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Detector | |||||||||
| MGFN | |||||||||
| UR-DMU | |||||||||
| RTFM | |||||||||
| DSANet | |||||||||
| VadCLIP | |||||||||
| Displacement | Detector response | ||
|---|---|---|---|
| Mask | MSE | (AUC points) | |
| irrelevant mask | |||
| relevant mask | |||
| anomalous videos | |||
| count-matched random | |||
| anomalous videos | |||
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| Prototypes | MSE wt. | Test Cosine | Root Util. | Action Util. | Outcome Util. |
|---|---|---|---|---|---|
| 16 | 0.2 | 0.591 | 88% | 100% | 94% |
| 24 | 1.0 | 0.734 | 100% | 100% | 100% |
| 32 | 0.5 | 0.712 | 100% | 100% | 100% |
| 48 | 0.5 | 0.435 | 50% | 100% | 42% |
| 64 | 0.5 | 0.581 | 39% | 75% | 69% |
| 128 | 0.5 | 0.521 | 24% | 20% | 23% |
| MSE Weight | Utilization | Cosine Sim. |
|---|---|---|
| 0.2 | Partial (88% to 100%) | 0.34 to 0.59 |
| 0.5 | Full (24 to 32 prototypes) | 0.71 to 0.74 |
| 1.0 | Full (24 to 32 prototypes) | 0.73 to 0.81 |
| Architecture / Strategy | Edge | Observation |
|---|---|---|
| Dot product | 0.18 | Insufficient capacity for asymmetric edges |
| MLP pair | 0.22 | Marginal improvement |
| Cross-attention | 0.25 | Treats pairs independently |
| MLP with , | 0.29 | Best baseline scoring mechanism |
| + Hard negative mining | 0.45 | Addresses class imbalance |
| + Hierarchical conditioning | 0.66 | Explicitly models edge directionality |
| Method | Edge | Linear Probe AUC | CF Gap |
|---|---|---|---|
| Mean Pooling | 0.690 | 0.893 | 0.969 |
| GIN (3-layer) | 0.322 | 0.924 | 0.004 |
| Virtual Node | 0.330 | 0.848 | 0.053 |
| Edge-Gated Pooling | 0.712 | 0.861 | 0.637 |
| VICReg | LP AUC | CF Gap | Edge |
|---|---|---|---|
| 0.00 | 0.843 | 1.031 | 0.697 |
| 0.25 | 0.883 | 0.936 | 0.697 |
| 0.50 | 0.900 | 0.957 | 0.687 |
| 1.00 | 0.932 | 1.067 | 0.697 |
| Metric | Joint-Phase Only (45K) | Full-Phase (60K) | Relative Change |
|---|---|---|---|
| CF Gap | 0.60 to 0.64 | 0.94 to 1.07 | +55% to +70% |
| Embed Gap | 0.02 to 0.06 | 0.18 to 0.41 | +300% to +700% |
| LP AUC | 0.86 to 0.88 | 0.88 to 0.93 | +2% to +5% |
| Edge | 0.71 to 0.73 | 0.69 to 0.70 | 2% to 3% |
| Masking Strategy | Cosine Sim. |
|---|---|
| Relevant (remove anomaly) | 0.123 |
| Irrelevant (remove background) | 0.944 |
| Semantic (replace scene) | 0.193 |
| CF Gap (Irrel Rel) | 1.067 |
| Dataset | Enc. | Node | Edge | Graph (Cos.) | |||
|---|---|---|---|---|---|---|---|
| Cos. | MSE | ||||||
| UCF | I3D | 0.891 | 0.016 | 0.68 | 0.25 | 0.96 | |
| CLIP | 0.885 | 0.017 | 0.64 | 0.26 | 0.97 | ||
| XDV | I3D | 0.864 | 0.021 | 0.70 | 0.15 | 0.99 | |
| CLIP | 0.862 | 0.020 | 0.68 | 0.15 | 0.98 | ||
| Detector | UCF-Crime (AUC) | XD-Violence (AP) | ||||
|---|---|---|---|---|---|---|
| Publ. | GT | Publ. | GT | |||
| I3D features | ||||||
| MIL ∗ | – | 83.49 | 2.37 | – | 63.84 | 0.95 |
| GlanceVAD | 91.96 | 91.76 | 1.66 | 89.40 | 89.18 | 0.20 |
| MGFN ∗ | 86.98 | 80.80 | 3.25 | 79.19 | 67.57 | 0.97 |
| RTFM ∗ | 84.30 | 79.69 | 1.93 | 77.81 | 74.03 | 0.70 |
| Setting | MGFN | UR-DMU | RTFM | DSANet | VadCLIP | GlanceVAD |
|---|---|---|---|---|---|---|
| , one node | ||||||
| Detector | |||||
|---|---|---|---|---|---|
| MGFN | |||||
| UR-DMU | |||||
| RTFM | |||||
| DSANet | |||||
| VadCLIP | |||||
| GlanceVAD |
| UCF-Crime (AUC) | XD-Violence (AUC) | |||||||
|---|---|---|---|---|---|---|---|---|
| Detector | GT | Mag. | Grad. | Rand. | GT | Mag. | Grad. | Rand. |
| UR-DMU | ||||||||
| RTFM | ||||||||
| MGFN | ||||||||
| GlanceVAD | ||||||||
| DSANet | – | – | ||||||
| FFN | Heads | Tokens | Params | ||
|---|---|---|---|---|---|
| 768 | 1536 | 8 | 4 | 38M | |
| 1024 | 2048 | 8 | 8 | 68M | |
| 512 | 1024 | 4 | 4 | 7M |
| Enc. | Cond. | vs. | vs. | ||
|---|---|---|---|---|---|
| Cos. | MSE | Cos. | MSE | ||
| I3D | 0.956 | 0.008 | – | – | |
| 0.954 | 0.009 | 0.998 | 0.001 | ||
| 0.887 | 0.043 | 0.931 | 0.053 | ||
| 0.856 | 0.068 | 0.899 | 0.063 | ||
| CLIP | 0.978 | 0.008 | – | – | |
| Enc. | Cond. | vs. Ground-Truth ( ) | vs. Full Graph ( ) | ||||
|---|---|---|---|---|---|---|---|
| Cos. Sim. | MSE | Cos. Sim. | MSE | ||||
| I3D | 0.942 | 0.010 | – | – | |||
| 0.940 | 0.011 | 0.997 | 0.001 | ||||
| 0.903 | 0.020 | 0.960 | 0.010 | ||||
| 0.500 | 0.315 | 0.536 | 0.304 | ||||
| CLIP | 0.966 | 0.015 | – | – | |||
| Quantity | MGFN | UR-DMU | RTFM | DSANet | VadCLIP | GlanceVAD |
|---|---|---|---|---|---|---|
| (i) ETU, test set videos, frame-AUC points | ||||||
| 95% lower | ||||||
| 95% upper | ||||||
| two-sided | ||||||
| ETU | Divergence-trained | |||
| Mask | MSE | MSE | ||
| videos of Table 3 of the main paper | ||||
| irrelevant mask | ||||
| relevant mask | ||||
| count-matched random | ||||
| anomalous videos | ||||
| Cosine to | to | |||||
|---|---|---|---|---|---|---|
| Mask | all | anom. | normal | all | anom. | normal |
| irrelevant mask | ||||||
| relevant mask | ||||||
| count-matched random | ||||||
| at | at | at | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Detector | 100 | 300 | 600 | 100 | 300 | 600 | 100 | 300 | 600 |
| MGFN | |||||||||
| UR-DMU | |||||||||
| RTFM | |||||||||
| DSANet | |||||||||
| VadCLIP | |||||||||
| Qwen3-VL / | Qwen3-VL / | InternVideo3 / | |
| InternVideo3 | Gemma-4 | Gemma-4 | |
| matched atom cosine, same video | 0.780 / 0.793 / 0.772 | 0.764 / 0.777 / 0.728 | 0.766 / 0.779 / 0.737 |
| same-class null | 0.704 / 0.729 / 0.715 | 0.705 / 0.722 / 0.687 | 0.709 / 0.731 / 0.693 |
| random-video null | 0.668 / 0.683 / 0.667 | 0.674 / 0.687 / 0.658 | 0.682 / 0.702 / 0.669 |
| paired win vs same-class null | 81.4% | 74.1% | 73.4% |
| edge , same video | 0.250 | 0.167 | 0.138 |
| Setting | MGFN | UR-DMU | RTFM | DSANet | VadCLIP | GlanceVAD |
|---|---|---|---|---|---|---|
| ETU , all 290 test videos | ||||||
| on-chain node | ||||||
| off-chain node | ||||||
| Divergence-trained generator , its own single-node cache, the same masks | ||||||
| on-chain node | ||||||
| off-chain node | ||||||
| Type | Concept | |||
|---|---|---|---|---|
| Action | Driving | |||
| Root | Car 1 | |||
| Outcome | Car 1 crash | |||
| Root | Pers. 2 | 0.092 | ||
| Outcome | Pers. 2 exit | 0.002 | 0.003 | |
| Root | Person 1 | 0.001 |
Interpretable Anomaly Detection
Interpretable anomaly detection identifies unusual data and explains the features, patterns, or conditions behind each alert.
See all 84 papers