Recent multimodal LLM (MLLM)-based anomaly detection (AD) methods can generate textual explanations that help interpret their predictions. However, reliably and efficiently evaluating these explanations remains challenging. While LLM judges can assess explanations through structured rubrics, they often require substantial token generation, making evaluation costly and difficult to scale. We instead propose to anchor evaluation in the available ground truth by extracting verifiable claims from each explanation and verifying them against the corresponding annotations. We instantiate this idea as AnomalyJev, a lightweight framework that extracts verifiable claims from contextualised representations using ground-truth-defined queries and pointer based heads, without token generation. The claims are then verified against the ground truth to score faithfulness and completeness. For evaluation, we construct a human-annotated set of 640 explanations with over 1,600 expert ratings across in-domain and cross-dataset settings. Despite its compact design, AnomalyJev achieves stronger agreement with human annotators than most open MLLM judges and approaches that of flagship MLLM judges, at a fraction of their runtime and resources.
Figures & tables
Figure 1: LLM/MLLM-as-a-judge vs. claim-based verification. (a) A judge takes the ground truth, evaluation rubric, and explanation as input and autoregressively generates a score. (b) AnomalyJev predicts one option per ground-truth attribute directly from contextualised representations, then verifies the resulting claims to produce deterministic scores, without token generation.
Figure 2: Overview of AnomalyJev. Training (orange): VLM explanations are augmented and weakly labelled by an LLM to supervise the LoRA module and query specific pointer heads. Inference (blue): the explanation is encoded once, question branches are processed in parallel, and the pointer head selects one option from each question’s option set without token generation. The extracted claims are verified against the ground truth to score faithfulness and completeness.
Figure 3: Agreement with human experts (Spearman ρ , faithfulness) against model size and runtime.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Screenshot of the interface used by human raters.
Figure 5: Instructions given to the two experts for the faithfulness (“correctness”) rating.
Figure 6: Instructions given to the two experts for the completeness rating.