cs.CVFeb 3, 2026

MM-SCALE: Evaluating Evidence-Grounded Moral Judgment in Vision-Language Models

Authors: Eunkyu Park, Wesley Hanwen Deng, Cheyon Jin, Matheus Kunzler Maldaner, Jordan Wheeler, Jason I. Hong, Hong Shen, Adam Perer, 

Organizations: Seoul National University · Carnegie Mellon University · University of Florida · Epic Games

Abstract

Vision-Language Models increasingly make socially consequential judgments from image-text inputs, yet current evaluations often stop at verdict-level accuracy: whether the final label is human-aligned, not whether it follows from the correct evidence. We introduce MM-SCALE (Multimodal Moral SCALE), a benchmark for evidence-grounded moral judgment built around a design choice: each image is paired with multiple action scenarios, so models must compare how different actions interact with the same visual context rather than score images in isolation. MM-SCALE contains 8,444 image contexts and 21,977 action scenarios, each annotated with a 5-point moral acceptability rating and a modality-grounding label indicating whether the human judgment relied on text, image, or both. Three tasks evaluate whether models (i) assign calibrated scalar moral scores, (ii) preserve human orderings within a shared visual context, and (iii) ground rationales in the evidence source humans found critical to judgment. Across models, aggregate metrics overstate alignment: NDCG@5 remains high across models, while within-image pairwise accuracy remains only 0.53-0.62 even on scenario pairs whose human mean ratings differ by at least one point. CoT has inconsistent effects on calibration and does not reliably improve within-image ordering. Evidence grounding evaluations further show that models often describe visual context without making it critical for the verdict. These results reveal an evidence-grounding failure in current VLMs and position MM-SCALE as an evaluation testbed for moral judgment under shared image contexts. This dataset includes potentially harmful and sensitive visual content. Images are intended solely for evaluation.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Visual Distraction Undermines Moral Reasoning in Vision-Language Models

    Mar 17, 2026Xinyi Yang, Chenheng Xu, Weijun Hong +4Multimodal Multi-Turn Safety AlignmentVLM Reasoning

  2. On the Context Sensitivity of LLM Moral Judgment

    Mar 24, 2026Adrian Sauter, Mona SchirmerMoral Reasoning in Language ModelsMoral Reasoning

  3. VIGOR: Benchmarking Visual Rationale Correctness in Multimodal Large Language Models

    Nov 9, 2026Yanlin Chen, Tang Li, Xi PengMLLM EvaluationVision-Language Grounding