MM-SCALE: Evaluating Evidence-Grounded Moral Judgment in Vision-Language Models
Authors: Eunkyu Park, Wesley Hanwen Deng, Cheyon Jin, Matheus Kunzler Maldaner, Jordan Wheeler, Jason I. Hong, Hong Shen, Adam Perer, Ken Holstein, Motahhare Eslami, Gunhee Kim
Organizations: Seoul National University · Carnegie Mellon University · University of Florida · Epic Games
Vision-Language Models increasingly make socially consequential judgments from image-text inputs, yet current evaluations often stop at verdict-level accuracy: whether the final label is human-aligned, not whether it follows from the correct evidence. We introduce MM-SCALE (Multimodal Moral SCALE), a benchmark for evidence-grounded moral judgment built around a design choice: each image is paired with multiple action scenarios, so models must compare how different actions interact with the same visual context rather than score images in isolation. MM-SCALE contains 8,444 image contexts and 21,977 action scenarios, each annotated with a 5-point moral acceptability rating and a modality-grounding label indicating whether the human judgment relied on text, image, or both. Three tasks evaluate whether models (i) assign calibrated scalar moral scores, (ii) preserve human orderings within a shared visual context, and (iii) ground rationales in the evidence source humans found critical to judgment. Across models, aggregate metrics overstate alignment: NDCG@5 remains high across models, while within-image pairwise accuracy remains only 0.53-0.62 even on scenario pairs whose human mean ratings differ by at least one point. CoT has inconsistent effects on calibration and does not reliably improve within-image ordering. Evidence grounding evaluations further show that models often describe visual context without making it critical for the verdict. These results reveal an evidence-grounding failure in current VLMs and position MM-SCALE as an evaluation testbed for moral judgment under shared image contexts. This dataset includes potentially harmful and sensitive visual content. Images are intended solely for evaluation.
Figures & tables
Figure 1: Overview of MM-SCALE . (A) Each image context is paired with multiple action scenarios. Each image–scenario pair has a human mean moral acceptability rating and a human modality-grounding label (whether the score depends on text, image, or both). (B) Three benchmark tasks. Task 1 evaluates scalar moral acceptability rating, which is compared to the mean human rating. Task 2 evaluates within-image moral ordering. Task 3 evaluates evidence-dependent rationale grounding: a verifier predicts which evidence source the rationale makes necessary for the verdict, and this predicted evidence label is compared with the human modality-grounding label.
Statistic
Value
Image contexts
8,444
Scenarios
21,977
Human judgment grounded on:
text
4,027
image
10,309
both
7,641
Table 1: MM-SCALE statistics. Modality and rounded mean-rating are computed over scenarios.
Task
Model output
Ground truth
Metrics
Evaluator
1. Scalar moral acceptability prediction
Rating y^i∈{1,…,5} for image–scenario pair (I,Si)
Mean human rating yˉi
aggregate NDCG@5, AUC-Acc, ECE
Deterministic comparison against human ratings
2. Within-image scenario ordering
Per-scenario ratings {y^i} for scenarios {S1,…,Sk} sharing image I
Pairwise human preference induced by yˉ within GI
Within-image pairwise accuracy at Δyˉ≥1 , modality-stratified pairwise accuracy
Image-conditioned verifier labels rationale as text , image , both , ungrounded , or hallucinated-visual
Table 2: MM-SCALE evaluation tasks. Tasks 1 and 2 compare model outputs deterministically against human scalar ratings and induced preferences over shared context. Task 3 uses an image-conditioned verifier to label the evidence source reasoning makes necessary for the verdict.
Open-weight
Proprietary
Metric
Prompt
LLaVA-OV
Qwen2.5-VL
Qwen3-VL
GPT-5.2
Claude Opus 4.7
Gemini 2.5 Pro
NDCG@5 ↑
ZS
0.937 [.931,.944]
0.942 [.936,.948]
0.941 [.935,.947]
0.944 [.935,.954]
0.946 [.936,.956]
0.948 [.939,.957]
ICL 3
0.936 [.930,.942]
0.941 [.934,.947]
0.939 [.933,.945]
0.950 [.940,.958]
0.948 [.938,.957]
0.951 [.941,.960]
CoT
0.937 [.931,.943]
0.937 [.930,.943]
0.939 [.933,.945]
0.948 [.938,.957]
0.943 [.933,.953]
0.941 [.931,.951]
Within-image pairwise acc. ( Δyˉ≥1 ) ↑
ZS
0.552 [.522,.584]
0.588 [.555,.620]
0.577 [.542,.610]
0.557 [.493,.617]
0.583 [.518,.641]
0.581 [.518,.646]
ICL 3
0.566 [.539,.596]
0.575 [.541,.610]
0.561 [.526,.594]
0.581 [.517,.646]
0.583 [.525,.645]
0.595 [.528,.660]
Table 3: Within-image scenario ordering across prompting regimes. We report aggregate NDCG@5 and within-image pairwise accuracy at Δyˉ≥1 (restricting to pairs whose human mean ratings differ by at least one point on the 1–5 scale; tied predictions count 0.5 ). Brackets are 95% bootstrap CIs over images ( B=2000 ). Aggregate NDCG@5 saturates at 0.93 – 0.95 across all six models, but within-image pairwise accuracy stays in 0.53 – 0.62 .
Variance filter
Human LOO [95% CI]
Model accuracy range
None
0.718[0.709,0.727]
0.530 – 0.620
Exclude SD >1.0
0.799[0.778,0.820]
0.583 – 0.664
Exclude top 10% by SD
0.747[0.726,0.768]
0.562 – 0.603
Exclude top 20% by SD
0.779[0.756,0.801]
0.579 – 0.658
Exclude rating range =4
0.738[0.716,0.758]
0.542 – 0.581
Table 4: Human leave-one-rater-out (LOO) reference and sensitivity to annotator rating disagreement on the matched within-image test evaluation. Ties receive 0.5 credit. Brackets denote 95% confidence intervals. The model range covers evaluated model configurations within each filtered subset. Filters are applied to scenario-level annotator ratings.
Figure 2: Modality-stratified within-image pairwise accuracy. (A) Same-modality vs. cross-modality accuracy by model. Same-modality pairs compare scenarios with the same human modality-grounding label ( T-T , I-I , B-B ); cross-modality pairs compare scenarios with different labels ( T-I , T-B , I-B ). (B) Accuracy by modality-pair type. Cross-modality accuracy is lower than same-modality accuracy, with the largest failures concentrated in T-I and I-B pairs, indicating that within-image ordering failures concentrate when scenarios depend on different evidence sources.
Model + Prompt
Grounding-F1 ↑
Over (%) ↓
Under (%) ↓
Hallu-Visual (%) ↓
LLaVA-OV CoT
0.260
44.3
55.0
5.7
Qwen2.5-VL CoT
0.254
36.1
64.4
3.7
Qwen3-VL CoT
0.284
41.5
54.4
1.9
GPT-5.2 CoT
0.252
34.0
65.8
0.7
Claude Opus 4.7 CoT
0.217
18.1
78.9
0.1
Gemini 2.5 Pro CoT
0.275
40.0
59.0
0.1
Table 5: Evidence-dependent rationale grounding on the MM-SCALE test split. The verifier predicts which evidence source the rationale makes necessary for the verdict, rather than which modality the rationale mentions. Grounding-F1 compares the verifier-predicted evidence label to the human modality-grounding label. Over-grounding is the fraction of human- text items whose rationales make visual or hallucinated visual evidence necessary for the verdict. Under-grounding is the fraction of human- image / both items whose rationales are judged text -grounded or ungrounded .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Judgment and modality collection page
Figure 4: Example annotation interface used in our MTurk task. Annotators were shown a target setting image and asked to rate the moral acceptability of multiple actions (scenarios) grounded in the image. They also selected which modality (image, text, or both) informed their decision.
Figure 5: Initial warning screen shown to participants before entering the annotation interface. The notice highlights that some scenarios may include sensitive or distressing content and emphasizes voluntary participation and informed consent.
Figure 6: Judgment interface used to collect scalar moral preferences and modality attribution. Participants rate the moral acceptability of a given action in context on a 5-point Likert scale and indicate whether their decision was based on the image, the text, or both
Figure 7: Input scenario creation page where participants propose morally relevant actions based on the displayed multimodal context. This step facilitates diverse scenario generation grounded in shared visual and textual prompts.
Audit axis
Metric / flag
Value
Corpus / filtering
Candidate generated image contexts
10,186
Image contexts retained after cleaning
8,444
Artifact audit
Minor distortions
9%
Lighting / texture issues
4%
Severe artifacts
2%
Artifact handling
Severe artifacts
Excluded or flagged for exclusion from visual analyses
Appendix
Table 6: Generated-image validity audits. We audit generated contexts for corpus filtering, visible artifacts, severe image failures, and robustness to synthetic-image style. The final released corpus retains 8,444 image contexts after filtering from 10,186 candidate generated contexts. Artifact rates are reported over the manually audited image subset.
Modality
↑ Accept.
Neutral
↓ Accept.
N
image
33.8%
18.1%
48.2%
10,309
text
34.5%
18.0%
47.5%
4,027
both
32.8%
17.6%
49.5%
7,641
Appendix
Table 7: Judgment shifts by attributed modality. Percentages are within each modality group. Shift directions compare the image-contextualized scalar judgment against the original text-only source label.
Model
Prompt
NDCG@5 ↑
Pairwise Acc. ↑
MAE ↓
AUC-Acc ↑
ECE ↓
(Δyˉ≥1)
Open-weight models
LLaVA-OV-7B
ZS
0.937 [.931,.944]
0.552 [.522,.584]
1.477
0.521
0.312
LLaVA-OV-7B
ICL 3
0.936 [.930,.942]
0.566 [.539,.596]
1.159
0.525
0.201
LLaVA-OV-7B
CoT
0.937 [.931,.943]
0.552 [.523,.584]
1.351
0.532
0.224
Qwen2.5-VL-7B
ZS
0.942 [.936,.948]
0.588 [.555,.620]
1.345
0.530
0.281
Appendix
Table 8: Full prompted evaluation on the MM-SCALE test split. NDCG@5 measures aggregate agreement between model-induced and human-rating-induced scenario rankings. Within-image pairwise accuracy at Δyˉ≥1 measures preference alignment on scenario pairs sharing the same image whose human mean ratings differ by at least one point on the 1–5 scale; tied predictions receive 0.5 credit, and brackets are 95% bootstrap CIs over images. MAE, AUC-Acc, and ECE are computed at the item level; AUC-Acc is the AUC of the low-acceptability classifier at threshold 2.5 (chance =0.5 ). Higher is better for NDCG@5, pairwise accuracy, and AUC-Acc; lower is better for MAE and ECE.
Model
T-T
I-I
B-B
T-I
T-B
I-B
Open-weight models
LLaVA-OV
.581 [.488,.669]
.556 [.507,.604]
.544 [.475,.610]
.478 [.379,.579]
.584 [.504,.660]
.502 [.439,.566]
Qwen2-VL
.581 [.488,.669]
.585 [.536,.632]
.553 [.485,.620]
.533 [.431,.631]
.634 [.554,.707]
.530 [.466,.593]
Qwen2.5-VL
.608 [.515,.694]
.604 [.556,.651]
.580 [.512,.645]
.516 [.416,.616]
.611 [.531,.685]
.517 [.453,.580]
Qwen3-VL
.599 [.506,.685]
.597 [.548,.644]
.558 [.490,.624]
.538 [.437,.636]
.604 [.524,.679]
.513 [.449,.576]
Proprietary models
Appendix
Table 9: Full modality-stratified within-image pairwise accuracy at Δyˉ≥1 . Each cell reports pairwise accuracy with 95% bootstrap CIs in brackets. Pairs are stratified by the human modality-grounding labels of the two scenarios. Same-modality pairs compare scenarios with the same evidence-source label (T-T, I-I, B-B); cross-modality pairs compare scenarios with different labels (T-I, T-B, I-B). Tied predictions receive 0.5 credit.
Model
Same-modality Δ (pp)
Cross-modality Δ (pp)
Qwen2.5-VL
−4.2
−2.0
LLaVA-OV
−0.6
+0.9
GPT-5.2
+1.0
+2.0
Claude Opus 4.7
−8.6
−0.6
Gemini 2.5 Pro
−4.0
−1.8
Appendix
Table 10: Change in within-image pairwise accuracy from zero-shot to standard CoT prompting, stratified by whether compared scenarios share an annotator-reported modality-grounding label. Entries are percentage points (CoT minus zero-shot).
Grounding-F1 ↑
Under-grounding ↓
Model
Direct
CoT
Direct
CoT
Qwen2.5-VL
.25
.27
70%
66%
LLaVA-OV
.25
.27
81%
77%
GPT-5.2
.24
.26
63%
60%
Claude Opus 4.7
.24
.23
57%
55%
Gemini 2.5 Pro
.27
.29
68%
64%
Appendix
Table 11: Matched direct-justification versus standard CoT comparison on 500 scenarios. Grounding-F1 and under-grounding are computed using the same evaluation procedure for both prompting conditions.
Evaluee
GPT-5.2 verifier
Claude Opus 4.7 verifier
LLaVA-OV
.271
.256
Qwen2.5-VL
.267
.249
Qwen3-VL
.296
.281
GPT-5.2
.264
.252
Claude Opus 4.7
.228
.236
Gemini 2.5 Pro
.286
.274
Appendix
Table 12: Grounding-F1 across evaluated VLMs when Task 3 outputs are re-scored using GPT-5.2 or Claude Opus 4.7 as rationale verifier.
Model + Prompt
Overall Shift (%) ↑
Text-grounded ↑
Image-grounded ↑
Both ↑
Hallu-Visual ↑
LLaVA-OV CoT
45.5
48.6
43.5
67.3
42.7
Qwen2-VL CoT
54.5
66.1
35.0
76.3
38.8
Qwen2.5-VL CoT
51.1
61.7
32.2
64.5
27.2
Qwen3-VL CoT
50.5
63.2
31.1
78.5
24.4
GPT-5.2 CoT
49.8
62.3
40.4
70.3
24.1
Claude Opus 4.7 CoT
58.8
63.1
34.5
78.8
22.1
Appendix
Table 13: Behavioral validation of evidence-dependent rationale labels. For each CoT trace, we ablate the evidence source that the verifier identifies as necessary for the verdict and re-query the model. We report the per-item rate at which the predicted rating shifts by at least one point on the 1–5 scale. Higher shift rates indicate stronger behavioral sensitivity to the identified evidence source. Per-modality columns are conditional on the verifier-predicted evidence label; Both OR-aggregates image and text ablations per item.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Moral reasoning is fundamental to safe Artificial Intelligence (AI), yet ensuring its consistency across modalities becomes critical as AI systems evolve from text-based assistants to embodied agents. Current safety techniques demonstrate success in textual contexts, but concerns remain about generalization to visual inputs. Existing moral evaluation benchmarks rely on textonly formats and lack systematic control over variables that influence moral decision-making. Here we show that visual inputs fundamentally alter moral decision-making in state-of-the-art (SOTA) Vision-Language Models (VLMs), bypassing text-based safety mechanisms. We introduce Moral Dilemma Simulation (MDS), a multimodal benchmark grounded in Moral Foundation Theory (MFT) that enables mechanistic analysis through orthogonal manipulation of visual and contextual variables. The evaluation reveals that the vision modality activates intuition-like pathways that override the more deliberate and safer reasoning patterns observed in text-only contexts. These findings expose critical fragilities where language-tuned safety filters fail to constrain visual processing, demonstrating the urgent need for multimodal safety alignment.
Xinyi Yang, Chenheng Xu, Weijun Hong +4
Institute for Artificial Intelligence, Peking University · School of Psychological and Cognitive Sciences, Peking University · State Key Lab of General Artificial Intelligence, Peking University +3
A human's moral decision depends heavily on the context. Yet research on LLM morality has largely studied fixed scenarios. We address this gap by introducing Contextual MoralChoice, a dataset of moral dilemmas with systematic contextual variations known from moral psychology to shift human judgment: consequentialist, emotional, and relational. Evaluating 22 LLMs, we find that nearly all models are context-sensitive, shifting their judgments towards rule-violating behavior. Comparing with a human survey, we find that models and humans are most triggered by different contextual variations, and that a model aligned with human judgments in the base case is not necessarily aligned in its contextual sensitivity. This raises the question of controlling contextual sensitivity, which we address with an activation steering approach that can reliably increase or decrease a model's contextual sensitivity. Code and data: https://github.com/adrian-sauter/contextual_moralchoice.
Multimodal large language models (MLLMs) are typically evaluated by whether they produce correct answers, but correctness alone does not reveal whether their outputs are supported by the right visual evidence. We introduce VIGOR, a benchmark for evaluating VIsually GrOunded Rationales in MLLMs. VIGOR represents visual rationales as explicit concepts paired with pixel-level groundings, covering multiple abstraction levels including low-level visual properties, intermediate structures, objects, parts and high-level motion, relations. To construct VIGOR at scale, we develop an automatic annotation framework that reuses existing dense supervision when available and generates missing concept groundings through proposal-based annotation followed by conservative multimodal verification. Using VIGOR, we evaluate whether MLLMs "look at what they say" by comparing model-derived visual evidence for each expressed concept against annotated ground-truth rationales. Our results show that current MLLMs exhibit limited visual rationale correctness and reveal distinct grounding failures across concept levels. VIGOR provides a benchmark and scalable data construction pipeline for studying whether MLLM outputs are supported by appropriate visual evidence. Code is available at https://github.com/deep-real/VIGOR