Organizations: School of Artificial Intelligence, Jilin University · International Center for Future Science, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China
As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.
Figures & tables
Benchmark
Proact.
MM
Vis. Att.
Detect
Diag.
Consist.
Strat.
Tax.
MoHoBench Zhu et al. (2025)
∙
∙
△
△
△
∘
R
4/-
Mis-prompt Zeng et al. (2025)
∙
∘
∘
∙
∙
∘
A, C
4/14
PCBench Li et al. (2025)
∙
∘
∘
∙
∙
△
-
4/-
ISEVAL Yang et al. (2025)
∙
∙
∘
∙
∙
∘
-
3/7
MiP Fan et al. (2025)
∙
∘
∘
∙
∘
△
R
1/4
QuestBench Wang et al. (2025)
∘
∘
∘
∙
∙
∘
C, R
1/4
Table 1: Comparison with related benchmarks. Proact. denotes the requirement for proactive error identification without explicit prompting, while MM indicates support for multimodal inputs. Vis. Att. involves visual premise manipulation, such as occlusion or noise. Regarding evaluation capabilities, Detect and Diag. measure error detection accuracy and diagnostic precision, respectively. Crucially, Consist. evaluates the logical consistency between the model’s internal reasoning and its final generated response. Strat. lists the remedial strategies covered ( C : Correction, Cl : Clarification, R : Refusal), and Tax. quantifies the granularity of the error taxonomy (Major Categories/Subcategories). Symbols: ∙ Fully supported, △ Partially supported, ∘ Not supported.
Figure 1: An illustrative failure case of multimodal reasoning under erroneous input. The input query requests the identification of an anatomical structure in the skull base, while the provided image is a purely geometric diagram, resulting in a semantic mismatch between textual premises and visual content. GPT-5.2 produces an overconfident but incorrect answer by ignoring the inconsistency. In contrast, Claude-Sonnet-4.5 (Thinking) detects the Mismatch error and requests clarification, highlighting the importance of Proactive Critique .
Figure 2: The Evaluation Pipeline. (a) Error Taxonomy: Classification of input defects into Expression Error, Premise Contradiction, Missing Premise, and Beyond Capability. (b) Construction Pipeline: A three-stage process featuring multi-source data sampling, multi-model error injection (12k samples), and automated/human filtration. (c) Evaluation Framework: Hierarchical assessment via EDA , DP , and SE metrics, integrated into the composite PCQ score using an LLM-as-a-Judge panel.
Models
Overall Performance
Category-wise PCQ (Score)
EDA (%)
DP (0-2)
SE (0-2)
PCQ
Expr.
Contr.
Miss.
Beyond
Closed-Source Models
Claude-Sonnet-4.5 (Thinking)
40.2
1.66
1.86
35.4
13.8
17.9
44.6
85.4
Gemini-3-Pro
37.4
1.46
1.79
30.4
13.6
39.2
24.9
64.2
Gemini-3-Flash
32.3
1.51
1.77
26.5
10.4
35.7
19.9
62.8
GPT-5.1 (high)
31.7
1.79
1.95
29.6
14.4
7.2
37.6
87.5
Table 2: Main Results of Proactive Critique Ability. Evaluation of 14 MLLMs. Metrics include Error Detection Accuracy (EDA, %) , Diagnostic Precision (DP, 0–2) , Strategic Effectiveness (SE, 0–2) , and the composite Proactive Critique Quality (PCQ, 0–100) . The right block details PCQ scores across four error categories: Expression Error (Expr.) , Premise Contradiction (Contr.) , Missing Premise (Miss.) , and Beyond Capability (Beyond) . Bold denotes the best result globally, and underlined denotes the second best. The bottom row shows the average performance across all evaluated models.
Model
EDA R ( ↑ )
EDA F ( ↑ )
DDR ( ↓ )
DQD ( ↓ )
Claude-Sonnet-4.5 (Thinking)
0.51
0.40
0.28
0.11
Gemini-3-Pro
0.51
0.37
0.34
0.17
Gemini-3-Flash
0.47
0.32
0.37
0.17
GLM-4.6V
0.42
0.16
0.63
0.19
Doubao-Seed-1.6-vision
0.44
0.15
0.68
0.37
Average
0.47
0.28
0.46
0.20
Table 3: Consistency Analysis between Reasoning and Response. EDAR and EDAF denote Error Detection Accuracy on the reasoning chain and final response, respectively. DDR (Detection Drop Rate): proportion of reasoning-detected errors absent from the final response (lower is better). DQD (Diagnostic Quality Drop): average decline in diagnostic precision when both components detect the error (lower is better).
Figure 3: Distribution of response strategies across 14 MLLMs. The chart illustrates the breakdown of Correction (Blue), Clarification (Orange), Refusal (Red), Unsure (Teal), and None (Grey). Values under 4.5% are omitted for visual clarity.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
Error Category
# of Instance
Expression Error
584
- Ambiguity
294
- Unclear Intent
290
Premise Contradiction
862
- Cross-Modal Contradiction
284
- Inter-Visual Contradiction
288
Appendix
Table 4: Dataset composition and coverage. Counts report the number of benchmark instances by primary error type, fine-grained subtype, and subject/domain.
Human
H-LLM
H-LLM
Metric
Fleiss’ κ
Cohen’s κ
Acc.
EDA
0.79
0.77
88%
DP
0.75
0.71
75%
SE
0.72
0.67
73%
Overall
0.76
0.71
78%
Appendix
Table 5: Per-metric agreement between human annotators and the LLM-as-a-Judge panel on 180 stratified samples. Human Fleiss’ κ measures inter-annotator agreement; H-LLM Cohen’s κ and H-LLM Acc. measure agreement and accuracy against the majority-vote human consensus.
Model
PCQ
95% CI
Sig.
Qwen3-VL-235B-A22B-Instruct
47.4
[45.9, 48.9]
∗
Qwen3-VL-30B-A3B-Instruct
40.4
[38.9, 41.9]
∗
Qwen3-VL-8B-Instruct
38.7
[37.1, 40.2]
∗
Claude-Sonnet-4.5 (Thinking)
35.4
[33.6, 37.0]
∗
Gemini-3-Pro
30.4
[28.9, 31.9]
n.s. ( p=0.37 )
GPT-5.1 (high)
29.6
[28.1, 31.2]
∗
Appendix
Table 6: PCQ scores with 95% bootstrap confidence intervals ( B=1000 ). The Sig. column reports pairwise bootstrap significance between each model and the next-ranked model: ∗p<0.05 (significant gap); n.s. p≥0.05 (non-significant gap); — for the last-ranked model.
Model
Prompt
EDA(%)
PCQ
Δ EDA
Δ PCQ
Qwen3-VL-235B-A22B-Instruct
orig.
55.3
37.0
− 0.0
− 0.0
formal
55.3
39.2
− 0.0
− 2.2
concise
52.7
37.6
− 2.7
− 0.5
GPT-5-mini
orig.
18.3
17.5
− 0.0
− 0.0
formal
20.0
18.2
− 1.7
− 0.7
concise
19.7
16.5
− 1.3
− 1.0
Appendix
Table 7: Prompt sensitivity of three representative models on a stratified 300 sample subset. Orig. replicates the main-experiment prompt; formal rewrites it in an academic register; concise reduces it to a minimal direct instruction. Δ EDA and Δ PCQ are absolute differences (in percentage points) relative to orig. per model.
Reasoning effort
EDAR
EDAF
DDR
DQD
Low
47.7%
39.3%
27.0%
-0.10
Medium
53.0%
41.3%
25.7%
0.02
High
53.7%
46.7%
19.0%
0.04
Appendix
Table 8: Reasoning-effort ablation for Claude-Sonnet-4.5 (Thinking), averaged over three maximum-token budgets and 120 samples. EDAR and EDAF are reasoning- and final-response Error Detection Accuracy; DDR and DQD are Detection Drop Rate and Diagnostic Quality Drop.
Reasoning effort
Maximum budget
EDAR
EDAF
DDR
DQD
High
4,096
54%
48%
18%
0.06
High
8,192
54%
45%
21%
0.00
High
16,384
53%
47%
18%
0.06
Appendix
Table 9: Reasoning-budget ablation for Claude-Sonnet-4.5 (Thinking) at high effort on 120 samples. EDAR and EDAF are reasoning- and final-response Error Detection Accuracy; DDR and DQD are Detection Drop Rate and Diagnostic Quality Drop.
Model
EDA/Recall
FAR
Precision
F1
PCQ-F1
Qwen3-VL-235B-A22B-Instruct
58.8
18.8
75.8
66.3
52.1
Claude-Sonnet-4.5 (Thinking)
39.6
1.56
96.2
56.1
49.5
Qwen3-VL-30B-A3B-Instruct
55.7
20.3
73.3
63.3
47.6
Gemini-3-Pro
38.3
1.25
96.8
54.9
45.3
Qwen3-VL-8B-Instruct
52.6
21.8
70.7
60.3
44.2
Gemini-3-Flash
34.3
2.03
94.4
50.3
41.7
Appendix
Table 10: False-alarm analysis on the filtered valid/error paired split. EDA/Recall is Error Detection Accuracy on error inputs; FAR, Precision, F1, and PCQ-F1 denote False-Alarm Rate, precision, F1, and the PCQ-weighted F1 score.
Model
Valid-input URR-V (%) ↓
Qwen3-VL-235B-A22B-Instruct
0.16
Qwen3-VL-30B-A3B-Instruct
0.16
Qwen3-VL-8B-Instruct
0.00
Claude-Sonnet-4.5 (Thinking)
0.00
Gemini-3-Pro
0.00
Gemini-3-Flash
0.00
Appendix
Table 11: Unfounded refusal rate on the 639-item valid-input split (URR-V). The table reports the percentage of valid inputs that received a full refusal without an underlying error.
Model
Flawed-input Refusal (%)
Qwen3-VL-235B-A22B-Instruct
13.57
Qwen3-VL-8B-Instruct
13.41
Qwen3-VL-30B-A3B-Instruct
9.28
GPT-5.1 (high)
8.90
Claude-Sonnet-4.5 (Thinking)
8.17
GPT-5.2
7.53
Appendix
Table 12: Full-refusal rates on the 3,146 flawed inputs. Values denote the proportion of responses assigned the refusal strategy.
Judge
Correction
Clarification
Refusal
GPT-5.2
1.28
1.90
1.87
Gemini-3-Pro
1.79
1.99
1.98
Claude-Sonnet-4
1.66
1.93
1.88
Appendix
Table 13: Mean Strategic Effectiveness (SE) scores assigned to clarification, correction, and refusal responses by the three LLM judges.
Response model
GPT judge
Gemini judge
Claude judge
GPT-5.1 (high)
1.718
1.873
1.562
GPT-5-mini
1.683
1.825
1.451
GPT-5.2
1.620
1.770
1.472
Gemini-3-Pro
1.565
1.708
1.199
Gemini-3-Flash
1.601
1.755
1.245
Claude-Sonnet-4.5 (Thinking)
1.548
1.820
1.544
Appendix
Table 14: Judge robustness for Diagnostic Precision (DP). Rows identify the model that generated the response and columns identify the judge model; entries are mean DP scores on the reported evaluation subset (0–2 scale).
Model
MEDA
DP-cov.
SE
PCQ
Type cov.
All det.
Partial det.
Qwen3-VL-8B-Instruct
76
1.114
1.105
42.2
44
16
72
Qwen3-VL-30B-A3B-Instruct
74
1.157
0.889
37.6
46
27
65
Qwen3-VL-235B-A22B-Instruct
74
1.090
0.811
35.2
42
16
68
GPT-5-mini
46
1.087
1.261
27.0
25
8
42
Gemini-3-Flash
30
1.178
1.200
17.8
19
10
28
Appendix
Table 15: Results on the 50-sample dual-error subset. MEDA denotes Multi-Error Detection Accuracy; DP-cov., Diagnostic-Point Coverage; SE, Strategic Effectiveness; and PCQ, Proactive Critique Quality. Type cov. is average ground-truth error-type coverage; All det. and Partial det. indicate detection of both and at least one ground-truth error type, respectively.
Model
Expression Error
Premise Contradiction
Missing Premise
Beyond Capability
Ambiguity
Unclear Intent
Cross-Modal
Inter-Visual
Intra-Textual
Visual Light
Visual Medium
Visual Severe
Missing Image
Missing Instruction
Mismatch
Beyond Capability
Claude-Sonnet-4.5 (Thinking)
15.7
21.0
21.8
3.1
41.0
8.0
11.0
19.8
70.7
69.7
74.0
87.1
Doubao-Seed-1.6-vision
5.1
5.5
17.6
2.4
24.1
9.3
7.5
5.0
9.3
6.3
26.7
50.2
GLM-4.6V
2.7
7.9
11.3
1.0
16.9
4.0
4.0
2.2
40.3
8.3
18.0
58.0
GPT-5-mini
7.1
24.1
9.5
1.0
15.9
5.3
8.7
14.8
44.7
46.3
30.3
67.5
GPT-5.1 (high)
5.4
28.6
10.6
0.4
17.9
12.7
8.1
15.9
47.0
63.0
53.0
89.1
Appendix
Table 16: Error Detection Accuracy (EDA, %) across fine-grained error subtypes and four primary error categories.
Model
Expression Error
Premise Contradiction
Missing Premise
Beyond Capability
Ambiguity
Unclear Intent
Cross-Modal
Inter-Visual
Intra-Textual
Visual Light
Visual Medium
Visual Severe
Missing Image
Missing Instruction
Mismatch
Beyond Capability
Claude-Sonnet-4.5 (Thinking)
1.09
1.45
1.41
1.52
1.63
0.89
0.89
1.66
1.87
1.25
1.88
1.95
Doubao-Seed-1.6-vision
0.76
0.88
1.35
0.38
1.29
0.21
0.21
0.56
0.71
1.14
1.25
1.60
GLM-4.6V
0.67
0.68
1.38
0.11
1.32
0.39
0.10
0.08
1.72
1.11
1.39
1.85
GPT-5-mini
1.19
1.52
1.11
1.56
1.23
1.88
1.93
1.91
1.98
1.65
1.91
1.91
GPT-5.1 (high)
1.00
1.63
1.32
0.33
1.21
1.65
1.86
1.91
1.99
1.65
1.97
1.94
Appendix
Table 17: Diagnostic Precision (DP) scores across fine-grained error subtypes and four primary error categories (0–2 scale).
Model
Expression Error
Premise Contradiction
Missing Premise
Beyond Capability
Ambiguity
Unclear Intent
Cross-Modal
Inter-Visual
Intra-Textual
Visual Light
Visual Medium
Visual Severe
Missing Image
Missing Instruction
Mismatch
Beyond Capability
Claude-Sonnet-4.5 (Thinking)
1.66
1.74
1.61
1.83
1.72
1.39
1.32
1.81
1.89
1.97
1.91
1.98
Doubao-Seed-1.6-vision
1.48
1.32
1.69
1.24
1.58
1.25
0.91
1.39
1.24
1.73
1.42
1.79
GLM-4.6V
1.56
1.17
1.47
0.33
1.67
0.78
0.98
1.12
1.79
1.55
1.50
1.92
GPT-5-mini
1.75
1.91
1.80
2.00
1.75
2.00
1.98
2.00
1.99
1.94
1.98
1.95
GPT-5.1 (high)
1.57
1.92
1.80
0.00
1.78
1.84
2.00
1.97
1.98
1.99
1.98
1.98
Appendix
Table 18: Strategic Effectiveness (SE) scores across fine-grained error subtypes and primary error categories (0–2 scale).
Model
Expression Error
Premise Contradiction
Missing Premise
Beyond Capability
Ambiguity
Unclear Intent
Cross-Modal
Inter-Visual
Intra-Textual
Visual Light
Visual Medium
Visual Severe
Missing Image
Missing Instruction
Mismatch
Beyond Capability
Claude-Sonnet-4.5 (Thinking)
10.8
16.8
16.5
2.6
34.4
4.6
6.1
17.2
66.3
56.2
70.3
85.4
Doubao-Seed-1.6-vision
2.9
3.0
13.4
1.0
17.3
3.4
2.1
2.4
4.6
4.5
17.8
42.6
GLM-4.6V
1.5
3.7
8.0
0.1
12.6
1.2
1.1
0.7
35.4
5.5
13.0
54.6
GPT-5-mini
5.2
20.7
6.9
0.9
11.8
5.2
8.5
14.5
44.3
41.5
29.5
65.0
GPT-5.1 (high)
3.5
25.4
8.2
0.0
13.4
11.1
7.8
15.4
46.7
57.4
52.3
87.5
Appendix
Table 19: Proactive Critique Quality (PCQ, %) across fine-grained error subtypes and primary error categories.
Figure 4: Per-subcategory PCQ (%) across 12 subcategories, grouped into three modality regions based on which modality carries the error signal (Text-only, Cross-modal, Visual-only). Each colored line connects the within-family average across subcategories, and the black dashed line shows the overall average across all 14 models. Subcategory names follow the format "Primary Error Type. Subcategory Name" (e.g., Expr. Ambiguity for Expression Error, Ambiguity subcategory).
Figure 5: An example of a failed response to an Ambiguity sample by GPT-5.2. The red text in the Question indicates the segment containing the ambiguity part.
Figure 6: An example of a successful response to a Beyond Capability sample by GPT-5-mini. The red text in the Question indicates the segment containing the beyond capability part. The blue text indicates the key part that highlights the presence of successful error detection.
Figure 7: An example generated by Claude-Sonnet-4.5 (Thinking) in which the internal reasoning and final response do not align. The final response and internal reasoning address a Cross-Modal Contradiction sample. The red text in the Question indicates the segment containing the cross-modal contradiction. The blue text in Chain of Thought indicates the key part that highlights successful error detection.
Figure 8: Prompt for evaluating Error Detection Accuracy
Figure 9: Prompt for evaluating Diagnostic Precision
Figure 10: Prompt for evaluating Strategic Effectiveness
Figure 11: Prompt for generating samples of the Ambiguity type
Figure 12: Prompt for generating samples of the Beyond Capability type
Figure 13: Prompt for generating samples of the Inter-Visual Contradiction type
Figure 14: Prompt for generating samples of the Intra-Textual Contradiction type
Figure 15: Prompt for generating samples of the Cross-Modal Contradiction type
Figure 16: Prompt for generating samples of the Unclear Intent type
Data Category
Question
Error Reason
Ambiguity
Instruction: Can you identify the pathogen affecting the potato sample located on the side in Image ? Image:
The problem lacks a necessary premise by failing to specify which potato sample is being referred to. The phrase "the potato sample located on the side" is ambiguous because it does not indicate left/right or a specific position. The ambiguity cannot be resolved from the image, since there are samples on both sides.
Beyond Capability
Instruction: Delete the minimum number from the given leftist heap. Which one of the following statements is TRUE about the exact wall-clock time taken by that deletion on the computer used to generate this diagram? Image:
The problem asks for the wall-clock time of a delete operation on the computer that produced the diagram, but that cannot be determined from the diagram alone because it provides no information about the machine’s hardware, software environment, or execution logs.
Intra-Textual Contradiction
Instruction: Which of the following responses will an increase in government spending to boost Aggregate Demand (AD), specifically defined as a reduction in government expenditure , as demonstrated in Image , trigger according to the theory of rational expectations? Image:
The problem text contains an internal contradiction by presenting both "increase government spending" and "decrease government spending" as definitions for the same policy action (mutually exclusive and yielding opposite answers); this contradiction is explicit in the text (verifiable without the image), both premises are stated on equal footing with no priority to resolve, and the direction of fiscal spending is a necessary premise for the AD response under rational expectations, so the question cannot be answered without clarifying which direction is intended.
Appendix
Table 20: Examples of Ambiguity, Beyond Capability, and Intra-Textual Contradiction. Red text marks the error-containing segment in the question.
Data Category
Question
Error Reason
Inter-Visual Contradiction
Instruction: In an aluminum (Al) bar of square cross section, a square hole is drilled and is filled with iron (Fe) as shown in the figure. The electrical resistivities of Al and Fe are 2.7×10−8Ωm and 1.0×10−7Ωm , respectively. The electrical resistance between the two faces P and Q of the composite bar is Image1 , Image2 Image1: Image2:
The two figures show inconsistent length labels (50 mm vs 66 mm ); since R=ρL/A and resistance is proportional to length, changing L alters the computed resistance and yields different answers
Missing Instruction
Image:
The problem references textual premise but omits essential textual question which leads to unanswerable question.
Missing Image
Instruction: Compute the reactions at supports A and E of the three-hinged parabolic arch in Figure P6.23. Next compute the axial load at points B and D, located at the quarter points.
The problem references "Figure P6.23" but omits essential numerical and structural data—arch geometry (span and rise), load magnitudes and distributions, and hinge/joint positions—without which the quantitative calculation of support reactions for a three-hinged parabolic arch and the axial force at the quarter point cannot be performed.
Appendix
Table 21: Examples of Inter-Visual Contradiction, Missing Instruction, and Missing Image.
Data Category
Question
Error Reason
Missing Visual Premise Light
Instruction: Image Calculate the missing item. Image:
The image’s central region is significantly occluded, obscuring critical structural connections and key-type information required to determine the number of keys; this essential visual premise cannot be inferred from the text or the remaining image, and no answer choices allow the missing information to be deduced.
Missing Visual Premise Medium
Instruction: Payback Consider the following projects: Image Calculate the discounted payback period for each project Image:
The cash-flow table in the image is damaged/blurred, rendering multiple key numeric values unreadable (e.g., projects A and B initial cash flows C0, several period cash flows, and certain column headers/symbols) that are required to calculate the discounted payback period. The problem statement does not provide a discount rate. The missing information cannot be reliably inferred from the text, the remaining visible data, or the provided answer options.
Missing Visual Premise Severe
Instruction: Room registrations in the Toronto Towers Plaza Hotel have been recorded for the past 9 years. To project future occupancy, management would like to determine the mathematical trend of guest registration. This estimate will help the hotel determine whether future expansion will be needed. Given the following time-series data, develop a regression equation relating registrations to time (e.g., a trend equation). Then forecast year 11 registrations. Room registrations are in the thousands: Image Image:
The time-series table is severely corrupted: several yearly registration values and labels are unreadable. These values are required to fit the regression trend and forecast year 11, so the requested equation and forecast cannot be determined reliably from the remaining image.
Appendix
Table 22: Examples of Missing Visual Premise at three information-loss levels: Light, Medium, and Severe.
Data Category
Question
Error Reason
Cross-Modal Contradiction
Instruction: Blood glucose level is regulated by two antagonistic hormones, glucagon and insulin. The following graph shows the fluctuations in an individual’s blood glucose level over a five hour period, where the blood glucose level at point B is lower than that at point E : Image Image:
The description "B is lower than E" directly contradicts the image, which shows B≈1.6mg/ml and E≈1.2mg/ml . The relative heights in the image are a necessary visual premise that affects subsequent judgments, and the problem does not specify whether the text or the image takes precedence.
Mismatch
Instruction: In the Base of Skull, identify the 205 structure. Image Image:
The question requires identifying the structure labeled "205" on a skull - base image, but the provided image is unrelated to the skull base and its labels do not match, so the question cannot be answered.
Unclear Intent
Instruction: Janet Ludlow’s firm requires all its analysts to use a two-stage dividend discount model (DDM) and the capital asset pricing model (CAPM) to value stocks. Using the CAPM and DDM, Ludlow has valued QuickBrush Company at $63 per share. She now must value SmileWhite Corporation. Image Image:
The question omits a specific task instruction, creating ambiguity about whether to calculate the CAPM required return or to value intrinsic value; additionally, the dividend data needed to compute intrinsic value are missing.
Appendix
Table 23: Examples of Cross-Modal Contradiction, Mismatch, and Unclear Intent. Red text marks the error-containing segment in the question.
Table 24: Model provenance and access links. A check mark in the Reasoning column indicates that the official documentation or model card provides an explicit reasoning/thinking mode; a cross indicates no dedicated reasoning mode is documented.
Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can MLLMs exhibit a similar "proactive" behavior by requesting simple user interventions? To investigate this, we introduce ProactiveBench, a benchmark built from seven repurposed datasets that tests proactiveness across different tasks such as recognizing occluded objects, enhancing image quality, and interpreting coarse sketches. We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains. Surprisingly, we found that conversation histories and in-context learning introduce negative biases, hindering performance. Finally, we explore a simple fine-tuning strategy based on reinforcement learning: its results suggest that proactiveness can be learned, even generalizing to unseen scenarios. We publicly release ProactiveBench as a first step toward building proactive multimodal models.
Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2
University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1
Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and introduce MM-JudgeBias, a benchmark for evaluating it. MM-JudgeBias introduces controlled perturbations across Query, Image, and Response, and evaluates model behavior via two complementary metrics: Bias-Deviation (BD) for sensitivity and Bias-Conformity (BC) for stability. Our dataset of over 1,800 curated and refined multimodal samples, drawn from 29 source benchmarks, enables a fine-grained diagnosis of nine bias types across diverse tasks and domains. Experiments on 26 state-of-the-art MLLMs reveal systematic modality neglect and asymmetric evaluation tendencies, underscoring the need for more reliable judges.
Sua Lee, Sanghee Park, Jinbae Im
Seoul National University · NAVER Cloud AI · KAIST AI
Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommendation and test whether they can recognize flawed recommendation requests. Existing recommender benchmarks mainly assess ranking, generation, or preference satisfaction, while existing error-detection benchmarks are usually not grounded in recommendation-specific user and candidate evidence. To address this gap, we introduce RPCBench, a benchmark for evaluating Recommender-Premise Critique: the ability to detect, diagnose, and properly handle faulty premises in natural-language recommendation requests. RPCBench contains evidence-grounded test instances from five recommendation domains and covers ten types of premise failures. Each instance provides a visible recommendation context and a corrupted user query. We further design a fine-grained evaluation framework that measures proactive detection, error localization, post-detection handling strategy, and evidence faithfulness. Through a systematic evaluation of 11 LLMs, we find that proactive detection is the main bottleneck in Recommender-Premise Critique, and models perform worst on underspecified-premise errors. We also observe that target-critical information density matters more than redundant evidence, and that longer reasoning does not monotonically improve critique quality: performance peaks at intermediate reasoning length, while overly long reasoning is accompanied by an overthinking penalty. The code is available at https://github.com/ZhongruChen/RPCBench.
Zhongru Chen, Yuan Wu, Yi Chang
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University