MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

Jinzhe Li; Gengxu Li; Jinnan Li; Yuan Wu; Yi Chang

cs.AIAug 29, 2026

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

Authors: Jinzhe Li, Gengxu Li, Jinnan Li, Yuan Wu, Yi Chang

Organizations: School of Artificial Intelligence, Jilin University · International Center for Future Science, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China

Abstract

As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructions but also on validating them. We define Proactive Critique as the model's autonomous ability to identify, analyze and fix faulty user inputs without extra prompts. However, evaluations mainly test models under ideal circumstances or simple refusal behaviors, largely ignoring active error processing. To fill this gap, we propose MMPCBench, a comprehensive framework for evaluating MLLMs' proactive critique competence. It features a fine-grained taxonomy of 4 primary error types spanning 12 subcategories, ranging from cross-modal contradictions to missing visual premises. We adopt a hierarchical evaluation protocol to measure models' error detection, diagnosis and resolution performance, and apply alignment-aware metrics to assess the coherence between internal reasoning and final responses. Tests on 14 mainstream MLLMs show obvious weaknesses in proactive critique, especially in dealing with subtle visual anomalies. Notably, we identify a pervasive "consistency gap": reasoning models can often correctly identify and analyze errors during internal reasoning yet suppress these valid insights in final outputs to prioritize response compliance. The code and data is available at https://github.com/ALIENS32/MMPCBench.

Figures & tables

Benchmark	Proact.	MM	Vis. Att.	Detect	Diag.	Consist.	Strat.	Tax.
MoHoBench Zhu et al. (2025)	$\bullet$	$\bullet$	$\triangle$	$\triangle$	$\triangle$	$\circ$	R	4/-
Mis-prompt Zeng et al. (2025)	$\bullet$	$\circ$	$\circ$	$\bullet$	$\bullet$	$\circ$	A, C	4/14
PCBench Li et al. (2025)	$\bullet$	$\circ$	$\circ$	$\bullet$	$\bullet$	$\triangle$	-	4/-
ISEVAL Yang et al. (2025)	$\bullet$	$\bullet$	$\circ$	$\bullet$	$\bullet$	$\circ$	-	3/7
MiP Fan et al. (2025)	$\bullet$	$\circ$	$\circ$	$\bullet$	$\circ$	$\triangle$	R	1/4
QuestBench Wang et al. (2025)	$\circ$	$\circ$	$\circ$	$\bullet$	$\bullet$	$\circ$	C, R	1/4

Table 1: Comparison with related benchmarks. Proact. denotes the requirement for proactive error identification without explicit prompting, while MM indicates support for multimodal inputs. Vis. Att. involves visual premise manipulation, such as occlusion or noise. Regarding evaluation capabilities, Detect and Diag. measure error detection accuracy and diagnostic precision, respectively. Crucially, Consist. evaluates the logical consistency between the model’s internal reasoning and its final generated response. Strat. lists the remedial strategies covered ( C : Correction, Cl : Clarification, R : Refusal), and Tax. quantifies the granularity of the error taxonomy (Major Categories/Subcategories). Symbols:

\bullet

Fully supported,

\triangle

Partially supported,

\circ

Not supported.

Figure 1: An illustrative failure case of multimodal reasoning under erroneous input. The input query requests the identification of an anatomical structure in the skull base, while the provided image is a purely geometric diagram, resulting in a semantic mismatch between textual premises and visual content. GPT-5.2 produces an overconfident but incorrect answer by ignoring the inconsistency. In contrast, Claude-Sonnet-4.5 (Thinking) detects the Mismatch error and requests clarification, highlighting the importance of Proactive Critique .

Figure 2: The Evaluation Pipeline. (a) Error Taxonomy: Classification of input defects into Expression Error, Premise Contradiction, Missing Premise, and Beyond Capability. (b) Construction Pipeline: A three-stage process featuring multi-source data sampling, multi-model error injection (12k samples), and automated/human filtration. (c) Evaluation Framework: Hierarchical assessment via EDA , DP , and SE metrics, integrated into the composite PCQ score using an LLM-as-a-Judge panel.

Closed-Source Models
Models	Overall Performance				Category-wise PCQ (Score)
Models	EDA (%)	DP (0-2)	SE (0-2)	PCQ	Expr.	Contr.	Miss.	Beyond
Claude-Sonnet-4.5 (Thinking)	40.2	1.66	1.86	35.4	13.8	17.9	44.6	85.4
Gemini-3-Pro	37.4	1.46	1.79	30.4	13.6	39.2	24.9	64.2
Gemini-3-Flash	32.3	1.51	1.77	26.5	10.4	35.7	19.9	62.8
GPT-5.1 (high)	31.7	1.79	1.95	29.6	14.4	7.2	37.6	87.5

Table 2: Main Results of Proactive Critique Ability. Evaluation of 14 MLLMs. Metrics include Error Detection Accuracy (EDA, %) , Diagnostic Precision (DP, 0–2) , Strategic Effectiveness (SE, 0–2) , and the composite Proactive Critique Quality (PCQ, 0–100) . The right block details PCQ scores across four error categories: Expression Error (Expr.) , Premise Contradiction (Contr.) , Missing Premise (Miss.) , and Beyond Capability (Beyond) . Bold denotes the best result globally, and underlined denotes the second best. The bottom row shows the average performance across all evaluated models.

Model	EDA R ( $\uparrow$ )	EDA F ( $\uparrow$ )	DDR ( $\downarrow$ )	DQD ( $\downarrow$ )
Claude-Sonnet-4.5 (Thinking)	0.51	0.40	0.28	0.11
Gemini-3-Pro	0.51	0.37	0.34	0.17
Gemini-3-Flash	0.47	0.32	0.37	0.17
GLM-4.6V	0.42	0.16	0.63	0.19
Doubao-Seed-1.6-vision	0.44	0.15	0.68	0.37
Average	0.47	0.28	0.46	0.20

Table 3: Consistency Analysis between Reasoning and Response.

\text{EDA}^{R}

and

\text{EDA}^{F}

denote Error Detection Accuracy on the reasoning chain and final response, respectively. DDR (Detection Drop Rate): proportion of reasoning-detected errors absent from the final response (lower is better). DQD (Diagnostic Quality Drop): average decline in diagnostic precision when both components detect the error (lower is better).

Figure 3: Distribution of response strategies across 14 MLLMs. The chart illustrates the breakdown of Correction (Blue), Clarification (Orange), Refusal (Red), Unsure (Teal), and None (Grey). Values under 4.5% are omitted for visual clarity.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Error Category	# of Instance
Expression Error	584
- Ambiguity	294
- Unclear Intent	290
Premise Contradiction	862
- Cross-Modal Contradiction	284
- Inter-Visual Contradiction	288

Appendix

Table 4: Dataset composition and coverage. Counts report the number of benchmark instances by primary error type, fine-grained subtype, and subject/domain.

	Human	H-LLM	H-LLM
Metric	Fleiss’ $\kappa$	Cohen’s $\kappa$	Acc.
EDA	0.79	0.77	88%
DP	0.75	0.71	75%
SE	0.72	0.67	73%
Overall	0.76	0.71	78%

Appendix

Table 5: Per-metric agreement between human annotators and the LLM-as-a-Judge panel on 180 stratified samples. Human Fleiss’

\kappa

measures inter-annotator agreement; H-LLM Cohen’s

\kappa

and H-LLM Acc. measure agreement and accuracy against the majority-vote human consensus.

Model	PCQ	95% CI	Sig.
Qwen3-VL-235B-A22B-Instruct	47.4	[45.9, 48.9]	∗
Qwen3-VL-30B-A3B-Instruct	40.4	[38.9, 41.9]	∗
Qwen3-VL-8B-Instruct	38.7	[37.1, 40.2]	∗
Claude-Sonnet-4.5 (Thinking)	35.4	[33.6, 37.0]	∗
Gemini-3-Pro	30.4	[28.9, 31.9]	n.s. ( $p=0.37$ )
GPT-5.1 (high)	29.6	[28.1, 31.2]	∗

Appendix

Table 6: PCQ scores with 95% bootstrap confidence intervals (

B=1000

). The Sig. column reports pairwise bootstrap significance between each model and the next-ranked model:

{}^{*}\,p<0.05

(significant gap); n.s.

p\geq 0.05

(non-significant gap); — for the last-ranked model.

Model	Prompt	EDA(%)	PCQ	$\Delta$ EDA	$\Delta$ PCQ
Qwen3-VL-235B-A22B-Instruct	orig.	55.3	37.0	$-$ 0.0	$-$ 0.0
	formal	55.3	39.2	$-$ 0.0	$-$ 2.2
	concise	52.7	37.6	$-$ 2.7	$-$ 0.5
GPT-5-mini	orig.	18.3	17.5	$-$ 0.0	$-$ 0.0
	formal	20.0	18.2	$-$ 1.7	$-$ 0.7
	concise	19.7	16.5	$-$ 1.3	$-$ 1.0

Appendix

Table 7: Prompt sensitivity of three representative models on a stratified 300 sample subset. Orig. replicates the main-experiment prompt; formal rewrites it in an academic register; concise reduces it to a minimal direct instruction.

\Delta

EDA and

\Delta

PCQ are absolute differences (in percentage points) relative to orig. per model.

Reasoning effort	$\mathrm{EDA}^{R}$	$\mathrm{EDA}^{F}$	$\mathrm{DDR}$	$\mathrm{DQD}$
Low	47.7%	39.3%	27.0%	-0.10
Medium	53.0%	41.3%	25.7%	0.02
High	53.7%	46.7%	19.0%	0.04

Appendix

Table 8: Reasoning-effort ablation for Claude-Sonnet-4.5 (Thinking), averaged over three maximum-token budgets and 120 samples.

\mathrm{EDA}^{R}

and

\mathrm{EDA}^{F}

are reasoning- and final-response Error Detection Accuracy; DDR and DQD are Detection Drop Rate and Diagnostic Quality Drop.

Reasoning effort	Maximum budget	$\mathrm{EDA}^{R}$	$\mathrm{EDA}^{F}$	$\mathrm{DDR}$	$\mathrm{DQD}$
High	4,096	54%	48%	18%	0.06
High	8,192	54%	45%	21%	0.00
High	16,384	53%	47%	18%	0.06

Appendix

Table 9: Reasoning-budget ablation for Claude-Sonnet-4.5 (Thinking) at high effort on 120 samples.

\mathrm{EDA}^{R}

and

\mathrm{EDA}^{F}

are reasoning- and final-response Error Detection Accuracy; DDR and DQD are Detection Drop Rate and Diagnostic Quality Drop.

Model	EDA/Recall	FAR	Precision	F1	PCQ-F1
Qwen3-VL-235B-A22B-Instruct	58.8	18.8	75.8	66.3	52.1
Claude-Sonnet-4.5 (Thinking)	39.6	1.56	96.2	56.1	49.5
Qwen3-VL-30B-A3B-Instruct	55.7	20.3	73.3	63.3	47.6
Gemini-3-Pro	38.3	1.25	96.8	54.9	45.3
Qwen3-VL-8B-Instruct	52.6	21.8	70.7	60.3	44.2
Gemini-3-Flash	34.3	2.03	94.4	50.3	41.7

Appendix

Table 10: False-alarm analysis on the filtered valid/error paired split. EDA/Recall is Error Detection Accuracy on error inputs; FAR, Precision, F1, and PCQ-F1 denote False-Alarm Rate, precision, F1, and the PCQ-weighted F1 score.

Model	Valid-input URR-V (%) $\downarrow$
Qwen3-VL-235B-A22B-Instruct	0.16
Qwen3-VL-30B-A3B-Instruct	0.16
Qwen3-VL-8B-Instruct	0.00
Claude-Sonnet-4.5 (Thinking)	0.00
Gemini-3-Pro	0.00
Gemini-3-Flash	0.00

Appendix

Table 11: Unfounded refusal rate on the 639-item valid-input split (URR-V). The table reports the percentage of valid inputs that received a full refusal without an underlying error.

Model	Flawed-input Refusal (%)
Qwen3-VL-235B-A22B-Instruct	13.57
Qwen3-VL-8B-Instruct	13.41
Qwen3-VL-30B-A3B-Instruct	9.28
GPT-5.1 (high)	8.90
Claude-Sonnet-4.5 (Thinking)	8.17
GPT-5.2	7.53

Appendix

Table 12: Full-refusal rates on the 3,146 flawed inputs. Values denote the proportion of responses assigned the refusal strategy.

Judge	Correction	Clarification	Refusal
GPT-5.2	1.28	1.90	1.87
Gemini-3-Pro	1.79	1.99	1.98
Claude-Sonnet-4	1.66	1.93	1.88

Appendix

Table 13: Mean Strategic Effectiveness (SE) scores assigned to clarification, correction, and refusal responses by the three LLM judges.

Response model	GPT judge	Gemini judge	Claude judge
GPT-5.1 (high)	1.718	1.873	1.562
GPT-5-mini	1.683	1.825	1.451
GPT-5.2	1.620	1.770	1.472
Gemini-3-Pro	1.565	1.708	1.199
Gemini-3-Flash	1.601	1.755	1.245
Claude-Sonnet-4.5 (Thinking)	1.548	1.820	1.544

Appendix

Table 14: Judge robustness for Diagnostic Precision (DP). Rows identify the model that generated the response and columns identify the judge model; entries are mean DP scores on the reported evaluation subset (0–2 scale).

Model	MEDA	DP-cov.	SE	PCQ	Type cov.	All det.	Partial det.
Qwen3-VL-8B-Instruct	76	1.114	1.105	42.2	44	16	72
Qwen3-VL-30B-A3B-Instruct	74	1.157	0.889	37.6	46	27	65
Qwen3-VL-235B-A22B-Instruct	74	1.090	0.811	35.2	42	16	68
GPT-5-mini	46	1.087	1.261	27.0	25	8	42
Gemini-3-Flash	30	1.178	1.200	17.8	19	10	28

Appendix

Table 15: Results on the 50-sample dual-error subset. MEDA denotes Multi-Error Detection Accuracy; DP-cov., Diagnostic-Point Coverage; SE, Strategic Effectiveness; and PCQ, Proactive Critique Quality. Type cov. is average ground-truth error-type coverage; All det. and Partial det. indicate detection of both and at least one ground-truth error type, respectively.

Model	Expression Error		Premise Contradiction			Missing Premise						Beyond Capability
Model	Ambiguity	Unclear Intent	Cross-Modal	Inter-Visual	Intra-Textual	Visual Light	Visual Medium	Visual Severe	Missing Image	Missing Instruction	Mismatch	Beyond Capability
Claude-Sonnet-4.5 (Thinking)	15.7	21.0	21.8	3.1	41.0	8.0	11.0	19.8	70.7	69.7	74.0	87.1
Doubao-Seed-1.6-vision	5.1	5.5	17.6	2.4	24.1	9.3	7.5	5.0	9.3	6.3	26.7	50.2
GLM-4.6V	2.7	7.9	11.3	1.0	16.9	4.0	4.0	2.2	40.3	8.3	18.0	58.0
GPT-5-mini	7.1	24.1	9.5	1.0	15.9	5.3	8.7	14.8	44.7	46.3	30.3	67.5
GPT-5.1 (high)	5.4	28.6	10.6	0.4	17.9	12.7	8.1	15.9	47.0	63.0	53.0	89.1

Appendix

Table 16: Error Detection Accuracy (EDA, %) across fine-grained error subtypes and four primary error categories.

Model	Expression Error		Premise Contradiction			Missing Premise						Beyond Capability
Model	Ambiguity	Unclear Intent	Cross-Modal	Inter-Visual	Intra-Textual	Visual Light	Visual Medium	Visual Severe	Missing Image	Missing Instruction	Mismatch	Beyond Capability
Claude-Sonnet-4.5 (Thinking)	1.09	1.45	1.41	1.52	1.63	0.89	0.89	1.66	1.87	1.25	1.88	1.95
Doubao-Seed-1.6-vision	0.76	0.88	1.35	0.38	1.29	0.21	0.21	0.56	0.71	1.14	1.25	1.60
GLM-4.6V	0.67	0.68	1.38	0.11	1.32	0.39	0.10	0.08	1.72	1.11	1.39	1.85
GPT-5-mini	1.19	1.52	1.11	1.56	1.23	1.88	1.93	1.91	1.98	1.65	1.91	1.91
GPT-5.1 (high)	1.00	1.63	1.32	0.33	1.21	1.65	1.86	1.91	1.99	1.65	1.97	1.94

Appendix

Table 17: Diagnostic Precision (DP) scores across fine-grained error subtypes and four primary error categories (0–2 scale).

Model	Expression Error		Premise Contradiction			Missing Premise						Beyond Capability
Model	Ambiguity	Unclear Intent	Cross-Modal	Inter-Visual	Intra-Textual	Visual Light	Visual Medium	Visual Severe	Missing Image	Missing Instruction	Mismatch	Beyond Capability
Claude-Sonnet-4.5 (Thinking)	1.66	1.74	1.61	1.83	1.72	1.39	1.32	1.81	1.89	1.97	1.91	1.98
Doubao-Seed-1.6-vision	1.48	1.32	1.69	1.24	1.58	1.25	0.91	1.39	1.24	1.73	1.42	1.79
GLM-4.6V	1.56	1.17	1.47	0.33	1.67	0.78	0.98	1.12	1.79	1.55	1.50	1.92
GPT-5-mini	1.75	1.91	1.80	2.00	1.75	2.00	1.98	2.00	1.99	1.94	1.98	1.95
GPT-5.1 (high)	1.57	1.92	1.80	0.00	1.78	1.84	2.00	1.97	1.98	1.99	1.98	1.98

Appendix

Table 18: Strategic Effectiveness (SE) scores across fine-grained error subtypes and primary error categories (0–2 scale).

Model	Expression Error		Premise Contradiction			Missing Premise						Beyond Capability
Model	Ambiguity	Unclear Intent	Cross-Modal	Inter-Visual	Intra-Textual	Visual Light	Visual Medium	Visual Severe	Missing Image	Missing Instruction	Mismatch	Beyond Capability
Claude-Sonnet-4.5 (Thinking)	10.8	16.8	16.5	2.6	34.4	4.6	6.1	17.2	66.3	56.2	70.3	85.4
Doubao-Seed-1.6-vision	2.9	3.0	13.4	1.0	17.3	3.4	2.1	2.4	4.6	4.5	17.8	42.6
GLM-4.6V	1.5	3.7	8.0	0.1	12.6	1.2	1.1	0.7	35.4	5.5	13.0	54.6
GPT-5-mini	5.2	20.7	6.9	0.9	11.8	5.2	8.5	14.5	44.3	41.5	29.5	65.0
GPT-5.1 (high)	3.5	25.4	8.2	0.0	13.4	11.1	7.8	15.4	46.7	57.4	52.3	87.5

Appendix

Table 19: Proactive Critique Quality (PCQ, %) across fine-grained error subtypes and primary error categories.

Figure 4: Per-subcategory PCQ (%) across 12 subcategories, grouped into three modality regions based on which modality carries the error signal (Text-only, Cross-modal, Visual-only). Each colored line connects the within-family average across subcategories, and the black dashed line shows the overall average across all 14 models. Subcategory names follow the format "Primary Error Type. Subcategory Name" (e.g., Expr. Ambiguity for Expression Error, Ambiguity subcategory).

Figure 5: An example of a failed response to an Ambiguity sample by GPT-5.2. The red text in the Question indicates the segment containing the ambiguity part.

Figure 6: An example of a successful response to a Beyond Capability sample by GPT-5-mini. The red text in the Question indicates the segment containing the beyond capability part. The blue text indicates the key part that highlights the presence of successful error detection.

Figure 7: An example generated by Claude-Sonnet-4.5 (Thinking) in which the internal reasoning and final response do not align. The final response and internal reasoning address a Cross-Modal Contradiction sample. The red text in the Question indicates the segment containing the cross-modal contradiction. The blue text in Chain of Thought indicates the key part that highlights successful error detection.

Figure 8: Prompt for evaluating Error Detection Accuracy

Figure 9: Prompt for evaluating Diagnostic Precision

Figure 10: Prompt for evaluating Strategic Effectiveness

Figure 11: Prompt for generating samples of the Ambiguity type

Figure 12: Prompt for generating samples of the Beyond Capability type

Figure 13: Prompt for generating samples of the Inter-Visual Contradiction type

Figure 14: Prompt for generating samples of the Intra-Textual Contradiction type

Figure 15: Prompt for generating samples of the Cross-Modal Contradiction type

Figure 16: Prompt for generating samples of the Unclear Intent type

Data Category	Question	Error Reason
Ambiguity	Instruction: Can you identify the pathogen affecting the potato sample located on the side in Image ? Image:	The problem lacks a necessary premise by failing to specify which potato sample is being referred to. The phrase "the potato sample located on the side" is ambiguous because it does not indicate left/right or a specific position. The ambiguity cannot be resolved from the image, since there are samples on both sides.
Beyond Capability	Instruction: Delete the minimum number from the given leftist heap. Which one of the following statements is TRUE about the exact wall-clock time taken by that deletion on the computer used to generate this diagram? Image:	The problem asks for the wall-clock time of a delete operation on the computer that produced the diagram, but that cannot be determined from the diagram alone because it provides no information about the machine’s hardware, software environment, or execution logs.
Intra-Textual Contradiction	Instruction: Which of the following responses will an increase in government spending to boost Aggregate Demand (AD), specifically defined as a reduction in government expenditure , as demonstrated in Image , trigger according to the theory of rational expectations? Image:	The problem text contains an internal contradiction by presenting both "increase government spending" and "decrease government spending" as definitions for the same policy action (mutually exclusive and yielding opposite answers); this contradiction is explicit in the text (verifiable without the image), both premises are stated on equal footing with no priority to resolve, and the direction of fiscal spending is a necessary premise for the AD response under rational expectations, so the question cannot be answered without clarifying which direction is intended.

Appendix

Table 20: Examples of Ambiguity, Beyond Capability, and Intra-Textual Contradiction. Red text marks the error-containing segment in the question.

Data Category	Question	Error Reason
Inter-Visual Contradiction	Instruction: In an aluminum (Al) bar of square cross section, a square hole is drilled and is filled with iron (Fe) as shown in the figure. The electrical resistivities of Al and Fe are $2.7\times 10^{-8}$ $\Omega$ $m$ and $1.0\times 10^{-7}$ $\Omega$ $m$ , respectively. The electrical resistance between the two faces $P$ and $Q$ of the composite bar is Image1 , Image2 Image1: Image2:	The two figures show inconsistent length labels (50 $mm$ vs 66 $mm$ ); since $R=\rho L/A$ and resistance is proportional to length, changing $L$ alters the computed resistance and yields different answers
Missing Instruction	Image:	The problem references textual premise but omits essential textual question which leads to unanswerable question.
Missing Image	Instruction: Compute the reactions at supports A and E of the three-hinged parabolic arch in Figure P6.23. Next compute the axial load at points B and D, located at the quarter points.	The problem references "Figure P6.23" but omits essential numerical and structural data—arch geometry (span and rise), load magnitudes and distributions, and hinge/joint positions—without which the quantitative calculation of support reactions for a three-hinged parabolic arch and the axial force at the quarter point cannot be performed.

Appendix

Table 21: Examples of Inter-Visual Contradiction, Missing Instruction, and Missing Image.

Data Category	Question	Error Reason
Missing Visual Premise Light	Instruction: Image Calculate the missing item. Image:	The image’s central region is significantly occluded, obscuring critical structural connections and key-type information required to determine the number of keys; this essential visual premise cannot be inferred from the text or the remaining image, and no answer choices allow the missing information to be deduced.
Missing Visual Premise Medium	Instruction: Payback Consider the following projects: Image Calculate the discounted payback period for each project Image:	The cash-flow table in the image is damaged/blurred, rendering multiple key numeric values unreadable (e.g., projects A and B initial cash flows C0, several period cash flows, and certain column headers/symbols) that are required to calculate the discounted payback period. The problem statement does not provide a discount rate. The missing information cannot be reliably inferred from the text, the remaining visible data, or the provided answer options.
Missing Visual Premise Severe	Instruction: Room registrations in the Toronto Towers Plaza Hotel have been recorded for the past 9 years. To project future occupancy, management would like to determine the mathematical trend of guest registration. This estimate will help the hotel determine whether future expansion will be needed. Given the following time-series data, develop a regression equation relating registrations to time (e.g., a trend equation). Then forecast year 11 registrations. Room registrations are in the thousands: Image Image:	The time-series table is severely corrupted: several yearly registration values and labels are unreadable. These values are required to fit the regression trend and forecast year 11, so the requested equation and forecast cannot be determined reliably from the remaining image.

Appendix

Table 22: Examples of Missing Visual Premise at three information-loss levels: Light, Medium, and Severe.

Data Category	Question	Error Reason
Cross-Modal Contradiction	Instruction: Blood glucose level is regulated by two antagonistic hormones, glucagon and insulin. The following graph shows the fluctuations in an individual’s blood glucose level over a five hour period, where the blood glucose level at point B is lower than that at point E : Image Image:	The description "B is lower than E" directly contradicts the image, which shows $B\approx 1.6mg/ml$ and $E\approx 1.2mg/ml$ . The relative heights in the image are a necessary visual premise that affects subsequent judgments, and the problem does not specify whether the text or the image takes precedence.
Mismatch	Instruction: In the Base of Skull, identify the 205 structure. Image Image:	The question requires identifying the structure labeled "205" on a skull - base image, but the provided image is unrelated to the skull base and its labels do not match, so the question cannot be answered.
Unclear Intent	Instruction: Janet Ludlow’s firm requires all its analysts to use a two-stage dividend discount model (DDM) and the capital asset pricing model (CAPM) to value stocks. Using the CAPM and DDM, Ludlow has valued QuickBrush Company at $63 per share. She now must value SmileWhite Corporation. Image Image:	The question omits a specific task instruction, creating ambiguity about whether to calculate the CAPM required return or to value intrinsic value; additionally, the dividend data needed to compute intrinsic value are missing.

Appendix

Table 23: Examples of Cross-Modal Contradiction, Mismatch, and Unclear Intent. Red text marks the error-containing segment in the question.

Model	Reasoning	Model Link
Proprietary Models
Gemini-3-Pro	✓	https://ai.google.dev/gemini-api/docs/gemini-3?hl=zh-cn
GPT-5.1 (high)	✓	https://platform.openai.com/docs/models/gpt-5.1
Claude-Sonnet-4.5 (Thinking)	✓	https://www.anthropic.com/news/claude-sonnet-4-5
Doubao-Seed-1.6-vision	✓	https://www.volcengine.com/docs/82379/1799865?lang=zh
Gemini-3-Flash	✓	https://ai.google.dev/gemini-api/docs/gemini-3?hl=zh-cn

Appendix

Table 24: Model provenance and access links. A check mark in the Reasoning column indicates that the official documentation or model card provides an explicit reasoning/thinking mode; a cross indicates no dedicated reasoning mode is documented.

Language Model Error Detection

Language model error detection identifies unreliable answers, reasoning steps, and actions so systems can flag, verify, or repair them.

See all 91 papers

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models
Mar 19, 2026Thomas De Min, Subhankar Roy, Stéphane Lathuilière +2Multimodal Large Language Models AI Agent Benchmarks
University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK +1
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLLM Evaluation Large Language Model-Based Recommendation
School of Artificial Intelligence, Jilin University · Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China · International Center of Future Science, Jilin University

Mar 19, 2026J/K move · Enter open · S save

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

Thomas De Min, Subhankar Roy, Stéphane Lathuilière, Elisa Ricci+1

University of Trento · University of Bergamo · Inria, Univ. Grenoble Alpes, CNRS, LJK+1

Multimodal Large Language Models AI Agent Benchmarks Proactive Assistance Multimodal Foundation Models Multimodal Agents Multimodal Model Evaluation+1

Open paper AskPDF arXiv

Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can MLLMs exhibit a similar "proactive" behavior by requesting simple user interventions? To investigate this, we introduce ProactiveBench, a benchmark built from seven repurposed datasets that tests proactiveness across different tasks such as recognizing occluded objects, enhancing image quality, and interpreting coarse sketches. We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains. Surprisingly, we found that conversation histories and in-context learning introduce negative biases, hindering performance. Finally, we explore a simple fine-tuning strategy based on reinforcement learning: its results suggest that proactiveness can be learned, even generalizing to unseen scenarios. We publicly release ProactiveBench as a first step toward building proactive multimodal models.

Abstract

Language Model Error Detection

Explore similar work

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge

RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation