When Figures Disagree with Text: Evaluating Autonomous Visual Evidence Verification in Scientific Papers
Authors: Guijia Zhang, Hongbo Qiu, Jiajun Zha, Guohao Cai, Hao Zheng, Huihao Jing, Kexu Cheng, Shipeng Luo, JinYang Huang, Shing-Chi Cheung, Changsheng Lu, Haoxuan Che, Harry Yang
Organizations: The Hong Kong University of Science and Technology · Shenzhen University · xgenlabs · Institute of Computing Technology, Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Central South University
Scientific figures carry evidence for the claims that cite them, so reviewing a paper means checking that a figure agrees with the text, and not only describing what the figure shows. Benchmarks of figure-text verification typically hand the model the figure, the claim, or the knowledge that an inconsistency exists, which leaves untested whether a model can find the figure worth checking in a complete paper, relate a visual element to the claim it conflicts with, and recognize the conflict. We study this autonomous setting on 31 full manuscripts, each carrying a reviewer-documented figure-text inconsistency from PRISMM-Bench that is hidden from the model. Six multimodal models review each PDF under a neutral prompt, and we then add one input at a time to the same manuscripts. Of 1,445 open-ended reviews, 60.6% mention the flawed figure, yet a grading rubric accepts only 3.3% as diagnosing its contradiction; a blind audit finds the rubric lenient, so we read these as rubric-accepted rates and compare conditions rather than absolute values. In a separate controlled run, naming the figure, without stating that it is flawed, raises rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues do not. When the contradiction is visible only in the rendered figure, text-only review yields no accepted diagnosis, while review of the rendered pages with the figure named reaches 15.8%. A baseline that lets the model choose which figures to check, ranked by centrality, stays at 4.8%. Deciding what to verify is thus a large part of the difficulty, and the selection rule we tested does not resolve it. Multimodal document understanding should be evaluated on whether a model can propose what to verify, not only answer questions about a figure it is handed.
Figures & tables
Figure 1: A figure–text contradiction that exists only between pixels and a caption clause, and where autonomous review stops. (a) Page 5 of 16 in an unmodified manuscript. The caption assigns textual input to the third setup, but the diagram routes a photograph into it. (b) The bar shows the 1,445 open-ended reviews, which pool four input conditions including figure borders. The three cards give one observation for each challenge: shortlist recall (Section 4.3 ), the border and crop contrasts (Sections 4.1 and 4.2 ), and the exploratory shortlist-subset rate (Section 4.3 ). They illustrate the challenges and do not measure them separately. (c) Rubric-accepted diagnosis in four arms of the controlled run (186 reviews each). Naming the figure helps, a consistency instruction alone does not, and a model-chosen shortlist stays near the unprompted level. Whiskers show 95% paper-clustered intervals.
Analysis
Section
Reviews and papers
Checked by
Open-ended cohort
4.1
1,445 reviews, 31 papers; four input conditions pooled
Table 1: What each analysis covers. All analyses use the same 31 manuscripts, except the paired excerpt, which needs a second concrete side to supply. The blind human audit sampled reviews from the open-ended cohort only. Controlled arms rely on the model rubric, with a reference-free panel as a second check (supplement).
Figure 2: Reviews mention the flawed figure but rarely diagnose it, and naming the figure shifts the remaining misses toward the wrong relation. (a) How far each model’s open-ended reviews get in the cohort, ordered by rubric-accepted diagnosis. (b) Rubric-accepted diagnosis for the same models in the controlled run, unprompted and with the figure named (31 reviews per model and arm). (c) Grader label of near-miss records: open-ended cohort (637 grader records), controlled unprompted arm (51), and controlled arm with the figure named (208).
Arm
Target
Verify
S2
S3
95% CI
Unprompted PDF
No
No
19.89
4.30
[1.08, 8.06]
Global-consistency
No
Yes
23.12
6.99
[2.69, 12.37]
Target-pointed
Yes
No
82.26
20.43
[13.44, 28.49]
Target-consistency
Yes
Yes
88.71
17.20
[10.22, 25.27]
Paired S3 contrasts (pp, K=31 )
Target, no framing
+16.13
[ +9.14 , +24.19 ]
Table 2: Naming the target figure and asking for a consistency check, crossed. Each cell runs on the identical 31 manuscripts and six reviewers, with 186 reviews on the unmodified full PDF. S2 is the concern rate and S3 the rubric-accepted diagnosis rate, with paper-clustered 95% CIs. Paired contrasts are per-paper differences in percentage points over K=31 .
Figure 3: Naming the figure helps, while border highlights, crop placement, and a model-chosen shortlist leave diagnosis near the unprompted level. (a) A second manuscript: the whole-figure border (cyan) lies a median 49.5 pt from the disputed caption band (magenta) and touches none of its pixels across 62 edits. (b) Rubric-accepted diagnosis for each arm of the controlled run (186 reviews, 31 manuscripts), with 95% paper-clustered intervals. (c) Text-only, unprompted, and figure-named reviews by evidence type, on the 20 manuscripts whose evidence lies only in the figure and the 11 whose evidence is also stated in the text.
Table A.1 : Provider model identifiers across roles in the audit.
Model
Role
Returned API identifier(s)
Request parameters
Calls
Window (UTC)
Claude Sonnet 4.6
Reviewer, cohort
claude-sonnet-4-6
temp 0.7; max 16384
242/248
2026-11-03
GPT-5.4
Reviewer, cohort
gpt-5.4 , gpt-5.4-2026-03-05
reasoning low; max 16384
241/248
2026-11-03
GPT-5 mini
Reviewer, cohort
gpt-5-mini-2025-08-07
reasoning low; max 16384
243/248
2026-11-03
GPT-5 nano
Reviewer, cohort
gpt-5-nano-2025-08-07
reasoning low; max 16384
239/248
2026-11-03
Gemini 3.1 Pro Preview
Reviewer, cohort
gemini-3.1-pro-preview
temp 0.7; max 16384
241/248
2026-11-03
Gemini 3.1 Flash Lite
Reviewer, cohort
gemini-3.1-flash-lite
temp 0.7; max 16384
239/248
2026-11-03
Appendix
Table A.2 : Returned API identifiers, request parameters, call counts, and execution windows by model and role. Every cell is read from the local call logs. Calls are completed calls, shown against scheduled calls where the two differ. Two identifiers in one cell mean that the provider returned both strings for the same requested identifier.
Selection funnel
Step
Inconsistencies
Papers
Public PRISMM-Bench release
384
353
Source PDF embedded in release
204
191
Names figure or table (benchmark category )
192
179
Names figure or table (parsed source)
198
186
One issue per paper (candidate papers)
186
186
Appendix
Table A.3 : Selection funnel and representativeness of the full-PDF audit cohort. Page counts exist for the 191 release papers with an embedded source PDF; the remaining papers carry none. The final step is a manual, outcome-blind, batch-wise enrollment, and the number of non-enrolled candidates that were examined is not recorded in any local file.
Reviewer
n
S1 (%)
S2 (%)
S3 (%)
S3 95% CI
Primary cohort ( K=31 papers, n=1,445 reviews)
claude-sonnet-4-6
242
86.78
42.98
15.29
[7.53, 24.37]
gemini-3.1-pro-preview
241
44.40
23.65
2.90
[0.41, 6.25]
gpt-5.4
241
75.52
34.85
0.83
[0.00, 2.51]
gemini-3.1-flash-lite
239
28.45
12.97
0.42
[0.00, 1.27]
gpt-5-mini
243
73.25
24.28
0.00
[0.00, 0.00]
Appendix
Table C.1 : Per-reviewer rates across all 31 papers and under leave-one-paper-out ( K=30 ). Intervals are 10,000-draw paper-clustered 95% bootstrap CIs.
ID
Paper ID
Ret.
P
U
ID
Paper ID
(A)
(B)
(C)
(D)
(E)
(F)
01
8TyGCAuCGd
48
10
4
01
8TyGCAuCGd
8/8
0/8
2/8
0/8
0/8
0/8
02
MazxSMs6Hs
48
7
10
02
MazxSMs6Hs
6/8
0/8
1/8
0/8
0/8
0/8
03
U9sHVjidYH
48
6
10
03
U9sHVjidYH
3/8
0/8
3/8
0/8
0/8
0/8
04
LXnTFMvn8A
48
5
1
04
LXnTFMvn8A
2/8
1/8
0/8
0/8
0/8
2/8
05
BeOEmnmyFu
45
3
11
05
BeOEmnmyFu
3/8
0/7
0/8
0/7
0/7
0/8
06
QO4bF6MHza
43
3
1
06
QO4bF6MHza
3/8
0/7
0/7
0/7
0/7
0/7
Appendix
Table D.1 : Left: paper-level returned, accepted ( P ), and unresolved ( U ) counts. Right: accepted/returned counts by reviewer (A–F).
Stratum
Complete
Missing
κ
ΔP (pp)
claude-sonnet-4-6
214
28
0.591
−2.80
gemini-3.1-flash-lite
210
29
0.241
+0.00
gemini-3.1-pro-preview
207
34
0.318
−0.48
gpt-5-mini
214
29
0.012
+0.00
gpt-5-nano
220
19
0.294
+0.00
gpt-5.4
214
27
0.353
+0.00
Appendix
Table F.1 : Primary vs. independent replay agreement across reviewers and visual conditions ( n=1,279 ).
Independent replay label
Primary label
Replay P
Replay N
Replay U
Total
Primary P
34
0
8
42
Primary N
0
1,062
26
1,088
Primary U
1
118
30
149
Total
35
1,180
64
1,279
Appendix
Table F.2 : Primary vs. independent replay label matrix ( n=1,279 complete pairs). Three-way P/N/U confusion matrix between the primary rubric and the independent replay; per-stratum κ and ΔP are in Table F.1 .
Figure G.1 : Claim verifiability strata and reference-free near-miss grounding breakdown across the open-ended cohort. (a) Authentic high-resolution source crops contrasting Type-I literal mismatches ( K=13 , S3=4.94% ) against Type-II interpretive disputes ( K=18 , S3=2.03% ). (b) Reference-free conflict check across 301 order-stable concerns (100.0% accepted diagnoses vs. 30.4% near-misses). (c) Failure stage breakdown across 637 near-miss grader records and 289 rejected concern reviews.
K=31 papers
K=30 papers
Def.
Contrast
Endpt.
Cells
Diff.
95% CI
p
Cells
Diff.
95% CI
p
nested
Target − Orig.
S1
349
−1.88
[−6.09,+2.57]
0.210
337
−1.67
[−6.00,+2.85]
0.286
nested
Target − Orig.
S2
349
+3.84
[−0.10,+7.67]
0.263
337
+3.13
[−0.75,+6.88]
0.359
nested
Target − Orig.
S3
349
+0.87
[−0.37,+2.49]
0.625
337
+0.62
[−0.61,+2.22]
1.000
nested
Target − Random
S1
351
+1.19
[−2.99,+5.25]
0.286
339
+1.23
[−3.00,+5.33]
0.286
nested
Target − Random
S2
351
+5.64
[+0.83,+10.50]
0.064
339
+4.99
[+0.17,+9.90]
0.093
Appendix
Table G.1 : Paired visual-intervention contrasts under nested and strict definitions for K=31 papers (left) and K=30 excluding 8TyGCAuCGd (right).
Panel label
Primary label
P
N
U
P
6
0
3
N
1
28
0
U
0
5
1
Total
7
33
4
Appendix
Table I.1 : Reference-anchored panel vs. the primary rubric on the 44-item packet. Rows are the primary label, columns the panel’s majority label (a tied panel is counted U ).
Arm
n
S3 (%)
Unprompted full PDF
186
4.30
Target-pointed
186
20.43
Target-consistency
186
17.20
Global-consistency
186
see Table 2
Two-stage index-then-verify
186
see Table J.1
Crop placement, three arms
558
2.69
Appendix
Table I.2 : Arms reported by the main paper. Every arm runs on all 31 manuscripts and up to six reviewers with the unmodified full PDF unless stated; n is the number of graded reviews, and rubric-accepted diagnosis S3 carries a paper-clustered 95% CI. Full per-arm results are in Table L.1 and Table 2 .
Two-stage quantity
Value
95% CI
Stage 1 target recall@3
34.41%
[23.66, 46.24]
Stage 1 target recall@5
57.53%
[43.55, 71.51]
End-to-end S3 ( n=186 )
4.84%
[1.61, 8.60]
S3 gain over unprompted
+0.54 pp
[ −3.76 , +5.38 ]
Oracle gain recovered
3.3%
[ −33.3 , +27.3 ]
Tokens vs. single pass
2.04 ×
Appendix
Table J.1 : Two-stage diagnostic baseline in aggregate. Recall is the share of 186 paper-model cells whose Stage 1 shortlist contains the registered element; end-to-end S3 counts a cell without a usable identifier as a non-diagnosis. Token counts are as logged.
Rate (%)
Δ vs. baseline (pp)
Arm
Metric
n
K
Rate
95% CI
Δ
95% CI
p
Unprompted full (baseline)
S1
186
31
40.86
[32.80, 48.92]
–
–
–
S2
19.89
[13.98, 26.34]
–
–
–
S3
4.30
[1.08, 8.06]
–
–
–
Target-pointed
S1
186
31
100.00
[100.00, 100.00]
+59.14
[ +51.08 , +67.20 ]
<0.001
S2
82.26
[76.88, 87.10]
+62.37
[ +53.76 , +70.43 ]
<0.001
Appendix
Table L.1 : Complete decoupled-probe results with bootstrap intervals and exact sign tests.
Arm
n
K
Affirmative rate
95% CI
Δ vs. baseline (pp)
p
Unprompted full (baseline)
186
31
22.58 & [18.82, 26.34]
–
–
Target-pointed
186
31
44.09 & [37.63, 50.00]
+21.51 & [ +13.98 , +29.03 ]
<0.001
Consistency-prompt
186
31
53.76 & [48.39, 59.68]
+31.18 & [ +24.19 , +38.71 ]
<0.001
Paired excerpt
84
14
38.10 & [23.81, 53.57]
+16.67 & [ +1.19 , +33.33 ]
0.227
Text-only
186
31
36.02 & [31.18, 40.86]
+13.44 & [ +8.06 , +18.82 ]
<0.001
OCR-only
186
31
33.33 & [28.49, 38.17]
+10.75 & [ +5.38 , +16.13 ]
0.001
Appendix
Table M.1 : Reference-free conflict check: share of reviews in which the panel affirms that two manuscript elements contradict each other, with paper-clustered bootstrap intervals and exact sign tests against the unprompted-full baseline.
Table O.1 : Rubric-accepted diagnosis stays near three percent across cohort exclusions, anchor definitions, and verifiability strata. The cohort pools four input conditions, including figure borders. All 1,445 returned reviews remain in n , including 175 unresolved U labels. Intervals are 10,000 -draw paper-clustered bootstrap 95% CIs over K paper clusters.
Table O.2 : Observed diagnosis rates hold between 2.74 and 3.74 percent across grading panels, whereas counting every unresolved review as accepted gives 15.36 percent. The human audit, which covers the open-ended cohort only, confirmed about half of the rubric-accepted reviews it sampled. Top: primary rubric, panel-external two-model replay over n=1,279 pairs, source-anchored 44-item adjudication over items H001 to H044 , pilot source check, and assumption-based bounds. Bottom: replay matrix.
Contrast
Cells
Diff.
95% CI
p
Empirical null: byte-identical repeat
Orig. − Repeat, S1
350
+1.39
[−3.65,+6.47]
0.678
Orig. − Repeat, S2
350
+0.44
[−4.89,+5.57]
0.690
Orig. − Repeat, S3
350
−0.22
[−2.64,+2.25]
1.000
Executed visual intervention
Target − Orig., S1
349
−1.88
[−6.09,+2.57]
0.210
Appendix
Table O.3 : Whole-figure visual highlights yield small shifts within empirical repeat variability and leave rubric-accepted diagnosis statistically indistinguishable from baseline. Paired visual-intervention contrasts over K=31 papers with 10,000 -draw paper-clustered bootstrap 95% CIs, exact per-paper sign-test p , and minimum Holm-Bonferroni pHolm=0.767 across all 12 contrasts, as detailed in Supplementary Table G.1 .
Figure O.1 : Whole-figure highlights produce small shifts within natural per-paper repeat variability and leave rubric-accepted diagnosis unchanged. Panels a to c plot paired paper-level contrasts on S1 mention, S2 concern, and S3 rubric-accepted diagnosis for Orig. minus Repeat, Target minus Orig., and Target minus Random across K=31 papers, with shaded empirical repeat variability bands of ±10.6 , ±11.6 , and ±3.5 pp.