cs.CVOct 19, 2026
SaveSciFigBench: A Benchmark for Scientific Method Figure Understanding
Organizations: VCIP, CS & AAIS, Nankai University · NKIARI, Shenzhen Futian · Li Auto.
Abstract
Scientific method figures often provide researchers with a clear visual entry point into a paper's methodology and core contributions, yet their dense layouts and complex information flows are difficult for Vision Language Models (VLMs) to interpret. Existing benchmarks predominantly focus on coarse-grained tasks, such as context-based retrieval or holistic comprehension, overlooking the fine-grained content and inter-module relations within complex methodological diagrams. To address these overlooked aspects, we introduce Scientific Figure Benchmark (SciFigBench), a benchmark for evaluating core capabilities required for scientific method figure understanding. Specifically, it targets three capabilities: dense content perception, visual relation comprehension, and information-flow tracing. For this purpose, we curate 4,352 scientific method figures from real-world research papers and construct 40,670 QA pairs to operationalize these capabilities as measurable question-answering tasks. All annotations are generated by dedicated pipelines and then manually verified. We further propose SciFigCue, a training-free visual prompting module that externalizes arrow-induced dependencies as visual relational evidence for method-figure reasoning. Experiments across open-source and proprietary VLMs show that even strong models remain substantially limited in visual relation comprehension, while SciFigCue improves performance across several tasks without parameter updates. The code and related resources are available at https://github.com/LeNcyer/SciFigBench.
Figures & tables
Figure 1: Why diagnostic method-figure QA is needed. A coarse-grained question may expose an incorrect answer, but cannot distinguish whether the failure comes from missing a component label or misreading a structural connection. SciFigBench instead asks fine-grained diagnostic questions to make these failure sources observable.
Figure 2: Source distribution of SciFigBench method figures. SciFigBench contains 4,352 method figures from papers published at seven AI conferences in 2024–2025.
Figure 3: Overview of the SciFigBench construction pipeline. (a) Starting from 20,000 AI conference papers, we extract candidate method figures and retain 4,352 scientific method figures after filtering. (b) Content annotation combines OCR detection with VLM-based verification to produce localized text fields. (c) Structural annotation extracts node boxes and head/tail-labeled arrow endpoints through segmentation and post-processing to build directed structural relations. Collectively, these annotations enable the construction of the diagnostic QA tasks in SciFigBench.
Figure 4: QA distribution in SciFigBench. The 40,670 QA pairs are distributed across five diagnostic tasks grouped into three evaluation dimensions: dense content perception, visual relation comprehension, and information-flow tracing.
Figure 5: Two visual-prompt formats in SciFigCue. Visual cues are extracted by the automated parsing pipeline used in SciFigBench construction and rendered in two formats: an auxiliary arrow-mask view for multi-target queries and an original-figure overlay for single-target queries.
| Model | Size | Dense Content Perception | Edge-Level Verification | Node-Level Degree Counting | Path-Level Reachability | Information-Flow Tracing |
|---|---|---|---|---|---|---|
| RandomChoice | – | – | 50.0 | – | 50.0 | – |
| Closed-source models | ||||||
| Gemini-2.5-Flash | – | 72.7 | 66.0 | 42.3 | 55.3 | 42.4 |
| Gemini-3.1-Flash-Lite | – | 71.2 | 67.3 | 55.8 | 54.1 | 46.1 |
| GPT-5-mini | – | 51.3 | 49.8 | 24.8 | 49.3 | 26.0 |
| GPT-5.5 | – | 76.8 | 55.8 | 34.7 | 51.8 | 38.6 |
Table 1: Main results on SciFigBench. All values are percentages; higher is better. Best and second-best results in each column are shown in bold and underlined , respectively. Colored parentheses in SciFigCue rows indicate absolute changes over the corresponding base model. Overall, current VLMs remain limited in method figure understanding, while SciFigCue improves performance by making connection cues explicit.
| Method | DCP | Edge | Degree | Reach. | IFT |
|---|---|---|---|---|---|
| Qwen3.5-VL-27B | 70.9 | 59.2 | 50.9 | 50.2 | 43.6 |
| AutoParse+Symbolic | 96.3 | 77.1 | 57.0 | 65.2 | 48.4 |
| Verified Oracle | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 |
Table 2: Automatic parsing and symbolic reasoning. AutoParse+Symbolic relies only on outputs from the automated parsing pipeline used in SciFigBench construction, with query regions matched to parsed nodes by geometry. Verified Oracle uses human-verified annotations. Both VLMs and AutoParse+Symbolic remain substantially below the oracle.
| Model | Avg. | Gap | |
|---|---|---|---|
| Qwen3.5-VL-9B | 76.0 | 45.5 | 30.5 |
| + SciFigCue | 84.0 | 61.5 | 22.5 |
| Qwen3.5-VL-27B | 75.9 | 65.8 | 10.1 |
| + SciFigCue | 85.2 | 79.7 | 5.5 |
Table 3: -wise analysis of Dense Content Perception. Avg. averages , and Gap denotes its difference from . SciFigCue consistently narrows the performance gap between the two settings.
| Setting | Edge | Degree | Reach. |
|---|---|---|---|
| Raw | 50.8 | 41.4 | 46.3 |
| Box only | 48.3 | 40.7 | 47.9 |
| + Arrow location | 53.8 | 37.8 | 48.7 |
| + Randomized arrow | 49.9 | 41.3 | 46.8 |
| SciFigCue | 56.6 | 44.1 | 52.0 |
Table 4: Controlled cue ablation on Qwen3.5-VL-9B. Raw denotes the standard SciFigBench input. Within each task, the prompted variants share the same rendering design and differ only in cue content. SciFigCue includes both arrow location and direction. For Degree, Box only strengthens the target highlight without adding arrow cues. Full SciFigCue performs best.
| Model | Cue | Degree | IFT |
|---|---|---|---|
| Qwen3.5-VL-9B | Auto | 44.1 | 36.0 |
| Qwen3.5-VL-9B | GT | 46.1 | 36.9 |
| Qwen3.5-VL-27B | Auto | 51.1 | 42.1 |
| Qwen3.5-VL-27B | GT | 52.7 | 43.6 |
Table 5: Sensitivity to arrow cue quality. Auto uses automatically parsed cues, while GT uses human-verified cues. GT cues further improve SciFigCue performance, eliminating the IFT degradation observed with Auto cues on Qwen3.5-VL-27B.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| 1 | 2 | 3 | 4 | 5 | 6 | Total | |
|---|---|---|---|---|---|---|---|
| Count | 4,352 | 730 | 1,014 | 811 | 623 | 837 | 8,367 |
Table 6: Dense Content Perception case distribution. Examples are grouped by the number of highlighted text regions . denotes the single-region special case, while denotes flow-path multi-region reading.
| Case | Count |
| Positive | 4,341 |
| Negative / easy | 2,152 |
| Negative / reverse | 493 |
| Negative / sibling source | 317 |
| Negative / sibling target | 397 |
| Negative / spatial close | 523 |
Table 7: Edge-Level Verification case distribution. Positive examples contain a directed edge from the red-boxed node to the blue-boxed node. Negative examples are grouped by construction type, including easy negatives and hard negatives.
| Direction | 0 | 1 | 2 | 3 | 4+ | Total |
|---|---|---|---|---|---|---|
| In | 746 | 1,141 | 1,152 | 635 | 279 | 3,953 |
| Out | 717 | 1,335 | 1,053 | 556 | 234 | 3,895 |
| Total | 1,463 | 2,476 | 2,205 | 1,191 | 513 | 7,848 |
Table 8: Node-Level Degree Counting case distribution. Examples are grouped by query direction and edge-count bucket, where 0 denotes no edge in the queried direction and 4+ denotes denser local connectivity.
| Case | Count |
|---|---|
| Positive / length=1 fallback | 162 |
| Positive / length=2 | 1,782 |
| Positive / length=3 | 1,173 |
| Positive / length=4 | 710 |
| Positive / length=5+ | 409 |
| Negative / easy | 1,753 |
Table 9: Path-Level Reachability case distribution. Positive examples are grouped by shortest directed path length, and negative examples are grouped by construction type. Length-one positives are used only as fallbacks for papers without any multi-hop positive path.
| Target type | Query direction | Answer type | Count |
| Both | Incoming | Non-empty | 2,485 |
| Both | Outgoing | Non-empty | 2,485 |
| In only | Incoming | Non-empty | 669 |
| In only | Outgoing | Empty | 669 |
| Out only | Incoming | Empty | 669 |
| Out only | Outgoing | Non-empty | 669 |
Table 10: Information-Flow Tracing case distribution. Counts are reported at the query level after each eligible target is expanded into one incoming and one outgoing query. Both-type targets produce two non-empty queries, whereas in-only and out-only targets produce one empty query in the missing direction.
| Component | Precision | Recall | F1 |
|---|---|---|---|
| OCR | 95.46 | 95.72 | 95.59 |
| Node segmentation | 57.01 | 71.80 | 63.56 |
| Arrow endpoint | 67.78 | 87.07 | 76.23 |
| Overall | 76.96 | 88.23 | 82.21 |
Table 11: Quality of automatic annotations before human verification. Primary metrics use the matching criteria defined above, with additional semantic correctness requirements for OCR and arrow endpoints. The automatic pipeline achieves an overall primary F1 of 82.21%.
| Arrow endpoint diagnostic | Value |
|---|---|
| Endpoint match precision | 73.49 |
| Endpoint match recall | 94.41 |
| Endpoint match F1 | 82.65 |
| Head/tail accuracy | 92.23 |
| Nearest-node accuracy | 71.33 |
| Mean endpoint distance | 0.58 px |
Table 12: Diagnostics of automatic arrow endpoint parsing. Endpoint matching uses a 15-pixel distance threshold. Head/tail and nearest-node accuracy evaluate directional labeling and endpoint-to-node association on matched endpoints, respectively. All matching and accuracy values are percentages.
| Model | Size | Dense Content Perception | Edge-Level Verification | Node-Level Degree Counting | Path-Level Reachability | Information-Flow Tracing |
|---|---|---|---|---|---|---|
| RandomChoice | – | – | 50.0 | – | 50.0 | – |
| Closed-source models | ||||||
| Gemini-2.5-Flash | – | 72.7 | 66.0 | 42.3 | 55.3 | 42.4 |
| Gemini-3.1-Flash-Lite | – | 71.2 | 67.3 | 55.8 | 54.1 | 46.1 |
| GPT-5-mini | – | 51.3 | 49.8 | 24.8 | 49.3 | 26.0 |
| GPT-5.5 | – | 76.8 | 55.8 | 34.7 | 51.8 | 38.6 |
Table 13: Results on the shared 2,000-QA evaluation subset. All values are percentages, using the same metrics as Table 1 . All closed- and open-source models are evaluated on the same subset. Best and second-best results in each column are shown in bold and underlined , respectively.
| Group | DCP | Edge | Degree | Reach. | IFT | Mean |
|---|---|---|---|---|---|---|
| Open-source | +1.5 | +1.2 | -0.1 | -0.2 | +2.5 | +0.9 |
Table 14: Average changes from full-test to subset evaluation. Values are percentage-point differences computed as subset score minus full-test score. Mean denotes the corresponding change in the weighted aggregate across tasks.
| (a) Dense Content Perception | ||||||
|---|---|---|---|---|---|---|
| Model | ||||||
| GPT-5.5 | 83.4 | 68.4 | 65.6 | 75.5 | 68.2 | 71.6 |
| Qwen3-VL-32B | 69.8 | 44.4 | 47.8 | 54.1 | 63.7 | 66.5 |
| Qwen3.5-VL-9B | 76.0 | 47.1 | 41.9 | 42.8 | 48.9 | 46.6 |
| + SciFigCue | 84.0 | 66.7 | 56.6 | 59.0 | 64.3 | 61.1 |
| Qwen3.5-VL-27B | 75.9 | 58.0 | 60.9 | 65.4 | 73.6 | 71.2 |
Table 15: Task-specific diagnostic analysis. All values are percentages. (a) DCP is grouped by sequence length . (b) Edge reports answer labels and representative negative subtypes. (c) Degree is grouped by the degree count in the queried direction. (d) Reach. reports answer labels and shortest-path length for positive examples; rare length-one fallbacks are omitted from the breakdown. (e) IFT is grouped by the number of ground-truth neighbors. GPT-5.5 uses the 2,000-QA subset, while the remaining models use the full test set. These breakdowns reveal distributed-reading difficulty, direct/indirect relation confusion, dense-connectivity failures, and answer biases.
| Model | Scheme | Edge | Degree | Reach. | Weighted | Weighted |
|---|---|---|---|---|---|---|
| Qwen3.5-VL-9B | Red/blue | 50.8 | 41.4 | 46.3 | 46.3 | +0.0 |
| Green/purple | 54.7 | 40.3 | 52.3 | 49.3 | +3.0 | |
| Cyan/orange | 50.8 | 36.5 | 49.4 | 45.8 | -0.5 | |
| InternVL3.5-14B | Red/blue | 47.7 | 35.7 | 46.3 | 43.4 | +0.0 |
| Green/purple | 51.5 | 35.3 | 49.2 | 45.6 | +2.2 | |
| Cyan/orange | 48.4 | 34.5 | 47.4 | 43.7 | +0.3 |
Table 16: Robustness to query-box color schemes. All values are percentages. Weighted denotes the task-size-weighted aggregate across Edge, Degree, and Reach., and Weighted is the absolute change relative to the original red/blue scheme.
| Scheme | Boxes | Low-contrast | Fraction | Median | 10th pct. |
|---|---|---|---|---|---|
| Red/blue | 41,466 | 0 | 0.0000% | 113.37 | 107.54 |
| Green/purple | 41,466 | 0 | 0.0000% | 96.76 | 96.17 |
| Cyan/orange | 41,466 | 4 | 0.0096% | 59.17 | 56.00 |
Table 17: Local visibility of query boxes under different color schemes. Larger indicates stronger local box/background contrast. The 10th percentile summarizes lower-visibility cases.
Figure 6: Dataset annotation example 1. Content and structural annotations for a real method figure.
Figure 7: Dataset annotation example 2. Content and structural annotations for a real method figure.
Figure 8: Dataset annotation example 3. Content and structural annotations for a real method figure.
Figure 9: Prompt for Dense Content Perception. The prompt asks the model to read highlighted text regions in a fixed color order and return the transcriptions as a JSON list.
Figure 10: Prompt for Edge-Level Verification. The prompt asks whether a directed edge exists from the red-boxed node to the blue-boxed node under the specified edge-direction rules.
Figure 11: Prompt for Node-Level Degree Counting. The prompt asks the model to count incoming or outgoing edges of the red-boxed node under the specified degree-counting rules.
Figure 12: Prompt for Path-Level Reachability. The prompt asks whether a directed path exists from the red-boxed node to the blue-boxed node under the specified path-following rules.
Figure 13: Prompt for Information-Flow Tracing. The prompt asks the model to list the text of direct incoming or outgoing neighbors for a uniquely named target node.
Figure 14: Dense Content Perception failure for Qwen3.5-VL-27B. The query asks the model to read the highlighted regions in the specified color order. The model correctly recognizes all three text contents but swaps the order of the latter two regions, returning an incorrectly ordered sequence. Ground truth: ["Temporal Convolution Layer", "Equivariant GNN Layer", "Equivariant Un-Pooling"] . Model answer: ["Temporal Convolution Layer", "Equivariant Un-Pooling", "Equivariant GNN Layer"] .
Figure 15: Edge-Level Verification failure for InternVL3.5-8B. The query asks whether a directed edge exists from the red-boxed node to the blue-boxed node. The figure instead contains the reverse blue-to-red connection, but the model answers "true" rather than the ground-truth "false" , indicating a failure to correctly follow arrow direction.
Figure 16: Node-Level Degree Counting failure for Qwen3.5-VL-9B. The query asks for the out-degree of the red-boxed node. The node has one incoming edge but no outgoing edge, yet the model answers 1 instead of the ground-truth 0 , indicating confusion between incoming and outgoing connections.
Figure 17: Path-Level Reachability failure for Qwen3.5-VL-9B. The query asks whether a directed path exists from the red-boxed node to the blue-boxed node. A multi-hop directed path connects the two nodes through intermediate components, yet the model answers "false" instead of the ground-truth "true" , indicating a failure to compose multiple directed relations.
Figure 18: Information-Flow Tracing failure for Qwen3.5-VL-9B. The query asks for the direct neighbor reachable from the target node Value Iteration along the specified information-flow direction. The figure contains a direct edge from Value Iteration to Equivalent LP , yet the model returns an empty list instead of the ground-truth ["Equivalent LP"] , indicating a failure to trace the outgoing information flow correctly.
Figure 19: Parser-induced cue error in Information-Flow Tracing for Qwen3.5-VL-9B. The canonical input is shown above and the SciFigCue input is shown below. The model answers correctly under the canonical input, but fails under SciFigCue because incorrectly parsed arrow cues are overlaid near the selected node and obscure valid arrow evidence in the original figure. Ground truth: ["LLM temporal structuring"] . Canonical answer: ["LLM temporal structuring"] . SciFigCue answer: ["List of dense, multi-event captions"] .
Figure 20: Residual reasoning failure in Edge-Level Verification for Qwen3.5-VL-9B. The canonical input is shown above, and SciFigCue augments it with the additional visual input shown below; both images are provided to the model. This example is a negative two-hop case asking whether a direct edge exists from the red-boxed node to the blue-boxed node. SciFigCue provides correct structural cues, but the two nodes are connected only through an intermediate node rather than by a direct edge. The model nevertheless answers "true" instead of the ground-truth "false" , indicating persistent confusion between direct and indirect relations despite correct cues.