cs.CVSep 30, 2026

When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning

Authors: Yunbei Zhang, Janet Wang, Jihun Hamm, Chandan K Reddy

Organizations: Tulane University · Virginia Tech

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to use information that is present. We introduce a diagnostic protocol to distinguish these explanations. Using the same solver model and generation settings, we compare three input conditions: the original image, question-blind structure extracted by a vision-language model, or gold structure derived from the diagram source. Validity-triggered recovery tests truncation and schema failure, question-relevant fidelity measures preservation of answer-critical structure, and matched edge interventions test the effect of error location. On a reserved holdout of 240 public FlowGen diagrams, evaluated under a frozen protocol, gold structure reaches 87% accuracy while direct vision and learned text both remain below 30%. The aggregate comparison includes source-derived relation labels that may not be printed in the image and uses different learned and gold graph encodings, so it does not isolate extraction error alone. Retrying only invalid extractions makes nearly every public representation schema-valid yet leaves accuracy essentially unchanged. The public learned-text deficit relative to gold more than doubles with structural difficulty. Question-relevant topology predicts correctness better than whole-graph topology. In an exposed intervention study, a single answer-relevant edge edit reduces the primary solver's original-answer accuracy to near zero, while matched irrelevant edits largely preserve it. Supplied structure requires fewer solving tokens than vision, but learned acquisition removes this advantage at single use. These comparisons motivate evaluating acquired text by the answer-relevant evidence it preserves and by the solver's ability to use that representation.

Figures & tables

Figure 1: Schema-valid text can still omit answer-critical structure. For the question shown, the diagram and source give {A,B}\{A,B\} . A learned representation can retain every node yet omit D ⁣→ ⁣BD\!\to\!B , yielding only {A}\{A\} . The highlighted relation determines whether the textual representation preserves the answer. The example motivates evaluating text substitution by the structure needed for the question, rather than node coverage or parse validity alone. This is an illustrative example; edge lists are schematic rather than recorded model outputs.
Input / solverValid / 240CorrectMissingQA (%)
Direct vision–1961127.2–28.8
Fixed learned196164022.8
Recovered learned238169023.5
Gold structure–627087.1
Fixed learned →\rightarrow 27B196157021.8
Table 1: Supplied structure substantially outperforms learned acquisition. Public holdout: 240 charts, 720 questions per arm. Invalid acquisitions count as QA failures; direct bounds cover 11 missing responses. The secondary 27B solver receives the same fixed extraction.
Figure 2: Recovery closes the controlled deficit, not the public gap. (a) Separate exposed and held-out FlowGen cohorts, each 240 charts and 720 questions. Whiskers bound missing outcomes (holdout: 11 direct; exposed: 3 direct, 2 gold), not confidence intervals. (b) Post-confirmation recovery on controlled program flowcharts and Boolean circuits. The four conditions compare the original image, fixed learned text, recovered learned text, and source-derived gold. Recovery retries only invalid extractions; already-valid text is unchanged. Both panels use Qwen3.5-122B-A10B and an 8,192-token solver cap.
Figure 3: The public acquisition gap widens with chart size and branch depth. QA on the 240-chart public holdout, grouped by prespecified node-count and branch-depth bins. All conditions use the same Qwen3.5-122B-A10B solver at an 8,192-token cap; direct shading denotes missing-outcome bounds. Fixed and recovered learned curves nearly overlap, while gold remains above the 70% floor (dotted) in every occupied bin. Lines connect bins; they are not fitted curves.
ExtractionFidelity metricWholeRelevantΔ\DeltaPaired CI 
Fixed (588 Q)Topology exact0.14060.1147−- 0.0259 [−- 0.0479,−- 0.0047]†
 Edge F10.13050.1240−- 0.0065 [−- 0.0162,0.0032] 
Recovered (714 Q)Topology exact0.12420.1058−- 0.0184 [−- 0.0346,−- 0.0029] 
 Edge F10.11650.1134−- 0.0031 [−- 0.0115,0.0054] 
Table 2: Relevant topology improves QA prediction. Lower Brier loss is better; Δ\Delta is relevant minus whole on matched cases and folds. Intervals are 98.33% for the primary test ( †\dagger ), 95% otherwise. Recovery and edge F1 are secondary; edge-F1 intervals cross zero.
    4B27B122B
EditmmChartsQRel.Irrel.Rel.Irrel.Rel.Irrel.
Reverse1601725.278.54.793.63.592.4
Reverse2581223.368.91.692.60.091.0
Drop1501480.079.10.093.90.093.9
Drop2491000.078.00.093.00.094.0
Redirect1501480.774.30.092.60.093.2
Table 3: Relevant corruption is more damaging than matched irrelevant corruption. Exposed FlowGen, 512-token solver cap; QA (%) is scored against original answers. Each row matches chart/question pairs and edit count mm . The 4B solver is lower even on irrelevant edits.
InputQA (%)Tokens/Q, K=1K=1Tokens/Q, K=3K=3Comparable QA?
Direct vision27.2–28.86,4236,423–
Fixed learned22.87,5513,201No
Recovered learned23.510,4134,397No
Gold structure87.11,1691,169Yes
Table 4: Reuse saves tokens without comparable QA. Means over 720 intended holdout questions include failed acquisitions. K=3K=3 reuses the same representation and answers; gold excludes acquisition. Only acquisition is amortized; direct and gold retain their per-question solving costs. The lower paired 95% text-minus-direct bound must exceed −2-2 points for comparability.
Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

CohortChartsQuestionsRole
Public FlowGen, reserved240720Confirmatory core
Public FlowGen, exposed240720Recovery / mechanism controls
Controlled OLD271271271Mixed controlled confirmation
Controlled NEW270270270Harder controlled extension
QZhou, exposed200600Secondary counterpoint
Appendix
Table 5: Cohorts used in the reported comparisons. Charts and questions are not independent replicate counts. The controlled cohorts are combined only for their diagnostic; interventions reuse eligible exposed FlowGen cases rather than a new holdout.
Figure 4: Controlled tasks require executing the provided structure. OLD271: (a) Starting with x=−3,y=1x=-3,y=1 , report final yy at END. (b) Compute OUT from the printed input bits and NOT/XOR gates. The crops retain the connected executable components; disconnected arithmetic fragments in (a) and an input in (b) are outside the display.
Figure 5: The harder controlled task and the secondary public task ask different questions. Original-image excerpts. (a) This 20-node, depth-four NEW270 program asks for final yy from x=0,y=−3x=0,y=-3 ; execution continues beyond the crop. (b) QZhou chart 534 has Chinese labels and loops. Its native questions ask about isolated nodes, node count, and direct successors of the practice/activity node (English glosses only).
Cohort / comparisonSolverExtractorSolve capExtraction cap / input
Public holdout, primary122B-A10B122B-A10B8,192Fixed 1,024
Public holdout, recovery122B-A10B122B-A10B8,1921,024 →\to 2,048 →\to 8,192
Public holdout, compatibility27B122B-A10B8,192Same fixed text
Exposed public recovery122B-A10B122B-A10B8,192Fixed / recovered
Controlled diagnostic122B-A10B122B-A10B8,192Fixed / recovered
Exposed perturbations4B / 27B / 122B–512Edited source structure
Appendix
Table 6: A shared model family, with explicitly separated caps and roles. Output caps are limits, not measured usage. Fixed and recovered extraction are distinct policies. All local arms use BF16, temperature zero and thinking disabled; confirmatory sampling uses seed 427. The 27B confirmatory arm receives exactly the fixed 122B extraction, not recovered text.
  QA (%)Fixed −- Gold (pp)
BinChartsFixedRecoveredGoldGap95% CI
Node count
≤\leq 104955.855.894.6−- 38.8 [−- 49.7,−- 27.2]
11–2013720.020.485.2−- 65.2 [−- 71.0,−- 59.1]
21–40540.01.985.2−- 85.2 [−- 90.1,−- 80.2]
>> 400–––––
Appendix
Table 7: Prespecified difficulty bins and paired gaps. QA columns are percentages on all intended questions. Every chart has three questions. Gap is fixed learned minus gold in pp; the descriptive 95% chart-cluster interval is in a separate column. All occupied bins pass the 70% gold floor; no post-outcome merging is applied.
Outcome / contrastAxisEstimateCI
Primary tests (multiplicity-adjusted 98.33% CI)
Fixed −- Gold QANode count−- 23.1 [−- 30.7,−- 15.6]
Fixed −- Gold QABranch depth−- 11.2 [−- 14.8,−- 7.6]
Relevant −- whole Brier–−- 0.0259 [−- 0.0479,−- 0.0047]
Secondary difficulty slopes (95% CI)
Direct QANode count−- 25.0 [−- 29.3,−- 20.7]
Appendix
Table 8: Primary tests confirm a widening gap and improved relevant-topology prediction. QA slopes are pp per occupied ordinal bin, with equal chart weight and fixed gold-floor eligibility. The Brier contrast uses the same valid fixed-extraction cases. Estimates and paired chart-cluster CIs are separate; group headings give the level of each interval. All entries are exports of the frozen analysis.
Figure 6: Paired holdout and exposed-cohort comparisons. Points are paired gaps (pp) with 95% chart-cluster intervals; gray bands are full-sample missing-outcome bounds, not confidence intervals. Holdout direct comparisons use 709 complete pairs (11 unknown observations); other holdout comparisons use 720 pairs, and the valid-input 27B comparison uses the identical 588 valid fixed-input questions. Exposed-cohort intervals are descriptive, at the aligned 8,192-token solver cap.
Figure 7: Lenient scoring preserves gold–direct–learned ordering. (a) Five arms, 720 intended questions, 8,192-token solver cap. Set-F1 is macro-averaged; Jaccard uses a per-question threshold of 0.5. Direct whiskers denote missing-outcome bounds. (b) Paired gaps on 709 complete pairs with chart-cluster intervals. Both are post-hoc sensitivities.
Figure 8: Matched fidelity prediction favors relevant topology. (a) Whole (open) and relevant (filled) predictors use identical cases and folds; negative Δ\Delta favors relevance. Recovered and all-intended rows are sensitivities; labeled-edge F1 uses valid source-relation cases. (b) Secondary AUC, higher is better; no paired AUC interval was estimated.
Figure 9: Relevant corruption reduces original-answer QA across matched edits. Exposed source-input interventions at a 512-token solver cap. Gaps are relevant minus irrelevant with paired 95% chart-cluster intervals; Q counts pairs and Rel./Irrel. are QA percentages. The 4B solver is lower even under irrelevant edits.
   Fixed learned  
DomainChartsDirect1,024 cap2,048 capRecoveredGold
Circuits13599.385.289.699.394.1
Flowcharts40699.027.375.199.599.8
Appendix
Table 9: Post-confirmation recovery approaches controlled direct-vision QA. QA (%) at an 8,192-token solver cap; fixed columns retain original extractions, recovery includes normalization. All 541 recovered charts are valid, versus 111/406 flowcharts and 116/135 circuits at fixed 1,024.
Figure 10: Actual native-token costs on the public holdout. Means include all 720 intended questions and failed acquisition attempts. Light segments are acquisition (per chart in (a), amortized over the three questions in (b)); full-shade segments are solving; totals are printed at the bar ends. The identical fixed extraction is shared across the two solver arms.
 Calls per questionSeconds per question
Input / solverK=1K=1K=3K=3K=1K=1K=3K=3
Direct vision1.001.0012.6312.63
Fixed learned1.821.1519.3510.71
Recovered learned2.201.3925.8613.86
Gold structure1.001.007.127.12
Fixed learned →\rightarrow 27B1.821.1552.8044.16
Appendix
Table 10: Logical request counts and serial request latency. Entries are per intended question under single-use or three-question reuse. Latencies reflect the execution hardware and serving conditions; they are not a hardware-controlled solver-speed ranking. Cached reuse does not imply new billed inference.
Figure 11: Shortest-path gold falls below the interpretive floor. Exploratory QA by question family on intended holdout questions. Direct whiskers denote missing-outcome bounds; the dotted line marks the 70% gold floor. These slices do not add confirmatory hypotheses.
Figure 12: QZhou shows a different source-input pattern. Exposed, secondary QA at a 512-token solver cap, counting invalid inputs as failures. Bars are paired-chart bootstrap 95% intervals. Filled/open circles denote learned extraction at 1,024/2,048 tokens.
Figure 13: Valid extraction omits an answer-critical outgoing edge. FlowGen easy chart 63 (8 nodes, 8 edges). The source has opposite directions between Underwriting and Legal Review of Claims; extraction retains only the incoming direction. Display-only relation excerpts.
Figure 14: The two answer-critical incoming edges are preserved. FlowGen easy chart 290 (8 nodes, 8 edges). Extraction preserves both incoming neighbors of Digital Resource Management despite reversing an irrelevant edge. Display-only excerpts omit labels and other relations.
Figure 15: Valid JSON still confuses node and container destinations. FlowGen hard chart 108 (18 nodes, 38 edges). Gray edges are retained, dashed pink edges omitted, and the solid pink edge incorrectly added to a container. Display-only union of source and recovered outgoing neighbors.
Input / solverAcceptedCap hitsHit rate (%)Invalid skipsMissing
Direct vision709354.9011
Fixed learned588111.91320
Recovered learned714162.260
Gold structure72081.100
Fixed learned →\rightarrow 27B588193.21320
Appendix
Table 11: Residual 8,192-cap hits and missing responses on the public holdout. A cap hit is a length finish or an output token count at the cap. Rates use accepted solver responses. Invalid skips are observed QA failures; missing responses remain unknown.
Figure 16: Cap-hit counts by prespecified structural bin. Each cell is hits/accepted solver responses, not hits/intended questions; shading shows the hit rate. The empty >40>40 node bin has no observations.
Adjudicated outcomeItems
QZhou source–image consistency (30 charts)
Match28
Cosmetic difference only1
Material mismatch0
Unresolved1
QZhou answer relevance (90 questions)
Appendix
Table 12: Targeted human checks bound secondary interpretations. Counts are adjudicated item outcomes; three reviews per item do not multiply sample size. Enriched sampling precludes population prevalence estimates.

Explore similar work

CardsList
Apr 28, 2026cs.CV

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Recent vision-language models (VLMs) often achieve high answer accuracy on these tasks, yet correct answers do not guarantee that models ground their reasoning in the diagram regions that support the prediction. Models may instead rely on textual correlations or dataset artifacts without identifying the visual evidence required to verify the answer. This limitation prevents reliable evaluation of diagram reasoning and reduces interpretability. We introduce DRAGON, a benchmark for evaluating evidence-grounded visual reasoning in diagrams. Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer. These evidence regions may include answer-bearing components, textual labels, legends, axes, connectors, and other supporting structures involved in the reasoning process. The DRAGON dataset contains 11,664 annotated question instances from six diagram QA datasets: ChartQA, Circuit-VQA, InfographicsVQA, MapIQ, MapWise, and AI2D, with a 2,445-instance test set carrying human-verified evidence annotations and a standardized evaluation framework. Our evaluation on recent VLMs reveals that even Claude Opus 4.6 achieves a Grounding IoU of only 23.8% on AI2D, showing that faithful visual grounding remains an open challenge. DRAGON supports future research on models that ground their predictions in visual evidence.

Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4

Arizona State University · Adobe Research · University of Maryland

DiagramsVisual ReasoningDiagnostic BenchmarkReasoning GraphsRecent Vision-Language Models
AskPDF
Apr 29, 2026cs.CL

DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA

Diagram question answering (Diagram QA) requires reasoning-level attribution that links each question-answer pair to all visual regions needed to derive the answer, rather than only the region containing the final response. Creating such structured evidence across diagrams, charts, maps, circuits, and infographics is time-consuming, and existing annotation tools tightly couple their interfaces to dataset-specific formats. We present DIAGRAMS, a lightweight, schema-driven review framework that decouples interface logic from dataset-specific JSON structures through an internal meta-schema and dataset adapters. Given an image and QA pair with optional candidate regions, the system performs QA-conditioned evidence selection and proposes the regions required for reasoning. When QA pairs or candidate regions are missing, it generates them and supports human verification and refinement. Across six Diagram QA datasets, model-suggested evidence achieves 85.39% precision and 75.30% recall against reviewer-final selections (micro-averaged). These results indicate that the review-first framework reduces manual region creation while maintaining high agreement with final reasoning-level attributions. We release a public demo and installable package to support dataset auditing, grounded supervision creation, and grounded evaluation.

Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Manan Suri +4

Arizona State University · University of Maryland · IIITDM +1

DiagramsReasoning GraphsQuestion-Answer PairsTable Reasoning DatasetsReview
AskPDF
Sep 1, 2026cs.CV

From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding

Vision-language models (VLMs) have demonstrated strong performance in visual question answering with natural images. However, they continue to struggle with scientific diagrams, which are designed to convey functional or relational meaning rather than literal scenes. We therefore introduce a framework for generating large-scale diagram-grounded instruction data by leveraging terminology derived from scientific curricula. Our approach systematically extracts domain concepts, synthesizes atomic facts, retrieves relevant diagrams from the web, and generates multimodal supervision in the form of diagram captions and multiple-choice questions. Using this pipeline, we construct SciGram, a dataset of over 194K diagrams and 1.4M visual instructions across life, earth, and physical sciences. Despite relying on noisy web data and synthetic annotations, models fine-tuned on SciGram achieve substantial improvements on diagram-centric benchmarks, including TQA, ScienceQA, and AI2D, outperforming or matching state-of-the-art VLMs while using fewer training instances. Furthermore, augmenting existing models such as LLaVA OneVision with SciGram establishes new state-of-the-art performance on diagram question answering. Our results highlight the effectiveness of terminology-grounded instruction generation as a general strategy for improving vision-language reasoning in scientific domains. To support future research in scientific diagram understanding, we release both the SciGram dataset and models.

Raul Ortega, José Manuel Gómez-Pérez

Language Technology Research Laboratory Expert.ai 17 Henri Dunant, 28036 Madrid, Spain

DiagramsReasoning GraphsVisual GenerationVision Datasets
AskPDF