Understanding and operating with scientific charts is a standard approach in data analysis, and while humans are good at interpreting such charts, open vision-language models (VLMs) are known to perform poorly when it comes to question answering based on such data types. Motivated by an initial observation that many of their errors are due to inexact answering, we raise the following natural question: how much accuracy can we recover without changing the model at all? In this paper, we study this question by searching over inference programs and the code around a "frozen" version of the model. Using ideas from autonomous evolution (famously used in AlphaEvolve), we first start from a baseline inference program, and in each round we show an LLM judge the code together with a few failures on training questions, and ask for an improving evolutionary move. As usual in autonomous evolution approaches, the candidate programs are kept in a small pool ranked by training accuracy, and the next edit is performed on top-scorers from this pool. We run this approach on three VLMs and four popular benchmarks on scientific charts, and we measure the fraction of descriptive questions whose answer exactly matches the ground truth. The accuracy improves in all experimental settings. On average, it rises from 16.6% with the starting program to 41.3% with the evolved one, and the largest single increase is 55.2 percentage points. As a side benefit, our evolved program is 3.8x faster than the starting program, because short answers are easier to grade and faster to generate. Our results demonstrate that a large part of the observed errors arise due to the inference pipeline, and that optimizing it can improve the "frozen" VLM without the need to retrain it.
Figures & tables
Figure 1: Our autonomous evolution framework for better VLMs. Fixed inputs (left) feed an evolution engine (middle) that outputs deployable inference programs (right). The model id, greedy decoding, and the train/holdout split never change during a run; only the code inside the EVOLVE-BLOCK of a candidate is being mutated. An LLM mutator (in our case Gemini-2.5-flash) proposes edits, a validator rejects broken candidate programs before any GPU work, an evaluator scores survivor programs by exact-match accuracy, and a population manager keeps the best-by-holdout program. The same engine produces all twelve runs by swapping the seed program and the benchmark evaluator.
Figure 2: Seed vs. our evolved descriptive exact-match accuracy on all 12 model-benchmark pairs. Annotations give the absolute gain in percentage points by our method. As observed, every pair improves with the largest improvement being +55.2 percentage points. The largest jumps are on Qwen3-VL-2B, whose naive seed produced verbose, wrongly formatted answers; the smallest is SmolVLM2/CharXiv, whose seed was already clean (EM stands for Exact-Match).
CharXiv
ChartQA
EvoChartQA
PlotQA
Model
Seed
Evolved
Seed
Evolved
Seed
Evolved
Seed
Evolved
Qwen3-VL-2B
38.3
66.4
0.5
55.7
1.0
47.4
0.0
33.3
InternVL3
23.4
52.3
49.5
62.0
6.8
36.5
10.9
13.5
SmolVLM2
35.2
37.5
11.5
41.1
9.9
28.6
12.5
21.4
Table 1: Descriptive exact-match accuracy (%) of the seed and the evolved program for all 12 runs , computed on the full descriptive portion of each benchmark. Mean over runs: seed 16.6%, evolved 41.3%.
Figure 3: The evolution loop. In every generation, the split loader feeds the train and holdout windows to the evaluator. The LLM-mutator proposes edits, the validator checks them before any GPU work is performed, and the population manager selects the next parent and writes the best-by-holdout candidate to disk. Every five generations, a crossover step combines two strong parents into one child.
#
Invariant
Check
1
Markers
EVOLVE-BLOCK present
2
Deterministic decode
no do_sample=True
3
Model unchanged
backbone id preserved
4
Signature
vlm_inference(...) intact
5
Syntax
ast.parse() succeeds
Table 2: The five validator invariants. Checks 1–4 are cheap string tests run before the costlier syntax and import stage; the whole gate runs before any GPU work.
Question
Gold
Seed output ( × )
Evolved
Highest labeled y -tick?
60
“The y -axis label is … the highest tick value is 58 …”
60
Year of largest blue–green gap?
2018
“… we calculate: 2005:∣43−46∣=3 , 2016… ”
2018
% supporting TikTok ban, Sept.
38
“… the value is 28 . Therefore, 28% ”
38
% with prior military service, 2023
17
“ 16% ”
17
Table 3: Examples of incorrect instances (abbreviated). The seed baseline perceives incorrectly and often returns a long, verbose response; the evolved program returns the exact answer. Wrong values in the seed output are underlined. Examples are drawn from Qwen3-VL-2B, InternVL3 and SmolVLM2 runs.
Program
Desc. EM
Rsn. EM
QPS
Overall
seed (starting)
38.3%
4.7%
0.33
0.061
hand-tuned (human)
56.3%
17.2%
1.20
0.198
evolved
58.6%
9.4%
0.78
0.138
best_accuracy
58.6%
10.9%
0.85
0.148
best_speed
58.6%
14.1%
1.24
0.205
best_overall
57.0%
9.4%
1.22
0.201
Table 4: Qwen3-VL-2B / ChartQA, all six programs (128 descriptive and 64 reasoning samples). best_speed is both the fastest and the highest on the aggregate score, beating the hand-tuned baseline.
Figure 4: Accuracy vs. throughput, Qwen3-VL-2B/ChartQA . Evolution moves the program up and to the right of the seed; best_speed dominates on both axes.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Absolute gain in descriptive exact match for each of the 12 runs, sorted. Color denotes model. Gains cluster by whether the seed’s failure was one of formatting (large, recoverable) or perception (small headroom).
Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large language and multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/datasets/ReadingTimeMachine/visual\_qa\_multipanel
Daniel J. Evans, J. P. Naiman, JooYoung Seo +1
School of Information Sciences, University of Illinois, Urbana-Champaign, 61820, USA
Vision-Language Models (VLMs) have demonstrated remarkable progress in chart understanding, largely driven by supervised fine-tuning (SFT) on increasingly large synthetic datasets. However, scaling SFT data alone is inefficient and overlooks a key property of charts: charts are programmatically generated visual artifacts, where small, code-controlled visual changes can induce drastic shifts in semantics and correct answers. Learning this counterfactual sensitivity requires VLMs to discriminate fine-grained visual differences, yet standard SFT treats training instances independently and provides limited supervision to enforce this behavior. To address this, we introduce ChartCF, a data-efficient training framework designed to enhance counterfactual sensitivity. ChartCF consists of: (1) a counterfactual data synthesis pipeline via code modification, (2) a chart similarity-based data selection strategy that filters overly difficult samples for improved training efficiency, and (3) multimodal preference optimization across both textual and visual modalities. Experiments on five benchmarks show that ChartCF achieves superior or comparable performance to strong chart-specific VLMs while using significantly less training data.
Jianzhu Bao, Haozhen Zhang, Kuicai Dong +5
1Nanyang Technological University · 2Aumovio Singapore Pte. Ltd.
Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.
Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell +1
University of Waterloo · Vector Institute · Layer 6 AI