RE-SciVQA: Improving Scientific VQA With Context-Relevant Retrieval and Lightweight MLLM Adaptation
Authors: Sheikh Tahmim Mehdi, Muminul Islam Jami
Organizations: Department of Electrical and Computer Engineering, North South University, Dhaka 1229, Bangladesh · Department of Computer Science and Engineering, United International University (UIU), Dhaka 1212, Bangladesh
A deep understanding of scientific literature often relies on interpreting figures that contain key information beyond the main text. Leveraging Large Language Models (LLMs) to extract insights from these figures can significantly accelerate knowledge discovery. Existing works on Scientific Visual Question Answering rely on resource-intensive training pipelines that are often inaccessible and require large computational resources. Instead of designing complex architectures, we propose RE- SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training. Using a simple query expansion step, relevant section headers are appended to the original question to improve the retrieval coverage. In addition, a simple adjustment of similarity scores is applied using figure references and caption alignment. On the SciVQA benchmark, RE-SciVQA achieves a ROUGE-1 F1 score of 73.9%, outperforming chart-focused models such as UniChart (14.7%), ChartGemma (6.9%), and Matcha-ChartQA (6.3%). Our relevance-enhanced RAG setup also improves the exact match by 11.9% and the fuzzy match by 29.5% compared with a standard RAG baseline, resulting in an overall gain of 14.3%. We also examine closed-ended infinite visual questions, a category in which chart-understanding models often struggle. Our experiments show that using plot digitization instead of synthetic captions leads to an 18.5% improvement for this question type.
Figures & tables
Fig. 1: Comparative analysis of model performance on the RE-SciVQAbenchmark. The chart plots the performance of various models based on the ROUGE-1 F1 score [ 17 ] (%). RE-SciVQAachieves a peak performance of 73.9%, significantly outperforming other evaluated models such as UniChart (14.7%), ChartGemma (6.9%), and Matcha-ChartQA (6.3%).
Notation
Description
q,q′,Qexp
Original query, single expanded query, and the full set of expanded queries.
Φ(⋅),V
Text embedding function and the pre-computed document block embeddings.
Cpool,Cbest
Initial retrieved candidate pool and the final selected top- K context blocks.
N,K
Count of initially retrieved blocks ( N ) and final selected blocks ( K ).
ci
The i -th candidate text block in the retrieval pool.
Srel(ci)
Cosine similarity score between the expanded query and candidate ci .
TABLE I: Summary of key notations used in the RE-SciVQAframework.
Fig. 2: RE-SciVQAArchitecture diagram
Fig. 3: Structure of the prompt used in RE-SciVQA.
Dataset
Number of Samples
Train
15,100
Validation
1,680
Test
4,200
Total
20,980
TABLE II: Train, validation and test split for the SciVQA dataset.
Model
Size
R-1 F1
R-L F1
RE-SciVQA
4B
0.739
0.738
ChartGemma
3B
0.069
0.066
Matcha-Chartqa
282M
0.063
0.063
UniChart
140M
0.147
0.147
With Chain of Thought (CoT)
InternVL3 [ 22 ]
8B
0.740
0.740
TABLE III: Performance comparison of RE-SciVQAvs other models with various sizes, comprising with and without Chain-of-Thought (CoT) on SciVQA dataset.
Question Type
Qwen-3 4B (Zero-Shot + caption)
RE-SciVQA
Baseline
with RAG
without RAG
with RAG
R1-F1
RL-F1
R1-F1
RL-F1
R1-F1
RL-F1
R1-F1
RL-F1
Closed-ended infinite (visual)
0.488
0.488
0.497
0.496
0.518
0.517
0.541
0.541
Closed-ended infinite (non-visual)
0.655
0.653
0.647
0.643
0.672
0.669
0.676
0.673
Closed-ended finite (binary visual)
0.735
0.735
0.736
0.736
0.738
0.738
0.771
0.771
Closed-ended finite (binary non-visual)
0.761
0.761
0.761
0.761
0.793
0.793
0.821
0.821
TABLE IV: RE-SciVQAvs. Qwen-3 4B Zero-Shot performace comparison across different question types using R1-F1 and RL-F1 Metrics.
Fig. 4: Qualitative comparison of RE-SciVQAwith other baselines.
Metric
Baseline RAG
Heuristic Scoring
Answer Presence in Contexts
Exact Match
27.8%
31.1%
Fuzzy Match
4.4%
5.7%
Total Found
32.2%
36.8%
TABLE V: Comparison of retrieval quality between baseline RAG vs our proposed retrieval mechanism with heuristic scoring along with query expansion. Answer presence measures the frequency of the ground truth answer string appearing within the retrieved context.
Scientific method figures often provide researchers with a clear visual entry point into a paper's methodology and core contributions, yet their dense layouts and complex information flows are difficult for Vision Language Models (VLMs) to interpret. Existing benchmarks predominantly focus on coarse-grained tasks, such as context-based retrieval or holistic comprehension, overlooking the fine-grained content and inter-module relations within complex methodological diagrams. To address these overlooked aspects, we introduce Scientific Figure Benchmark (SciFigBench), a benchmark for evaluating core capabilities required for scientific method figure understanding. Specifically, it targets three capabilities: dense content perception, visual relation comprehension, and information-flow tracing. For this purpose, we curate 4,352 scientific method figures from real-world research papers and construct 40,670 QA pairs to operationalize these capabilities as measurable question-answering tasks. All annotations are generated by dedicated pipelines and then manually verified. We further propose SciFigCue, a training-free visual prompting module that externalizes arrow-induced dependencies as visual relational evidence for method-figure reasoning. Experiments across open-source and proprietary VLMs show that even strong models remain substantially limited in visual relation comprehension, while SciFigCue improves performance across several tasks without parameter updates. The code and related resources are available at https://github.com/LeNcyer/SciFigBench.
Lijia Wu, Zheng Li, Yuxuan Liu +3
VCIP, CS & AAIS, Nankai University · NKIARI, Shenzhen Futian · Li Auto.
Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large language and multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/datasets/ReadingTimeMachine/visual\_qa\_multipanel
Daniel J. Evans, J. P. Naiman, JooYoung Seo +1
School of Information Sciences, University of Illinois, Urbana-Champaign, 61820, USA
Scientific charts encode quantities in axes, legends, and geometric marks, yet large vision-language models still treat them as natural photographs. Visual in-context examples do not expose the coordinate frame; unconstrained chain-of-thought can name a plausible number that was never read from a bar. We present STEER (Structure-Token Evidence-anchored Reasoning), which freezes a Llama-3.2-Vision encoder and inserts three modules: a chart structure graph encoder (CSGE) that binds ticks, legend items, and marks; evidence-anchored step reasoning (EASR) that forces every arithmetic step to cite a graph node; and weak-parser strong-reasoner alignment (WPSR) that uses a specialized table extractor only as a teacher of node attributes. On ChartQA, STEER reaches 82.70 average relaxed accuracy versus 80.16 for ChartGemma and 76.40 for a LLaVA-CoT backbone trained on the same mix. Gains widen on CharXiv reasoning (33.60 vs. 29.20 InternVL Chat V1.5) and ChartQAPro CoT (40.70 vs. 37.17 Qwen2-VL-7B), where OCR shortcuts disappear. Ablations show that dropping node serialization or numeric candidate constraints undoes most of the reasoning lift.
Alberlucia Rafael Soarez, Camila Ferreira, Daniel Kim +2