cs.AIOct 30, 2026

RE-SciVQA: Improving Scientific VQA With Context-Relevant Retrieval and Lightweight MLLM Adaptation

Authors: Sheikh Tahmim Mehdi, Muminul Islam Jami

Organizations: Department of Electrical and Computer Engineering, North South University, Dhaka 1229, Bangladesh · Department of Computer Science and Engineering, United International University (UIU), Dhaka 1212, Bangladesh

Abstract

A deep understanding of scientific literature often relies on interpreting figures that contain key information beyond the main text. Leveraging Large Language Models (LLMs) to extract insights from these figures can significantly accelerate knowledge discovery. Existing works on Scientific Visual Question Answering rely on resource-intensive training pipelines that are often inaccessible and require large computational resources. Instead of designing complex architectures, we propose RE- SciVQA, which focuses on improving the quality of retrieved context and fine-tuning small LLMs with parameter-efficient LoRA training. Using a simple query expansion step, relevant section headers are appended to the original question to improve the retrieval coverage. In addition, a simple adjustment of similarity scores is applied using figure references and caption alignment. On the SciVQA benchmark, RE-SciVQA achieves a ROUGE-1 F1 score of 73.9%, outperforming chart-focused models such as UniChart (14.7%), ChartGemma (6.9%), and Matcha-ChartQA (6.3%). Our relevance-enhanced RAG setup also improves the exact match by 11.9% and the fuzzy match by 29.5% compared with a standard RAG baseline, resulting in an overall gain of 14.3%. We also examine closed-ended infinite visual questions, a category in which chart-understanding models often struggle. Our experiments show that using plot digitization instead of synthetic captions leads to an 18.5% improvement for this question type.

Figures & tables

Explore similar work

Oct 19, 2026

SciFigBench: A Benchmark for Scientific Method Figure Understanding

Scientific method figures often provide researchers with a clear visual entry point into a paper's methodology and core contributions, yet their dense layouts and complex information flows are difficult for Vision Language Models (VLMs) to interpret. Existing benchmarks predominantly focus on coarse-grained tasks, such as context-based retrieval or holistic comprehension, overlooking the fine-grained content and inter-module relations within complex methodological diagrams. To address these overlooked aspects, we introduce Scientific Figure Benchmark (SciFigBench), a benchmark for evaluating core capabilities required for scientific method figure understanding. Specifically, it targets three capabilities: dense content perception, visual relation comprehension, and information-flow tracing. For this purpose, we curate 4,352 scientific method figures from real-world research papers and construct 40,670 QA pairs to operationalize these capabilities as measurable question-answering tasks. All annotations are generated by dedicated pipelines and then manually verified. We further propose SciFigCue, a training-free visual prompting module that externalizes arrow-induced dependencies as visual relational evidence for method-figure reasoning. Experiments across open-source and proprietary VLMs show that even strong models remain substantially limited in visual relation comprehension, while SciFigCue improves performance across several tasks without parameter updates. The code and related resources are available at https://github.com/LeNcyer/SciFigBench.
Oct 9, 2026

Do you "see" what I "see"? A Multi-panel Visual Question Answer Dataset for Large Language Model Chart Analysis

Digital libraries increasingly host scientific publications whose figures encode primary research findings, yet making this visual content searchable, accessible, and reusable remains an open challenge. Large language and multimodal models (LLMs/LMMs) provide the possibility to automate many figure parsing tasks such as determining data trends or extracting data from plots. In this work, we consider if these models can reliably support task-oriented visual data analysis when presented with synthetic, but realistic, scientific figures that contain multiple charts. Our Visual Question Answering (VQA) process scaffolds our tasks along a four-level description model from basic conceptual answers to interpretive and analytical understanding. Additionally, we outline our process for synthetically generating our single and multi-panel (charts in which there are multiple panels present) plots and testing the data with various LMMs using quantitative analysis. In general, we find LMMs become less accurate with increasing figure complexity as the number of panels, number of plot elements, and difficulty of questions increase. Finally, because LMM accuracy is expected to change rapidly, our analysis also considers cost relative to performance as a metric to inform future user decisions. We provide the detailed question-answer pairs used in this analysis, along with the code used to generate random plots, question-answer pairs and detailed bounding-box metadata for chart elements. The code and data are available at https://github.com/ReadingTimeMachine/LLM_VQA_MultiPanel and https://huggingface.co/datasets/ReadingTimeMachine/visual\_qa\_multipanel
Sep 7, 2026

Structure-Token Evidence-Anchored Reasoning for Scientific Chart Understanding

Scientific charts encode quantities in axes, legends, and geometric marks, yet large vision-language models still treat them as natural photographs. Visual in-context examples do not expose the coordinate frame; unconstrained chain-of-thought can name a plausible number that was never read from a bar. We present STEER (Structure-Token Evidence-anchored Reasoning), which freezes a Llama-3.2-Vision encoder and inserts three modules: a chart structure graph encoder (CSGE) that binds ticks, legend items, and marks; evidence-anchored step reasoning (EASR) that forces every arithmetic step to cite a graph node; and weak-parser strong-reasoner alignment (WPSR) that uses a specialized table extractor only as a teacher of node attributes. On ChartQA, STEER reaches 82.70 average relaxed accuracy versus 80.16 for ChartGemma and 76.40 for a LLaVA-CoT backbone trained on the same mix. Gains widen on CharXiv reasoning (33.60 vs. 29.20 InternVL Chat V1.5) and ChartQAPro CoT (40.70 vs. 37.17 Qwen2-VL-7B), where OCR shortcuts disappear. Ablations show that dropping node serialization or numeric candidate constraints undoes most of the reasoning lift.