cs.CLMar 12, 2026
SaveSciMDR: Advancing Scientific Multimodal Document Reasoning
Organizations: University of Chicago · Yale University · TCS Research
Abstract
Constructing scientific multimodal document reasoning datasets for foundation model training involves an inherent trade-off among scale, faithfulness, and realism. To address this challenge, we introduce the synthesize-and-reground framework, a two-stage pipeline comprising: (1) Claim-Centric QA Synthesis, which generates faithful, isolated QA pairs and reasoning on focused segments, and (2) Document-Scale Regrounding, which programmatically re-embeds these pairs into full-document tasks to ensure realistic complexity. Using this framework, we construct SciMDR, a large-scale training dataset for cross-modal comprehension, comprising 300K QA pairs with explicit reasoning chains across 20K scientific papers. We further construct SciMDR-Eval, an expert-annotated benchmark to evaluate multimodal comprehension within full-length scientific workflows. Experiments demonstrate that models fine-tuned on SciMDR achieve significant improvements across multiple scientific QA benchmarks, particularly in those tasks requiring complex document-level reasoning.
Figures & tables
Figure 1: The Faithfulness-Realism Dilemma in scientific data synthesis and our proposed solution. Existing approaches face an inherent trade-off: simplifying context ensures faithfulness but lacks real-world complexity, while generating directly from full documents ensures realism but risks hallucination. We resolve this by decoupling the objectives into a two-stage synthesize-and-reground framework. By first generating verified QA pairs on atomic contexts and subsequently re-embedding them into full-document tasks, we achieve a dataset that simultaneously satisfies Scale, Faithfulness, and Realism .
| Category | Data | CoT | Q-Gen | Num QA | Source | Domain | Full Text | Visual |
| Bench. | QASPER Dasigi et al. (2021) | human | 5K | 1.5K papers | NLP | × | × | |
| QASA Lee et al. (2023) | human | 1.8K | 112 papers | AI/ML | ✓ | × | ||
| ArgSciChat Ruggeri et al. (2023) | human | 41 | 20 papers | NLP | ✓ | × | ||
| MMLongBench-Doc Ma et al. (2024) | - | human + llms | 2.5K | 1612 charts | STEM | ✓ | ✓ | |
| CharXiv Wang et al. (2024a) | human | 11.5K | 2.3K charts | STEM | × | ✓ | ||
| ChartQAPro Masry et al. (2025) | human + llms | 1.9K | 1.3K charts | STEM | × | ✓ |
Table 1: Comparison of Scientific QA Benchmarks & Datasets. Unlike prior works that rely on sanitized contexts or lack reasoning annotations, SciMDR integrates Full-Text understanding, Visual modality, and explicit chain-of-thought reasoning at scale , bridging the gap between faithful synthesis and realistic document complexity.
Table 2: Detailed statistics of the SciMDR-Eval benchmark (left) and SciMDR training dataset (right). SciMDR-Eval is categorized by reasoning type, a taxonomy that also guides the synthesis of multi-modal samples in SciMDR ; while SciMDR is categorized by modality
Figure 2: Overview of the synthesize-and-reground framework. The pipeline operates in two stages: Claim-Centric QA Synthesis ensures faithfulness by extracting atomic claims and employing backward reasoning to generate QA pairs with chain-of-thought; Document-Scale Re-grounding ensures realism by re-embedding these pairs into full-document contexts and injecting information localization steps to create hard training instances.
| Model | ChartQA | CharXiv-D | CharXiv-R | SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval |
| GPT-5.1 | - | 90.9 | 58.3 | 79.4 | 79.8 | 71.6 | 47.2 |
| GPT-5.2 | - | 95.2 | 73.1 | 79.9 | 75.4 | 74.0 | 49.9 |
| Qwen-3-VL-8B | 87.4 | 74.2 | 40.1 | 73.2 | 64.0 | 62.3 | 34.2 |
| LLaVA-OV-1.5-8B | 85.9 | 66.3 | 32.9 | 66.0 | 62.7 | 51.1 | 15.5 |
| InternVL-3-8B | 86.2 | 66.7 | 34.6 | 59.6 | 46.9 | 40.8 | 16.8 |
| Qwen2.5-VL-7B | 84.6 | 65.0 | 37.7 | 66.4 | 56.6 | 48.9 | 19.8 |
Table 3: Main results on scientific QA benchmarks. Fine-tuning with SciMDR outperforms the base model and the recent synthetic dataset across most metrics, particularly on complex reasoning tasks.
| Model | SciMDR-Eval |
| GPT-5.2 | 49.9 |
| GPT-5.1 | 47.2 |
| GPT-4o | 24.7 |
| Qwen2.5-VL-7B | 19.8 |
| + SciMDR | 49.1 +29.3 |
Table 4: Performance comparison on SciMDR-Eval against advanced models. Despite having only 7B parameters, our model matches the performance of GPT-5.2 and GPT-5.1 on this domain-specific task.
| Input | SciMDR-Eval |
| Standard | 19.8 |
| Oracle | 32.9 |
| Full-Paper | 12.8 |
Table 7: Challenge of Attention Dilution. Effect of context noise on accuracy. Performance degrades as the amount of irrelevant context increases.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Failure Type | Qwen | SciMDR |
| Incorrect Evidence Localization | 18 | 5 |
| Reasoning / Logic Error | 6 | 9 |
| Hallucination of Context | 11 | 3 |
| Incomplete Synthesis | 8 | 7 |
| Total Errors | 43 | 24 |
Table 8: Failure type comparison on 100 randomly sampled SciMDR-Eval questions.
Figure 3: TQA generation prompt. This prompt generates questions testing deep understanding of scientific content without visual evidence.
Figure 4: LLM judge prompt. This prompt evaluates model responses based on text citation (0.30), image citation (0.30), and answer accuracy (0.40).
Figure 5: Claim extraction prompt. This prompt guides the LLM to distill paragraphs into structured, verifiable claims serving as blueprints for QA generation.
Figure 6: Visual grounding prompt. This prompt matches textual claims with visual evidence, determining relationship types (Supports, Quantifies, Illustrates, Elaborates, Contradicts).
Figure 7: MQA generation prompt. This prompt generates questions requiring synthesis of textual and visual information across five reasoning types (EEQ, CIM, HVI, CAC, ARS).
Figure 8: VQA generation prompt. This prompt generates questions answerable solely from visual information across eight reasoning categories.
Figure 9: Example of EEQ (Evidence-Based Explanation & Quantification) type question. This example demonstrates how the model must explain how visual patterns (correlation matrix) support textual claims with quantitative analysis, integrating statistical interpretation from the figure with conceptual explanations from the text.
Figure 10: Example of CIM (Concept-to-Instance Mapping) type question. This example shows how the model links abstract architectural components (encoder, decoder, ResidualLSTM) described in text to their concrete visual representations in the system diagram, tracing information flow across modules.
Figure 11: Example of HVI (Hypothesis Validation & Inferential Reasoning) type question. This example illustrates inferential reasoning where the model analyzes distributional patterns in violin plots alongside textual explanations to infer underlying factors explaining behavioral differences across models.
Figure 12: Example of CAC (Critical Analysis & Consistency Check) type question. This example demonstrates critical evaluation of whether textual claims are accurately supported by visual data, requiring careful assessment of evidence strength and potential discrepancies.
Figure 13: Example of ARS (Argumentative Role & Synthesis) type question. This example shows how the model synthesizes visual evidence and textual arguments to articulate the overall scientific contribution and understand the role of visual elements in supporting the main thesis.
More in this topic
Multimodal Reasoning
Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.
See all 508 papers