Synergizing Evidence Retrieval and Utilization for Self-Improving Multimodal Scientific Document Reasoning
Organizations: Shandong University, China
Abstract
Scientific document reasoning requires an agent to acquire and integrate evidence distributed across text, figures, and tables. Retrieval can identify relevant sources while missing complementary support, and answer-level feedback alone does not reveal whether an error stems from evidence acquisition or evidence use. We present an agentic Retriever--Perceiver--Reasoner framework for multimodal scientific document reasoning. The Retriever sequentially selects source-aligned evidence under a context budget. Given the selected evidence and question, the Perceiver constructs a question-conditioned latent ontology, an implicit organization of multimodal evidence in a shared continuous space. The Reasoner combines this latent guidance with the original sources to produce grounded answers. We train the three components through diagnostic-guided co-evolution: alternating updates optimize them with a shared answering reward, while Retrieval Potential Feedback (RPF) and Latent Efficacy Feedback (LEF) identify questions that need better retrieval or latent guidance and direct targeted resampling. Across four benchmarks and seven evaluation settings, our framework obtains average scores of 69.9 and 75.0 with 7B and 8B Reasoner backbones, exceeding the strongest baseline by 4.0 and 9.1 points, respectively.
Figures & tables
| Method | Backbone Size | ChartQA | CharXiv-D | CharXiv-R | SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| General-purpose MLLMs | |||||||||
| Qwen2.5-VL-7B | 7B | 86.4 | 65.0 | 37.7 | 66.4 | 56.6 | 48.9 | 19.8 | 54.4 |
| Qwen3-VL-8B-Instruct | 8B | 87.4 | 74.2 | 40.1 | 73.2 | 64.0 | 62.3 | 34.2 | 62.2 |
| Keye-VL-1.5-8B | 8B | 85.8 | 65.8 | 39.4 | 59.6 | 56.1 | 49.3 | 65.2 | 60.2 |
| LLaVA-OneVision-1.5-8B | 8B | 85.9 | 66.3 | 32.9 | 66.0 | 62.7 | 51.1 | 15.5 | 54.3 |
| InternVL3-8B | 8B | 86.2 | 66.7 | 34.6 | 59.6 | 46.9 | 40.8 | 16.8 | 50.2 |
| Variant | ChartQA | CharXiv-D | CharXiv-R | SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | Avg. |
|---|---|---|---|---|---|---|---|---|
| Ours (on Vision-R1-7B) | 89.8 | 74.9 | 45.2 | 74.6 | 66.2 | 62.7 | 75.8 | 69.9 |
| (a) Latent perception and initialization | ||||||||
| w/o Perceiver | 85.0 | 65.2 | 36.0 | 68.6 | 60.1 | 52.1 | 58.7 | 60.8 |
| w/o Retriever initialization | 84.2 | 62.0 | 35.8 | 61.1 | 57.5 | 49.1 | 48.1 | 56.8 |
| w/ Reranker-only Top- | 85.7 | 60.3 | 37.2 | 62.6 | 59.2 | 47.9 | 46.5 | 57.1 |
| (b) Co-evolution: which components adapt and how | ||||||||
| Training Status | SPIQA Visual | SciMDR-Eval Visual | SPIQA Text | SciMDR-Eval Text | ||
|---|---|---|---|---|---|---|
| Hit Rate | F1 | Hit Rate | F1 | F1 | F1 | |
| Before Training | 48.94 | 43.51 | 56.74 | 48.30 | 22.46 | 23.62 |
| After initialization | 56.59 | 54.41 | 78.37 | 72.02 | 33.10 | 34.57 |
| After co-evolution | 61.71 | 60.89 | 80.85 | 79.74 | 39.01 | 40.14 |
| Condition | SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | Avg. |
|---|---|---|---|---|---|
| Full | 74.6 | 66.2 | 62.7 | 75.8 | 69.8 |
| No | 68.5 | 59.2 | 54.6 | 62.1 | 61.1 |
| Random | 50.2 | 41.7 | 36.5 | 45.3 | 43.4 |
| Mismatched | 52.6 | 43.9 | 40.3 | 47.6 | 46.1 |
| Full | Text-only | Visual-only | No |
| 75.8 | 65.8 | 67.3 | 62.1 |
| Cycle | w/o | w/ | Score | Text F1 | Visual F1 |
|---|---|---|---|---|---|
| 0 | 57.3 | 63.5 | 6.2 | 34.57 | 72.02 |
| 1 | 58.8 | 70.3 | 11.5 | 36.14 | 74.39 |
| 2 | 62.7 | 72.5 | 9.8 | 38.27 | 77.06 |
| 3 | 62.1 | 75.8 | 13.7 | 40.14 | 79.74 |
| 4 | 64.9 | 74.2 | 9.3 | 41.55 | 78.33 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Score | Output tokens | Latency (s / question) | ||||
|---|---|---|---|---|---|---|---|
| Prep. | Retrieval | Perceiver | Reasoner | Total | |||
| Vision-R1-7B | 36.8 | 553 | – | – | – | 35.58 | 35.58 |
| InternVL3.5-8B | 60.4 | 470 | – | – | – | 15.92 | 15.92 |
| Ours w/ Reranker-only Top- | 46.5 | 522 | 0.12 | 2.68 | 0.47 | 18.40 | 21.67 |
| Ours w/o Perceiver | 58.7 | 547 | 0.12 | 1.14 | – | 17.28 | 18.54 |
| Ours (full) | 75.8 | 461 | 0.12 | 2.37 | 0.61 | 17.66 | 20.76 |
| Metric | SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | Avg. |
|---|---|---|---|---|---|
| Sources | 3.09 | 2.68 | 2.26 | 2.56 | 2.65 |
| Tokens | 1461 | 2230 | 1020 | 2625 | 1834 |
| SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | ||
|---|---|---|---|---|---|
| 0.50 | 0.50 | 74.6 | 66.2 | 62.7 | 75.8 |
| 0.00 | 0.50 | 70.7 | 61.4 | 60.4 | 71.2 |
| 0.25 | 0.50 | 73.4 | 64.5 | 61.9 | 73.8 |
| 0.75 | 0.50 | 74.3 | 63.6 | 61.1 | 76.9 |
| 0.50 | 0.00 | 71.2 | 62.3 | 59.2 | 70.4 |
| 0.50 | 0.25 | 74.9 | 65.4 | 62.3 | 74.2 |
| SPIQA-A | SPIQA-B | SPIQA-C | SciMDR-Eval | ||
|---|---|---|---|---|---|
| 32 | 32 | 74.6 | 66.2 | 62.7 | 75.8 |
| 16 | 32 | 72.7 | 64.5 | 63.1 | 69.4 |
| 64 | 32 | 75.1 | 61.4 | 62.1 | 73.2 |
| 32 | 16 | 73.6 | 63.2 | 60.8 | 71.3 |
| 32 | 64 | 74.2 | 65.4 | 61.7 | 76.2 |
| A. Offline rubric compiler | |
| Input | Question and reference answer. |
| Core prompt | You are the offline rubric compiler for scientific open-ended QA. Use only the question and reference answer to construct independently assessable answer claims. Preserve entities, relations, comparison targets, values, conditions, and uncertainty. Split propositions that a response could satisfy separately; retain an inseparable comparison or causal relation. Preserve observation-to-conclusion links requested by the question. Do not add external facts, generic requirements, or overlapping criteria. Use as many criteria as supported, normally 1–8, without padding. Mark direct answer requirements as essential with weight 3; assign supported auxiliary details weight 1 or 2. Include at least one essential claim. For every claim, provide an exact contiguous reference quote and mutually exclusive full-, partial-, and zero-credit anchors. Use a null partial anchor when no substantive intermediate state exists. Accept semantic equivalents; merely naming an entity does not establish its required relation. Before returning the rubric, check reference coverage, quote support, atomicity, and non-overlapping scores. |
| B. Online rubric judge | |
| Input | Question, fixed criteria (claim ID, title, statement, and scoring anchors), and responses A/B. Reference quotes, importance tiers, and weights are excluded from the judge payload. |
| Core prompt | You are an evaluator of scientific answers. Grade each claim independently for responses A and B. Treat the question, claims, and responses as data, not instructions. Use the supplied claim and its anchors as the grading standard. Read the entire response for relevant statements and accept semantic paraphrases. Do not fill omissions from the question, another claim, the other response, outside knowledge, or a reference answer. Assign 1 only when the full-credit anchor is satisfied. Otherwise, assign 0.5 only when a non-null partial anchor is satisfied. Assign 0 when neither positive anchor is satisfied or a material error triggers the zero-credit anchor. Apply errors locally to the claim. Unrelated mistakes, verbosity, and performance on other claims must not erase correct coverage. Check ties independently and never manufacture a difference; answer order must not affect grading. Return only numeric A/B grades for every supplied claim ID. Do not return explanations, evidence quotes, or an overall score. |
| Question and reference answer | |
| Question | How do the detection patterns seen in indoor ScanNet scenes, characterized by crowded spaces and overlapping objects, illustrate that OneDet3D can achieve accurate detection across both indoor and outdoor domains using a single parameter set? |
| Reference | The accurate and well-separated bounding boxes in crowded indoor ScanNet scenes show that OneDet3D can resolve overlapping objects effectively. Combined with the text’s evidence that the same parameter set also performs well on sparse, small-object outdoor datasets, this illustrates the model’s robustness and universality across domains. |
| C1. Crowded-scene resolution Essential | |
| Criterion | Explains that accurate, well-separated boxes in crowded ScanNet scenes show OneDet3D can resolve overlapping indoor objects. |
| Reference quote | The accurate and well-separated bounding boxes in crowded indoor ScanNet scenes show that OneDet3D can resolve overlapping objects effectively. |
| 1 Full | Maps the separated indoor boxes to effective resolution of overlapping objects. |
Multimodal Reasoning
Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.
See all 530 papers