Question-Guided Evidence Acquisition for Multimodal Visual Question Answering

Alin-Ionut Popa

cs.CVAug 20, 2026

Question-Guided Evidence Acquisition for Multimodal Visual Question Answering

Authors: Alin-Ionut Popa

Organizations: Amazon Inc.

Abstract

Multimodal LLMs can see a document, but they often can't read it reliably. Small text, tables, visual cues, and topological elements still trip them up under direct visual inference, even when the page is already sitting in the model's context. Most document-VQA systems treat perception as fixed: they encode the page once, ask the question, and answer from whatever the model happened to extract in that single fast pass. We think document VQA needs slower, more deliberate perception: rather than answering from one fixed encoding, the model should spend a bit of extra compute at inference time working out what to look at next, and only then answer. We build this into Q-Guide, a small agent that reads a question, works out what evidence it is still missing, and calls targeted tool(s) to recover it---reading text where text is needed, zooming in where detail is needed, or grounding a region where position matters. On DocVQA2026 and Manga109, Q-Guide outperforms both direct prompting and recent multi-agent document systems ( $65.0\%$ vs.\ $40.0\%$ on DocVQA2026, $32.4\%$ vs.\ $24.4\%$ on Manga109), and the improvement holds across three Claude backbones (Opus 4.6, Sonnet 4.6, and Opus 4.5). We find that accuracy scales with the perception budget---most of the gain appears within two to three deliberate rounds---and that the gain comes from directing perception to the right place, not from complex control logic: adding planners, routers, or multiple collaborating agents does not help.

Figures & tables

Figure 1: From uniform perception to question-conditioned evidence recovery. Direct MLLM inference ( Left ) processes the document uniformly and answers from incomplete perception. Q-Guide ( Right ) directs perception toward the question—locating landmarks, reading targeted labels, and verifying spatial relationships—before committing to an answer.

Figure 2: Overview of Q-Guide. Given unstructured multimodal content and a user question, Q-Guide makes perception question-conditioned: the agent inspects the document, reasons about what evidence is still missing, selects perceptual recovery tools to fill the gap, and updates its evidence state. The loop continues until the recovered evidence supports an answer.

Figure 3: M109NC task example. For visualization, we show 3 of the 6 context pages (orange border) that reference the query character; the query page highlights the target (red overlay). The task requires cross-page reasoning: reading Japanese dialogue, associating names with visual appearances, and matching the target’s identity across scenes.

Configuration	T	V	Q	S	P	Acc.	Drop
Q-Guide (full)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$65.0\%$	—
$-$ Text	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$51.9\%$	$\downarrow\,13.1$
$-$ Visual	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$47.8\%$	$\downarrow\,17.2$
$-$ Query	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$48.8\%$	$\downarrow\,16.2$
$-$ Structure	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$57.8\%$	$\downarrow\,7.2$
$-$ Spatial	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$61.1\%$	$\downarrow\,3.9$

Table 1: Tool importance on DocVQA2026. Leave-one-out over the five recovery tools (T: Text, V: Visual Inspection, Q: Targeted Query, S: Structure, P: Spatial), reporting accuracy and the drop when each is removed. Visual Inspection and Targeted Query are the most impactful; analyzed in Section 5.3 .

Figure 4: Q-Guide in action. A science-poster example where the answer depends on both table structure and color. The agent locates the relevant region, recovers the table and red cell values, and submits the answer only after the evidence supports it. Each evidence node $e_{i}$ shows the observation $o_{i}$ added at that step (Section 3.1 ); together they form the accumulated evidence state.

Direct prompting baselines (single LLM call, no tools)
		Document Category
Method	Type	Bus. Report	Comics x	Eng. Draw.	Infogr. x	Maps x	Sci. Paper	Sci. Poster	Slides x	Overall
Visual Prompt	Direct	$20.0\%$	$40.0\%$	$20.0\%$	$20.0\%$	$20.0\%$	$30.0\%$	$40.0\%$	$30.0\%$	$27.5\%$
OCR Prompt	Direct	$30.0\%$	$20.0\%$	$10.0\%$	$40.0\%$	$50.0\%$	$20.0\%$	$50.0\%$	$40.0\%$	$32.5\%$
Visual+OCR Prompt	Direct	$40.0\%$	$20.0\%$	$20.0\%$	$50.0\%$	$30.0\%$	$50.0\%$	$50.0\%$	$50.0\%$	$38.8\%$
Agentic baselines (multi-step, tool use)

Table 2: Results on the DocVQA2026 validation split (80 questions, 8 categories). Q-Guide achieves the best overall accuracy, with the largest gains on engineering drawings and science posters, where precise local evidence is critical. Even at this sample size, Q-Guide’s Wilson [ 47 ]

95\%

confidence interval on the overall column (

[54.1,74.5]

) sits clearly above those of the strongest baselines (Visual+OCR

[28.8,49.7]

, DocAgent/MDocAgent

[30.0,51.0]

); per-category cells (

n=10

) are noted for trends only, and the gaps are robust under paired [ 32 , 10 ] and bootstrap [ 13 ] tests.

Figure 5: Per-book accuracy on M109NC. Q-Guide (red) covers the largest area, indicating consistent gains across diverse manga styles.

Direct prompting baselines
	Text Source
Method	Claude-OCR	GT-OCR
Visual Prompt	$21.8\%$	$21.8\%$
Visual+OCR Prompt	$23.1\%$	$24.9\%$
Agentic approaches
DocAgent [ 41 ]	$24.4\%$	$29.3\%$

Table 3: Character naming on M109NC (

n=504

). Q-Guide benefits strongly from higher-quality text: ground-truth OCR lifts its accuracy to

53.7\%

. The two Wilson

95\%

intervals—

[28.4,36.5]

under Claude-OCR and

[49.4,58.1]

under GT-OCR—do not overlap, so this text-quality effect is well separated from sampling noise.

Figure 6: Orchestration and tool-use analysis. Left : Q-Guide performs best with a compact iterative tool-use loop, while heavier planning and routing do not improve accuracy. Right : tool usage changes naturally across document categories, showing that the agent adapts evidence acquisition without explicit category routing.

Direct prompting baselines
	DocVQA2026			M109NC
Method	Opus 4.6	Sonnet 4.6	Opus 4.5	Opus 4.6	Sonnet 4.6	Opus 4.5
Visual Prompt	$27.5\%$	$30.0\%$	$27.1\%$	$21.8\%$	$21.8\%$	$21.8\%$
Visual+OCR Prompt	$38.8\%$	$31.4\%$	$37.1\%$	$23.1\%$	$22.9\%$	$22.9\%$
Agentic baselines
DocAgent	$40.0\%$	$34.3\%$	$37.1\%$	$24.4\%$	$24.2\%$	$22.5\%$

Table 4: Accuracy across MLLM backbones. Q-Guide consistently achieves the best accuracy regardless of the underlying model. On M109NC, direct prompting baselines plateau around

22\%

irrespective of backbone, suggesting the bottleneck is the reasoning architecture rather than perceptual capability.

Figure 7: M109NC accuracy by context page count. Q-Guide leads at every context size and exploits additional naming pages that single-call baselines cannot, exceeding $50\%$ with $12$ or more naming pages while the baselines stay nearly flat.

Panel of Figure 8 — Figure 8: Cost and budget analysis. Left: accuracy vs. token cost on M109NC; Q-Guide breaks above the single-call MLLM ceiling (dashed, $+26$ pp) by spending tokens on active investigation rather than context stuffing. Right: accuracy vs. iteration budget ( $T_{\max}$ ) on DocVQA2026; most gain is captured within 2–3 rounds, with diminishing returns beyond $T_{\max}=3$ .

Method	In	Out	Total
Visual Only	$2{,}336$	$166$	$2{,}502$
Visual+Text	$3{,}941$	$422$	$4{,}364$
DocAgent	$3{,}452$	$603$	$4{,}055$
Q-Guide	$\mathbf{9{,}668}$	$\mathbf{683}$	$\mathbf{10{,}351}$

Table 5: Measured per-question token cost on M109NC (mean over

50

questions, Claude Opus 4.6). Q-Guide’s overhead is input-dominated (

93\%

Document QA

Document question answering uses AI to answer questions from text, tables, images, and structured content in documents.

See all 92 papers

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Apr 15, 2026Yuanlei Zheng, Pei Fu, Hang Li +9Document QA Long-Document VQA
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3Multimodal QA Multimodal Understanding
LG AI Research
VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Oct 14, 2026Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12Long-Document VQA Document QA
University of Southern California · Oracle AI · Turing Enterprise Inc +1

Apr 15, 2026J/K move · Enter open · S save

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang+8

School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology+1

Document QA Long-Document VQA VLM Reasoning

Open paper AskPDF arXiv

Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc-

V^*

, an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc-

V^*

begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc-

V^*

balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc-

V^*

outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.

Abstract

Document QA

Explore similar work

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents