In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.
Figures & tables
Figure 1: Comparison with existing tasks and datasets. (a) DocVQA tasks mainly return textual answers or evidence regions, while our task requires textual attributes and corresponding images. (b) Existing datasets mainly rely on academic papers or general documents, which are insufficient for evaluating our task.
Figure 2: Distributions of ITJoint under the query-level and instance-level taxonomies.
Statistic
Number
Documents
Total pages
2,455
Avg. pages
59.88
Max / Min pages
178 / 22
Avg. images / page
1.77
Set Queries
Table 1: Statistics of our collected ITJoint.
Figure 3: Overview of Q2IT: a three-agent framework for query-driven image-text joint extraction. The Evidence Agent performs evidence collection by collecting textual attribute values and image-reference clues, the Page Agent performs page localization by identifying target pages and coarse spatial descriptions, and the Figure Agent completes image-text binding by selecting target image regions and producing final answer instances.
Model
Single Pos.
Single Neg.
Set Pos.
Set Neg.
1-to-0
1-to-1
1-to-N
Indep.
Comp.
Intra
Cross
Exp.
Imp.
Standalone VLMs
Gemini-3.1-Pro-Preview
0.447
0.714
0.400
0.545
0.744
0.482
0.291
0.498
0.182
0.502
0.344
0.452
0.412
GPT-5.4
0.099
0.393
0.104
0.310
0.541
0.113
0.024
0.113
0.007
0.084
0.105
0.074
0.131
Qwen3-VL-235B-A22B-Instruct
0.249
0.309
0.225
0.298
0.387
0.230
0.100
0.226
0.093
0.235
0.149
0.231
0.147
GLM-4.5V
0.074
0.304
0.046
0.158
0.273
0.034
0.078
0.050
0.015
0.046
0.041
0.043
0.045
VLMs within Q2IT
Table 2: Main results on ITJoint under the Strict Joint setting. We evaluate VLMs in two settings: standalone direct inference and as the backbone of Q2IT.
Model
Text
Page
Image
Joint
Standalone VLMs
Gemini-3.1-Pro-Preview
0.615
0.881
0.518
0.507
GPT-5.4
0.306
0.808
0.187
0.185
Qwen3-VL-235B-A22B-Instruct
0.414
0.776
0.268
0.263
GLM-4.5V
0.243
0.629
0.131
0.128
VLMs within Q2IT
Table 3: Decomposed results on Text, Page, Image, and Joint metrics.
Model
Text Acc.
Joint Acc.
Gap ↓
Standalone VLMs
Gemini-3.1-Pro-Preview
0.849
0.744
0.105
GPT-5.4
0.767
0.541
0.226
Qwen3-VL-235B-A22B-Instruct
0.714
0.387
0.327
GLM-4.5V
0.597
0.273
0.324
VLMs within Q2IT
Table 4: Results on 1-to-0 instances.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Interface of ITLabel.
Figure 5: Ground-truth annotation example for a single-instance query in ITJoint.
Figure 6: Instance-level taxonomy examples in ITJoint.
Figure 7: Medical document examples from ITJoint (English).
Figure 8: Medical document examples from ITJoint (Chinese).
Figure 9: Archaeological document examples from ITJoint (English).
Figure 10: Archaeological document examples from ITJoint (Chinese).
Figure 11: Agricultural document examples from ITJoint (English).
Figure 12: Biology document examples from ITJoint (Chinese-English).
Figure 13: Art document examples from ITJoint (English).
Figure 14: Qualitative example of image-text alignment hallucination.
Model
Single
Single
Set
Set
1-to-0
1-to-1
1-to-N
Indep.
Comp.
Intra
Cross
Exp.
Imp.
Pos.
Neg.
Pos.
Neg.
Closed-source Models
Gemini-3.1-Pro-Preview
0.466
0.714
0.410
0.545
0.744
0.482
0.333
0.509
0.183
0.505
0.364
0.458
0.429
Gemini-3.1-Pro-Preview (Q2IT)
0.727 +0.261
0.943 +0.229
0.583 +0.173
0.722 +0.177
0.876 +0.132
0.563 +0.081
0.596 +0.263
0.592 +0.083
0.475 +0.292
0.550 +0.046
0.600 +0.236
0.607 +0.149
0.498 +0.069
GPT-5.4
0.099
0.393
0.107
0.310
0.541
0.113
0.042
0.118
0.007
0.091
0.105
0.080
0.131
GPT-5.4 (Q2IT)
0.649 +0.550
0.805 +0.412
0.624 +0.517
0.701 +0.391
0.802 +0.262
0.536 +0.423
0.621 +0.578
0.579 +0.461
0.453 +0.445
0.555 +0.464
0.556 +0.451
0.623 +0.543
0.420 +0.290
Appendix
Table 5: Additional results of direct inference and Q2IT on ITJoint under the Soft Joint setting.
Setting
1-to-0
1-to-1
1-to-N
Indep.
Comp.
Intra
Cross
Exp.
Imp.
Page
Page Agent Ablation
Full Q2IT
0.802
0.534
0.452
0.545
0.387
0.530
0.495
0.574
0.400
0.872
w/o Page Checker
0.531 -0.271
0.457 -0.078
0.355 -0.096
0.453 -0.092
0.350 -0.037
0.491 -0.039
0.349 -0.146
0.510 -0.064
0.282 -0.118
0.799 -0.072
w/o Page Search
0.548 -0.254
0.436 -0.099
0.307 -0.145
0.431 -0.114
0.299 -0.088
0.523 -0.007
0.234 -0.261
0.469 -0.105
0.282 -0.118
0.740 -0.132
External Tool Ablation
Full Q2IT
0.802
0.534
0.452
0.545
0.387
0.530
0.495
0.574
0.400
0.872
Appendix
Table 6: Ablation results of Q2IT on ITJoint with GPT-5.4 as the backbone model.
Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We isolate this choice, holding the prompt, judge, and scoring pipeline fixed, across four commercial model endpoints, two corpora, and two context regimes (gold evidence pages and the full document). On documents that fit the image budget, page images lead on accuracy at every document length on both corpora, but this advantage carries a growing latency and cost premium: text latency stays roughly flat as documents lengthen while image latency rises steadily. Text and images also fail on different questions, with exactly one representation correct on 19--25% of items across the reported cells, so neither subsumes the other. Exploiting this complementarity, a lightweight TF-IDF router that reads only the question text gains 2.6 points over always-text while cutting median latency 30% relative to always-vision, on a document-disjoint held-out split.
Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.
A. Said Gurbuz, Ahmed Nassar, Christoph Auer +8
IBM Research Zurich · ETH Zurich · IBM Research Zurich, Rüschlikon, Switzerland +1
Multimodal document question answering requires retrieving dispersed evidence from visually rich long documents and performing reliable reasoning over heterogeneous information. Existing multimodal RAG systems remain limited by two bottlenecks: static retrieval that ignores query complexity, and end-to-end Vision-Language Models (VLMs) that couple visual perception with logical reasoning, leading to inefficient computation and unstable answer generation. We propose AutoThinkRAG, a complexity-aware inference architecture for multimodal document QA. It has two components: (1) a Query Complexity Router that analyzes query difficulty and structure to adaptively select retrieval and reasoning paths; and (2) a Perception--Reasoning Decoupling architecture that uses a lightweight VLM as a high-fidelity visual interpreter to convert query-relevant visual cues into textual representations, which are then passed to an LLM for logical reasoning and answer synthesis. This design improves both efficiency and robustness, especially on long-document and unanswerable queries. Experiments on DocBench and MMLongBench show that AutoThinkRAG achieves 82.13% and 51.29% overall accuracy, respectively, while reducing per-query token consumption by 18.9% and monetary cost by 18.2%. Further analyses show that the gains are most pronounced on complex queries requiring adaptive retrieval and multi-step reasoning.
Jiashu Yang, Chi Zhang, Abudukelimu Wuerkaixi +5
Dalian University of Technology · Meituan LongCat Interaction Team · Tsinghua University