ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering

Aymen Lassoued; Mohamed Ali Souibgui; Yousri Kessentini

cs.CVMar 2, 2026

ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering

Authors: Aymen Lassoued, Mohamed Ali Souibgui, Yousri Kessentini

Organizations: Digital Research Center of Sfax, SMARTS Laboratory, Sfax, Tunisia. · ´Ecole Polytechnique de Tunisie, University of Carthage · Computer Vision Center, Universitat Aut`onoma de Barcelona

Abstract

Document Visual Question Answering (DocVQA) remains challenging for existing Vision-Language Models (VLMs), especially under complex reasoning and multi-step workflows. Current approaches struggle to decompose intricate questions into manageable sub-tasks and often fail to leverage specialized processing paths for different document elements. We present ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering, a novel multi-agent framework that addresses these limitations through strategic agent coordination and iterative refinement. ORCA begins with a reasoning agent that decomposes queries into logical steps, followed by a routing mechanism that activates task-specific agents from a specialized agent dock. Our framework leverages a set of specialized AI agents, each dedicated to a distinct modality, enabling fine-grained understanding and collaborative reasoning across diverse document components. To ensure answer reliability, ORCA employs a debate mechanism with stress-testing, and when necessary, a thesis-antithesis adjudication process. This is followed by a sanity checker to ensure format consistency. Extensive experiments on three benchmarks demonstrate that our approach achieves significant improvements over state-of-the-art methods, establishing a new paradigm for collaborative agent systems in vision-language reasoning.

Figures & tables

Figure 1: Comparison of different approaches for DocVQA . Single-model VLMs and reasoning-enhanced VLMs lack critical capabilities such as adaptivity and self-verification. In contrast, ORCA introduces a feature-oriented, multi-agent design achieving improved DocVQA performance as well as the missing capabilities in one unified framework.

Figure 2: Overview of ORCA : A reasoning-guided multi-agent framework for Document Visual Question Answering operating through five stages: (1) Context Understanding: A thinker agent analyzes the question and document to generate both a reasoning path and initial answer ( $a_{T}$ ). (2) Collaborative Agent Execution: A router selects relevant specialized agents from a dock of nine expert types (OCR, Layout, Table/List, Figure/Diagram, Form, Free Text, Image/Photo, Yes/No, and General), which an orchestrator sequences for optimal execution to produce an expert answer ( $a_{E}$ ). (3) Stress Testing: When $a_{E}$ differs from $a_{T}$ , a debate agent generates challenging questions to stress-test the specialized agent’s confidence, with an evaluation agent assessing the responses to produce $a_{D}$ . (4) Multi-turn Conversation: If stress testing indicates uncertainty, thesis and antithesis agents engage in structured three-turn debate under judge supervision to resolve conflicts and generate $a_{C}$ . (5) Answer Refinement: A sanity checker performs final formatting corrections to ensure consistency with document conventions, producing the final answer ( $a_{F}$ ).

Document Understanding Models
Model	Open-source	DocVQA	InfoVQA	Avg.
LayoutLMv2 LARGE [ 68 ]	✓	86.7	28.3	57.5
Text-Monkey [ 32 ]	✓	66.7	28.6	47.7
DocOwl-1.5 [ 20 ]	✓	82.2	50.7	66.5
General-Purpose VLMs
Qwen2-VL [ 5 ]	✓	96.7	84.7	90.7

Table 1: Performance comparison across DocVQA benchmarks. Models are categorized by architectural paradigm.

Open-source LMMs
Model	Rec	Ref	Spo	Ext	Par	Cal	Und	Rea	Avg
LLaVA-Next-8B* [ 37 ]	41.3	18.8	0	49.5	21.2	17.3	55.2	48.9	31.5
LLaVA-OV-7B* [ 29 ]	46.0	20.8	0.1	58.3	25.3	23.3	64.4	53.0	36.4
TextMonkey* [ 32 ]	39.1	0.7	0	19.0	12.2	19.0	61.1	40.2	23.9
Molmo-7B* [ 15 ]	52.4	21.3	0.1	45.5	7.6	28.5	65.3	55.0	34.5
Cambrian-34B* [ 59 ]	45.3	21.5	0	53.6	19.2	19.5	63.5	55.5	34.7

Table 2: OCRBench-v2 performance comparison between baseline VLMs and ORCA across model scales. The results marked with * are not reproduced, they are given by the papers’ authors.

Configuration	Latency (s)	DocVQA	InfoVQA
Baseline (Qwen3VL-8B)	0.3–0.8	96.1	83.1
ORCA Early-Termination	2.9–4.5	96.7	86.9
ORCA Full Pipeline	9.6–13.1	97.2	88.0

Table 3: Inference latency comparison on 4

\times

NVIDIA L4 GPUs with vLLM optimization. Early-termination applies when thinker and expert agents agree, bypassing Stages 3–5 (77% of cases).

Individual Component Ablations
Configuration	DocVQA	InfoVQA	OCRBench-v2
ORCA (Full)	97.2	88.0	67.1
w/o Stage 1 (Reasoning)	96.5 (-0.7)	84.9 (-3.1)	66.1 (-1.0)
w/o Stage 2 (Collaborative Agents)	96.3 (-0.9)	84.1 (-3.9)	66.0 (-1.1)
w/o Stage 3 (Stress Testing)	97.0 (-0.2)	87.5 (-0.5)	66.9 (-0.2)
w/o Stage 4 (Multi-turn Debate)	96.9 (-0.3)	87.2 (-0.8)	66.7 (-0.4)

Table 4: Stage-wise ablation study on Qwen3VL-8B. Each row isolates the impact of removing a specific stage from the complete ORCA framework.

Figure 3: A case study demonstrating ORCA ’s multi-agent reasoning pipeline on a complex visual document question. Better viewed with zoom. Question: What publication detail accompanies the Genealogical Society entry?. GT answer: “GSU, 1977”.

Component Variation	DocVQA	InfoVQA	OCRBench-v2
No reasoning masking	96.5	87.6	66.4
Reasoning masking (Full)	97.2	88.0	67.1
Performance gain	+0.7	+0.4	+0.7

Table 5: Reasoning path masking ablation on Qwen3VL-8B. Masking prevents confirmation bias while preserving strategic guidance.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Latency	DocVQA	InfoVQA	OCRBench-v2
Single Model	0.3–0.8s	96.1	83.1	65.4
ORCA-Lite	3.2–5.3s	96.8	87.0	66.4
ORCA Full	9.6–13.1s	97.2	88.0	67.1

Appendix

Table 6: ORCA-Lite vs. full pipeline accuracy–latency trade-off.

Open-source models results
Model	Fig/Diag	Form	Table/List	Layout	Free Text	Img/Photo	Handwr.	Yes/No	Others	Score
LayoutLMv2 LARGE	65.7	89.5	87.7	87.9	87.1	72.9	67.3	55.2	81.0	86.7
Qwen2-VL	92.1	98.2	97.0	96.8	96.2	91.4	94.4	96.6	95.4	96.7
InternVL2-Pro	88.9	97.1	94.9	95.8	94.5	89.1	92.8	96.6	94.1	95.1
Molmo-72B	88.2	95.5	93.9	94.1	91.0	86.9	92.0	92.0	92.3	93.5
DeepSeek-VL2	88.5	95.8	93.6	93.1	92.1	86.9	89.9	89.7	90.1	93.3

Appendix

Table 7: Detailed performance breakdown on DocVQA benchmark for open-source models. Results are shown across different document types and question categories.

Open-source models results
		Answer type				Evidence					Operation
Method	Score	Image span	Question span	Multiple spans	Non span	Table/List	Textual	Visual object	Figure	Map	Comparison	Arithmetic	Counting
LayoutLMv2 LARGE	28.3	34.3	27.6	6.4	11.1	24.5	38.6	14.4	26.0	31.1	19.0	11.3	11.6
Qwen2-VL	84.7	87.4	87.1	77.8	74.2	86.0	94.3	78.3	81.7	75.9	73.0	89.8	57.9
InternVL2-Pro	83.3	86.8	89.3	73.5	69.7	83.4	92.6	77.6	80.9	71.9	73.0	85.8	53.7
Molmo-72B	81.9	85.1	88.3	68.2	70.4	81.8	91.4	80.6	79.5	69.6	70.5	81.9	59.3

Appendix

Table 8: Detailed performance breakdown on Infographics VQA benchmark for open-source models. Results are categorized by answer type, evidence source, and required operations.

Figure 4: ORCA demonstrates robust multi-stage reasoning on a document containing ambiguous textual references and visually challenging OCR content. While baseline VLMs fail due to misidentification and shallow pattern matching, ORCA decomposes the task into OCR parsing, cell-level localization, cross-reference verification, and answer consistency checking. Through iterative agent collaboration and critical evidence consolidation, ORCA resolves ambiguity, corrects earlier misinterpretations, and converges on the correct entity with high confidence

Figure 5: ORCA successfully handles a structurally complex form where precise line indexing, noisy OCR text, and subtle vocabulary variations mislead baseline VLMs. By combining layout-aware processing, content-aware sequence reasoning, and downstream sanity validation, ORCA incrementally narrows the search space and suppresses earlier incorrect hypotheses. The multi-agent pipeline enables reliable disambiguation and robust extraction even under OCR artifacts and positional uncertainty.

Failure Mode	Proportion	Description
Reasoning errors	43%	Thinker agent generates incorrect reasoning path, misleading all subsequent agents
Router errors	27%	Incorrect agent selection causes missing evidence or mismatched specialist
Agent coordination failures	18%	Error propagation through sequential execution from early agents
Over-refinement	12%	Verification stages introduce errors by over-analyzing initially correct answers

Appendix

Table 9: Error attribution by originating stage across 100 analyzed failure cases.

Visual Question Answering

Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.

See all 253 papers

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Apr 15, 2026Yuanlei Zheng, Pei Fu, Hang Li +9Document QA Long-Document VQA
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1
DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding
Oct 15, 2026Zhuowen Liang, Xiaotian Lin, Yuyu Luo +2Document QA Multimodal Agents
The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA
AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction
Mar 5, 2026Jiashu Yang, Chi Zhang, Abudukelimu Wuerkaixi +5Multimodal RAG Multimodal QA
Dalian University of Technology · Meituan LongCat Interaction Team · Tsinghua University

Apr 15, 2026J/K move · Enter open · S save

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang+8

School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology+1

Document QA Long-Document VQA VLM Reasoning

Open paper AskPDF arXiv

Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc-

V^*

, an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc-

V^*

begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc-

V^*

balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc-

V^*

outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.

Abstract

Visual Question Answering

Explore similar work

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding

AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction