cs.CVAug 19, 2026

DocClaw: A Unified Agentic System for Intelligent Document Processing

Authors: Siqi Xiang, Zhipeng Xu, Yufei Liu, Junhao Ji, Qing Liu, Zulong Chen, Zhibo Yang, Chunyan Miao, 

Organizations: Nanyang Technological University · Alibaba Group · Qwen Team, Alibaba Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Intelligent document processing (IDP) encompasses a broad range of tasks, including optical character recognition (OCR), document question answering (DocQA), and key information extraction (KIE). Despite their distinct objectives, these tasks share a common need to perceive document content, acquire task-relevant information, and progressively refine intermediate results. However, they are typically formulated as separate prediction problems and addressed by task-specific models or processing pipelines. We introduce DocClaw, a unified agentic system that formulates diverse intelligent document processing tasks as a shared process of interaction between an agent and a document. Given a document and a task-specific query, DocClaw follows an appropriate document skill to iteratively identify the information required, invoke relevant tools, and integrate the resulting observations into the desired output. Throughout this process, a structured document state organizes reusable document knowledge and task-specific interaction context, allowing the agent to accumulate, revisit, and progressively refine information as the interaction proceeds. Under this formulation, task-specific requirements are captured by the agent's interpretation of the query objective and the corresponding document skill, while the underlying interaction loop, tool space, and document state are shared across tasks. Extensive experiments across multiple intelligent document processing benchmarks demonstrate that DocClaw effectively handles diverse tasks within a single agentic framework and achieves competitive performance compared with both general-purpose VLMs and task-specific methods.

Figures & tables

Figure 1: Unlike existing single-pass and multi-step paradigms, the proposed DocClaw formulates intelligent document processing as interactions between an agent and a document, leveraging iterative tool invocation and knowledge accumulation across queries.
Figure 2: Overview of the proposed DocClaw. Given an input document and a user query, DocClaw first selects a relevant task-oriented document skill and initializes the document state for the current query. It then iteratively plans and executes document processing tools, updating the document state until the final task-specific output is generated.
(a) Optical Character Recognition
TypeMethodOverall ↑\uparrowText Edit↓{}^{\mathrm{Edit}}\downarrowFormula CDM↑{}^{\mathrm{CDM}}\uparrowTable TEDS↑{}^{\mathrm{TEDS}}\uparrowTable TEDS-S↑{}^{\mathrm{TEDS\text{-}S}}\uparrowOrder Edit↓{}^{\mathrm{Edit}}\downarrow
General VLMsGPT-5.286.590.11488.2182.9587.930.193
 Qwen3-VL-235B89.780.06392.5583.0786.750.166
 Gemini-3-Pro92.910.06495.9989.1592.960.165
Specialized VLMsYoutu-Parsing93.740.04493.6392.0295.000.116
 Qianfan-OCR93.900.04095.0890.5393.310.130
Table 1: Main results across optical character recognition, document question answering, and key information extraction. Panels (a) to (c) report results on OmniDocBench v1.6, MMLongBench-Doc, and the KIE subsets of OCRBench and OCRBench v2, respectively. For document question answering, TXT, LAY, CHA, TAB, FIG, and UNA denote text, layout, chart, table, figure, and unanswerable questions. For key information extraction, ALL averages the results across OCRBench KIE and OCRBench v2 subsets. The best and second-best results for each metric are highlighted in bold and underlined, respectively.
Figure 3: Ablation of document state and document skills. Left: overall score and average latency when using document state, document skills, or both. Right: latency across 30 consecutive queries on the same document.
Figure 4: Ablation of OCR refinement. Left: performance improvement attributed to zoom, crop, and rotate operations. Right: number of pages refined by each operation.
Figure 5: Ablation of OCR and verification for KIE. Left: KIE performance of the vanilla VLM and DocClaw with OCR and verification operations. Right: percentage of vanilla VLM extraction errors corrected by the verification operation.
More in this topic

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 290 papers

Explore similar work

CardsList
Feb 26, 2026

IDP Accelerator: Agentic Document Intelligence from Extraction to Compliance Validation

Understanding and extracting structured insights from unstructured documents remains a foundational challenge in industrial NLP. While Large Language Models (LLMs) enable zero-shot extraction, traditional pipelines often fail to handle multi-document packets, complex reasoning, and strict compliance requirements. We present IDP (Intelligent Document Processing) Accelerator, a framework enabling agentic AI for end-to-end document intelligence with four key components: (1) DocSplit, a novel benchmark dataset and multimodal classifier using BIO tagging to segment complex document packets; (2) configurable Extraction Module leveraging multimodal LLMs to transform unstructured content into structured data; (3) Agentic Analytics Module, compliant with the Model Context Protocol (MCP) providing data access through secure, sandboxed code execution; and (4) Rule Validation Module replacing deterministic engines with LLM-driven logic for complex compliance checks. The interactive demonstration enables users to upload document packets, visualize classification results, and explore extracted data through an intuitive web interface. We demonstrate effectiveness across industries, highlighting a production deployment at a leading healthcare provider achieving 98% classification accuracy, 80% reduced processing latency, and 77% lower operational costs over legacy baselines. IDP Accelerator is open-sourced with a live demonstration available to the community.

Md Mofijul Islam, Md Sirajus Salekin, Joe King +8

Amazon Web Services

Document Information Extraction
AskPDF
Jul 21, 2026

DocAtlas: Long-Document Understanding as Mutable-State Interaction

Long-document understanding requires models to find and combine evidence across many pages, layouts, tables, figures, and charts. Existing retrieval-augmented systems usually select evidence from a static index before generation, while recent agentic systems add multi-turn tool use but often rely on frozen proprietary backbones whose behavior is set by prompts. We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents. With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7%, compared with a 54.4% direct-input baseline, showing that mutable document-harness design can improve compact document agents by a large margin.

Hongchen Wei, Yuanzhe Wang, Bei Liu +8

Wuhan University · Microsoft

Document UnderstandingLong-Context QALong-Context ModelingAgentic RAGDocument QA
AskPDF
Oct 15, 2026

DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding

Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that supports expressive orchestration and makes complex document solving learnable by small language models (SLMs). DocPilot orchestrates evidence seeking through page-level navigation and fine-grained evidence localization, and integrates local visual inspection, code-based numerical reasoning, and cross-modal reasoning for multimodal understanding. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings. The code is available at https://github.com/HKUSTDial/DocPilot.

Zhuowen Liang, Xiaotian Lin, Yuyu Luo +2

The Hong Kong University of Science and Technology (Guangzhou) · EvenUp, USA

Document QAMultimodal AgentsMultimodal QALLM Agent TrainingLong-Context VLMs
AskPDF