DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting

Md Mofijul Islam; Md Sirajus Salekin; Nivedha Balakrishnan; Vincil C. Bishop; Niharika Jain; Spencer Romo; Bob Strahan; Boyi Xie

cs.CLFeb 17, 2026

DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting

Authors: Md Mofijul Islam, Md Sirajus Salekin, Nivedha Balakrishnan, Vincil C. Bishop, Niharika Jain, Spencer Romo, Bob Strahan, Boyi Xie,

Organizations: Amazon Web Services

Abstract

Document understanding in real-world applications often requires processing heterogeneous, multi-page document packets containing multiple documents stitched together. Despite recent advances in visual document understanding, the fundamental task of document packet splitting, which involves separating a document packet into individual units, remains largely unaddressed. We present the first comprehensive benchmark dataset, DocSplit, along with novel evaluation metrics for assessing the document packet splitting capabilities of large language models. DocSplit comprises five datasets of varying complexity, covering diverse document types, layouts, and multimodal settings. We formalize the DocSplit task, which requires models to identify document boundaries, classify document types, and maintain correct page ordering within a document packet. The benchmark addresses real-world challenges, including out-of-order pages, interleaved documents, and documents lacking clear demarcations. We conduct extensive experiments evaluating multimodal LLMs on our datasets, revealing significant performance gaps in current models' ability to handle complex document splitting tasks. The DocSplit benchmark datasets and proposed novel evaluation metrics provide a systematic framework for advancing document understanding capabilities essential for legal, financial, healthcare, and other document-intensive domains. We release the datasets to facilitate future research in document packet processing.

Figures & tables

Figure 1: The document packet splitting task, where an unordered sequence of pages from multiple documents (shown at top) must be correctly grouped and reordered into their constituent documents (shown at bottom). The colors represent individual documents, while the numbers indicate the original page indices from the top input sequence, demonstrating how pages from the same document can be shuffled and interleaved in the input packet.

		Proposed				Classical
Benchmark	Model	Clustering	Ordering	Packet	Page	Page+Split	Page+Split+Order
DocSplit-Mono-Seq	Claude Haiku 4.5	0.8433	0.9950	0.9191	0.7919	0.7326	0.7316
	Claude Sonnet 4.5	0.8542	0.9960	0.9251	0.8239	0.7656	0.7656
	DeepSeek V3.1	0.8473	0.9953	0.9213	0.8242	0.6831	0.6821
	Gemma 3 27B	0.6093	0.9940	0.8016	0.7257	0.3134	0.3134
	Qwen 3 VL 235B	0.8831	0.9960	0.9395	0.8418	0.8036	0.8026

Table 2: Experimental results of document packet splitting task across all benchmarks. Clustering and Ordering are the components of the proposed Packet score, Page = Page level class accuracy, Page+Split = Page and split level accuracy, Page+Split+Order = Page and split level accuracy with order. Best results per benchmark shown in bold . For the Proposed metrics

\alpha

\beta

w

= 0.5 were used.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Test Case	Description
Perfect	All predictions match ground truth
Misclassification Only	Correct grouping and ordering, but class labels swapped
Wrong Grouping Only	Correct classification and ordering, but group IDs swapped
Wrong Ordering Only	Correct classification and grouping, but page order scrambled
Split Groups	One document split into two groups (over-segmentation)
Merged Groups	Two documents merged into one group (under-segmentation)

Appendix

Table 3: Edge case test scenarios. Each case isolates a specific error type: classification errors (Misclassification, Partial Misclass), boundary detection errors (Split Groups, Merged Groups, Wrong Grouping), ordering errors (Wrong Ordering, Reverse Order, Duplicate Page Nums), or combinations thereof (Multiple Errors). The Perfect case serves as the baseline with all predictions matching ground truth.

Test Case	$\mathbf{S_{\text{packet}}}$	$\mathbf{S_{\text{clustering}}}$	V-measure	RI	$\mathbf{S_{\text{ordering}}}$
Perfect	1.0000	1.0000	1.0000	1.0000	1.0000
Misclassification Only	0.7974	0.5949	0.5897	0.6000	1.0000
Wrong Grouping Only	1.0000	1.0000	1.0000	1.0000	1.0000
Wrong Ordering Only	0.1667	1.0000	1.0000	1.0000	$-0.6667$
Split Groups	0.8428	0.6856	0.6713	0.7000	1.0000
Merged Groups	0.6000	0.2000	0.0000	0.4000	1.0000

Appendix

Table 4: Proposed metrics results (

\alpha=\beta=w=0.5

S_{\text{packet}}

is the combined score,

S_{\text{clustering}}

combines V-measure and Rand Index (RI), and

S_{\text{ordering}}

uses Kendall’s Tau (

\tau

). Key observations: (1) classification errors reduce

S_{\text{clustering}}

proportionally (Misclassification: 0.59), (2) ordering errors produce negative

S_{\text{ordering}}

(Reverse Order:

-1.0

), (3) swapped group IDs receive full credit when structure is preserved (Wrong Grouping: 1.0), and (4) combined errors yield low or negative

S_{\text{packet}}

(Multiple Errors:

-0.04

Test Case	Page	Page+Split	Page+Split+Order
Perfect	100.00%	100.00%	100.00%
Misclassification Only	0.00%	0.00%	0.00%
Wrong Grouping Only	100.00%	100.00%	100.00%
Wrong Ordering Only	100.00%	100.00%	0.00%
Split Groups	100.00%	0.00%	0.00%
Merged Groups	100.00%	0.00%	0.00%

Appendix

Table 5: Classical metrics results using exact matching at three strictness levels: Page (classification only), Page+Split (classification and boundary detection), and Page+Split+Order (all three aspects). Key limitations: (1) binary outcomes with no partial credit (Misclassification: 0% despite correct grouping), (2) Split Groups and Merged Groups both score 0% despite different error severities, and (3) Reverse Order scores identically to Wrong Ordering (0%) despite being more severe.

Test Case	Classical	Proposed
Misclassification	All = 0% (complete failure)	$S_{\text{clustering}}$ = 0.59, $S_{\text{packet}}$ = 0.80
Split Groups	Page+Split = 0%	$S_{\text{clustering}}$ = 0.69, $S_{\text{packet}}$ = 0.84
Merged Groups	Page+Split = 0%	$S_{\text{clustering}}$ = 0.20, V-measure = 0
Reverse Order	Page+Split+Order = 0%	$S_{\text{ordering}}$ = $-1.0$ (captures direction)
Partial Misclass	Page+Split = 50%	$S_{\text{packet}}$ = 0.89 (localized penalty)

Appendix

Table 6: Comparative analysis of classical vs. proposed metrics. The proposed framework provides: (1) continuous scores instead of binary pass/fail, (2) proportional penalties reflecting error severity (Split vs. Merged receive different scores), (3) directional ordering information (

S_{\text{ordering}}

distinguishes scrambled from reversed), and (4) localized impact for partial errors.

Dataset	Size	Data Source	Domain	Task	OCR	Layout
IIT-CDIP Lewis et al. (2006)	35.5M	UCSF-IDL	Industry	P	✗	✗
RVL-CDIP Harley et al. (2015a)	400K	UCSF-IDL	Industry	DC	✗	✗
RVL-CDIP-N Larson et al. (2022)	1K	Doc Cloud	Industry	DC	✗	✗
TAB Mungmeeprued et al. (2022)	44.8K	UCSF-IDL	Industry	DC	✗	✗
FUNSD Jaume et al. (2019)	199	UCSF-IDL	Industry	KIE	✓	✗
SP-DocVQA Mathew et al. (2020)	12K	UCSF-IDL	Industry	QA	✓	✗

Appendix

Table 7: Document understanding benchmarks with corresponding data sources, domains, and task specifications. DPS: Document Packet Splitting, DC: Document Classification, DLA: Document Layout Analysis, KIE: Key Information Extraction, QA: Question Answering, TSR: Table Structure Recognition, P: Pretrain

Dataset Type	Size	Split	Total	Pages/	Subdocs/	Types/	Pages/
			Packets	Packet	Packet	Packet	Subdoc
PolySeq	Large	Test	500	69.86	12.31	7.97	5.67
		Train	1864	127.05	16.04	8.77	7.92
		Validation	199	79.51	11.10	7.50	7.16
	Small	Test	500	14.66	4.05	3.66	3.62
		Train	7840	14.89	4.31	3.85	3.46

Appendix

Table 8: Packet statistics across DocSplit benchmark variants. Large variants contain 45 to 127 pages per packet with 5 to 16 subdocuments, while Small variants contain 11 to 15 pages with 3 to 4 subdocuments, enabling evaluation across different complexity scales. Poly variants (PolySeq, PolyInt, PolyRand) incorporate 7 to 8 document types per packet in Large configurations and 3.5 to 3.8 types in Small configurations, testing cross-type boundary detection. Mono variants (MonoSeq, MonoRand) contain exactly one document type per packet (Types/Packet = 1.0), isolating intra-type boundary detection from classification. The pages per subdocument ratio indicates that Mono variants contain longer individual documents (8 to 16 pages) compared to Poly variants (5 to 8 pages), reflecting different real-world scenarios where homogeneous document batches often contain lengthier materials.

Size	Split	Budget	Email	Form	Handwritten	Invoice	Language	Letter	Memo	News Article	Questionnaire	Resume	Sci. Pub.	Spec.
DocSplit-Poly-Seq
Large	Test	323	288	330	322	314	292	301	322	281	294	287	341	292
	Train	1240	1225	1361	1342	1196	1224	1232	1215	1246	1265	1236	1359	1212
	Validation	135	98	118	135	109	100	112	113	111	98	119	149	95
Small	Test	129	139	136	138	135	130	123	135	138	136	160	167	165
	Train	2346	2246	2045	2207	2344	2683	2316	2317	2393	2322	2371	2231	2334

Appendix

Table 9: Document type distribution across DocSplit benchmark variants showing counts for 13 categories: Budget, Email, Form, Handwritten, Language, Invoice, Letter, News Article, Questionnaire, Memo, Resume, Scientific Publication, and Specification. Poly variants (PolySeq, PolyInt, PolyRand) exhibit balanced representation across all document types, with each category contributing 100 to 350 instances in Large test splits, ensuring unbiased evaluation across diverse formats. Mono variants (MonoSeq, MonoRand) exclude the Language category due to limited source samples, as evidenced by zero or near-zero counts. The stratified splitting maintains proportional type distributions across train, validation, and test partitions, with Form, Letter, and Resume being most prevalent in Mono variants. This distribution enables comprehensive evaluation of both cross-type discrimination in Poly benchmarks and same-type boundary detection in Mono benchmarks.

Panel of Table 10 — Table 10: Model configuration parameters for document splitting baseline approach

Panel of Table 11 — Table 11: System prompt for document splitting baseline approach

Panel of Table 12 — Table 12: Task prompt for document splitting baseline approach - Document structure and terminology

Panel of Table 13 — Table 13: Task prompt for document splitting baseline approach - Splitting guidance and critical instructions

Panel of Table 14 — Table 14: Task prompt for document splitting baseline approach - Classification process and local document ID

Figure 2: Test Studio interface with pre-populated benchmark datasets for document processing evaluation.

Figure 3: Test Studio evaluation dashboard displaying benchmark results with performance metrics and score distribution.

Figure 4: Test Studio cost estimation interface displaying itemized costs across pipeline stages (Classification, Extraction, OCR, Granular Assessment, Summarization) with per-API token and invocation tracking.

Figure 5: Custom dataset import interface displaying the required structure with input/ folder for packet PDFs and baseline/ folder containing ground truth annotations in result.json files.

Document Image Understanding

Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.

See all 148 papers

Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms
Jul 23, 2026Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan +7Document Chunking Multilingual VLM Evaluation
Wichita State University, USA · CCDS, Independent University, Bangladesh · Amazon GenAI, USA
MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing
May 21, 2026Bangbang Zhou, Hangdi Xing, Yifan Chen +8Document Parsing Benchmarks Document Parsing
University of Science and Technology of China · Tongyi Lab, Alibaba Group
Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing
May 31, 2026Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22VLM Evaluation Document Parsing Benchmarks
2077AI · University of Arizona · University of Southern California +8

Jul 23, 2026J/K move · Enter open · S save

Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms

Abu Tyeb Azad, Fahim Ahmed, Ishita Sur Apan, Ezharuddin Jubaer+6

Wichita State University, USA · CCDS, Independent University, Bangladesh · Amazon GenAI, USA

Document Chunking Multilingual VLM Evaluation Document Image Understanding

Open paper AskPDF arXiv

Document packets, multiple documents concatenated into a single file, are common in government and administrative workflows, yet splitting them into their constituent documents is difficult, especially for low-resource languages. We introduce Khondo (Bangla for split/segment), the first benchmark for document packet splitting on Bangladeshi government forms. Unlike prior English and OCR-text-based datasets, Khondo is bilingual (Bangla--English) and vision-native; where models operate directly on page images. It spans five concatenation schemes, from sequential to fully shuffled, across 14 administrative domains, with ground-truth boundaries, domain types, and page order. Zero-shot evaluation of MLLMs shows they cluster pages into their source documents fairly well but struggle in restoring the original page order once shuffled. To isolate what drives this difficulty, we run two controlled analyses, varying the prompt instruction and then the packet language. Both primarily affect ordering rather than clustering: (a) explicit page-order instructions are necessary but insufficient, and (b) English packets are ordered more reliably than Bangla, making page arrangement the dominant challenge and language a secondary but consistent factor. Khondo establishes page-order reconstruction as a key open problem in vision-based, low-resource document understanding, and provides a controlled benchmark for measuring progress toward solving it. Our dataset and code is available at https://huggingface.co/datasets/Mausul/khondo

Abstract

Document Image Understanding

Explore similar work

Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms

MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing