The COTe score: A decomposable framework for evaluating Document Layout Analysis models

Jonathan Bourne; Mwiza Simbeye; Ishtar Govia

cs.CVMar 13, 2026

The COTe score: A decomposable framework for evaluating Document Layout Analysis models

Authors: Jonathan Bourne, Mwiza Simbeye, Ishtar Govia

Organizations: THE 3TC AI · University College London · Amagi Brain Health

Abstract

Document Layout Analysis (DLA) is the process by which a page is parsed into meaningful elements, often using machine learning models. Typically, the quality of a model is judged using general machine vision metrics such as IoU, F1 or mAP. However, these metrics are designed for images that are 2D projections of 3D space, not for the natively 2D imagery of printed media. This discrepancy can result in misleading or uninformative interpretation of model performance. To encourage more robust, comparable, and nuanced DLA, we introduce: The Structural Semantic Unit (SSU), a relational labelling approach that shifts the focus from the physical to the semantic structure of the content; and the Coverage, Overlap, Trespass, and Excess (COTe) score, a decomposable metric for measuring page parsing quality. We demonstrate the value of these methods through case studies and by evaluating 5 common DLA models on 3 DLA datasets. We show that the COTe score is more informative than traditional metrics and reveals distinct failure modes across models, such as breaching semantic boundaries or repeatedly parsing the same region. We find that, under granularity differences between model and ground truth, the COTe score is substantially more robust than the F1. Even in the worst case, comparing character-level predictions against paragraph-level ground truth with otherwise perfect parsing, COTe returns 0.68 where F1 returns 0. Notably, we find that, on real datasets, the COTe's granularity robustness largely holds even without explicit SSU labelling, reducing the barrier to entry. Finally, we release an SSU labelled dataset and a Python library for applying COTe in DLA projects.

Figures & tables

Panel of Figure 1 — Figure 1 : The relationship between the data, model, and metric. Labelling differences between model train and model test data can lead to pragmatic failure of the metric evaluation of the model which presents as misleading values (shown as dashed arrow).

Paradigm	Machine Vision	Page Parsing
Spatial Model	2D projection of 3D space	Natively 2D (Tessellation)
Primary Unit	Bounding Box	SSU
Structure	Atomic	Composite
Assignment	1-to-1	many-to-1-to-many
Failure Modes	Undifferentiated	Decomposable
Core Metric	mAP	COTe

Table 1 : Comparison of the Machine Vision and Page Parsing evaluation paradigms.

Panel of Figure 2 — Figure 2 : The SSU as the shared pivot of two independent many-to-one mappings. Ground-truth region labels (left) are grouped into a single SSU, and any number of predictions (right) are independently assigned to that same SSU. The two sides need not match in count, which is the source of the framework’s granularity robustness.

Figure 3 : Each limerick comprises its own semantic unit, the titles and text create their own SSU as titles create visual structure for the reader, column breaks also create new SSUs as it requires a new bounding box.

Metric	Definition	Impact
Coverage	How much of the ground truth is covered by predictions	Rewards covering all content
Overlap	How much of the ground truth has more than one prediction covering the same area	Penalises stacked prediction, impossible in 2D space
Trespass	How much of the ground truth is covered by a prediction that belongs to a different SSU	Penalises breaches of the tessellation logic
Excess	How much area outside the ground truth is covered by predictions	Contextualises the core metrics

Table 2 : Evaluation metrics definitions

Figure 4 : Example providing a visual representation of the COTe score. The example shows how predictions and groundtruth interact.

Dataset	Image Type	Images	Regions	Region Type	SSU
NCSE	Newspapers	31	358	Bounding Box	Yes
HNLA2013	Newspapers	50	2668	Polygon	Yes
DocLayNet	Multi Format	4999	66531	Bounding Box	No

Table 3 : Details on the datasets used for evaluation, HNLA2013 and DocLayNet data are for their test sets, the NCSE dataset is only 31 pages in total

Model	Type	Parameters	Framework
DocLayout-YOLO	CNN	15.4M	PyTorch
Heron	DETR-based	42.9M	PyTorch
PP-DocLayout-L	DETR-based	30.94M	PaddlePaddle
PP-DocLayout-M	PicoDet-M	5.65M	PaddlePaddle
PP-DocLayout-S	PicoDet-S	1.21M	PaddlePaddle

Table 4 : Comparison of Document Layout Analysis Models

Metric	GT: Line Pred: Para	GT: Para Pred: Line
True Positive	4	4
False Positive	3	14
False Negative	14	3
Precision	0.57	0.22
Recall	0.22	0.57
F1	0.32	0.32

Table 5 : Comparison of COTe vs traditional metrics on granularity-mismatched predictions from Figure 3 . Traditional metrics suggest almost complete failure despite perfect parsing. However, the COTe correctly identifies excellent performance.

	COTe	Coverage	Overlap	Trespass	Excess	IoU	F1
Perfect	1.00	1.00	0.00	0.00	0.00	1.00	1.00
Figure 4	0.67	0.91	0.12	0.12	0.07	0.64	0.76

Table 6 : COTe performance for the predictions shown in Figure 4

Class	$\mathcal{C}^{\textrm{share}}_{k}$	$\mathcal{O}^{\textrm{share}}_{k}$	$\mathcal{T}^{\textrm{share}}_{k}$
headline	0.05	0.04	0.00
text	0.95	0.96	1.00

Table 7 : Share of total Coverage, Overlap and Trespass attributed to each predicted class.

GT $\downarrow$ / Pred $\rightarrow$	headline	text
headline	1.00	0.05
text	0.00	0.95

Table 8 : Overlap confusion matrix between classes. Each column is a predicted class and sums to one.

GT $\downarrow$ / Pred $\rightarrow$	headline	text
headline	0.00	0.09
text	0.00	0.91

Table 9 : Trespass confusion matrix between classes. Each column is a predicted class and sums to one.

GT $\downarrow$ / Pred $\rightarrow$	Character	Word	Line	Paragraph
Character	1.00	1.00	1.00	1.00
Word	1.00	1.00	1.00	1.00
Line	0.90	0.90	1.00	1.00
Paragraph	0.68	0.68	0.76	1.00

Table 10 : Mean COTe when one ground truth granularity is scored against another.

GT $\downarrow$ / Pred $\rightarrow$	Character	Word	Line	Paragraph
Character	1.00	0.06	0.00	0.00
Word	0.06	1.00	0.01	0.00
Line	0.00	0.01	1.00	0.07
Paragraph	0.00	0.00	0.07	1.00

Table 11 : Mean F1@0.5 when one ground truth granularity is scored against another.

Model	COTe	Coverage	Overlap	Trespass	Excess	IoU	mAP	F1@50
Heron	0.59	0.87	0.28	0.00	0.03	0.75	0.56	0.51
PPDoc-L	0.72	0.80	0.05	0.03	0.03	0.70	0.38	0.57
PPDoc-M	0.42	0.76	0.05	0.29	0.12	0.36	0.09	0.26
PPDoc-S	0.25	0.87	0.21	0.41	0.12	0.44	0.14	0.29
YOLO	0.59	0.92	0.17	0.16	0.06	0.72	0.37	0.55

Table 12 : Model results for NCSE

Model	COTe	Coverage	Overlap	Trespass	Excess	IoU	mAP	F1@50
Heron	0.80	0.98	0.18	0.01	0.13	0.81	0.62	0.82
PPDoc-L	0.73	0.83	0.05	0.05	0.16	0.61	0.46	0.74
PPDoc-M	0.25	0.69	0.00	0.44	0.45	0.06	0.01	0.02
PPDoc-S	0.18	0.81	0.08	0.55	0.48	0.09	0.02	0.06
YOLO	0.85	0.97	0.07	0.05	0.15	0.74	0.44	0.68

Table 13 : Model results for HNLA2013

Model	COTe	Coverage	Overlap	Trespass	Excess	IoU	mAP	F1@50
Heron	0.90	0.95	0.04	0.02	0.03	0.85	0.76	0.90
PPDoc-L	0.78	0.91	0.05	0.08	0.07	0.66	0.42	0.71
PPDoc-M	0.63	0.78	0.06	0.09	0.07	0.51	0.29	0.61
PPDoc-S	0.62	0.77	0.05	0.09	0.07	0.47	0.29	0.61
YOLO	0.71	0.96	0.15	0.11	0.08	0.72	0.49	0.76

Table 14 : Model results for DocLayNet

Model	SSU	No SSU	Difference
Heron	0.00	0.01	0.01
PPDoc-L	0.03	0.05	0.02
PPDoc-M	0.29	0.35	0.06
PPDoc-S	0.41	0.49	0.08
YOLO	0.16	0.23	0.07

Table 15 : Trespass for NCSE with and without SSU grouping

Model	SSU	No SSU	Difference
Heron	0.01	0.03	0.02
PPDoc-L	0.05	0.07	0.02
PPDoc-M	0.44	0.54	0.09
PPDoc-S	0.55	0.68	0.12
YOLO	0.05	0.09	0.03

Table 16 : Trespass for HNLA2013 with and without SSU grouping

Dataset	IoU	F1@50	mAP
DocLayNet	0.90	0.90	0.80
HNLA2013	0.80	0.60	0.60
NCSE	0.50	0.90	0.60
All	0.76	0.91	0.79

Table 17 : Spearman correlation between dataset-level COTe and the other metrics, per dataset and pooled

Dataset	COTe	Coverage	Overlap	Trespass	IoU	mAP	F1@50
DocLayNet	-0.59	-0.29	0.13	0.31	-0.67	-0.67	-0.83
HNLA2013	-0.83	0.21	-0.46	0.59	-0.88	-0.88	-0.93
NCSE	-0.51	0.18	0.11	0.24	-0.51	-0.53	-0.82

Table 18 : Page-level Spearman correlation between

|\log_{2}\text{Ratio}|

of predicted to ground truth regions and each metric, per dataset (pages pooled over all five models)

Figure 5 : It is clear in visual analysis that the models can behave very differently on the same image, and have very different failure modes.

Model	COTe	Coverage	Overlap	Trespass	Excess	mIoU	F1
YOLO	-0.60	0.98	0.59	0.98	0.10	0.25	0.00
Heron	0.66	0.77	0.12	0.00	0.02	0.44	0.06
PPDoc-L	0.65	0.65	0.00	0.00	0.02	0.45	0.12

Table 19 : Per-model metrics for TTW_1868-05-16_page_5 , bold is best in column

Font	Layout	Columns
Ewert, Great Vibes, Kablammo, Roboto, UnifrakturMaguntia	Centered, Justified, Left aligned	1, 2, 3

Table S1 : Parameters used in text generation process

Panel of Figure 26 — Figure S1 : Example pages of text use for measuring granularity robustness.

Panel of Figure 27 — Figure S2 : Different parsing issues and errors

Prediction set	COTe	Cov	Ovl	Tre	Exc	F1@0.5	mIoU	CER
Missing regions	0.85	0.85	0.00	0.00	0.00	0.83	0.71	0.17
Duplicate regions	0.39	1.00	0.61	0.00	0.00	0.87	1.00	0.59
Granularity, line GT	1.00	1.00	0.00	0.00	0.15	0.32	0.36	0.00
Granularity, SSU GT	0.84	0.84	0.00	0.00	0.00	0.32	0.61	0.00
Complex errors	0.67	0.91	0.12	0.12	0.07	0.76	0.64	0.36

Table S2 : There is a substantial range of outcomes across the case studies and metrics demonstrating the importance of decomposable errors. We also see that parsing errors induce apparent OCR errors even when there are none.

Panel of Figure 29 — Figure S3 : Man sitting on howdah and riding an Elephant.

Panel of Figure 30 — Figure S4 : Image of a train station

Model	COTe	Coverage	Overlap	Trespass	Excess	COTe F1	PQ	SQ	RQ
Mask2Former	0.96	0.97	0.00	0.01	0.03	0.91	0.76	0.76	1.00
Yolo26	0.96	0.96	0.00	0.00	0.04	0.90	0.78	0.78	1.00

Table S3 : COTe and panoptic quality on the COCO elephant image (things scope). Overlap, trespass and excess are lower-is-better.

Model	COTe	Coverage	Overlap	Trespass	Excess	COTe F1	PQ	SQ	RQ
Mask2Former	0.91	0.93	0.00	0.02	0.01	0.93	0.27	0.32	0.42
Yolo26	0.11	0.92	0.78	0.02	0.02	0.92	0.58	0.58	0.75

Table S4 : COTe and panoptic quality on the COCO train image (things scope). Overlap, trespass and excess are lower-is-better.

Explore similar work

May 19, 2026cs.CL

How Do Document Parsers Break? Auditing Structural Vulnerability in Document Intelligence

Document Layout Analysis (DLA) pipelines provide structured page representations for retrieval-augmented generation, long-document question answering, and related applications. Yet their robustness evaluation remains largely area-centric. We identify this Footprint Bias and propose ProSA, a lightweight output-level auditing framework that decouples controlled probing, policy-driven targeting, and structure-aware diagnosis. ProSA combines Block-level Structural Loss Rate (B-SLR), granularity-aware exposure descriptors, and pathway attribution to analyze where structural identity is lost, at what exposure granularity failures emerge, and how failures propagate. Across MinerU and PP-StructureV3 on 1,000 pages, affected area weakly tracks perturbation-induced OCR instability (

R^2=0.384/0.110

), whereas B-SLR aligns much more closely with it (

R^2=0.727/0.916

). Exposure descriptors further separate occlusion- and topology-dominant pathways, while matched-footprint structural probes cause much larger downstream QA/retrieval drops than area-matched erasure. These results shift DLA robustness evaluation from footprint-based measurement toward structure-aware vulnerability auditing.

Yue Chen, Yihao Wang, Ziyi Tang +2

Sun Yat-sen University · Chaotic Pendulum Lab · Nanyang Technological University

Document Parsing Model Auditing Optical Character Recognition Exposure Geometric Bias

Ask PDF

Jun 22, 2026cs.CV

RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild

Accurate document layout analysis remains a critical bottleneck for document parsing systems, due to the intricate coupling among heterogeneous document layout elements, geometric distortions (\eg, paper warping and bending, perspective variations), and reading order within diverse layout structures. Existing approaches typically rely on fragmented multi-stage pipelines or computationally heavy generative Transformer architectures, leading to error propagation and limited efficiency. In this paper, we present RT-DocLayout, a highly efficient end-to-end framework for document layout analysis, designed as a front-end for document parsing tasks. The proposed model unifies classification, detection, pixel-level segmentation, and reading order prediction for layout elements within a single 33M-parameter architecture. Built upon the RT-DETR, our key contribution is a unified multi-task formulation within a single query-based decoder that simultaneously classifies, regresses bounding box, generates masks, and constructs relationship to reason reading order. By jointly learning geometric and structural representations, RT-DocLayout introduces multi-task optimization that substantially improves robustness under real-world document distortions. Extensive experiments on public benchmarks demonstrate state-of-the-art performance in document layout analysis while maintaining real-time inference speed(132.1 FPS). When coupled with downstream OCR engines, RT-DocLayout significantly improves full-document reconstruction quality, providing a scalable and practical foundation for real-world document intelligence systems.

Cheng Cui, Tingquan Gao, Xueqing Wang +11

PaddlePaddle Team, Baidu Inc.

Document Parsing Order Matters Generative Pretrained Transformers Reconstruction Fidelity

Ask PDF

May 19, 2026cs.CV

Structured Layout Priors for Robust Out-of-Distribution Visual Document Understanding

Vision-Language Models (VLMs) parse documents end-to-end but frequently break down on layouts unlike those seen in training. We attribute this to a two-hop bottleneck: before the decoder can extract content (Hop 2), it must first classify and localize the enclosing layout entity (Hop 1), and when the first hop fails the second collapses into omissions, malformed structure, or autoregressive repetition. We pre-resolve Hop 1 outside the decoder by running a lightweight RT-DETR detector, serializing its outputs in the parser's native DocTags vocabulary, and injecting them into the prompt alongside the full page image. Unlike analyze-then-parse approaches that crop the page, or prior prompt-level priors written in plain text, our prior shares the decoder's generation space and leaves the global image in view as a fallback when detections are noisy. On a 10k-page structural out-of-distribution benchmark, markdown F1 rises from

0.37

0.92

; on the Chinese subset of OmniDocBench, table TEDS rises from

0.01

0.36

; and on the 26k-page ViDoRe V3 benchmark, infinite-loop decoding failures drop across every industrial domain tested. These gains cost

15\%

wall-clock latency and a median of

74

prompt tokens, with no architectural change to the base VLM. An attention-level analysis further reveals a bimodal phase shift in which the decoder attends to injected layout tokens when emitting structure and to image patches when emitting content, consistent with the two-hop bottleneck being alleviated. Model weights will be released to support reproducibility.

Peter El Hachem, Ahmed Nassar, A. Said Gurbuz +2

ETH Zurich Rämistrasse, Switzerland · IBM Research Rüschlikon, Switzerland

Visual Document Retrieval Document Parsing Object Tokens Out-Of-Distribution Prior Knowledge Integration Concept Bottleneck Models

Ask PDF