LayoutMamba: Efficient Document Layout Analysis with Hybrid Mamba-Transformer Networks

Kehua Chen; Mingyi Chen

cs.CVOct 26, 2026

LayoutMamba: Efficient Document Layout Analysis with Hybrid Mamba-Transformer Networks

Authors: Kehua Chen, Mingyi Chen

Abstract

Document layout analysis (DLA) faces a long-standing dilemma: existing methods either capture rich local details at prohibitive computational cost (e.g., Transformer-based approaches with quadratic complexity) or achieve efficient global modeling but sacrifice fine-grained structural awareness (e.g., state-space models). This trade-off hinders deployment in resource-constrained scenarios. We present LayoutMamba, a hybrid framework that synergistically combines sparse-attention Transformers for localized feature extraction (O(n) complexity) and the recurrent state-space mechanism of Mamba for dilution-free long-range dependency modeling across the 2D document canvas. Our two-stage paradigm first uses a lightweight Transformer encoder to hierarchically distill salient local semantics; second, Mamba propagates global contextual dependencies without attention dilution, enabling precise, efficient, and scalable layout understanding. We further identify a critical yet overlooked issue in existing DLA benchmarks: the inherent low-overlap distribution bias of layout objects, which we mitigate through a novel Geometric Overlap Penalty Loss (GOPLoss), a dual-optimization objective that minimizes redundant bounding box overlaps and encourages geometrically consistent predictions. Extensive experiments on PubLayNet and D4LA show that LayoutMamba surpasses state-of-the-art AP on PubLayNet with 8x fewer parameters and maintains leading performance on D4LA, ranking top-3 among published methods, achieving an unprecedented balance between accuracy, efficiency, and scalability. Our work pioneers a cognition-inspired paradigm for document analysis, advancing practical deployment in edge computing and resource-limited environments.

Figures & tables

Figure 1: Comprehensive overview of the proposed efficient document layout analysis framework leveraging hybrid Mamba-Transformer architectures.

Models	Year	#Param	Text	Title	List	Table	Figure	mAP
VSR ( Zhang et al., 2021 )	2021	–	0.967	0.931	0.947	0.974	0.964	0.957
TRDLU ( Yang and Hsu, 2022 )	2022	–	0.958	0.921	0.975	0.976	0.966	0.959
DiT-L ( Li et al., 2022a )	2022	304M	0.944	0.893	0.960	0.978	0.972	0.949
UDoc ( Gu et al., 2023 )	2022	272M	0.939	0.885	0.937	0.973	0.964	0.939
VGT ( Da et al., 2023 )	2023	174M	0.950	0.939	0.968	0.981	0.971	0.962
DETR ( Shehzadi et al., 2023 )	2023	–	0.947	0.918	0.964	0.981	0.975	0.957

Table 1: Document Layout Detection mAP @ IoU [0.50:0.95] on PubLayNet validation set.

Models	#Param	DocTitle	ListText	LetterHead	Question	RegionList	TableName	FigureName	Footer	Number
DiT-B ( Li et al., 2022a )	87M	73.1	70.6	82.2	55.0	80.1	68.4	51.8	81.2	83.2
LayoutLMv3-Base ( Huang et al., 2022 )	133M	66.8	56.5	78.5	39.3	72.1	64.3	32.1	72.2	82.1
VGT ( Da et al., 2023 )	174M	72.6	71.3	82.3	63.9	80.2	68.4	46.6	79.7	83.2
DoPTA-HR ( SR et al., 2024 )	85M	73.11	72.46	82.07	77.42	79.32	67.08	56.86	77.88	83.15
LayoutMamba-T	6.9M	74.2	69.5	85.3	68.8	82.1	61.7	49.5	77.2	81.4
LayoutMamba-B	22M	74.6	72	83.9	64.9	82.4	65.2	52	78.5	81.5

Table 2: Document Layout Detection mAP @ IoU [0.50:0.95] on

D^{4}LA

validation set.

GOPLoss Weight	0.0	1.0	3.0	5.0	7.5
mAP	57.9	58.2	58.2	58.4	57.6

Table 3: Document Layout Detection mAP @ IoU [0.50:0.95] on

D^{4}LA

validation set with different GOPLoss weight.

VRSSBlock	Attention	#Param	mAP
		5.8	60.7
✓		6.2	61.2
✓	✓	6.9	63.3

Table 4: Document Layout Detection mAP @ IoU [0.50:0.95] on

D^{4}LA

validation set.

Panel of Figure 2 — Figure 2: Visual comparison of detection results under different GOPLoss weights.

Document Layout Analysis

Document layout analysis identifies and organizes page elements such as text, tables, figures, and forms.

See all 64 papers

Mar 10, 2026

PromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue

Document Layout Analysis (DLA) is crucial for document artificial intelligence and has recently received increasing attention, resulting in an influx of large-scale public DLA datasets. Existing work often combines data from various domains in recent public DLA datasets to improve the generalization of DLA. However, directly merging these datasets for training often results in suboptimal model performance, as it overlooks the different layout structures inherent to various domains. These variations include different labeling styles, document types, and languages. This paper introduces PromptDLA, a domain-aware Prompter for Document Layout Analysis that effectively leverages descriptive knowledge as cues to integrate domain priors into DLA. The innovative PromptDLA features a unique domain-aware prompter that customizes prompts based on the specific attributes of the data domain. These prompts then serve as cues that direct the DLA toward critical features and structures within the data, enhancing the model's ability to generalize across varied domains. Extensive experiments show that our proposal achieves state-of-the-art performance among DocLayNet, PubLayNet, M6Doc, and D

^4

LA. Our code is available at https://github.com/Zirui00/PromptDLA.

Zirui Zhang, Yaping Zhang, Lu Xiang +4

Columbia University · Institute of Automation, Chinese Academy of Sciences · University of the Chinese Academy of Sciences +1

Document Layout Analysis Domain Generalization Prompt Learning

Ask PDF

Jun 22, 2026

RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild

Accurate document layout analysis remains a critical bottleneck for document parsing systems, due to the intricate coupling among heterogeneous document layout elements, geometric distortions (\eg, paper warping and bending, perspective variations), and reading order within diverse layout structures. Existing approaches typically rely on fragmented multi-stage pipelines or computationally heavy generative Transformer architectures, leading to error propagation and limited efficiency. In this paper, we present RT-DocLayout, a highly efficient end-to-end framework for document layout analysis, designed as a front-end for document parsing tasks. The proposed model unifies classification, detection, pixel-level segmentation, and reading order prediction for layout elements within a single 33M-parameter architecture. Built upon the RT-DETR, our key contribution is a unified multi-task formulation within a single query-based decoder that simultaneously classifies, regresses bounding box, generates masks, and constructs relationship to reason reading order. By jointly learning geometric and structural representations, RT-DocLayout introduces multi-task optimization that substantially improves robustness under real-world document distortions. Extensive experiments on public benchmarks demonstrate state-of-the-art performance in document layout analysis while maintaining real-time inference speed(132.1 FPS). When coupled with downstream OCR engines, RT-DocLayout significantly improves full-document reconstruction quality, providing a scalable and practical foundation for real-world document intelligence systems.

Cheng Cui, Tingquan Gao, Xueqing Wang +11

PaddlePaddle Team, Baidu Inc.

Multi-Task Learning Document Layout Analysis Document Image Understanding

Ask PDF

Oct 20, 2026

DocLayout-YOLO: Axis-Attentive Document Layout Detector with Mesh-Guided Synthetic Pretraining

Accurate and efficient document layout analysis is critical for modern document parsing systems. However, although the YOLO framework represents a leading paradigm for the real-time detector, its adaptation to heterogeneous documents remains underexplored. To address this gap, we propose DocLayout-YOLO, a document-specialized real-time detector that integrates diverse synthetic document pretraining and page-axis context modeling. From the data perspective, we introduce MeshLayout-1M, a synthetic pretraining corpus generated by the proposed Mesh-candidate BestFit algorithm, which provides large-scale and diverse document layouts for pretraining and improves adaptation to heterogeneous document scenarios. From the model perspective, we propose Page-Axis Layout Attention (PALA), which captures long-range dependencies along the horizontal and vertical page axes while remaining computationally efficient. Extensive experiments demonstrate that DocLayout-YOLO achieves state-of-the-art performance on four heterogeneous document benchmarks, obtaining 72.4%/75.5%/96.0%/73.1% mAP50:95 on D4LA, M6Doc, HJDataset, and PRIMA-LAD, respectively. Compared with existing real-time detectors, DocLayout-YOLO establishes a superior accuracy-efficiency trade-off in document layout analysis. Code, model, and data are available at https://github.com/opendatalab/DocLayout-YOLO/tree/release-v2.

Zhiyuan Zhao, Hengrui Kang, Guang Liang +2

Shanghai AI Laboratory

Document Layout Analysis Efficient Object Detection Attention Mechanisms Synthetic Data Generation

Ask PDF

Abstract

Document Layout Analysis

Explore similar work

PromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue

RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild

DocLayout-YOLO: Axis-Attentive Document Layout Detector with Mesh-Guided Synthetic Pretraining