cs.CVMar 10, 2026
SavePromptDLA: A Domain-aware Prompt Document Layout Analysis Framework with Descriptive Knowledge as a Cue
Organizations: Columbia University · Institute of Automation, Chinese Academy of Sciences · University of the Chinese Academy of Sciences · Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd.
Abstract
Document Layout Analysis (DLA) is crucial for document artificial intelligence and has recently received increasing attention, resulting in an influx of large-scale public DLA datasets. Existing work often combines data from various domains in recent public DLA datasets to improve the generalization of DLA. However, directly merging these datasets for training often results in suboptimal model performance, as it overlooks the different layout structures inherent to various domains. These variations include different labeling styles, document types, and languages. This paper introduces PromptDLA, a domain-aware Prompter for Document Layout Analysis that effectively leverages descriptive knowledge as cues to integrate domain priors into DLA. The innovative PromptDLA features a unique domain-aware prompter that customizes prompts based on the specific attributes of the data domain. These prompts then serve as cues that direct the DLA toward critical features and structures within the data, enhancing the model's ability to generalize across varied domains. Extensive experiments show that our proposal achieves state-of-the-art performance among DocLayNet, PubLayNet, M6Doc, and DLA. Our code is available at https://github.com/Zirui00/PromptDLA.
Figures & tables
Fig. 1: Examples of different domain differences across ( a ) Different document types caused variations in layout structure and element distribution (financial report, manual, patent) ( b ) different language types, and ( c )Inconsistent labeling styles. Note that the ”text” and ”list” items in DocLayNet are labeled as smaller individual units while they are integrated as a whole in DocBank.
Fig. 2: Overview of the PromptDLA method for domain-aware layout prediction. A Domain-Aware Prompter encodes domain information into a prompt vector, which is prepended to the sequence of image patch embeddings. This augmented input is processed by a vision backbone. Multi-scale features are extracted from the backbone and refined by a FPN before being passed to a detection head for final layout prediction.
Fig. 3: Comparison with Pre-training Paradigms in DLA
| Dataset | Document Type | A.M. | Format | #Class | Language | #Images |
| PubLayNet [ 14 ] | Articles | Automatic | 5 | English | 364232 | |
| DocLayNet [ 16 ] | Financial Reports, Manuals, Scientific Articles, Laws & Regulations, Patents, Government Tenders. | Manual | 11 | English, German, French, Japanese | 80863 | |
| M6Doc [ 17 ] | Scientific articles, Textbooks, Books, Test papers, Magazines, Newspapers, Notes | Manual | PDF, Scanned, Photographed | 74 | English,Chinese | 9080 |
| D 4 LA [ 18 ] | Scientific report, Email, Form, Invoice, Letter, Specification, News article, Presentation, Resume, Scientific publication, Budget, Memo | Manual | PDF, Scanned, | 27 | English | 11092 |
| MLDLA(ours) | Magazine, Newspaper, Government Reports | Manual | PDF, Scanned, Photographed | 5 | Hindi, Kazakhstan, Vietnam, Turkey, Persia, Laos, Khmer | 17505 |
TABLE I: The Detail of document types in different DLA datasets.
Fig. 4: Framework of Prompt Generator.
Fig. 5: Framework of Fusion Layer.
| Method | Pretraining | Caption | Footnote | Formula | List-item | Page-footer | Page-header | Picture | Section-header | Text | Table | Title | mAP |
| Mask R-CNN [ 17 ] | ✗ | 71.5 | 71.8 | 63.4 | 80.8 | 59.3 | 70.0 | 72.7 | 69.3 | 82.9 | 85.8 | 80.4 | 73.5 |
| Faster R-CNN [ 17 ] | ✗ | 70.1 | 73.7 | 63.5 | 81.0 | 58.9 | 72.0 | 72.0 | 68.4 | 82.2 | 85.4 | 79.9 | 73.4 |
| YOLOv5 [ 17 ] | ✗ | 77.7 | 77.2 | 66.2 | 86.2 | 61.1 | 67.9 | 77.1 | 74.6 | 86.3 | 88.1 | 82.7 | 76.8 |
| TransDLANet [ 17 ] | ✗ | 68.2 | 74.7 | 61.6 | 81.0 | 54.8 | 68.2 | 68.5 | 69.8 | 82.4 | 83.8 | 81.7 | 72.3 |
| SwinDocSegmenter [ 5 ] | ✗ | 83.6 | 64.8 | 62.3 | 82.3 | 65.1 | 66.4 | 84.7 | 66.5 | 87.4 | 88.2 | 63.3 | 76.9 |
| SelfDocSeg [ 5 ] | ✓ | - | - | - | - | - | - | - | - | - | - | - | 74.3 |
TABLE II: Comparison with state-of-the-art methods on DocLayNet. Our method uses domain prompts from human knowledge and prompt engineering, with a ViT backbone, CLIP text encoder, and Cascade Mask R-CNN detection head. All baseline results are taken directly from previous papers.
| Method | Text | Title | Figure | List | Table | mAP |
| DiT | 77.9 | 58.6 | 54.5 | 75.6 | 76.1 | 68.5 |
| PromptDLA | 78.7 | 59.1 | 57.3 | 74.5 | 77.9 | 69.5 |
| \color[rgb]{0,0,1}{+0.8} | \color[rgb]{0,0,1}{+0.5} | \color[rgb]{0,0,1}{+2.8} | {\color[rgb]{1,0,0}-1.1} | \color[rgb]{0,0,1}{+1.8} | \color[rgb]{0,0,1}{+1.0} |
TABLE III: Experiments on a real multi-language dataset.
Fig. 6: Inconsistent labeling relationship between DocLayNet and PubLayNet.
| DocLayNet | PubLayNet | |||||
| Baselines | Joint | PromptDLA | Baselines | Joint | PromptDLA | |
| Text | 86.9 | 87.0( +0.1) | 87.5 ( +0.6) | 94.4 | 94.5( +0.1) | 94.5 ( +0.1) |
| Title | 71.2 | 72.1( +0.9) | 73.7 ( +2.5) | 88.9 | 89.4( +0.5) | 89.5 ( +0.6) |
| List | 84.0 | 83.5( -0.5 ) | 84.4 ( +0.4) | 94.8 | 95.5( +0.7) | 95.6 ( +0.8) |
| Table | 87.1 | 87.2( +0.1) | 87.7 ( +0.6) | 97.6 | 97.8( +0.2) | 97.8( +0.2) |
| Figure | 77.2 | 75.7( -1.5 ) | 76.3 ( -0.9 ) | 96.9 | 96.9( +0.0) | 97.0 ( +0.1) |
TABLE IV: Experiments on inconsistent labeling styles.
| Tag | Method | mAP | |
| (a) | DiT | 62.68 | |
| (b) | DiT + Human knowledge | 64.23 | +1.55 |
| (c) | DiT + LVLM | 63.93 | +1.25 |
TABLE V: OOD result on Manuals from DocLayNet
| Tag | Method | mAP | |
| (a) | ViT | 76.4 | |
| (b) | ViT + PromptDLA | 78.7 | +2.3 |
| (c) | Swin | 78.7 | -0.12 |
| (d) | Swin + PromptDLA | 79.7 | +1.0 |
| (e) | ResNet50 (DETR) | 77.0 | |
| (f) | ResNet50 + PromptDLA | 77.7 | +0.7 |
TABLE VI: Performance comparison of different backbone architectures with and without PromptDLA on the DocLayNet dataset. indicates the absolute mAP improvement achieved by adding PromptDLA compared to the respective baseline model
| Tag | Method | mAP | |
| (a) | Faster R-CNN (R50) | 66.8 | |
| (b) | Faster R-CNN + PromptDLA | 68.4 | +1.6 |
| (c) | DETR (R50) | 77.0 | |
| (d) | DETR + PromptDLA | 77.7 | +0.7 |
TABLE VII: Effect of PromptDLA on different detection heads. denotes the absolute mAP improvement. We re-implemented baseline models for fair comparison.
| Model | mAP@IOU[0.50:0.95] | ||
| DocLayNet | M6Doc | D 4 LA | |
| TransDLA [ 17 ] | 72.3 | 63.8 | - |
| SwinDocSegmenter [ 5 ] | 76.9 | - | - |
| VGT [ 18 ] | - | - | 68.8 |
| LayoutLMv3 | 75.7 | 60.5 | 62.6 |
| +PromptDLA | 76.4( +0.7) | 61.3( +0.8) | 63.1( +0.5) |
TABLE VIII: Performance of the PromptDLA with different pre-trained models on DocLayNet, M6Doc, and D 4 LA, and comparison with the current state-of-the-art method
Fig. 7: Different pre-trained text encoder for PromptDLA.
| mAP | ||
| DiT [ 34 ] | 76.4 | |
| w/o Pre-trained Text Encoder | 76.3 | |
| CLIP Text Encoder [ 19 ] | 78.7 | |
| BLIP2 Text Encoder [ 27 ] | 79.0 | |
| LLAMA2 Text Encoder [ 20 ] | 77.8 |
TABLE IX: Experiments on different Pre-trained model Text Encoders (w/o Pre-trained Text Encoder means random initializing prompt without using a Pre-trained Text Encoder).
| Tag | Method | mAP | |
| (a) | Human-Knowledge Prompt (Baseline) | 78.69 | - |
| (b) | LVLM-Generated Prompt | 78.68 | -0.01 |
| (c) | Human-Knowledge Prompt w/ Classifier | 78.57 | -0.12 |
| (d) | Hybrid Prompt (Human-Guided LVLM) | 79.02 | +0.33 |
| (e) | Concatenated Prompts (a) + (b) | 78.71 | +0.02 |
TABLE X: Analysis of different prompt generation strategies on the DocLayNet dataset. The column measures the mAP change relative to the baseline Human-Knowledge Prompt (a).
| Tag | Method | mAP | |
| (a) | PromptDLA | 78.69 | |
| (b) | PromptDLA + OCR | 78.53 | -0.16 |
TABLE XI: Comparison of Visual-Only vs Textual and Visual Modalities
Fig. 8: Details of the shallow and deep prompt.
Fig. 9: Qualitative comparison between DiT and PromptDLA on Financial Reports (1st row) and Laws (2nd row) domain from DocLayNet. It is best viewed in color and zooming out.
| Tag | Method | mAP | |
| (a) | PromptDLA | 78.69 | |
| (b) | w/o pretraining | 70.66 | -8.03 |
| (c) | VLP-pretraining(Layoutlmv3) | 76.38 | -2.31 |
| (d) | w/o CLIP text encoder | 76.33 | -2.36 |
TABLE XII: Ablation study for pretraining methods.
| Tag | Method | mAP | |
| (a) | Baselines | 76.43 | |
| (b) | SPE | 78.41 | +1.98 |
| (c) | DPE | 78.69 | +2.26 |
| (d) | DPE with share MLP Layer | 78.44 | +2.01 |
TABLE XIII: Ablation study for prompt location.
| Prompt | Accuracy | mAP |
| w/o prompt | - | 76.43 |
| A page comes from {Domain Class} | 44.05 | 77.49 |
| A document page of {Domain Class} | 44.05 | 77.53 |
| A piece of paper concerning with {Domain Class} | 44.44 | 77.55 |
| A piece of paper comes from {Domain Class} | 46.99 | 77.97 |
| A document page comes from {Domain Class} | 48.45 | 78.12 |
TABLE XIV: Ablation study for different domain prompts.
| Caption | Footnote | Formula | List-item | Page-footer | Page-header | Picture | Section-header | Table | Text | Title | mAP | ||
| Finacial reports | w/o | 55.7 | 10.6 | - | 69.4 | 52.6 | 46.1 | 64.0 | 64.4 | 90.4 | 85.1 | 54.5 | 59.3 |
| w | 55.4 | 23.9 | - | 73.8 | 60.2 | 55.4 | 67.7 | 67.6 | 91.4 | 86.2 | 60.5 | 64.2 | |
| Government tenders | w/o | 15.0 | 82.0 | 65.3 | 90.0 | 51.5 | 75.6 | 75.3 | 78.0 | 95.4 | 85.1 | 39.1 | 68.4 |
| w | 27.1 | 88.3 | 100.0 | 91.8 | 50.9 | 80.4 | 77.8 | 81.7 | 95.0 | 88.3 | 61.3 | 76.6 | |
| Laws and regulations | w/o | 28.0 | 91.6 | 16.7 | 81.6 | 42.9 | 71.8 | 49.5 | 68.0 | 70.4 | 84.1 | 84.4 | 62.6 |
| w | 28.5 | 96.1 | 18.5 | 82.0 | 42.4 | 64.4 | 50.4 | 69.5 | 67.7 | 84.7 | 84.6 | 62.6 |
TABLE XV: Experiments result on each document type in DocLayNet (“w/o” and “w” denote DiT baseline and PromptDLA, respectively).
| DocLayNet | M6Doc | |
| #Document Types | 6 | 7 |
| Accuracy | 48.45% | 54.55% |
TABLE XVI: Zero-shot document classification performance of CLIP.
| Model | mAP@IOU[0.50:0.95] | FPS |
| DiT-Base | 76.4 | 6.75 |
| +PromptDLA | 78.7(+2.3) | 6.62(-0.13) |
| LayoutLMv3 | 75.7 | 4.44 |
| +PromptDLA | 76.4(+0.7) | 4.41(-0.03) |
TABLE XVII: Computational overhead of PromptDLA.
Fig. 10: Appendix: More visualization examples on the DocLayNet dataset. From left to right, columns show Ground Truth annotations, predictions without PromptDLA, and predictions with PromptDLA, highlighting document layout segmentation accuracy improvements.
More in this topic
Document Layout Analysis
Document layout analysis identifies and organizes page elements such as text, tables, figures, and forms.
See all 60 papers