AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing

Boyu Niu; Yingjie Xiong; Shurui Li; Linhan Pang; Yuntong Zhang; Han Wang; Hao Wu; Kai Zuo

cs.AINov 4, 2026

AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing

Authors: Boyu Niu, Yingjie Xiong, Shurui Li, Linhan Pang, Yuntong Zhang, Han Wang, Hao Wu, Kai Zuo,

Organizations: Shanghai Jiao Tong University · Theseus Labs · Xiaohongshu

Abstract

Traditional document parsing systems use Markdown as their output representation, which is effective for linear text but does not fully preserve fine-grained spatial structure, table geometry, visual hierarchy, or rendering semantics. We show that HTML is advantageous for heterogeneous and layout-sensitive documents whose representations must remain editable, inspectable, and reusable, and introduce AmbiHTMLBench to systematically study this setting. In this benchmark, given a real-world document, a model must recover executable HTML that preserves its content, organization, and visual appearance. The benchmark contains 1,115 diverse pages with structured annotations and evaluates predictions in two different ways: The objective protocol aligns ground-truth and predicted rendered blocks and measures table fidelity, page-level text error, and a two-dimensional layout consistency; a complementary reference-anchored protocol measures perceived fidelity by aligning LLM judges with human preferences. Across ten primary multimodal systems, the best objective score is 0.8092, while the subjective winner is different, demonstrating that automatic decomposition and human-perceived fidelity capture complementary properties. We further fine-tune a small-parameter expert model and find that task-specific adaptation yields good performance on document-to-HTML reconstruction task. These results position AmbiHTMLBench as a diagnostic testbed for building smarter and more logical document parsing systems.

Figures & tables

Figure 1: From document images to reusable HTML. Complete pages from realistic genres (top) are reconstructed as HTML and rendered in browser (middle), more cases are shown in appendix C . The bottom row lists potential uses of the generated content.

Figure 2: AmbiHTMLBench focuses on three level of fidelity in HTML based document parsing, with both objective and reference-anchored subjective protocols.

Figure 3: Pilot evidence that HTML representations support spatial binding and table verification.

Figure 4: The overall AmbiHTMLBench Construction process.

Figure 5: Human calibration and deployment of the frozen preference proxy.

Rank	Model	Text ED $\downarrow$	TEDS $\uparrow$	TEDS-S $\uparrow$	Layout $\uparrow$	Obj. score $\uparrow$
1	Gemini 3.1 Pro Preview	0.1754	0.6518	0.6650	$0.8410$	0.8092
2	Claude Opus 5.5	$0.1994$	$0.6433$	$0.6553$	$0.8574$	$0.8048$
3	Gemini 3.8 Flash	$0.2070$	$0.6415$	$0.6530$	$0.8288$	$0.7888$
4	Kimi-K3	$0.2399$	$0.6253$	$0.6389$	$0.8307$	$0.7732$
5	Claude Opus 5	$0.2537$	$0.6290$	$0.6465$	$0.8387$	$0.7712$
6	Qwen 3.8 max	$0.2470$	$0.6095$	$0.6235$	$0.8198$	$0.7633$

Table 1: Objective leaderboard on 1,115 pages for fifteen primary systems. Text ED is lower-is-better; all other columns are higher-is-better. The weighted score uses table weight 0.3 and text and layout weights 1.0. TEDS and TEDS-S are averaged over the 263 table-bearing pages.

Model	Method	Weighted $\uparrow$	Text $\uparrow$	TEDS $\uparrow$	Layout $\uparrow$
Qwen3.5-9B	Base	$0.6623$	$0.6321$	$0.5440$	$0.7281$
Qwen3.5-9B	Distill SFT	0.7064	0.6761	0.5827	0.7737
Qwen3.5-4B	Base	$0.6786$	$0.6323$	$0.5382$	$0.7670$
Qwen3.5-4B	Distill SFT	0.6934	0.6556	0.5722	0.7675

Table 2: Task-specific fine-tuning results on AmbiHTMLBench. All reported metrics are higher-is-better; “Text” is the normalized text score and the weighted score is the primary objective metric.

Rank	Model	BT score (95% CI)			Soft pref. $\uparrow$	Rank CI
Rank	Model	Score	Low	High	Soft pref. $\uparrow$	Rank CI
1	GPT 5.6 Sol	$154.70$	$140.59$	$168.82$	$71.64$ %	[1, 1]
2	GPT 5.6 Terra	$102.87$	$90.18$	$116.25$	$64.76$ %	[2, 3]
3	Gemini 3.1 Pro Preview	$100.86$	$89.88$	$112.74$	$64.49$ %	[2, 3]
4	Qwen3.7 Plus	$56.04$	$45.63$	$66.45$	$58.12$ %	[4, 4]
5	Kimi K3	$9.62$	$-2.16$	$21.71$	$51.32$ %	[5, 6]

Table 3: Final subjective leaderboard from the frozen preference proxy. BT scores are centered; soft preference averages predicted win probability over scheduled appearances. Intervals use page-block bootstrap. The 8,128 additional comparisons are proxy-scored rather than independently human-labelled.

Figure 6: Page-level reversal on page 0309: objective metrics favor Opus (A), whereas recorded subjective judgments favor Sol (B). Insets contrast faithful wording with preserved visual emphasis.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Activation	Axis	$n$	Visual criteria
Fixed	Global layout	1,872	Regions, columns, spacing, grouping, reading order, proportions.
Fixed	Text presentation	1,872	Wrapping, typography, density, alignment, hierarchy, legibility.
Conditional	Table	497	Visible row/column organization and cell alignment.
Conditional	Formula	61	Fractions, scripts, radicals, matrices, and mathematical alignment.
Conditional	Algorithm	12	Block organization, indentation, and step hierarchy.
Conditional	Vertical text	90	Vertical flow, orientation, and reading organization.

Appendix

Table 4: Diagnostic axes in the 1,872-comparison human set. Counts denote comparisons, not distinct pages. The rarest axes support case analysis but not precise population comparisons.

Judge	Coefficient $\beta_{j}$
Claude Opus 5	0.09045204815695966
Claude Sonnet 5	0.10401091278902799
Gemini 3.1 Pro Preview	0.37576678499354577
GPT-5.6-Sol	0.13742077090225210
GPT-5.6-Terra	0.16012498207363500

Appendix

Table 5: Frozen preference-proxy coefficients. The deployment applies clipping to the weighted sum; the coefficients are not normalized.

	Pilot			Holdout
Committee	Brier	Coverage	Rank	Brier	Coverage	Rank
S+G+T (v1)	0.1258	77.81%	1	0.1293	78.99%	4
O+G+T	0.1302	78.44%	2	0.1263	79.90%	2
S+G+L	0.1369	79.06%	3	0.1267	79.45%	3
O+G+L (v2)	0.1404	79.06%	4	0.1240	80.35%	1

Appendix

Table 6: Historical cross-family committee audit. Pilot

n=320

; holdout

n=1,552

. Coverage is the fraction resolved. O, S, G, L, and T denote Opus, Sonnet, Gemini, Sol, and Terra. These discrete committees are separate from the deployed preference proxy.

Figure 7: Text occlusion. Claude Opus 5.5 keeps the section heading between the two tables. In GPT-6.1 Sol, the heading overlaps the preceding table, disrupting the separation between sections. This makes the document difficult to read visually in terms of layout.

Figure 8: Table structure degradation. GPT-6.1 Sol aligns the final column header with the table body. In Claude Opus 5.5, the header extends beyond the right edge of the body, leaving the column misaligned, and the header of the second-to-last column has not been properly recognized.

Figure 9: Text reproduction error. Claude Sonnet 5.5 substitutes a different character; The distinctive color and doodle style of social media notes have also been eliminated. In this case, GPT-6.1 Sol towers over.

Figure 10: Information loss. Gemini 3.8 Flash retains the tuition and examination information in the highlighted row. Claude Sonnet 5.5 leaves both corresponding cells empty.

Figure 11: Formula rendering error. Claude Opus 5.5 renders the highlighted calculation as two-dimensional fractions. Qwen3.8-Max displays raw formula commands in several lines, obscuring the mathematical structure.

Figure 12: Code formatting error. Gemini 3.8 Flash retains the line breaks and indentation of the reference code block. Qwen3.8-Max merges separate statements and loses indentation, making the code structure harder to read.

Document Parsing Benchmarks

See all 18 papers

May 31, 2026

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Optical Character Recognition (OCR) and document parsing benchmarks are increasingly limited in coverage and difficulty: many focus on common document genres or uniformly sampled pages where modern parsers already perform strongly, while offering limited annotation for expert-domain structures such as chemical formula, music notation, complex tables, and cross-page layouts. We introduce Dr. DocBench, a difficulty-aware benchmark for expert-level document parsing. Built from a large-scale multilingual book corpus, Dr. DocBench spans 52 BISAC subject domains and selects challenging documents through parser-failure-based sampling, targeting cases where multiple state-of-the-art systems struggle. It contains 4,514 annotated pages from long documents averaging around 100 pages, with 65k high-quality page- and block-level annotations for layout, reading order, hierarchical relations, and domain-specific visual contents. Evaluations of pipeline-based parsers and general-purpose VLMs show that strong performance on existing benchmarks does not transfer to our expert-level document parsing. Our analysis reveals substantial failures across subjects, content types, and structural attributes, highlighting Dr. DocBench as a comprehensive testbed for diagnosing and advancing document intelligence.

Minglai Yang, Xinyan Velocity Yu, Pengyuan Li +22

2077AI · University of Arizona · University of Southern California +8

VLM Evaluation Document Parsing Benchmarks Document Parsing Document Layout Analysis Document Image Understanding

Ask PDF

May 8, 2026

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

Document parsing models are improving rapidly, but existing benchmarks are still insufficient for a comprehensive evaluation of their capabilities. We introduce PureDocBench, a benchmark with 10 domains, 66 subcategories, and 1,475 pages. Each page has matched Clean, Digital, and Real versions, covering clean renders, simulated degradation, and real capture or sharing conditions. The resulting 4,425 images have source-derived annotations for text, formulas, tables, and reading order, checked through automated validation and human review. Keeping page content and annotations fixed allows direct comparisons across image conditions. We evaluate 58 models, including pipeline-based parsers, end-to-end parsers, and general vision-language models. The best three-track average score is 79.69 out of 100, and the mean across models is 63.84. Scores drop by 3.34 points on Digital and 11.61 points on Real on average, relative to Clean. TeleOCR leads on Clean, GLM-5.3-Flash on Digital, and Gemini 3.6 Flash on Real. The benchmark dataset, source documents, generation tools, and evaluation code are publicly available at https://github.com/zhihengli-casia/PureDocBench.

Zhiheng Li, Zongyang Ma, Jiaxian Chen +12

CASIA · UCAS · Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information +6

Document Parsing Benchmarks Document Parsing Document Image Understanding

Ask PDF

May 21, 2026

MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing

Document parsing converts visually rich documents into machine-readable structured representations, forming a crucial foundation for information systems. Although many benchmarks have been proposed for document parsing, they remain inadequate for realistic scenarios. Existing benchmarks either focus on specific tasks or assess only single-page, text-centric settings, making them insufficient for practical multi-page parsing. Moreover, they lack fine-grained evaluation of semantic continuity, hierarchical structure recovery, and visual content preservation. To address these gaps, we propose MPDocBench-Parse, a benchmark for multi-page document parsing in real-world applications. It contains 433 manually annotated documents with 3,246 pages, covering 15 document types in English and Chinese, with diverse layout styles, and supports document-level end-to-end evaluation. We further design a comprehensive protocol for content fidelity and logical structure, covering text, table, and formula recognition, truncated text and table merging, figure extraction, reading order, and heading hierarchy recovery. Experiments show that, while existing models perform well on basic text extraction, they still suffer clear limitations in semantic continuity integration, visual content parsing, and hierarchical structure recovery. MPDocBench-Parse provides a unified foundation for advancing document parsing toward more realistic scenarios.

Bangbang Zhou, Hangdi Xing, Yifan Chen +8

University of Science and Technology of China · Tongyi Lab, Alibaba Group

Document Parsing Benchmarks Document Parsing Document Layout Analysis Document Image Understanding

Ask PDF

Abstract

Document Parsing Benchmarks

Explore similar work

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

How Far Is Document Parsing from Solved? PureDocBench: A Source-Traceable Benchmark across Clean, Degraded, and Real-World Settings

MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing