HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents

Lexin Wang; Shenghua Liu; Yiwei Wang; Yujun Cai; Yuyao Ge; Jiayu Yao; Jiafeng Guo; Xueqi Cheng

cs.CVMar 25, 2026

HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documents

Authors: Lexin Wang, Shenghua Liu, Yiwei Wang, Yujun Cai, Yuyao Ge, Jiayu Yao, Jiafeng Guo, Xueqi Cheng

Organizations: Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · University of Queensland

Abstract

Visual markups such as highlights, underlines, and bold text are common in table-centric documents. Although multimodal large language models (MLLMs) have made substantial progress in document understanding, their ability to treat such cues as explicit logical directives remains under-explored. More importantly, existing evaluations cannot distinguish whether a model fails to see the markup or fails to reason with it. This creates a key blind spot in assessing markup-conditioned behavior over tables. To address this gap, we introduce HighlightBench, a diagnostic benchmark for markup-driven table understanding that decomposes evaluation into five task families: Markup Grounding, Constrained Retrieval, Local Relations, Aggregation & Comparison, and Consistency & Missingness. We further provide a reference pipeline that makes intermediate decisions explicit, enabling reproducible baselines and finer-grained attribution of errors along the perception-to-execution chain. Experiments show that even strong models remain unstable when visual cues must be consistently aligned with symbolic reasoning under structured output constraints.

Figures & tables

Figure 1: Overview of HighlightBench . The benchmark contains five complementary task families with question counts. This design gives transparent coverage of markup-conditioned reasoning.

Figure 2: Pipeline overview. The reference pipeline converts the input image into a unified docgraph, uses two-stage routing to determine the solving path, and executes the routed result as a DSL plan on the docgraph to produce the final structured output and its intermediate trace.

Figure 3: Representative examples of the five task families in HighlightBench. Each card shows the core capability tested by a task family, together with a typical question format and its expected structured output.

Model	Subset	Accuracy (%)
Model	Subset	T1	T2	T3	T4	T5	All
Claude Sonnet 4	Synthetic	32.7	81.9	40.0	37.5	67.1	47.6
Claude Sonnet 4	Real-world	56.7	59.6	42.2	60.7	68.2	58.7
Gemini2.5-Flash	Synthetic	51.9	90.3	89.1	67.8	93.9	73.9
Gemini2.5-Flash	Real-world	81.3	75.3	62.7	72.2	78.5	75.3
InternVL3.5-8B	Synthetic	29.0	81.2	17.6	24.7	71.0	40.5

Table 1: Main results on HighlightBench. We report exact-match accuracy (%) for each task family and the overall score on two subsets: Synthetic and Real-world.

Task	Family	Brief example
Bold Presence	T1	Is a queried cell boldfaced?
Underline Presence	T1	Is a queried cell underlined?
Highlight Presence	T1	Is a queried cell highlighted?
Cell Retrieval	T2	Read the value at a specified row/column.
Extremum Reasoning	T4	Return the item/value with the column maximum.

Table 2: Representative sub-tasks used in the counterfactual analysis.

Figure 4: Extremum Reasoning accuracy under the counterfactual variants.

Panel of Figure 5 — Figure 5: Perception-task accuracy under counterfactual variants. Rows correspond to Bold Presence, Underline Presence, and Highlight Presence.

Panel of Figure 6 — Figure 6: Mechanism probes under counterfactual variants. (a) Marker-hit rate versus value-correct rate across variants. (b) Error attribution of failed cases based on observable outcomes.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Shared table image for the qualitative failure cases.

Figure 8: Per-subtask performance heatmap across models on HighlightBench. Rows denote models and columns denote the 21 subtasks. Color indicates average exact-match accuracy on the full benchmark.

Symbol	Meaning
$I$	Table image.
$M$	Visual markups in $I$ that are relevant to reasoning, including highlights, underlines, bold text, arrows, and color-based emphasis.
$Q$	Question associated with $I$ .
$A$	Target structured answer under the task-specific output schema.
$z$	Structural evidence indicated by $M$ , such as marked cells, headers, rows, columns, or local neighborhoods.
$c$	Reasoning condition induced from $(z,Q)$ , such as a filtered subset, a local relation, or a comparison scope.

Appendix

Table 3: Notation used in HighlightBench and the reference pipeline.

Family	Subtasks
$T_{1}$ Markup Grounding	Marked Value Listing, Marker Counting, Highlight Presence, Marked Column Listing, Underline Presence, Bold Presence
$T_{2}$ Constrained Retrieval	Value Filtering, Color-Anchor Retrieval, Cell Retrieval
$T_{3}$ Local Relations	Neighbor Value Tracing, Marker-Anchor Neighbor, Neighbor Indexing
$T_{4}$ Aggregation & Comparison	Extremum Reasoning, Argmax Reasoning, Grouped Aggregation, Pairwise Comparison, Best Selection, Rank Selection
$T_{5}$ Consistency & Missingness	Column Missingness, Cell Missingness, Consistency Probing

Appendix

Table 4: List of the 21 subtasks grouped by the five task families.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
Jun 8, 2026Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee +2Document Understanding Table QA
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)
LTEM: LLM-Friendly Table-Enhanced Markdown for Table Understanding and Generation
Oct 21, 2026Eunbin Kim, Hyeonseok Kang, Sangkeun JungDocument Understanding Structured Output Generation
Department of Computer Science and Engineering, Chungnam National University, Republic of Korea · EurekaAI
TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables
Apr 4, 2026Xiaoyu Chen, Lu Dai, Hanqing Wang +6Table Reasoning Spatial Reasoning Benchmarks
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1

Jun 8, 2026J/K move · Enter open · S save

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

Momina Ahsan, Sarfraz Ahmad, Ming Shan Hee, Roy Ka-Wei Lee+1

1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · 2Singapore University of Technology and Design (SUTD)

Document Understanding Table QA LLM Evaluation Table Structure Recognition Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation remains under-explored. In practice, the same table content may appear in different structural formats, such as HTML, Markdown, and LaTeX, or as rendered images. However, existing evaluations often let content, format, layout, and modality vary together, making it difficult to isolate representation effects. We introduce TABVERSE, a controlled multimodal table benchmark that aligns the same table content across multiple structural formats and rendered images, with question category and difficulty tags. This design enables systematic evaluation of representation effects while holding table content fixed. We evaluate LLMs and VLMs across three tasks: Question Answering (QA), Structural Understanding Capability (SUC), and Structure Reconstruction (SR). Our results show that representation choice substantially affects table understanding. Models generally perform better with structured text than with rendered images, but the size of this gap depends on the task, model, and format. HTML is often the most robust text format, while row-sensitive structural tasks and syntactically usable LaTeX reconstruction remain challenging. These findings show that table representation is a key factor in reliable table evaluation.

Abstract

Multimodal Model Evaluation

Explore similar work

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

LTEM: LLM-Friendly Table-Enhanced Markdown for Table Understanding and Generation

TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables