MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables
Organizations: DSA Thrust, HKUST(GZ), Guangzhou, China
Abstract
The ability to generate multimodal reports integrating fluent narratives with faithful visualizations from tabular data, has become a cornerstone capability for modern vision-language models in practical domains such as finance, healthcare, and policy making. However, existing benchmarks are limited by single-modality focus, simplistic metrics, novelty evaluation, and visualization quality. We address these limitations with MM-ReportBench, a multimodal report evaluation benchmark covering 386 tasks across 185 real-world tables. Each task is paired with an expert-curated reference report featuring standardized chapter structures and verifiable key insight points, constructed through a hybrid pipeline of multimodal reports generation based on Monte Carlo Tree Search and rigorous human quality control. For robust evaluation, we design a dual-judge evaluation framework: Text Judge Model assesses textual quality along structural completeness, grounded insight verification with numerical fact-checking against source tables, and novelty analysis that distinguishes genuine analytical discovery from trivial paraphrasing; Visualization Judge Model evaluates generated visualizations across data fidelity, expressiveness, aesthetics, and technical correctness. They also evaluate cross-modal consistency. Extensive experiments on 20 state-of-the-art models reveal critical weaknesses: even the strongest model achieves only 68.9 overall score, with numerical hallucination affecting 38.5% of reports and genuinely novel insights remaining rare. Our ablation studies further demonstrate that single-judge evaluation consistently inflates scores by 8-12 points compared to dual-judge assessment, confirming that rigorous and reliable benchmarking demands multi-faceted, complementary evaluation probes.
Figures & tables
| Property | Value |
|---|---|
| Number of tables | 185 |
| Number of domains | 6 |
| Number of tasks | 386 |
| Average tasks per table | 2.09 |
| Average tables per multi-table task | 4.2 |
| Average rows per table | 285 |
| Model | SC | GIV | NA | CMC | T-Score | DF | EX | AE | TC | V-Score | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Text-only LLMs | |||||||||||
| GPT-4o (text) | 72.3 | 48.5 | 28.1 | 0.0 | 37.2 | — | — | — | — | 0.0 | 22.3 |
| Claude-3.5-Sonnet | 70.8 | 45.2 | 25.6 | 0.0 | 35.4 | — | — | — | — | 0.0 | 21.2 |
| DeepSeek-V3 | 68.4 | 46.1 | 27.3 | 0.0 | 35.5 | — | — | — | — | 0.0 | 21.3 |
| Qwen3-32B | 67.2 | 43.8 | 24.5 | 0.0 | 33.9 | — | — | — | — | 0.0 | 20.3 |
| Llama-3.3-70B | 65.9 | 41.2 | 22.7 | 0.0 | 32.5 | — | — | — | — | 0.0 | 19.5 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Pearson |
|---|---|
| Structural Completeness (SC) | 0.88 |
| Grounded Insight Verification (GIV) | 0.94 |
| Novelty Analysis (NA) | 0.82 |
| Data Fidelity (DF) | 0.91 |
| Expressiveness (EX) | 0.85 |
| Aesthetics (AE) | 0.79 |
| Error Type | Frequency (%) | Metric(s) |
|---|---|---|
| Numerical hallucination | 38.5% | GIV , DF |
| Missing key insights | 25.0% | GIV |
| Structure/organization issues | 18.5% | SC |
| Chart-data mismatch | 22.0% | DF |
| Poor chart readability | 16.5% | EX, AE, TC |
| Trivial/paraphrased insights | 31.0% | NA |
Multimodal Generation
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
See all 258 papers