Frame: Form-Restricted Authoring with Model-Based Evaluation
Organizations: Microsoft Office AI
Abstract
Many professional reports reuse the structure of prior editions while updating the content. We formalize this setting as form-restricted report generation: given a sample report and new grounding documents, generate a report that preserves the sample's form while accurately reflecting the new evidence. We introduce FRAME, a benchmark of 120 EDGAR- and arXiv-derived instances across Financial Quarterly, Financial Annual, Academic Simple, and Academic Complex categories, built with a temporal three-snapshot pipeline. We evaluate outputs with three human-validated metrics: structural similarity via semantic tree-edit distance, coverage via information unit recall, and faithfulness via claim-level grounding verification. Across five LLMs, open-source models nearly match proprietary models on structure but lag on coverage and faithfulness; all models perform better on financial than academic reports, and performance declines with longer outputs and more complex grounding.
Figures & tables
| Grounding Documents | Sample Report | Gold Report | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Category | Instances | Files | Tokens | Tables | Images | Tokens | Sec. | Tables | Tokens | Sec. | Tables |
| Financial Qtr. | 30 | 11.2 | 24.2k | 39.6 | — | 9.1k | 12.5 | 12.2 | 8.8k | 12.5 | 12.2 |
| Financial Ann. | 30 | 20.8 | 59.2k | 70.8 | — | 22.1k | 18.9 | 33.7 | 22.4k | 19.1 | 32.6 |
| Academic Simp. | 30 | 3.4 | 28.0k | 12.9 | 13.7 | 21.6k | 19.1 | 8.2 | 22.6k | 18.8 | 8.6 |
| Academic Comp. | 30 | 6.6 | 57.1k | 28.8 | 30.4 | 19.3k | 21.8 | 7.1 | 20.8k | 22.0 | 7.4 |
| Evaluation | ||||
|---|---|---|---|---|
| Coverage | 75.0 | 93.2 | 0.919 | 0.926 |
| Faithfulness | 85.2 | 97.4 | 0.663 | 0.664 |
| Metric | Data | GPT-5 | Claude-4.5-Sonnet | Qwen3-VL-A3B | Phi-4-multi | Kimi-VL-A3B |
|---|---|---|---|---|---|---|
| Structural | Overall | 0.298 | 0.314 | 0.315 | 0.253 | 0.204 |
| Financial Ann. | 0.331 | 0.392 | 0.390 | 0.241 | 0.177 | |
| Financial Qtr. | 0.414 | 0.406 | 0.413 | 0.361 | 0.317 | |
| Academic Comp. | 0.201 | 0.206 | 0.205 | 0.190 | 0.165 | |
| Academic Simp. | 0.248 | 0.251 | 0.250 | 0.222 | 0.158 | |
| Coverage | Overall | 0.779 | 0.770 | 0.548 | 0.258 | 0.298 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Evidence-Grounded Generation
Evidence-grounded generation produces answers and documents whose claims are supported by traceable sources.
See all 156 papers