Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
Organizations: University of Chinese Academy of Sciences · Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences
Abstract
Large language models (LLMs) are increasingly expected to extract and synthesize structured information from complex real-world documents, yet they typically produce unstructured, hard-to-verify paragraph-style outputs. We introduce AOE (Arranged and Organized Extraction), a bilingual benchmark that evaluates LLMs' ability to distill long-form, multi-document inputs (64k to 1.8M tokens) into structured tables. AOE comprises 12 tasks across four domains (academic, financial, legal, and medical), each grounded in genuine analytical needs, with schema guidance provided to isolate extraction quality from formatting variance. We evaluate 12 frontier LLMs (both open-weight and proprietary) under four generation settings of increasing autonomy: direct full-document input, RAG-based retrieval, offline document-browsing agent, and online web-search agent. Even the best-performing model achieves only 42.7% Cell F1, and retrieval-augmented approaches often degrade rather than improve performance, revealing that structured extraction from long contexts remains a fundamental challenge - an "Area of Effect" (AOE) that impacts all tested systems.
Figures & tables
| Metric | Academic | Financial | Legal | Medical |
| Basic Info | ||||
| Language | EN | ZH, EN | ZH | EN |
| Tables (#) | 74 | 224 | 75 | 74 |
| Document-Level | ||||
| Documents (#) | 257 | 944 | 625 | 340 |
| Avg Tokens | 72k | 435k | 8k | 23k |
| Example: Constructing a Medical Instance |
| 1. Review paper discovered via PubMed: Li et al. (2024b) contains Table 1: summary of cases of NSCLC-to-SCLC transformation (2017–2023). |
| 2. Ground-truth table extracted (11 columns): |
| source_doc, Age, Sex, Country, Smoking_status, |
| Mutational_status, Medication_before, Medication_after, |
| CNS_metastasis, OS_after_transformation, OS |
| Each row cites one independent case report; values curated by the review authors. |
| Model | Ctx. Limit | Pass Rate | Cell F1 by Domain | Overall Evaluation | |||||||
| Acad. | Legal | Fin-S | Fin-L | Med. | LLM | Col. | Row | Cell | |||
| Open-Weight Models | |||||||||||
| DeepSeek-V3.2 | 131k | 92.2 | 36.9 | 45.5 | 57.1 | 30.1 | 21.6 | 74.1 | 73.7 | 76.0 | 38.2 |
| Qwen-3.6-Plus | 131k | 94.4 | 52.2 | 48.2 | 48.8 | 22.7 | 26.0 | 75.5 | 74.3 | 78.2 | 39.6 |
| Kimi-K2.6 | 131k | 96.0 | 34.5 | 43.3 | 63.4 | 35.3 | 23.4 | 73.8 | 82.8 | 75.2 | 40.0 |
| GLM-5.1 | 200k | 96.2 | 42.1 | 44.4 | 52.9 | 33.4 | 28.9 | 74.9 | 78.0 | 78.2 | 40.3 |
| Model | Setting | CF1 | ColR | ColP | RowR | CSV |
| Claude- Sonnet-4.6 | Prompt-All | 19.6 | 71.5 | 63.9 | 92.6 | 64.9 |
| Schema-Guided | 28.4 | 93.1 | 83.9 | 92.1 | 95.9 | |
| +8.8 | +21.6 | +20.0 | 0.5 | +31.0 | ||
| DeepSeek- V3.2 | Prompt-All | 20.3 | 66.9 | 59.2 | 95.4 | 52.7 |
| Schema-Guided | 23.4 | 89.1 | 80.1 | 89.8 | 90.5 | |
| +3.1 | +22.2 | +20.9 | 5.6 | +37.8 |
| Table Construction | Cell Correctness | ||||||
| Model | Column | Row | Coverage | Num. | Calc. | Reason. | Text |
| DeepSeek-V3.2 | 67.6 | 89.8 | 50.6 | 90.0 | 43.9 | 41.6 | 25.7 |
| Kimi-K2.6 | 75.8 | 80.5 | 54.0 | 85.0 | 46.8 | 44.4 | 26.6 |
| Claude-Sonnet-4.6 | 77.0 | 84.1 | 55.8 | 86.2 | 45.7 | 27.1 | 30.2 |
| Gemini-3-Flash | 69.8 | 86.9 | 55.7 | 87.5 | 58.5 | 31.7 | 19.9 |
| GPT-5.4 | 73.8 | 88.6 | 54.0 | 91.2 | 45.1 | 36.2 | 19.2 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Context Window | Cost / Task |
| Claude-Opus-4.6 | 200k | $190.8 |
| GPT-5.4 | 128k | $74.4 |
| Gemini-3.1-Pro | 131k | $60.3 |
| DeepSeek-V4-Pro | 131k | $13.2 |
| Qwen-3.6-Plus | 131k | $11.8 |
| Domain | GT Cells | 32k | 128k | 256k |
| Academic | 1,037 | 15.5% | 1.7% | 0.0% |
| Financial | 6,031 | 24.4% | 19.0% | 9.9% |
| Legal | 4,075 | 0.0% | 0.0% | 0.0% |
| Medical | 3,283 | 0.3% | 0.0% | 0.0% |
| Overall | 14,426 | 11.4% | 8.1% | 5.6% |
| paper_name | method | result | settings |
| Paper A | GAN | 21.5 | Standard settings (Sec. 3) |
| Paper B | StyleGAN | 4.40 | Style mixing, 1024 2 , no trunc. |
| Paper C | ADM | 2.97 | Classifier guide, 250 DDPM steps |
| Model | Setting | Academic | Legal | Financial | Avg. Cell F1 | ||||||
| Col | Row | Cell | Col | Row | Cell | Col | Row | Cell | |||
| GPT-5.4 | Direct | 71.2 | 63.8 | 48.2 | 72.0 | 71.0 | 40.1 | 62.9 | 91.2 | 46.3 | 44.9 |
| RAG | 62.2 | 38.8 | 32.7 | 50.4 | 44.6 | 19.6 | 57.9 | 92.0 | 44.5 | 32.3 | |
| Claude Opus 4.6 | Direct | 71.2 | 57.1 | 36.3 | 70.5 | 73.1 | 39.0 | 60.9 | 90.3 | 46.7 | 40.7 |
| RAG | 67.9 | 37.1 | 23.1 | 51.8 | 60.6 | 28.6 | 57.0 | 95.7 | 45.0 | 32.2 | |
| Gemini 3.1 Pro | Direct | 71.0 | 60.4 | 45.7 | 69.8 | 76.3 | 49.7 | 51.7 | 89.6 | 41.6 | 45.7 |
| Model | Length | Cell F1 | Col F1 | Row F1 |
| Claude-Opus-4.6 | Short | 59.5 | 62.0 | 98.5 |
| Long | 41.8 | 51.7 | 92.9 | |
| DeepSeek-V3.2 | Short | 54.9 | 57.4 | 98.5 |
| Long | 37.8 | 48.9 | 88.0 | |
| Gemini-3.1-Pro | Short | 55.0 | 53.2 | 100.0 |
| Long | 43.8 | 46.6 | 100.0 |
| Query: Assess “shareholder value creation analysis” | |
| Required Evidence | Context Retrieved by RAG |
| Earnings per share (basic): 12.50; Total equity... | "Our board of directors remains fully committed to creating long-term shareholder value through strategic investments..." |
| Model | Domain | Pass Rate | Col Recall | Row F1 | Cell F1 |
| Claude-Opus-4.6 | Financial | 90.5 6.0 | 69.3 1.7 | 83.6 4.0 | 45.9 1.2 |
| Legal | 93.2 4.1 | 96.8 1.1 | 72.4 1.4 | 37.7 1.8 | |
| Academic | 100.0 0.0 | 100.0 0.0 | 57.0 0.8 | 39.1 1.7 | |
| DeepSeek-V3.2 | Financial | 90.0 3.2 | 64.6 1.8 | 85.2 2.0 | 42.4 1.0 |
| Legal | 93.8 2.8 | 91.8 2.2 | 73.4 2.3 | 43.6 2.3 | |
| Academic | 80.2 3.3 | 100.0 0.0 | 45.9 1.9 | 37.8 1.2 |
| Model | R0 | R1 | R2 | R3 | R4 | Range |
| Qwen-3.6-Plus | 1 | 3 | 3 | 2 | 2 | 1–3 |
| Kimi-K2.5 | 3 | 1 | 1 | 4 | 1 | 1–4 |
| GPT-5.4 | 2 | 2 | 2 | 1 | 3 | 1–3 |
| DeepSeek-V3.2 | 4 | 5 | 4 | 3 | 5 | 3–5 |
| Claude-Opus-4.6 | 5 | 4 | 5 | 5 | 4 | 4–5 |
| Value Type | Mean Cell F1 | SD |
| Numeric | 49.9 | 1.9 |
| Calculation | 41.8 | 2.1 |
| Reasoning | 29.6 | 3.4 |
| Text | 28.8 | 1.6 |
| Gold Standard | Raw Prediction (Cell F1 = 0.0) | Schema-Guided (Cell F1 = 0.64) | |
| Instance: fimmu-17-1806045 — Takayasu Arteritis with IBD (5 rows, 5 target columns) | |||
| Schema | No_references, Predominant_phenotype, Number_of_cases, Treatment, Outcome | LVV_type , IBD_type , Age , Sex , Diagnostic_order , Key_presentation , Treatment, Outcome | No_references, Predominant_phenotype, Number_of_cases, Treatment, Outcome |
| Col Recall | — | 40% (2/5 gold cols) | 100% (5/5 gold cols) |
| Row 1 | 3 (56), TAK+CD, 2, anti-TNF , Improved | TAK, CD, 16, M, Simultaneous, Abdominal pain, Prednisone, Asymptomatic | 3 (56), TAK+CD, 2, anti-TNF , Improved |
| Failure mode | — | Adds 6 unrequested columns; disaggregates summary rows into per-patient entries | Correct schema; minor value differences |
| System | Acad. | Legal | Med. | Fin. |
| Human | 42.4 | 85.8 | 34.0 | 92.9 |
| GPT-5.4 | 48.2 | 40.1 | 14.6 | 46.0 |
| DeepSeek-V4-Pro | 58.6 | 50.4 | 16.5 | 38.7 |
| Claude-Opus-4.6 | 36.3 | 39.0 | 21.5 | 46.6 |
| Gemini-3.1-Pro | 45.4 | 47.9 | 12.9 | 40.5 |
| Structure & Coverage | Cell F1 by Content Type | ||||||
| Model | Schema | Row | Coverage | Numeric | Calculation | Reasoning | Text |
| Open-Weight Models | |||||||
| DeepSeek-V3.2 | 67.6 | 89.8 | 50.6 | 90.0 | 43.9 | 41.6 | 25.7 |
| Qwen-3.6-Plus | 63.3 | 80.2 | 44.9 | 93.8 | 34.4 | 45.1 | 21.9 |
| Kimi-K2.6 | 75.8 | 80.5 | 54.0 | 85.0 | 46.8 | 44.4 | 26.6 |
| GLM-5.1 | 64.5 | 86.1 | 49.0 | 96.2 | 41.9 | 32.3 | 19.4 |
| Case | Field | Gold | Prediction (Claude-Opus-4.6) |
| Row 1 | Location | Middle fossa | Posterior clinoid |
| Age | 41 | 34 | |
| Sex | M | M | |
| Symptoms | Visual disturbance, diplopia | Visual disturbances, headache | |
| Treatment | PSR | PSR | |
| Improvement | No | Yes |
| Task ID | Task Objective | Input Documents | Primary Keys | Output Columns |
|---|---|---|---|---|
| Identify potential citation relationships between scholarly articles and extract the specific text segment that is being referenced. | Multiple research papers. | ["Cited paper title", "Referencing paper title"] | ["Referenced content", "Label"] | |
| Enumerate the reported {{Metrics}} of proposed methodologies within a research paper, specifically on the {{Dataset_Name}} dataset. | Multiple research papers. | ["paper_name", "method", "result"] | ["result", "models_and_settings"] | |
| Conduct an analysis of enterprise scale and growth trends. | Annual company reports from 2020-2023. | ["Filename"] | ["Reporting Period", "Total Assets (CNY)", "Asset Growth Rate (%)", "Revenue (CNY)", "Revenue Growth Rate (%)", "Total stockholders’ equity (CNY)", "Weighted Average Return on Equity (%)"] | |
| Perform a comprehensive assessment of profitability. | Annual company reports from 2020-2023. | ["Filename"] | ["Reporting Period", "Revenue (CNY)", "Net Profit (CNY)", "Net Profit Margin (%)", "Weighted Average Return on Equity (%)", "Net Profit Growth Rate (%)", "Net cash provided by operating activities (CNY)", "Net Cash Ratio (%)"] | |
| Execute a diagnostic analysis of financial risks. | Annual company reports from 2020-2023. | ["Filename"] | ["Reporting Period", "Total Liabilities (CNY)", "Liability-to-Asset Ratio (%)", "Operating Cash Flow Growth Rate (%)", "Total Assets (CNY)"] | |
| Analyze the creation of shareholder value. | Annual company reports from 2020-2023. | ["Filename"] | ["Reporting Period", "Share Capital (Shares)", "Total stockholders’ equity (CNY)", "Net Profit (CNY)", "Net Asset per Share (CNY/share)", "Net Profit per Share (CNY/share)", "Weighted Average Return on Equity (%)"] |
Document Information Extraction
Document Information Extraction turns text, tables, forms, and images into structured, searchable data.
See all 303 papers