WordInferBench: Can Agents Infer How to Edit from Word Document Context?
Organizations: Xi’an Jiaotong University · Microsoft · KAIST
Abstract
Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consistency without spelling out every attribute. We introduce WordInferBench, a benchmark comprising 280 base document-editing instances, each paired with three instruction conditions: descriptive (the instruction explicitly lists every attribute change), referential (the instruction points to another object as an exemplar), and inferential (the agent must examine document context to infer what attributes to apply). To evaluate contextual attribute inference, each base instance is constructed so that the target formatting is recoverable from an in-document exemplar. The instances are derived from real Word documents collected from Common Crawl and cover 13 object types and 81 attributes. We evaluate six recent coding-agent systems and find that the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.
Figures & tables
| Benchmark | Task Scenario | Word Tasks | Object Types | Attributes | Contextual Inference |
|---|---|---|---|---|---|
| OSWorld | Explicit formatting & content editing | 47 | 4 | 12 | ✗ |
| OfficeBench | Cross-app workflow | 80 | 1 | 0 | ✗ |
| DocFormEval | Explicit formatting | 1,911 | 1 | 19 | ✗ |
| WorldGUI | Explicit formatting & content editing | 63 | 7 | 7 | ✗ |
| Ours | Context-aware formatting modification | 280 | 13 | 81 | ✓ |
| Descriptive | Referential | Inferential | Average | |||||
|---|---|---|---|---|---|---|---|---|
| Agent | Pass Rate | Attr Acc | Pass Rate | Attr Acc | Pass Rate | Attr Acc | Pass Rate | Attr Acc |
| Human Baseline | 50.0% | 85.1% | 43.3% | 75.3% | 34.0% | 70.1% | 42.5% | 76.9% |
| Claude Code (Opus 4.6) | 62.9% | 82.3% | 77.1% | 87.5% | 40.7% | 59.3% | 60.2% | 76.4% |
| Claude Code (Sonnet 4.6) | 45.4% | 78.9% | 56.8% | 71.1% | 29.3% | 48.2% | 43.8% | 66.1% |
| Codex (GPT-5.5) | 55.4% | 70.0% | 43.6% | 58.7% | 25.7% | 43.7% | 41.5% | 57.5% |
| Claude Code (GLM 5.2) | 31.4% | 51.8% | 56.4% | 67.8% | 20.0% | 35.9% | 36.0% | 51.8% |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent System | Underlying Model | CLI Tool & Version |
|---|---|---|
| Claude Code | Claude Opus 4.6 | [email protected] |
| Claude Code | Claude Sonnet 4.6 | [email protected] |
| Codex | GPT-5.5 | [email protected] |
| Codex | GPT-5.4 | [email protected] |
| Claude Code | GLM 5.2 | [email protected] |
| Kimi CLI | Kimi K2.6 | [email protected] |
| Component | Specification |
|---|---|
| OS | Windows 11 Enterprise 25H2 |
| VM Resources | 8 GB RAM, 2 cores, 32 GB disk |
| Per-task Timeout | 600 seconds |
| Runs per Variant | 1; provider-default decoding settings |
| Office | Microsoft Office LTSC 2024 |
| Language / Locale | English (default Windows 11 install) |
| Object Type | Tasks |
|---|---|
| Plain paragraph | 33 |
| Listing | 31 |
| Table | 28 |
| Table of contents | 27 |
| Hyperlink | 23 |
| SmartArt | 23 |
| Attribute | Acc% | Attribute | Acc% | ||
|---|---|---|---|---|---|
| font_formatting.bold | 108 | 65.5 | contrast | 2 | 55.6 |
| font_formatting.color | 96 | 53.8 | data_table_show_legend_key | 7 | 41.3 |
| font_formatting.font | 148 | 59.6 | fill_back_color_rgb | 10 | 63.9 |
| font_formatting.font_size | 155 | 64.1 | fill_fore_color_rgb | 40 | 61.9 |
| font_formatting.italic | 49 | 64.4 | fill_gradient_angle | 7 | 71.4 |
| font_formatting.strike | 1 | 50.0 | fill_transparency | 15 | 58.5 |
| Group | Subset 1 | Subset 2 | Subset 3 |
|---|---|---|---|
| A | Referential | Descriptive | Inferential |
| B | Descriptive | Inferential | Referential |
| C | Inferential | Referential | Descriptive |
| Condition | Mean | Median |
|---|---|---|
| Descriptive | 424 | 301 |
| Referential | 362 | 230 |
| Inferential | 324 | 230 |
| Overall | 370 | 253 |
| Human | Agent | |||||
|---|---|---|---|---|---|---|
| Rank | Attribute | Acc. | Attribute | Acc. | ||
| 1 | Column width | 20.5 | 99 | Column width | 36.0 | 174 |
| 2 | Object style | 22.9 | 297 | Cell spacing | 36.1 | 61 |
| 3 | Cell margin | 28.8 | 171 | Series error bars | 38.7 | 31 |
| 4 | Object alignment | 37.5 | 72 | Data-table legend key | 46.4 | 112 |
| 5 | Hyperlink | 40.6 | 429 | Chart data table | 46.4 | 112 |
| Agent | Desc. | Ref. | Infer. | Avg. |
|---|---|---|---|---|
| Claude Code (Opus 4.6) | 92.1 | 90.0 | 89.6 | 90.6 |
| Claude Code (Sonnet 4.6) | 92.1 | 88.6 | 86.4 | 89.0 |
| Codex (GPT-5.5) | 65.4 | 52.9 | 70.0 | 62.7 |
| Claude Code (GLM 5.2) | 91.8 | 86.4 | 92.5 | 90.2 |
| Kimi CLI (Kimi K2.6) | 84.6 | 85.4 | 85.4 | 85.1 |
| Codex (GPT-5.4) | 54.6 | 51.8 | 60.7 | 55.7 |
Agent Evaluation
Agent evaluation measures how reliably AI systems act, use tools, and complete tasks across interactive environments.
See all 160 papers