Document editing is among the most common knowledge-work tasks, yet existing benchmarks for evaluating autonomous agents on Word document editing provide fully specified attribute-modification instructions. They do not evaluate agents' ability to infer the appropriate attributes from document context, a capability essential in real-world editing where users expect consistency without spelling out every attribute. We introduce WordInferBench, a benchmark comprising 280 base document-editing instances, each paired with three instruction conditions: descriptive (the instruction explicitly lists every attribute change), referential (the instruction points to another object as an exemplar), and inferential (the agent must examine document context to infer what attributes to apply). To evaluate contextual attribute inference, each base instance is constructed so that the target formatting is recoverable from an in-document exemplar. The instances are derived from real Word documents collected from Common Crawl and cover 13 object types and 81 attributes. We evaluate six recent coding-agent systems and find that the best-performing evaluated system, Claude Code with Opus 4.6, achieves a 62.9% task-level pass rate on descriptive variants but only 40.7% on inferential variants, indicating that current systems complete tasks less reliably when the required formatting changes must be determined from document context.
Figures & tables
Figure 1: Overview of WordInferBench. Left : Tasks are constructed from real Word documents by identifying same-type objects with identical formatting, resetting the target’s attributes to defaults, and generating three instruction variants. Right : To solve tasks, agents explore context, locate the target, infer the required changes, and execute the edits.
Benchmark
Task Scenario
Word Tasks
Object Types
Attributes
Contextual Inference
OSWorld
Explicit formatting & content editing
47
4
12
✗
OfficeBench
Cross-app workflow
80
1
0
✗
DocFormEval
Explicit formatting
1,911
1
19
✗
WorldGUI
Explicit formatting & content editing
63
7
7
✗
Ours
Context-aware formatting modification
280
13
81
✓
Table 1: Comparison of Word-related task coverage across open-source computer-use and document-formatting benchmarks. Contextual Inference denotes tasks requiring agents to infer the required formatting from document context.
Figure 2: A complete test case from WordInferBench. The inferential instruction specifies only the target object; the agent must infer what attribute modifications are needed from document context. Left : the input document whose target object has been reset to defaults. Right : the held-out reference document with original formatting. The evaluator compares the 9 listed attributes between the agent’s output and the reference document.
Descriptive
Referential
Inferential
Average
Agent
Pass Rate
Attr Acc
Pass Rate
Attr Acc
Pass Rate
Attr Acc
Pass Rate
Attr Acc
Human Baseline
50.0%
85.1%
43.3%
75.3%
34.0%
70.1%
42.5%
76.9%
Claude Code (Opus 4.6)
62.9%
82.3%
77.1%
87.5%
40.7%
59.3%
60.2%
76.4%
Claude Code (Sonnet 4.6)
45.4%
78.9%
56.8%
71.1%
29.3%
48.2%
43.8%
66.1%
Codex (GPT-5.5)
55.4%
70.0%
43.6%
58.7%
25.7%
43.7%
41.5%
57.5%
Claude Code (GLM 5.2)
31.4%
51.8%
56.4%
67.8%
20.0%
35.9%
36.0%
51.8%
Table 2: Overall results across three instruction conditions. Pass Rate is the proportion of tasks in which all evaluated changed attributes are correct. Attr Acc is the fraction of target attributes requiring modification that are correctly updated in each task, macro-averaged across tasks. Both metrics evaluate only the target object; non-target preservation is reported separately.
Figure 3: Average task-level pass rates (%) by object type across all three instruction conditions.
Table 6: All 81 evaluated formatting attributes with per-attribute accuracy averaged across all six agents. N = number of tasks (out of 280) that evaluate each attribute.
Group
Subset 1
Subset 2
Subset 3
A
Referential
Descriptive
Inferential
B
Descriptive
Inferential
Referential
C
Inferential
Referential
Descriptive
Appendix
Table 7: Counterbalanced assignment of instruction conditions to annotator groups and base-instance subsets.
Condition
Mean
Median
Descriptive
424
301
Referential
362
230
Inferential
324
230
Overall
370
253
Appendix
Table 8: Human editing time in seconds by instruction condition. No per-task time limit is enforced.
Human
Agent
Rank
Attribute
Acc.
N
Attribute
Acc.
N
1
Column width
20.5
99
Column width
36.0
174
2
Object style
22.9
297
Cell spacing
36.1
61
3
Cell margin
28.8
171
Series error bars
38.7
31
4
Object alignment
37.5
72
Data-table legend key
46.4
112
5
Hyperlink
40.6
429
Chart data table
46.4
112
Appendix
Table 9: Lowest macro-averaged attribute accuracies (%) for human and agent outputs. N is the number of scoreable output documents in which the attribute is evaluated.
Figure 4: Example from the visual attribute detection experiment. GPT-5.5 correctly identifies visually salient changes such as fill color and bold weight, but misses subtle attributes like line spacing, spacing after, and shadow parameters whose changes produce barely perceptible visual differences.
Agent
Desc.
Ref.
Infer.
Avg.
Claude Code (Opus 4.6)
92.1
90.0
89.6
90.6
Claude Code (Sonnet 4.6)
92.1
88.6
86.4
89.0
Codex (GPT-5.5)
65.4
52.9
70.0
62.7
Claude Code (GLM 5.2)
91.8
86.4
92.5
90.2
Kimi CLI (Kimi K2.6)
84.6
85.4
85.4
85.1
Codex (GPT-5.4)
54.6
51.8
60.7
55.7
Appendix
Table 10: Preservation pass rates (%) across the three instruction conditions, using all 280 tasks per condition as the denominator. The average is computed over all 840 tasks across the three conditions.
Microsoft Word is a central environment for office automation, and the faithful execution of its editing operations is an important capability for document agents. Unlike plain-text editing, editing in Word modifies a structured object model, where the requested effect is realized in object properties. Existing benchmarks, however, evaluate office workflows or desktop automation, treating word-processor documents as one task domain among many and verifying outcomes with keyword matching, per-task property checks, or screenshot comparison. We introduce WordEval, an operation-fidelity benchmark that instead centers evaluation on Word editing operations, testing whether each requested edit is realized as intended changes to Word objects and properties. It contains 642 cases across 15 Word feature areas, ranging from text and lists to charts, shape geometry, and SmartArt. 546 atomic cases cover 231 commands, 96 compositional cases combine multiple edits in a single document, and a Word-native oracle checks task-specific object properties and preservation conditions. Cases are constructed from editing-unit specifications, yielding controlled input documents, reference documents, and aligned oracle checks. Each compositional case is linked to corresponding atomic cases, supporting fine-grained diagnosis of editing failures. We evaluate eight agent systems and find that many struggle on the Atomic track, with performance varying across feature areas. Moreover, every system scores lower on Compositional than on Atomic, with gaps that vary substantially by system. Exact atomic-intent links and recorded failure diagnoses provide complementary views of this variation.
We introduce LEDGER to tackle the novel context engineering challenge of agentic document editing, where localized edits to long, structured documents must be applied efficiently without breaking cross-references or semantic consistency. LEDGER constructs a lightweight dependency graph that explicitly models document structure, including hierarchical organization, explicit references, implicit dependencies, and semantic relationships. For each edit, graph-guided retrieval selects only the necessary context, avoiding full-document processing while preserving consistency. We evaluate LEDGER on a curated benchmark of 1.9k test cases with various document types and lengths, spanning six state-of-the-art models: LEDGER improves consistency from 56% to 76% across all six models and test scenarios while reducing token usage. Notably, LEDGER with low reasoning effort matches baseline performance at high reasoning effort using fewer tokens, showing that explicit dependency representations can partially substitute for expensive internal reasoning in agentic document editing.
Large language model agents have made strong progress on software engineering, yet current systems suffer from a context coupling problem: the standard code editing interface conflates code inspection, modification planning, and edit execution within a single context window, forcing agents to interleave exploratory viewing with strictly formatted edit generation. Irrelevant context accumulates and edit reliability degrades. We propose SWE-Edit, which decomposes the editing interface into two specialized subagents: a Viewer that extracts task-relevant code on demand, and an Editor that executes modifications from high-level natural language plans -- letting the main agent focus on reasoning while delegating context-intensive operations to clean context windows. On SWE-Bench Verified, this decomposition raises resolve rate by 2.1 pp and cuts inference cost by 17.9%, with consistent gains across multiple reasoning-model families (Kimi-K2, MiniMax-M2.1, GLM-4.7). We further show that effective edit-format selection can be trained into a small model rather than requiring frontier-scale capacity: GRPO training on Qwen3-8B with an adaptive find-replace/whole-file-rewrite policy improves edit success by 12.5 pp and brings an 8B open-source editor to parity with GPT-5-nano on downstream SWE-Bench resolve rate. To enable rapid editor iteration, we release PR-Edit, a lightweight evaluation whose scores correlate strongly with SWE-Bench resolve rate. We release our code at https://github.com/microsoft/SWE-Edit.
Yikai Zhang, Jiaxin Pei, Kenan Li +9
1Microsoft, Redmond, Washington, United States · 3Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University · Department of Computer Sciences, University of Wisconsin–Madison