WordEval: Evaluating Word-Native Operation Fidelity in Document Editing
Organizations: 1KAIST · 2Microsoft · 3XJTU
Abstract
Microsoft Word is a central environment for office automation, and the faithful execution of its editing operations is an important capability for document agents. Unlike plain-text editing, editing in Word modifies a structured object model, where the requested effect is realized in object properties. Existing benchmarks, however, evaluate office workflows or desktop automation, treating word-processor documents as one task domain among many and verifying outcomes with keyword matching, per-task property checks, or screenshot comparison. We introduce WordEval, an operation-fidelity benchmark that instead centers evaluation on Word editing operations, testing whether each requested edit is realized as intended changes to Word objects and properties. It contains 642 cases across 15 Word feature areas, ranging from text and lists to charts, shape geometry, and SmartArt. 546 atomic cases cover 231 commands, 96 compositional cases combine multiple edits in a single document, and a Word-native oracle checks task-specific object properties and preservation conditions. Cases are constructed from editing-unit specifications, yielding controlled input documents, reference documents, and aligned oracle checks. Each compositional case is linked to corresponding atomic cases, supporting fine-grained diagnosis of editing failures. We evaluate eight agent systems and find that many struggle on the Atomic track, with performance varying across feature areas. Moreover, every system scores lower on Compositional than on Atomic, with gaps that vary substantially by system. Exact atomic-intent links and recorded failure diagnoses provide complementary views of this variation.
Figures & tables
| Benchmark | Interface | Word tasks | Edit operations | Verification oracle |
| OfficeBench [ 24 ] | python-docx API | 80 ∗ | 4 file actions | keyword, file existence |
| OdysseyBench [ 22 ] | python-docx API | 235 ∗ | 4 file actions [ 24 ] | keyword, file existence |
| OSWorld [ 25 ] | LibreOffice GUI | 23 | 23 property edits | text match, property checks |
| WindowsAgentArena [ 3 ] | LibreOffice GUI | 19 | 18 property edits [ 25 ] | text match, property checks |
| WorldGUI [ 27 ] | Word GUI | 63 † | 11 meta-tasks | screenshot task success |
| WordEval (ours) | Unrestricted | 642 | 231 commands | Word object properties, preservation |
| Category | Cases | Coverage | Definition |
| WordEval | 642 | 231 commands; 27 scenarios | Benchmark across 15 Word feature areas with task-specific object-state checks. |
| Atomic | 546 | 231 commands | Single editing unit over one Word command. |
| Feature areas | 546 | 15 feature areas ( Text , Paragraphs , Lists , Tables , Shape geometry , Shape appearance , Shape text , Pictures , Graphics , Charts , SmartArt , Equations , Page Structure , Annotations , References ) | Object-model coverage axis; feature areas group runtime objects, which expose checked object-property pairs. |
| Compositional | 96 | 27 scenarios | Multiple editing units in the same controlled document substrate. |
| Additive | 46 | 11 scenarios: font, paragraph, list, document enrichment/cleanup, chart, layout, picture, shape, SmartArt, table | Unit preconditions are jointly satisfied and edits do not invalidate one another. |
| Interaction-sensitive | 50 | 16 scenarios | Earlier edits affect later preconditions, target scope, or layout state. |
| Agent system | Atomic | Compositional | Overall | |||
| Case | Macro | Case | Macro | Case | Macro | |
| Claude Code (Opus 5) | 87.5 | 89.2 | 84.4 | 85.5 | 87.1 | 88.8 |
| Codex (GPT-5.6, high) | 67.6 | 68.2 | 61.5 | 60.3 | 66.7 | 67.3 |
| Kimi Code (K3, high) | 63.4 | 68.1 | 53.1 | 55.8 | 61.8 | 66.8 |
| OpenCode (Gemini 3.8 Flash, high) | 57.9 | 63.2 | 49.0 | 51.3 | 56.5 | 61.9 |
| ZCode (GLM-5.2, high) | 58.2 | 63.7 | 39.6 | 40.4 | 55.5 | 61.2 |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Count | Reduction |
| Ribbon control entries | 5,638 | |
| Normalized commands | 2,569 | drop entries without a command identifier; deduplicate repeated placements |
| Verifiable state-changing commands | 1,150 | exclude view, settings, clipboard, external, UI-structural, and selection commands ( ; categories below) |
| In-scope commands | 992 | remove excluded-by-design commands (OLE, forms, mail merge, macro/VBA, master documents, AutoCorrect/AutoFormat, proofing) |
| Released atomic commands | 231 | selected set spanning 15 feature areas; remaining in-scope commands define an extension space |
| Feature area | Runtime object | Object properties |
|---|---|---|
| Text | Border | Color, LineStyle, LineWidth |
| ColorFormat | RGB | |
| FillFormat | GradientAngle, Type | |
| Font | Bold, Color, Italic, Ligatures, Name, Size, StrikeThrough, Subscript, Superscript, Underline, UnderlineColor | |
| LineFormat | DashStyle, Weight | |
| Range | HighlightColorIndex, Text |
| Verification schema | Target postcondition | Preservation invariants |
| Run / character formatting | Range.Font.* (weight, style, size, color, highlight, text effects) on the target range | Non-target (“control”) text content and formatting; paragraph count |
| Paragraph formatting | ParagraphFormat.* (alignment, indents, line spacing, space before/after, borders) | Paragraph text; non-target paragraph properties; paragraph count |
| Lists | ListFormat (list string, list level) on the target paragraphs | Paragraph text; non-target list state; paragraph count |
| Shape / picture geometry & effects | Shape.{Left, Top, Width, Height, Rotation, ZOrderPosition} , wrap type, fill / line / shadow / glow / reflection | Document structure counts (shape, inline-shape, footnote, section); non-target shapes |
| Tables | Row and column counts, cell content, borders, style | Table dimensions; non-target cells |
| Fields & references (footnote, endnote, TOC, citation) | Field count, resolved content, reference position | Body text when the field is the sole mutation |
| Evaluator | Precision (%) | FPR (%) | FNR (%) |
| Word Object | 96.5 | 5.5 | 16.8 |
| Word Revision | 97.1 | 2.2 | 59.9 |
| SSIM | 77.3 | 45.1 | 16.2 |
| SSIM | 94.5 | 7.7 | 27.5 |
| agent judge | 93.0 | 12.1 | 12.0 |
| Agent system | Package | Version | Model identifier |
| Claude Code | @anthropic-ai/claude-code | – | Opus 5 |
| Codex | @openai/codex | 0.128.0 | gpt-5.6-sol |
| Kimi Code | kimi-cli | 1.41.0 | Kimi-K3 |
| OpenCode | opencode | 1.18.3 | gemini-3.8-flash |
| ZCode | zcode-agent-cli | 0.15.2 | GLM-5.2 |
| OpenCode | opencode | – | DeepSeek-V4-Pro |
| Participant | Atomic | Compositional | Overall | |||
| Case | Macro | Case | Macro | Case | Macro | |
| Human A01 | 83.0 | 85.4 | 71.9 | 69.6 | 81.3 | 83.7 |
| Human A02 | 90.1 | 90.9 | 78.1 | 76.8 | 88.3 | 89.4 |
| Human A03 | 82.4 | 83.9 | 58.3 | 58.8 | 78.8 | 81.3 |
| Class | Transfer | Transfer fail | Operation-limited | Coincidence | ||
| Additive | 79.7 | 91.2 | 8.8 | 9.7 | 10.7 | 1170 |
| Prerequisite | 70.8 | 93.5 | 6.5 | 8.2 | 21.0 | 219 |
| Scope-changing | 91.9 | 95.2 | 4.8 | 1.5 | 6.6 | 136 |
| Layout-coupled | 90.9 | 71.9 | 28.1 | 1.7 | 7.4 | 176 |
| Case level | Intent level | |||||||||
| System | Class | Pass gap | ||||||||
| Opus 5 | Additive | 80.4 | 82.6 | 89.2 | 37 | 97.4 | 93.8 | 3.6 | 93.7 | 190 |
| Prerequisite | 80.0 | 80.0 | 85.0 | 20 | 95.5 | 95.5 | 0.0 | 97.6 | 42 | |
| Scope-changing | 58.3 | 100.0 | 100.0 | 7 | 95.8 | 100.0 | -4.2 | 100.0 | 23 | |
| Layout-coupled | 100.0 | 84.6 | 84.6 | 13 | 100.0 | 93.8 | 6.2 | 93.8 | 32 | |
| GPT-5.6 | Additive | 37.0 | 56.5 | 58.8 | 17 | 73.2 | 90.9 | -17.7 | 91.7 | 120 |
| Class | closed (%) | open (%) | (pp) | 95% CI | clusters | ||
| Additive | 94.3 | 88.9 | 5.3 | [1.2, 10.2] | 384 | 542 | 44 |
| Prerequisite | 97.0 | 88.7 | 8.2 | [0.0, 18.1] | 66 | 71 | 19 |
| Scope-changing | 100.0 | 92.3 | 7.7 | [0.0, 18.4] | 47 | 78 | 12 |
| Layout-coupled | 92.2 | 56.5 | 35.7 | [19.5, 53.7] | 64 | 92 | 12 |
| Operation | |||||
| shadow | 19 | 89.5 | 26.3 | 29.4 | 17 |
| pic_shadow | 19 | 84.2 | 26.3 | 31.2 | 16 |
| reflection | 17 | 52.9 | 35.3 | 44.4 | 9 |
| rotate_angle | 46 | 78.3 | 52.2 | 47.2 | 36 |
| flip_direction | 17 | 100.0 | 58.8 | 58.8 | 17 |
| align_command | 73 | 100.0 | 72.6 | 72.6 | 73 |
| Word feature area | |||||
| Annotations | 51 | 84.3 | 98.0 | 97.7 | 43 |
| Charts | 171 | 88.9 | 95.3 | 97.4 | 152 |
| Equations | 75 | 61.3 | 92.0 | 93.5 | 46 |
| Graphics | 17 | 76.5 | 88.2 | 92.3 | 13 |
| Lists | 48 | 87.5 | 77.1 | 83.3 | 42 |
| Page Structure | 49 | 73.5 | 69.4 | 91.7 | 36 |
| Operation | System | Class | Compositional case | Atomic case |
| insert_equation | GPT-5.6 | Prerequisite | PrerequisiteEquation/variant_fraction | EquationInsertNew/apply |
| drop_cap | GLM-5.2 | Additive | AdditiveParagraphComposition/variant_center_dropcap | DropCapInsertGallery/dropped |
| outline_weight | MiniMax M3 | Additive | AdditiveFontComposition/variant_mixed_effects | TextOutlineWeightGallery/apply_1pt |
| outline_color | K3 | Additive | AdditiveFontComposition/variant_off_heavy | TextOutlineColorPickerAlternate/apply_Blue |
| rotate_angle | GPT-5.6 | Additive | AdditiveLayoutComposition/variant_rotate_cw_distribute_h | ObjectRotateGallery/apply_rotate_right_90 |
| table_style | Qwen3.8-27B | Additive | AdditiveTableComposition/variant_style_textdir_border | TableStylesGalleryWord/Grid_Table_4___Accent_1 |
| Word feature area | Source command | Instruction example |
|---|---|---|
| Text | Bold | Make the word ’Target’ bold. |
| Paragraphs | AlignCenter | Center the paragraph containing ’Target’. |
| Lists | BulletsGalleryWord | Apply bullet list formatting to all paragraphs in the document using checkmark bullets. |
| Tables | TableInsertGallery | Insert a 2x2 table. |
| Shape geometry | AlignDistributeHorizontally | Distribute all three rectangles evenly horizontally with equal spacing between them. |
| Shape appearance | ShapeFillColorPicker | Set the shape’s fill color to Blue (RGB: 0, 112, 192). |
| Composition class | Referenced atomic commands | Instruction example |
| Additive | AlignCenter ; ParagraphSpacingBefore ; ParagraphSpacingAfter ; DropCapInsertGallery | Center the paragraph containing the word ’Target’, set paragraph spacing to 0 points before and 0 points after for all paragraphs, and apply a dropped drop cap to the first paragraph. |
| Prerequisite | ChartInsert ; ChartDataLabel | Below the ’Quarterly Sales:’ heading, insert a clustered column chart with categories ’Q1’ and ’Q2’, and one series ’Revenue’ with values 100 and 150, then add data labels at the ’Outside End’ position. |
| Scope-changing | TableRowsInsertBelowWord ; TableColumnsInsertRight ; TableSplitTable ; TableColumnsDistribute ; TableRowsDistribute | Insert a new row below the first row and a new column to the right of the first column, split the table at the inserted row, then distribute the columns in the upper table and distribute the rows in the lower table. |
| Layout-coupled | TextWrapGallery ; PicturePositionGallery | Apply square wrapping to the cross shape (text wraps around the cross’s rectangular bounding box), then position it at the top-center of the page, using the Position preset (within default page margins). |
| Word feature area | Released commands |
|---|---|
| Text | Bold, ChangeCaseGallery, CharacterBorder, CharacterShading, ClearFormatting, Font, FontColorPicker, FontSize, FontSizeDecreaseWord, FontSizeIncreaseWord, Italic, LigatureGalleryWord, ShadingColorPicker, Strikethrough, Subscript, Superscript, SymbolInsertGallery, TextFillGradientGallery, TextHighlightColorPicker, TextOutlineColorPickerAlternate, TextOutlineDashesGallery, TextOutlineWeightGallery, TextReflectionGallery, UnderlineColorPicker, UnderlineGallery |
| Paragraphs | AlignCenter, AlignJustify, AlignLeft, AlignRight, BordersSelectionGallery, DropCapInsertGallery, IndentDecreaseWord, IndentIncreaseWord, LineSpacingGallery, ParagraphSpacingAfter, ParagraphSpacingBefore, QuickStylesGallery, TableOfContentsAddTextGallery |
| Lists | BulletsGalleryWord, ListLevelGallery, MultilevelListGallery, NumberingGalleryWord |
| Tables | BorderStylesGallery, MergeCells, QuickTablesInsertGallery, SortDialogClassic, SplitCells, TableAutoFitContents, TableAutoFitWindow, TableCellAlignBottomCenter, TableCellAlignBottomLeft, TableCellAlignBottomRight, TableCellAlignMiddleCenter, TableCellAlignMiddleLeft, TableCellAlignMiddleRight, TableCellAlignTopCenter, TableCellAlignTopLeft, TableCellAlignTopRight, TableColumnsDelete, TableColumnsDistribute, TableColumnsInsertLeft, TableColumnsInsertRight, TableDelete, TableInsertGallery, TableRowsDelete, TableRowsDistribute, TableRowsInsertAboveWord, TableRowsInsertBelowWord, TableSplitTable, TableStyleClear, TableStylesGalleryWord |
| Shape geometry | AlignDistributeHorizontally, AlignDistributeVertically, GalleryAllShapesAndCanvas, ObjectBringForward, ObjectBringToFront, ObjectRotateGallery, ObjectSendBackward, ObjectSendToBack, ObjectsAlignBottomSmart, ObjectsAlignCenterHorizontalSmart, ObjectsAlignLeftSmart, ObjectsAlignMiddleVerticalSmart, ObjectsAlignRightSmart, ObjectsAlignTopSmart, ObjectsGroup, ObjectsUngroup, PicturePositionGallery, ShapeChangeShapeGallery, TextBoxInsert, TextBoxInsertGallery, TextBoxInsertVerticalWord, TextWrapGallery |
| Shape appearance | BevelShapeGallery, GlowColorPicker, GradientGallery, ObjectEffectGlowGallery, ObjectEffectShadowGallery, ObjectEffectSoftEdgesGallery, ReflectionGallery, ShapeFillColorPicker, ShapeOutlineColorPicker, ShapeStylesGallery, _3DRotationGallery |