cs.AIOct 19, 2026
SaveStepBench: Diagnosing Real-World Procedural Understanding in AI Agents
Organizations: UCL AI Centre, University College London · School of Informatics, University of Edinburgh · Microsoft · Cohere Labs
Abstract
Evaluating an agent's planning ability remains challenging for open-domain tasks, where verifying procedures through execution is often impractical. Existing evaluations typically assess only the final plan and do not distinguish procedural errors such as missing, incorrect, or misordered actions. We introduce STEPBENCH, a benchmark of 6,885 verified procedures crawled from wikiHow across 81 open-domain categories. From each procedure, we derive 5 diagnostic tasks: four with deterministic answers involving withheld, incorrect, or shuffled steps, and one requiring next-step generation. We additionally evaluate full-plan generation to examine how diagnostic performance correlates with end-to-end plan validity. Using the same procedures across tasks lets us directly compare different planning capabilities. For generation, STEPEVAL identifies critical failures that prevent goal completion. Across 14 configurations of 12 models, we find substantial differences across planning capabilities, with step ordering proving the most difficult: the strongest model correctly orders only 63.0% of procedures, dropping to 33.1% for those with nine or more steps. Incorrect-step detection also predicts judged plan validity across models, suggesting a lower-cost alternative for evaluating generation quality. On STEPBENCH-Val, comprising 214 procedures published in 2026, comprehension scores and rankings remain stable, suggesting limited influence from memorisation.
Figures & tables
Figure 1: Constructing StepBench . wikiHow methods are crawled into an initial pool, filtered by GPT-4.1 for whether they describe a genuine procedural task, and the selected procedures (each a task query, category, and ordered steps) instantiate five diagnostic tasks: missing-step selection (T1–T2), erroneous-step detection and reordering (T3–T4), and next-step generation (T5). Full-plan generation is additionally evaluated to examine how diagnostic performance correlates with end-to-end plan validity.
| Task query and reference steps | Model-generated steps | StepEval |
|---|---|---|
| Query: [Cakes] How to make cake flour from all-purpose flour? Reference: 1. Measure one level cup of all-purpose flour. 2. Remove two tablespoons of flour. 3. Replace them with two tablespoons of cornstarch. 4. Sift the flour and cornstarch together five times. | 1. Measure one cup of all-purpose flour. 2. Sift into a bowl to remove lumps. 3. Sift again through a fine-mesh sieve. 4. Repeat sifting two or three more times until powdery and smooth. 5. Store the cake flour in an airtight container. | Missing critical action. The plan omits the cornstarch substitution. Repeated sifting leaves the ingredient composition unchanged and does not achieve the conversion described by the reference. |
| Query: [Computers and Electronics] How to blur your background in Zoom using the mobile app? Reference: 1. Open the Zoom app. 2. Join a meeting. 3. Tap the three-dot menu. 4. Tap Virtual Background or Background and Filters. 5. Tap Blur. | 1. Open the Zoom mobile app. 2. Join or start a meeting. 3. Tap More or the three-dot icon. 4. Select Virtual Background. 5. Choose an available background or upload your own. 6. Select the background to apply it. 7. Adjust it through the Background option as needed. | Failure to achieve the stated goal. The plan reaches the relevant menu but instructs the user to apply a replacement background, omitting the essential action of selecting Blur. |
| Query: [Computers] How to check your computer uptime on Linux? Reference: 1. Open the terminal. 2. Type uptime -p and press Enter. | 1. Open the terminal. 2. Type uptime and press Enter. 3. Read the displayed system uptime. | No critical failure. Omitting -p changes the output format, but the command still provides the requested uptime information. |
Table 1: Examples of critical-failure evaluation with StepEval on full-plan generation. Each example pairs a task query and reference steps with a plan generated by Llama 3.1 8B Instruct and the judgment of the StepEval . The examples show an omitted essential action, failure to satisfy the task goal, and an acceptable variation. Steps and explanations are abbreviated for readability, and bold text highlights the relevant differences.
Figure 2: Correlations between the five diagnostic tasks and end-to-end plan validity. Left: Spearman correlations with full-plan VPR across all 14 configurations, using accuracy for T1–T4 and VPR for T5. Right: reordering accuracy versus full-plan VPR, with each point representing one configuration.
| T1 | T2 | T3 | T4 | T5 | Full plan | |
| Change | -1.4 | -0.5 | -0.6 | -8.3 | -7.5 | -16.2 |
Table 3: Mean score change on StepBench -Val relative to StepBench (percentage points), after matching categories and reweighting procedure lengths. The five diagnostic tasks use accuracy (T1–T4) or VPR (T5), while full-plan generation uses VPR. Results are averaged over all 14 configurations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Source | Size | Gen. | Miss. | Err. | Ord. | Decomp. metric |
| Task decomposition | |||||||
| Zhang et al. (2021) | Wunderlist, wikiHow | 8,262 | ✓ | ✗ | ✗ | ROUGE, BERTScore | |
| TaskLAMA ( Yuan et al., 2023a ) | search logs, MSCT | 1,612 | ✓ | ✗ | ✗ | ROUGE, Hungarian | |
| TaskBench ( Shen et al., 2024 ) | synth. tool graphs | 17,331 | ✓ | ✗ | ✗ | ✗ | ROUGE only |
| UltraTool ( Huang et al., 2024 ) | expert queries | 5,824 | ✓ | ✗ | ✗ | ✗ | LLM judge (6 dim.) |
| Procedural comprehension | |||||||
Table 4: Selected benchmarks for task decomposition or procedural comprehension. Gen. : the model must produce step text. Miss. : the model must identify a step withheld from a given procedure. Err. : the model must localise an incorrect step injected into an otherwise correct procedure. Ord. : the model must recover or judge step order — marks benchmarks that predict pairwise dependencies over gold steps rather than requiring a full ordering. Among the task-decomposition benchmarks in this table, none evaluates recovery of missing or detection of incorrect steps, and none evaluates comprehension and generation over a shared procedure pool. Sizes are as reported by each paper; StepBench is the count of curated procedures, each of which yields an instance of every task.
Figure 3: StepBench corpus. Left : procedures per category, top 15 of 81; the distribution is long-tailed because how-to content is. Centre : procedures are short — a median of 4 steps — which is what makes exact-match reordering a demanding but not hopeless task. Right : editorial provenance of the source articles.
| Scale | Provenance of source articles | ||
|---|---|---|---|
| Procedures | 6,885 | Distinct articles | 2,920 |
| Reference steps | 30,975 | Expert co-authored | 2,806 (96.1%) |
| Categories | 81 | Editorially reviewed | 1,707 (58.5%) |
| Steps per procedure | 4.5 | Fact checked | 990 |
| median (range) | 4 (1–28) | Expert approved | 658 |
| Words per step | 7.27 | Tested | 59 |
Table 5: StepBench at a glance. Left : benchmark scale. Right : provenance of the wikiHow articles the procedures are drawn from. wikiHow attaches an authorship record to every article; the great majority of StepBench ’s sources were co-authored with a credentialed subject expert, and a further editorial process fact-checks, expert-approves, or physically tests a large subset. Readership and helpfulness ratings are reader-supplied signals that these are procedures people actually follow.
Figure 4: The five StepBench diagnostic tasks and full-plan generation on a shared procedure (“perform a glute bridge”). T1–T2 withhold a step for recovery from candidates; T3–T4 introduce incorrect or shuffled steps for detection or reordering; and T5 requires generating the next step. Full-plan generation is shown separately as an end-to-end evaluation setting. Circles indicate each step’s state: given, withheld, incorrect, shuffled, or generated.
Figure 5: Scores by procedure length for eight configurations. Left: incorrect-step detection accuracy. Centre: exact-order accuracy. Right: normalised edit distance, where higher scores indicate closer reference agreement. Gemini 3 Flash uses ; Qwen3 32B uses non-thinking mode.
Figure 6: Accuracy by relative step position for the same eight configurations as Figure 5 . Left: the replaced step in T3. Right: the missing step in T2. Position effects vary across tasks and models.
Figure 7: Mean scores across 14 configurations for the twenty largest categories, ordered by procedure count. Counts in parentheses refer to the source procedure pool; task-specific counts can differ. T1–T4 use accuracy and T5 and full-plan generation use VPR. Shading is normalised within each column.
| Diagnostic tasks | Full plan | |||||
|---|---|---|---|---|---|---|
| Configuration | T1 | T2 | T3 | T4 | T5 ( vpr ) | vpr |
| GPT-5.6-sol (T=0, no think) | -0.7 | -4.0 | +0.6 | -11.7 | -8.3 | -15.7 |
| Gemini 3 Flash (think, T=0) | -0.2 | -0.6 | +1.9 | -20.7 | -8.0 | -13.8 |
| Gemini 3 Flash (think, T=1) | -2.9 | -1.2 | -0.3 | -20.1 | -10.7 | -14.2 |
| Command A Reasoning (think) | -1.7 | -2.3 | -3.3 | -10.4 | -12.1 | -16.0 |
| Command A | -1.3 | +1.6 | +0.5 | -7.5 | -7.7 | -13.9 |
Table 7: Per-configuration change from StepBench to the 2026 validation set, after matching category composition and reweighting to the 2026 procedure-length distribution. T1–T4 use accuracy; T5 and full-plan generation use StepEval VPR. Values are percentage-point changes. Every configuration has lower scores on reordering, next-step generation, and full-plan generation. Amber marks a decrease of more than three points.