DynaRev: Benchmarking Web Agents under Dynamic Goal Revision

Shiqi Zhang; Guochen Zhou; Zesheng Wei; Hai Zhao; Zhisong Zhang

cs.AIOct 29, 2026

DynaRev: Benchmarking Web Agents under Dynamic Goal Revision

Authors: Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao, Zhisong Zhang

Organizations: City University of Hong Kong · Shanghai Jiao Tong University

Abstract

Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. DynaRev uses semantic milestones to deliver revisions at task-defined stages, covers additive revisions, unconditional overrides, and runtime-conditioned overrides, and evaluates correctness, responsiveness, and efficiency through trajectory-grounded evaluation. Five-model evaluations show that agents often satisfy only part of the task requirements, with performance varying across task sources, revision types, and models. Further analyses reveal varied milestone timing, mixed effects of dynamic delivery, and contrasting revision-handling outcomes, with gains from instruction rewriting while preserving browser state and losses from browser resets. DynaRev provides a testbed for studying how web agents adapt to evolving user goals. The code and data are available at https://github.com/zsq259/DynaRev.

Figures & tables

Figure 1: Example dynamic goal revision task.

Figure 2: Overview of benchmark construction, milestone-triggered execution, and revision-aware evaluation.

Source	Type	Tasks	Avg. Cons.	Avg. Rounds
OM2W	Additive	37	4.76	2.59
	Uncond.	39	4.56	2.92
	Cond.	31	5.74	3.26
M2W2	Additive	39	9.69	2.97
	Uncond.	23	10.04	4.22
	Cond.	12	9.75	4.92

Table 1: Dataset statistics by source and revision type. Avg. Cons. and Avg. Rounds denote average constraint and instruction-round counts. Uncond. and Cond. denote unconditional and conditional overrides.

	Correctness			Responsiveness	Efficiency
Model	Score $\uparrow$	Success $\uparrow$	MFFR $\uparrow$	WRA $\uparrow$	CE (Steps) $\uparrow$	CE (Tokens) $\uparrow$	Avg Steps $\downarrow$
DeepSeek-V4.1-Flash	0.698	43.1%	2.21	66.8%	0.482	0.637	43.0
GPT-6.1-Sol	0.598	28.2%	1.91	68.8%	0.477	0.554	24.9
Gemini-3.8-Flash	0.735	42.5%	2.28	69.4%	0.522	0.673	33.1
Qwen3.8-Flash	0.604	39.2%	2.09	68.4%	0.410	0.549	50.9
Claude-Opus-5-5	0.681	36.5%	2.16	68.3%	0.448	0.596	43.8

Table 2: Main results across all dynamic subsets.

	OM2W						M2W2
	Additive		Unconditional		Conditional		Additive		Unconditional		Conditional
Model	Score $\uparrow$	Steps $\downarrow$	Score $\uparrow$	Steps $\downarrow$	Score $\uparrow$	Steps $\downarrow$	Score $\uparrow$	Steps $\downarrow$	Score $\uparrow$	Steps $\downarrow$	Score $\uparrow$	Steps $\downarrow$
DeepSeek-V4.1-Flash	0.780	32.4	0.774	30.6	0.824	38.5	0.570	55.4	0.581	56.5	0.518	62.2
GPT-6.1-Sol	0.793	19.8	0.642	27.7	0.753	23.7	0.404	27.7	0.435	23.2	0.401	29.0
Gemini-3.8-Flash	0.795	29.3	0.759	26.2	0.898	30.8	0.606	38.6	0.647	40.9	0.642	40.5
Qwen3.8-Flash	0.804	33.7	0.739	37.1	0.727	39.2	0.394	67.4	0.419	74.3	0.275	80.0

Table 3: Performance breakdown by task source and revision type.

Figure 3: Task-level distribution of constraint satisfaction scores. Each point reports the proportion of tasks reaching the corresponding score threshold.

Figure 4: Cost–quality curves under action-step and token budgets.

	Revision Satisfaction $q_{i,k}$
Model	WRA $=1$	WRA $=0$
DeepSeek-V4.1-Flash	0.683	0.644
GPT-6.1-Sol	0.587	0.465
Gemini-3.8-Flash	0.720	0.696
Qwen3.8-Flash	0.609	0.577
Claude-Opus-5-5	0.677	0.628

Table 4: Revision-level analysis conditioned on WRA.

	OM2W		M2W2
Model	$\Delta$ Score	$\Delta$ Steps	$\Delta$ Score	$\Delta$ Steps
DeepSeek	$-0.025$	$+9.06$	$-0.035$	$+6.47$
GPT	$+0.079$	$-0.74$	$+0.122$	$+4.87$
Gemini	$-0.133$	$+12.54$	$+0.108$	$+7.53$
Qwen	$+0.161$	$+8.54$	$+0.115$	$-12.67$
Claude	$-0.107$	$+14.34$	$-0.169$	$+10.93$

Table 5: Changes from static to dynamic additive delivery on paired tasks.

\Delta

denotes dynamic minus static.

Figure 5: Representative failure cases under dynamic goal revision.

Strategy	Score $\uparrow$	Steps $\downarrow$	Tokens (k) $\downarrow$
Baseline	0.647	50.3	806.7
Rewrite	0.718	44.9	658.8
Clear	0.648	53.6	667.2
Clear + Rewrite	0.663	52.8	673.4
Reset	0.586	59.7	691.7
Reset + Rewrite	0.525	61.5	843.6

Table 6: Comparison of revision handling strategies. Rewrite consolidates instructions, Clear removes interaction history, and Reset resets the browser.

Figure 6: Relative revision-injection positions across tasks and agents, ordered by mean across agents.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Scenario distribution of retained seed tasks.

Model	Better	Worse	Unchanged
DeepSeek	8	9	13
GPT	15	9	6
Gemini	9	7	14
Qwen	11	3	16
Claude	5	14	11
All	48	42	60

Appendix

Table 7: Pairwise Score changes under dynamic delivery.

Model	$\geq$ 0.1	$\geq$ 0.2	$\geq$ 0.3	$\geq$ 0.4	$\geq$ 0.5	$\geq$ 0.6	$\geq$ 0.7	$\geq$ 0.8	$\geq$ 0.9	$\geq$ 1.0
DeepSeek-V4.1-Flash	81.8%	81.8%	79.6%	77.9%	76.2%	72.9%	68.0%	59.1%	45.9%	43.1%
GPT-6.1-Sol	86.7%	82.9%	74.6%	69.1%	64.6%	54.7%	49.7%	37.6%	28.7%	28.2%
Gemini-3.8-Flash	89.0%	87.3%	86.2%	81.8%	80.1%	78.5%	71.8%	56.4%	44.2%	42.5%
Qwen3.8-Flash	69.1%	69.1%	68.0%	67.4%	67.4%	64.1%	58.6%	51.4%	40.9%	39.2%
Claude-Opus-5-5	81.2%	80.7%	79.6%	79.0%	77.9%	74.0%	65.7%	53.0%	39.8%	36.5%

Appendix

Table 8: Task-level distribution of constraint satisfaction scores. Each column reports the percentage of tasks reaching the corresponding score threshold.

	OM2W				M2W2
	Score		Steps		Score		Steps
Model	Static	Dynamic	Static	Dynamic	Static	Dynamic	Static	Dynamic
DeepSeek	0.909	0.884	20.67	29.73	0.616	0.581	45.00	51.47
GPT	0.791	0.870	18.07	17.33	0.307	0.429	26.20	31.07
Gemini	0.953	0.820	16.13	28.67	0.534	0.642	30.47	38.00
Qwen	0.822	0.983	25.13	33.67	0.337	0.452	74.80	62.13

Appendix

Table 9: Full results for static and dynamic additive delivery on paired tasks.

Annotator	Early	Aligned	Late
Annotator 1	0.0%	90.0%	10.0%
Annotator 2	10.0%	76.7%	13.3%

Appendix

Table 10: Human judgments of milestone-trigger timing.

Judgment	Agreement	Cohen’s $\kappa$
Q1: Trigger state	90.0%	0.000
Q2: Previous state	90.0%	0.516
Three-way label	83.3%	0.438

Appendix

Table 11: Agreement between the two human annotators.

Model	Metric	$I_{0}$	$I_{1}$	$I_{2}$	$I_{3}$	$I_{4+}$
DeepSeek-V4.1-Flash	CSS	70.8%	68.2%	62.9%	59.3%	50.3%
DeepSeek-V4.1-Flash	WRA	–	64.8%	68.9%	61.3%	72.0%
GPT-6.1-Sol	CSS	63.1%	60.6%	50.0%	45.5%	28.5%
GPT-6.1-Sol	WRA	–	69.1%	63.9%	50.0%	46.7%
Gemini-3.8-Flash	CSS	74.3%	76.2%	64.2%	57.0%	68.7%
Gemini-3.8-Flash	WRA	–	69.9%	70.3%	61.3%	65.6%

Appendix

Table 12: Round-wise constraint satisfaction score (CSS) and window response accuracy (WRA).

Web Agent Benchmarks

Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.

See all 67 papers

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
Apr 8, 2026Jiwan Chung, JiHyuk Byun, Vibhav Vineet +1Process-Based Agent Evaluation Agent Evaluation
Yonsei University · Microsoft Research
WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents
Sep 28, 2026Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2Web Agent Benchmarks GUI Agents
DAIMLD · HSE University · NUST MISIS
TimeWarp: Evaluating Web Agents by Revisiting the Past
Mar 5, 2026Md Farhan Ishmam, Kenneth MarinoComputer-Use Agent Benchmarks Web Agent Benchmarks
University of Utah, USA

Apr 8, 2026J/K move · Enter open · S save

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

Jiwan Chung, JiHyuk Byun, Vibhav Vineet, Seon Joo Kim

Yonsei University · Microsoft Research

Process-Based Agent Evaluation Agent Evaluation AI Agent Evaluation Web Agents Web Agent Benchmarks State Tracking

Open paper AskPDF arXiv

Web agents act through long interaction sequences, yet existing benchmarks evaluate only terminal success, discarding all process information and offering little guidance on improvement. In this work, we conduct a process-level analysis of web agents. We introduce WebStep, a benchmark of 1,800 task instances with controlled difficulty and automatic semantic state tracking. Each website exposes a deterministic semantic MDP alongside the GUI: the agent operates on the interface, while the environment records high-level states and transitions in the background, enabling fine-grained analysis without manual annotation. Based on the semantic trajectory, we first show that process metrics reveal differences invisible to outcome evaluation: three agents whose success rates cluster within 34-37% diverge in exploration reach versus execution accuracy. Then, decomposing by skill characterizes the nature of these differences, exposing opposite per-skill rankings hidden within the same website: e.g., on Q&A, Claude CUA outperforms OpenAI CUA by 30% on navigation actions yet underperforms it by 6.7% on inspection, pinpointing a concrete skill to improve even within a domain. Bifurcation analysis further localizes the decisive error that loses the task and shows that this error is agent-specific rather than shared. Finally, these differences widen as tasks grow harder: success rate is similar on easy tasks but separates sharply as exploration becomes more demanding. Our process-level analysis opens a new avenue in web agent evaluation, providing fine-grained and actionable insight into where and how each agent should be improved. Project page: https://jiwanchung.github.io/webstep

Abstract

Web Agent Benchmarks

Explore similar work

Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking

WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

TimeWarp: Evaluating Web Agents by Revisiting the Past