AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

Shanshan Zhong; Kate Shen; Chenyan Xiong

cs.MAApr 13, 2026

AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

Authors: Shanshan Zhong, Kate Shen, Chenyan Xiong

Organizations: Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Anaxi Labs

Abstract

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs are released on https://github.com/cxcscmu/AgentWebBench.

Figures & tables

Figure 1: Overview of AgentWebBench. The benchmark models the Agentic Web paradigm where user agents coordinate with multiple autonomous content agents across information-seeking tasks. (a) User intents spanning four representative tasks: web search, web recommendation, question answering, and deep research. (b) User agent that analyzes intents, selects and queries content agents, and synthesizes responses. (c) Content agents that process requests and retrieve information from their proprietary domains. (d) Environment Components comprising 100 websites with 18.4M documents, each managed by an independent content agent.

Task	#Sample	Output	Metrics
Web Search	354	Ranked documents	NDCG@ $k$ , Recall@ $k$
Web Recommendation	281	Ranked documents	NDCG@ $k$ , Recall@ $k$
Question Answering	53	Short answer	Accuracy, F1
Deep Research	331	Long report	KPR, KPC, Clarity, Insight

Table 1: Summary of tasks, datasets, and evaluation metrics.

LLM	Method	Web Search				Web Recommendation				QA		Deep Research
		N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
	Classic IR	47.86	51.04	57.63	65.25	-	-	-	-	-	-	-	-	-	-
Qwen3-4B	Classical	47.65	47.65	54.80	54.80	0.80	0.80	1.07	1.07	18.87	21.43	52.74	1.48	71.45	63.84
	$\text{Tool}_{\text{E}}$	30.67	31.03	36.44	37.29	0.00	0.00	0.00	0.00	13.21	15.21	53.17	2.19	77.13	74.35
	$\text{Tool}_{\text{P}}$	32.62	33.21	35.31	36.72	0.58	0.58	0.71	0.71	13.21	16.13	47.83	1.37	67.79	61.99
	Multi-Agent	26.80	27.04	29.38	29.94	0.58	0.58	0.71	0.71	15.09	17.44	47.54	1.63	71.63	62.63

Table 2: Performance comparison across four tasks, including web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific. Classic IR is a non-agent baseline; “-” indicates not applicable. Bold and underline indicate the best and second-best results, respectively.

Figure 2: Impact of multi-agent coordination on web traffic. The charts show citation frequency for the 20 most cited sources (websites or content agents) for Qwen3-4B and Gemini-3 user agents on the deep research task. In each subplot, sources are ranked by the frequency difference (Multi-Agent minus Classical). The upper section highlights sources cited more often under multi-agent decentralized coordination, while the lower section shows sources cited more often in the Classical centralized setting.

Figure 3: Effect of test time scaling on coordination performance. Valid (%) measures the fraction of successfully content-agent parsed requests.

Figure 4: Interaction efficiency across tasks in AgentWebBench. Turn is the number of agent responses. #Agent and #Req. are the numbers of contacted content agents and retrieval requests. Valid (%) is the success rate of parsing requests or returning results.

LLM	Web Search		Web Rec.		QA
	User	Content	User	Content	User	Content
Qwen3-4B	59.59	40.41	48.75	51.25	84.44	15.56
Qwen3-14B	64.83	35.17	62.59	37.41	84.21	15.79
Qwen3-30B-A3B	56.41	43.59	52.86	47.14	83.33	16.67
Qwen3-80B-A3B	54.90	45.10	60.85	39.15	93.94	6.06
DeepSeek-V3.2	35.18	64.82	48.57	51.43	70.27	29.73

Table 3: Failure mode (%) analysis of incorrect predictions. Each failure is attributed exclusively to either the user agent or the content agents based on task-specific rules. Percentages sum to 100% within each task.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Setting	#Cite	No-Ref.	Word ↓	Pos. ↓	Overall ↓
Qwen3-4B	Classic	2.93	0.60	0.1642	0.1569	0.1675
Qwen3-4B	Multi-Agent	2.63	31.12	0.0948	0.0931	0.0960
Qwen3-14B	Classic	2.97	1.81	0.1540	0.1443	0.1611
Qwen3-14B	Multi-Agent	2.67	1.51	0.0748	0.0766	0.0877
Qwen3-30B-A3B	Classic	3.35	2.42	0.1667	0.1589	0.1731
Qwen3-30B-A3B	Multi-Agent	2.53	0.60	0.1386	0.1315	0.1442

Appendix

Table 4: Content agent visibility on the deep research task, measured by the standard deviation of GEO scores across cited content agents. No-Ref. (%) reports the percentage of responses without any citation. Word, Pos, and Overall correspond to word-based, position-based, and combined GEO variants, respectively (four decimal places to identify fine-grained differences).

Figure 5: Impact of the different coordination strategies on web traffic. The charts illustrate the citation frequency of 20 most cited sources (websites or content agents) for Gemini-3 user agents on the deep research task. Each subplot is sorted vertically by the frequency difference (Tool ${}_{\text{E}}$ / Tool ${}_{\text{P}}$ / Multi-Agent minus Classical). The upper section highlights sources more frequently cited under the multi-agent coordination, while the lower section shows those preferred by Classical.

LLM	Method	Web Search				Web Recommendation				QA		Deep Research
		N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
DeepSeek V3.2 (non-think)	Classical	49.37	50.66	57.63	60.73	1.07	1.07	1.07	1.07	15.09	12.40	65.26	1.36	88.73	80.15
	$\text{Tool}_{\text{E}}$	32.99	34.63	37.01	40.96	0.36	0.36	0.36	0.36	16.98	8.57	66.17	1.91	90.18	85.71
	$\text{Tool}_{\text{P}}$	20.09	21.50	23.73	27.12	0.00	0.00	0.00	0.00	9.43	5.94	57.91	1.63	79.61	69.79
	Multi-Agent	12.26	12.39	13.28	13.56	0.00	0.00	0.00	0.00	16.98	15.58	58.13	1.81	80.06	70.12
DeepSeek V3.2 (think)	Classical	49.02	49.37	55.37	56.21	0.40	0.40	0.71	0.71	24.53	19.51	66.54	1.27	87.16	80.48

Appendix

Table 5: Performance comparison across four tasks under different LLM-based methods. Tasks include web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific. “-” indicates not applicable. Bold and underline indicate the best and second-best results, respectively.

Task	User Agent				Content Agent
	Turn	#Agent	#Req.	Valid (%)	Turn	Valid (%)
LLM: Qwen3-4B
Web Search	2.61	2.61	2.79	88.58	2.53	78.65
Web Rec.	7.63	6.02	9.88	84.88	2.69	53.99
Question Answering	3.38	1.36	2.06	91.37	3.83	47.62
Deep Research	4.61	3.60	4.05	96.20	2.24	52.54

Appendix

Table 6: Mean interaction efficiency across tasks in AgentWebBench for Qwen3-4B and Gemini-3. Turn indicates the number of agent-generated responses; #Agent and #Req. denote the average number of contacted content agents and retrieval requests, respectively. Valid (%) reports the success rate of requests or returns.

Short Name	Full Name	Max Input Length	Max Output Length	Thinking
Qwen1.5-14B	Qwen/Qwen1.5-14B	32,768	8,192	×
Qwen2.5-14B	Qwen/Qwen2.5-14B	131,072	8,192	×
Qwen3-4B-Instruct	Qwen/Qwen3-4B-Instruct-2507	262,144	8,192	×
Qwen3-4B	Qwen/Qwen3-4B-Thinking-2507	262,144	8,192	$\checkmark$
Qwen3-14B	Qwen/Qwen3-14B	32,768	8,192	$\leftrightarrows$
Qwen3-30B-A3B-Instruct	Qwen/Qwen3-30B-A3B-Instruct-2507	262,144	8,192	×

Appendix

Table 7: All LLMs used in experiments with maximum input and default output lengths, including thinking capability. The Thinking column indicates model capability: × = thinking mode not supported;

\checkmark

= thinking mode only (forced step-by-step reasoning);

\leftrightarrows

= supports switching/dynamic control between thinking and non-thinking/fast modes.

LLM	Web Search				Web Recommendation				QA		Deep Research
	N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
Qwen1.5-14B	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.08	3.73	0.06	4.35	4.26
Qwen2.5-14B	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	5.66	4.01	7.32	0.18	6.74	6.89
Qwen3-14B	27.33	27.90	31.36	32.77	0.22	0.53	0.36	1.07	28.30	23.79	49.85	1.29	63.23	51.69

Appendix

Table 8: Performance comparison across four tasks under different version LLMs. Tasks include web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific.

Model	N@3	N@5	R@3	R@5
Qwen3-4B-QueryGen	0.00	0.00	0.00	0.00
Qwen3-14B-QueryGen	0.76	0.76	1.07	1.07
Qwen3-30B-A3B-QueryGen	0.18	0.32	0.36	0.71
Qwen3-80B-A3B-QueryGen	0.89	0.89	1.42	1.42
DeepSeek-V3.2-QueryGen	0.76	0.90	1.07	1.42
GPT-5-QueryGen	0.71	1.02	0.71	1.42

Appendix

Table 9: QueryGen performance (%) on our web recommendation setting. QueryGen is a non-agent baseline.

Model	Recall@10	NDCG@10	Recall@50	NDCG@50	Recall@100	NDCG@100
GPT-3.5-Turbo-QueryGen	0.88	0.58	1.56	0.73	2.54	0.89
GPT-4o-QueryGen	0.68	0.27	1.76	0.50	3.12	0.72
Gemini-2.5-Flash-QueryGen	0.68	0.42	1.46	0.58	2.64	0.77
GPT-4.1-QueryGen	1.07	0.50	1.95	0.68	2.54	0.77
Claude-Sonnet-4-QueryGen	0.68	0.32	1.66	0.52	2.15	0.60
DeepSeek-V3-QueryGen	1.27	0.82	2.64	1.11	3.71	1.29

Appendix

Table 10: Zero-shot ClueWeb-Reco benchmarking test results (%) on candidate item ranking ( He et al., 2025 ) .

Index	Name	Description	#Doc
1	www.amazon.com	Welcome to Amazon.com, the world’s largest online retailer and a pioneering force in e-commerce. Explore an unparalleled selection of products, from books and electronics to groceries and fashion, delivered conveniently to your door. Beyond shopping, Amazon also offers digital services like streaming media, cloud computing, and smart home devices.	1,749,804
2	en.wikipedia.org	en.wikipedia.org is the English-language version of Wikipedia, a free online encyclopedia that offers a vast and continually growing collection of knowledge. It is a massive collaborative project, with articles contributed and maintained by a global community of volunteers, covering nearly every topic imaginable.	1,406,613
3	stackoverflow.com	Stack Overflow is the definitive online community for developers to learn, share their knowledge, and build their careers. It’s a question-and-answer platform where millions of programmers find solutions to coding problems, ask new questions, and contribute their expertise. This vast, collaborative resource is indispensable for anyone working with code.	1,268,548
4	www.youtube.com	YouTube is the world’s largest online video-sharing platform, revolutionizing how we consume and create visual content. Millions of users visit daily to watch, upload, and share an incredibly diverse range of videos, from educational tutorials and music videos to vlogs and live streams. It serves as a global hub for creators to connect with audiences and for viewers to discover endless entertainment and information.	1,181,242
5	www.reddit.com	Reddit is a massive network of online communities where users from around the world converge to share content, discuss a myriad of topics, and engage in conversations. Often called “the front page of the internet,” it’s a dynamic platform where content is upvoted or downvoted by its users, shaping what becomes visible and popular across thousands of specialized subreddits.	654,802
6	www.quora.com	Quora is a leading question-and-answer platform where users can ask questions on virtually any topic and receive answers from a global community. It serves as a vast knowledge-sharing network, allowing individuals to learn, share insights, and connect with others based on shared interests and expertise.	651,607

Appendix

Table 11: The 100 websites in AgentWebBench with descriptions and document counts. Websites are selected from ClueWeb22-B based on document count, with each website assigned to one content agent. Descriptions are used by user agents for website selection in coordination strategies.

Web Agent Benchmarks

Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.

See all 59 papers

Oct 9, 2026

Web$hell: A Unified Interface for Agentic Web Access

Large language model (LLM) agents are becoming direct users of the open web. As they take on increasingly complex and information-intensive tasks, effective web access is becoming a core capability for agentic problem solving. Existing approaches have advanced agentic web access through different access models, including search-augmented agents, browser-based agents, and, more recently, filesystem-based agents. Nevertheless, as users expect agents to complete long-horizon, end-to-end tasks, they increasingly need to conduct broad information gathering, perform browser actions, and revisit acquired evidence within a single trajectory. Yet our preliminary analyses reveal that existing access models struggle to meet these demands. They primarily support either information gathering or browser interaction, while combining their capabilities can involve substantial costs or performance trade-offs. To this end, we present Web$hell, an interface that unifies web access under a single access model. Using a common schema defined by WebContract, WebCompiler compiles live webpages into addressable information and action objects. These objects are maintained in WebCache and handled through a shared set of operations, providing agents with consistent, operable, and persistent web access. Through extensive experiments across web search, web navigation, and deep research benchmarks, we show that Web$hell achieves strong task performance at lower cost. Our results further indicate the importance of interface design for effective web access, with Web$hell opening new possibilities for more capable and efficient LLM agents on the web.

Sunghwan Kim, Deogyong Kim, Sangam Lee +2

Yonsei University

Web Agents LLM Agent Harnesses Long-Horizon LLM Agents

Ask PDF

Jul 1, 2025

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

Foundation models now enable autonomous agents to interact with real-world websites, but existing benchmarks emphasize general-purpose browsing, underrepresent research-oriented environments and scholarly discovery workflows, and often depend on live sites whose changing content and structure undermine reproducibility. arXiv provides a realistic, reproducible, hierarchically structured, information-centric testbed without privacy-sensitive interactions. We introduce WebArxiv, a static-snapshot benchmark comprising 510 time-invariant tasks, each with a unique deterministic ground truth. Its diverse, realistic scholarly tasks go beyond simple information lookup and rule following to emphasize multi-constraint paper retrieval, fine-grained content extraction, and cross-paper comparison. Evaluations of a range of foundation-model-based web agents show that WebArxiv remains challenging. Behavioral analysis reveals that agents over-rely on fixed interaction histories, causing incomplete or repetitive reasoning. We therefore equip agents with a lightweight dynamic-memory mechanism for adaptive retrieval and reasoning over relevant context. The benchmark and code are available at https://anonymous.4open.science/r/74E4423BVNW/README.md.

Zihao Sun, Zijing Shi, Ling Chen

University of Technology Sydney, Australia

Web Agent Benchmarks Multimodal Agents IR Evaluation Web Agents

Ask PDF

Aug 28, 2026

AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics

User preferences in user-agent collaboration are rarely static and fully-specified upfront: preferences are formed, revealed, adjusted, and relaxed during interaction. Existing benchmarks for evaluating user-agent collaboration focus almost exclusively on resolving underspecified preferences, thereby failing to capture the richer dynamics of real-world interaction. We introduce AcCoRD, a user-agent collaboration benchmark requiring agents to handle diverse user preference dynamics in two domains: online shopping and travel planning. We evaluate five frontier LLMs under two prompting strategies: vanilla ReAct, and an uncertainty-guided variant that prompts models to identify and resolve ambiguity about user preferences. Our results reveal that frontier models can handle underspecification but struggle to satisfy preferences that emerge or evolve mid-interaction and require more sophisticated uncertainty modeling. Further, prompting alone fails to elicit the required uncertainty recognition. We release AcCoRD as a resource for developing agents that can navigate the full complexity of real-world user preferences.

Tejas Srinivasan, Shikib Mehri, Nandita Shankar Naik +3

University of Southern California · Contextual AI · Capital One +1

LLM Agent Evaluation Human-AI Collaboration LLM Agents AI Agent Benchmarks Preference Modeling User Preference Modeling

Ask PDF

Abstract

Web Agent Benchmarks

Explore similar work

Web$hell: A Unified Interface for Agentic Web Access

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics