AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

Shanshan Zhong; Kate Shen; Chenyan Xiong

cs.MAApr 13, 2026

AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web

Authors: Shanshan Zhong, Kate Shen, Chenyan Xiong

Organizations: Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Anaxi Labs

Abstract

Agentic Web is an emerging paradigm where autonomous agents help users use online information. As the paradigm develops, content providers are also deploying agents to manage their data and serve it through controlled interfaces. This shift moves information access from centralized retrieval to decentralized coordination. To study this setting, we introduce AgentWebBench, a benchmark that evaluates how well a user agent synthesizes answers by interacting with website-specific content agents. We evaluate four tasks that cover common web information needs, spanning ranked retrieval (web search, web recommendation) and open-ended synthesis (question answering, deep research). Across seven advanced LLMs and three coordination strategies, multi-agent coordination generally lags behind centralized retrieval as expected, because user agent cannot directly access the corpus, but the gap shrinks with model scale and can even outperform centralized retrieval on question answering. This benchmark also enables us to study properties of the emerging paradigm of the digital world. We find that decentralized access concentrates traffic toward a small set of websites, test time scaling improves both interaction reliability and task performance, and strong results require sufficient interactions guided by careful planning. Finally, our failure analysis suggests that user agents need better planning and answer synthesis, while content agents need more reliable retrieval and evidence quality. Code, data, and APIs are released on https://github.com/cxcscmu/AgentWebBench.

Figures & tables

Figure 1: Overview of AgentWebBench. The benchmark models the Agentic Web paradigm where user agents coordinate with multiple autonomous content agents across information-seeking tasks. (a) User intents spanning four representative tasks: web search, web recommendation, question answering, and deep research. (b) User agent that analyzes intents, selects and queries content agents, and synthesizes responses. (c) Content agents that process requests and retrieve information from their proprietary domains. (d) Environment Components comprising 100 websites with 18.4M documents, each managed by an independent content agent.

Task	#Sample	Output	Metrics
Web Search	354	Ranked documents	NDCG@ $k$ , Recall@ $k$
Web Recommendation	281	Ranked documents	NDCG@ $k$ , Recall@ $k$
Question Answering	53	Short answer	Accuracy, F1
Deep Research	331	Long report	KPR, KPC, Clarity, Insight

Table 1: Summary of tasks, datasets, and evaluation metrics.

LLM	Method	Web Search				Web Recommendation				QA		Deep Research
		N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
	Classic IR	47.86	51.04	57.63	65.25	-	-	-	-	-	-	-	-	-	-
Qwen3-4B	Classical	47.65	47.65	54.80	54.80	0.80	0.80	1.07	1.07	18.87	21.43	52.74	1.48	71.45	63.84
	$\text{Tool}_{\text{E}}$	30.67	31.03	36.44	37.29	0.00	0.00	0.00	0.00	13.21	15.21	53.17	2.19	77.13	74.35
	$\text{Tool}_{\text{P}}$	32.62	33.21	35.31	36.72	0.58	0.58	0.71	0.71	13.21	16.13	47.83	1.37	67.79	61.99
	Multi-Agent	26.80	27.04	29.38	29.94	0.58	0.58	0.71	0.71	15.09	17.44	47.54	1.63	71.63	62.63

Table 2: Performance comparison across four tasks, including web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific. Classic IR is a non-agent baseline; “-” indicates not applicable. Bold and underline indicate the best and second-best results, respectively.

Figure 2: Impact of multi-agent coordination on web traffic. The charts show citation frequency for the 20 most cited sources (websites or content agents) for Qwen3-4B and Gemini-3 user agents on the deep research task. In each subplot, sources are ranked by the frequency difference (Multi-Agent minus Classical). The upper section highlights sources cited more often under multi-agent decentralized coordination, while the lower section shows sources cited more often in the Classical centralized setting.

Figure 3: Effect of test time scaling on coordination performance. Valid (%) measures the fraction of successfully content-agent parsed requests.

Figure 4: Interaction efficiency across tasks in AgentWebBench. Turn is the number of agent responses. #Agent and #Req. are the numbers of contacted content agents and retrieval requests. Valid (%) is the success rate of parsing requests or returning results.

LLM	Web Search		Web Rec.		QA
	User	Content	User	Content	User	Content
Qwen3-4B	59.59	40.41	48.75	51.25	84.44	15.56
Qwen3-14B	64.83	35.17	62.59	37.41	84.21	15.79
Qwen3-30B-A3B	56.41	43.59	52.86	47.14	83.33	16.67
Qwen3-80B-A3B	54.90	45.10	60.85	39.15	93.94	6.06
DeepSeek-V3.2	35.18	64.82	48.57	51.43	70.27	29.73

Table 3: Failure mode (%) analysis of incorrect predictions. Each failure is attributed exclusively to either the user agent or the content agents based on task-specific rules. Percentages sum to 100% within each task.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Setting	#Cite	No-Ref.	Word ↓	Pos. ↓	Overall ↓
Qwen3-4B	Classic	2.93	0.60	0.1642	0.1569	0.1675
Qwen3-4B	Multi-Agent	2.63	31.12	0.0948	0.0931	0.0960
Qwen3-14B	Classic	2.97	1.81	0.1540	0.1443	0.1611
Qwen3-14B	Multi-Agent	2.67	1.51	0.0748	0.0766	0.0877
Qwen3-30B-A3B	Classic	3.35	2.42	0.1667	0.1589	0.1731
Qwen3-30B-A3B	Multi-Agent	2.53	0.60	0.1386	0.1315	0.1442

Appendix

Table 4: Content agent visibility on the deep research task, measured by the standard deviation of GEO scores across cited content agents. No-Ref. (%) reports the percentage of responses without any citation. Word, Pos, and Overall correspond to word-based, position-based, and combined GEO variants, respectively (four decimal places to identify fine-grained differences).

Figure 5: Impact of the different coordination strategies on web traffic. The charts illustrate the citation frequency of 20 most cited sources (websites or content agents) for Gemini-3 user agents on the deep research task. Each subplot is sorted vertically by the frequency difference (Tool ${}_{\text{E}}$ / Tool ${}_{\text{P}}$ / Multi-Agent minus Classical). The upper section highlights sources more frequently cited under the multi-agent coordination, while the lower section shows those preferred by Classical.

LLM	Method	Web Search				Web Recommendation				QA		Deep Research
		N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
DeepSeek V3.2 (non-think)	Classical	49.37	50.66	57.63	60.73	1.07	1.07	1.07	1.07	15.09	12.40	65.26	1.36	88.73	80.15
	$\text{Tool}_{\text{E}}$	32.99	34.63	37.01	40.96	0.36	0.36	0.36	0.36	16.98	8.57	66.17	1.91	90.18	85.71
	$\text{Tool}_{\text{P}}$	20.09	21.50	23.73	27.12	0.00	0.00	0.00	0.00	9.43	5.94	57.91	1.63	79.61	69.79
	Multi-Agent	12.26	12.39	13.28	13.56	0.00	0.00	0.00	0.00	16.98	15.58	58.13	1.81	80.06	70.12
DeepSeek V3.2 (think)	Classical	49.02	49.37	55.37	56.21	0.40	0.40	0.71	0.71	24.53	19.51	66.54	1.27	87.16	80.48

Appendix

Table 5: Performance comparison across four tasks under different LLM-based methods. Tasks include web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific. “-” indicates not applicable. Bold and underline indicate the best and second-best results, respectively.

Task	User Agent				Content Agent
	Turn	#Agent	#Req.	Valid (%)	Turn	Valid (%)
LLM: Qwen3-4B
Web Search	2.61	2.61	2.79	88.58	2.53	78.65
Web Rec.	7.63	6.02	9.88	84.88	2.69	53.99
Question Answering	3.38	1.36	2.06	91.37	3.83	47.62
Deep Research	4.61	3.60	4.05	96.20	2.24	52.54

Appendix

Table 6: Mean interaction efficiency across tasks in AgentWebBench for Qwen3-4B and Gemini-3. Turn indicates the number of agent-generated responses; #Agent and #Req. denote the average number of contacted content agents and retrieval requests, respectively. Valid (%) reports the success rate of requests or returns.

Short Name	Full Name	Max Input Length	Max Output Length	Thinking
Qwen1.5-14B	Qwen/Qwen1.5-14B	32,768	8,192	×
Qwen2.5-14B	Qwen/Qwen2.5-14B	131,072	8,192	×
Qwen3-4B-Instruct	Qwen/Qwen3-4B-Instruct-2507	262,144	8,192	×
Qwen3-4B	Qwen/Qwen3-4B-Thinking-2507	262,144	8,192	$\checkmark$
Qwen3-14B	Qwen/Qwen3-14B	32,768	8,192	$\leftrightarrows$
Qwen3-30B-A3B-Instruct	Qwen/Qwen3-30B-A3B-Instruct-2507	262,144	8,192	×

Appendix

Table 7: All LLMs used in experiments with maximum input and default output lengths, including thinking capability. The Thinking column indicates model capability: × = thinking mode not supported;

\checkmark

= thinking mode only (forced step-by-step reasoning);

\leftrightarrows

= supports switching/dynamic control between thinking and non-thinking/fast modes.

LLM	Web Search				Web Recommendation				QA		Deep Research
	N@3	N@5	R@3	R@5	N@3	N@5	R@3	R@5	Acc.	F1	KPR	KPC ↓	Clarity	Insight
Qwen1.5-14B	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	1.08	3.73	0.06	4.35	4.26
Qwen2.5-14B	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	5.66	4.01	7.32	0.18	6.74	6.89
Qwen3-14B	27.33	27.90	31.36	32.77	0.22	0.53	0.36	1.07	28.30	23.79	49.85	1.29	63.23	51.69

Appendix

Table 8: Performance comparison across four tasks under different version LLMs. Tasks include web search, web recommendation, question answering (QA), and deep research. Metrics (%) are task-specific.

Model	N@3	N@5	R@3	R@5
Qwen3-4B-QueryGen	0.00	0.00	0.00	0.00
Qwen3-14B-QueryGen	0.76	0.76	1.07	1.07
Qwen3-30B-A3B-QueryGen	0.18	0.32	0.36	0.71
Qwen3-80B-A3B-QueryGen	0.89	0.89	1.42	1.42
DeepSeek-V3.2-QueryGen	0.76	0.90	1.07	1.42
GPT-5-QueryGen	0.71	1.02	0.71	1.42

Appendix

Table 9: QueryGen performance (%) on our web recommendation setting. QueryGen is a non-agent baseline.

Model	Recall@10	NDCG@10	Recall@50	NDCG@50	Recall@100	NDCG@100
GPT-3.5-Turbo-QueryGen	0.88	0.58	1.56	0.73	2.54	0.89
GPT-4o-QueryGen	0.68	0.27	1.76	0.50	3.12	0.72
Gemini-2.5-Flash-QueryGen	0.68	0.42	1.46	0.58	2.64	0.77
GPT-4.1-QueryGen	1.07	0.50	1.95	0.68	2.54	0.77
Claude-Sonnet-4-QueryGen	0.68	0.32	1.66	0.52	2.15	0.60
DeepSeek-V3-QueryGen	1.27	0.82	2.64	1.11	3.71	1.29

Appendix

Table 10: Zero-shot ClueWeb-Reco benchmarking test results (%) on candidate item ranking ( He et al., 2025 ) .

Index	Name	Description	#Doc
1	www.amazon.com	Welcome to Amazon.com, the world’s largest online retailer and a pioneering force in e-commerce. Explore an unparalleled selection of products, from books and electronics to groceries and fashion, delivered conveniently to your door. Beyond shopping, Amazon also offers digital services like streaming media, cloud computing, and smart home devices.	1,749,804
2	en.wikipedia.org	en.wikipedia.org is the English-language version of Wikipedia, a free online encyclopedia that offers a vast and continually growing collection of knowledge. It is a massive collaborative project, with articles contributed and maintained by a global community of volunteers, covering nearly every topic imaginable.	1,406,613
3	stackoverflow.com	Stack Overflow is the definitive online community for developers to learn, share their knowledge, and build their careers. It’s a question-and-answer platform where millions of programmers find solutions to coding problems, ask new questions, and contribute their expertise. This vast, collaborative resource is indispensable for anyone working with code.	1,268,548
4	www.youtube.com	YouTube is the world’s largest online video-sharing platform, revolutionizing how we consume and create visual content. Millions of users visit daily to watch, upload, and share an incredibly diverse range of videos, from educational tutorials and music videos to vlogs and live streams. It serves as a global hub for creators to connect with audiences and for viewers to discover endless entertainment and information.	1,181,242
5	www.reddit.com	Reddit is a massive network of online communities where users from around the world converge to share content, discuss a myriad of topics, and engage in conversations. Often called “the front page of the internet,” it’s a dynamic platform where content is upvoted or downvoted by its users, shaping what becomes visible and popular across thousands of specialized subreddits.	654,802
6	www.quora.com	Quora is a leading question-and-answer platform where users can ask questions on virtually any topic and receive answers from a global community. It serves as a vast knowledge-sharing network, allowing individuals to learn, share insights, and connect with others based on shared interests and expertise.	651,607

Appendix

Table 11: The 100 websites in AgentWebBench with descriptions and document counts. Websites are selected from ClueWeb22-B based on document count, with each website assigned to one content agent. Descriptions are used by user agents for website selection in coordination strategies.

Web Agent Benchmarks

Web agent benchmarks test how reliably AI agents browse, navigate, and act on websites, including under long-horizon, ambiguous, and adversarial conditions.

See all 58 papers

Web$hell: A Unified Interface for Agentic Web Access
Oct 9, 2026Sunghwan Kim, Deogyong Kim, Sangam Lee +2Web Agents LLM Agent Harnesses
Yonsei University
WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks
Jul 1, 2025Zihao Sun, Zijing Shi, Ling ChenWeb Agent Benchmarks Multimodal Agents
University of Technology Sydney, Australia
AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
Aug 28, 2026Tejas Srinivasan, Shikib Mehri, Nandita Shankar Naik +3LLM Agent Evaluation Human-AI Collaboration
University of Southern California · Contextual AI · Capital One +1

Oct 9, 2026J/K move · Enter open · S save

Web$hell: A Unified Interface for Agentic Web Access

Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon+1

Yonsei University

Web Agents LLM Agent Harnesses Long-Horizon LLM Agents

Open paper AskPDF arXiv

Large language model (LLM) agents are becoming direct users of the open web. As they take on increasingly complex and information-intensive tasks, effective web access is becoming a core capability for agentic problem solving. Existing approaches have advanced agentic web access through different access models, including search-augmented agents, browser-based agents, and, more recently, filesystem-based agents. Nevertheless, as users expect agents to complete long-horizon, end-to-end tasks, they increasingly need to conduct broad information gathering, perform browser actions, and revisit acquired evidence within a single trajectory. Yet our preliminary analyses reveal that existing access models struggle to meet these demands. They primarily support either information gathering or browser interaction, while combining their capabilities can involve substantial costs or performance trade-offs. To this end, we present Web$hell, an interface that unifies web access under a single access model. Using a common schema defined by WebContract, WebCompiler compiles live webpages into addressable information and action objects. These objects are maintained in WebCache and handled through a shared set of operations, providing agents with consistent, operable, and persistent web access. Through extensive experiments across web search, web navigation, and deep research benchmarks, we show that Web$hell achieves strong task performance at lower cost. Our results further indicate the importance of interface design for effective web access, with Web$hell opening new possibilities for more capable and efficient LLM agents on the web.

Abstract

Web Agent Benchmarks

Explore similar work

Web$hell: A Unified Interface for Agentic Web Access

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics