A Baseline Assessment for the Agentic Web Technology Stack

Vagner Figueredo de Santana; Diogo Cortiz

cs.AIOct 9, 2026

A Baseline Assessment for the Agentic Web Technology Stack

Authors: Vagner Figueredo de Santana, Diogo Cortiz

Organizations: NIC.br, Brazil · NIC.br / PUC-SP, Brazil

Abstract

The fast pace in which protocols, frameworks, and technologies are being proposed for LLM-based agents results in a complex scenario for anyone aiming to create state-of-the-art AI solutions. Without an assessment on how different protocols, frameworks, and libraries work together to compose an Agentic Web solution, developers and organizations face different challenges (e.g., skilling, governance, infrastructure) even before becoming part of the Agentic Web ecosystem. In this context, we propose a system for performing Agentic Web evaluations called MASTIF (Multi-Agent System TestIng Framework) and present a baseline assessment of Agentic Web technologies, including 3 LLMs, 3 protocols, and 4 agent implementation frameworks, against 300 Mind2Web benchmark tasks. Results suggest that selecting a task-specific technology stack may result in gains of task completion rate, efficiency score, and social value stability scores; and reductions in verbosity and time to complete tasks. With the presented results, we expect to present a baseline in terms of metrics and results for this complex and fast evolving scenario for those aiming at designing, assessing, and developing multi-agent systems.

Figures & tables

Figure 1. Overview of Multi-Agent System TestIng Framework (MASTIF). Component diagram of the MASTIF (Multi-Agent System TestIng Framework), which combines frameworks, protocols, and frameworks.

Category	$TS_{model,protocol,framework}$
Models	$\bullet$ Mistral-7B-Instruct-v0.1 ( Jiang et al., 2023 ) $\bullet$ Qwen2.5-7B ( Team, 2024 ) $\bullet$ Llama-4-Scout-17B-16E-Instruct ( AI, 2025 )
Judge model	$\bullet$ GPT-4o-mini ( OpenAI, 2024 )
Protocols	$\bullet$ Standard call $\bullet$ A2A ( A2A, 2025 ) $\bullet$ MCP ( Anthropic, 2024 )
Frameworks	$\bullet$ LangGraph ( LangGraph, 2025 ) $\bullet$ LangChain ( LangChain, 2025 ) $\bullet$ CrewAI ( CrewAI, ) $\bullet$ Semantic Kernel ( SemanticKernel, 2025 )
Benchmark	$\bullet$ Mind2Web ( Deng et al., 2023b )

Table 1. Experiment setup overview.

$TS_{model,protocol,framework}$	$TU_{TS}$	$TA_{TS}$	$TCR_{TS}$ ( $\sigma$ )
Mistral-7B, standard, CrewAI	0.910	0.969	0.667 (0.186)
Mistral-7B, MCP, CrewAI	0.883	0.947	0.633 (0.222)
Mistral-7B, A2A, CrewAI	0.893	0.952	0.628 (0.196)
Mistral-7B, standard, LangChain	0.925	0.983	0.680 (0.207)
Mistral-7B, MCP, LangChain	0.918	0.987	0.659 (0.199)
Mistral-7B, A2A, LangChain	0.910	0.967	0.631 (0.215)

Table 2. Average task understanding (

TU_{TS}

), adherence (

TA_{TS}

), and task completion (

TCR_{TS}

) for different technology stacks, as provided by judge model given rubric detailed in Appendix C .

Figure 2. Task completion scores by model, protocol, and framework. Dunn’s Post-hoc pairwise significant differences are shown according to the following coding: *** for $p-value$ < 0.001, ** for $p-value$ < 0.01, and * for $p-value$ < 0.05. Task completion scores by model, protocol, and framework.

Figure 3. Convergence time in seconds grouped by model, protocol, and framework. Dunn’s Post-hoc pairwise significant differences are shown according to the following coding: *** for $p-value$ < 0.001, ** for $p-value$ < 0.01, and * for $p-value$ < 0.05. Convergence time by model, protocol, and framework.

Figure 4. Verbosity in tokens grouped by model, protocol, and framework. Dunn’s Post-hoc pairwise significant differences are shown according to the following coding: *** for $p-value$ < 0.001, ** for $p-value$ < 0.01, and * for $p-value$ < 0.05. Verbosity time by model, protocol, and framework.

Figure 5. Pareto analysis zoomed on the frontier region of technology stacks combining task completion, verbosity, and convergence time. Pareto analysis zoomed on the frontier region of technology stacks combining task completion, verbosity, and convergence time.

$TS_{model,protocol,framework}$	$\mathrm{CT_{TS}}(\sigma)$	$\mathrm{V}_{TS}(\sigma)$
Mistral-7B, standard, CrewAI	20.8 (13.3)	912.4 (273.6)
Mistral-7B, MCP, CrewAI	32.8 (19.1)	1186.6 (405.5)
Mistral-7B, A2A, CrewAI	32.8 (17.2)	1176.6 (332.1)
Mistral-7B, standard, LangChain	9.5 (3.2)	822.7 (212.5)
Mistral-7B, MCP, LangChain	13.1 (6.9)	1088.0 (210.1)
Mistral-7B, A2A, LangChain	13.1 (7.8)	1059.9 (177.8)

Table 3. Mean convergence time (

\mathrm{CT_{TS}}

) per task in seconds and mean verbosity

\mathrm{V}_{TS}

per task in tokens for different technology stacks.

Figure 6. Tool use success rate for Qwen2.5-7B by protocol and framework. Dunn’s Post-hoc pairwise significant differences are shown according to the following coding: *** for $p-value$ < 0.001, ** for $p-value$ < 0.01, and * for $p-value$ < 0.05. Tool use success rate by model, protocol, and framework.

$TS_{model,protocol,framework}$	$TUSR_{TS}$ ( $\sigma$ )
Qwen2.5-7B, standard, CrewAI	0.543 (0.442)
Qwen2.5-7B, MCP, CrewAI	0.523 (0.306)
Qwen2.5-7B, A2A, CrewAI	0.570 (0.330)
Qwen2.5-7B, standard, LangChain	0.638 (0.310)
Qwen2.5-7B, MCP, LangChain	0.670 (0.238)
Qwen2.5-7B, A2A, LangChain	0.577 (0.323)

Table 4. Tool use success rate for different technology stacks. Not shown technology stacks did not register tool usage.

Figure 7. Social value score by model, protocol, and framework. Dunn’s Post-hoc pairwise significant differences are shown according to the following coding: *** for $p-value$ < 0.001, ** for $p-value$ < 0.01, and * for $p-value$ < 0.05. Social value score by model, protocol, and framework.

Panel of Figure 8 — Figure 8. Comparison of mean frequency of social values retrieved by a human-curated prompt social value classifier ( Santana et al., 2025 ) . Input tasks (a, b) show that classifier is stable. Reasoning steps (c, d) show that intermediate reasoning steps are consistent reporting agents functioning, regardless of input. Social values represented in the final answer (e, f) are similar, differing in the ranking involving some values.

$TS_{model,protocol,framework}$	$SVS_{{TS}_{P,o_{a}}}$	$SVS_{{TS}_{reasoning}}$	$SVS_{TS}$ ( $\sigma$ )
Mistral-7B, standard, CrewAI	0.156	0.235	0.212 (0.0380)
Mistral-7B, MCP, CrewAI	0.129	0.228	0.189 ( 0.0116 )
Mistral-7B, A2A, CrewAI	0.105	0.228	0.189 (0.0123)
Mistral-7B, standard, LangChain	0.146	0.235	0.212 (0.0383)
Mistral-7B, MCP, LangChain	0.091	0.228	0.189 (0.0124)
Mistral-7B, A2A, LangChain	0.172	0.228	0.190 (0.0134)

Table 5. Social value stability considering values present in the prompt and final answer (

SVS_{{TS}_{P,o_{a}}}

), reasoning steps (

SVS_{{TS}_{reasoning}}

), and all steps from prompt, reasoning steps up to final answer (

SVS_{TS}

) for different technology stacks; the higher the better.

	Metric	Model	Protocol	Framework
Efficiency (RQ2) $~\Biggl\{$	Task Completion (RQ1)	Qwen2.5-7B	Standard	SemanticKernel
	Convergence Time	Qwen2.5-7B	MCP	LangChain
	Verbosity	Qwen2.5-7B	Standard	LangChain
	Tool Use Success (RQ3)	Qwen2.5-7B *	MCP	SemanticKernel
	Social Value Stability (RQ4)	Mistral-7B	A2A	SemanticKernel

Table 6. Best-performing technology stacks per metric (consolidated from Sections 5.1 to 5.4) and respective research questions (from RQ1 to RQ4)

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Short description	License	Stars	Forks	Contributors
AutoGPT ( Significant Gravitas, 2025 )	Framework for managing GPT-4 agents.	PolyForm 1.0 and MIT	179k	46k	781
LangChain ( LangChain, 2025 )	Framework for building applications through composability.	MIT	133k	21.9k	3,914
AutoGen ( Wu et al., 2023 )	Framework for creating MAS that can act autonomously or work alongside humans.	CC-BY-4.0	56.8k	8.5k	558
LlamaIndex ( Liu, 2022 )	Data framework for LLM application allowinf for covering core and plugin components.	MIT	48.4k	7.2k	1,855
Mem0 ( Chhikara et al., 2025 )	Memory layer for agents.	Apache 2.0	52.2k	5.8k	297
CrewAI ( CrewAI, )	Framework for orchestrating role-playing MAS.	MIT	48.3k	6.6k	302

Appendix

Table 7. LLM-agnostic python agent frameworks (sorted by GitHub stars).

Benchmark	Task	Environment	Modalities
AgentBench ( Liu et al., 2023 )	Multi-environment evaluation of LLMs acting as agents	Mixed (APIs, simulated env.)	Actions
AgentEval ( Arabzadeh et al., 2024 )	Multi-dimensional utility assessment framework for LLM-powered applications	Offline simulation-based task evaluation	Text, Solution traces
Agent-SafetyBench ( Zhang et al., 2025 )	Tasks to assess safety of agents	Simulated interaction env.	Text
AgentSims ( Lin et al., 2023 )	Sandbox for creating agent evaluation tasks	Simulated GUI sandbox	Text, Actions
ALFWorld ( Shridhar et al., 2020 )	Align text-based abstractions with embodied environments	Simulated	Text, Vision, Actions
ClemBench ( Chalamalasetti et al., 2023 )	Game-based interactive tests for chat-optimized LLMs	Simulated games	Text

Appendix

Table 8. Benchmarks used in agent evaluations (alphabetical order).

Web Agents

Web agents use language and vision models to navigate websites, gather information, and complete tasks through interactive interfaces.

See all 174 papers

MASEval: Extending Multi-Agent Evaluation from Models to Systems
Mar 9, 2026Cornelius Emde, Alexander Rubinstein, Anmol Goel +4AI Agent Evaluation AI Agent Benchmarks
Parameter Lab · University of Oxford · University of Tübingen +4
AI Planning Framework for LLM-Based Web Agents
Mar 13, 2026Orit Shahnovsky, Rotem DrorLLM Agent Planning Web Agents
Faculty of Computer and Information Science, University of Haifa, Haifa, Israel
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1

Mar 9, 2026J/K move · Enter open · S save

MASEval: Extending Multi-Agent Evaluation from Models to Systems

Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl+3

Parameter Lab · University of Oxford · University of Tübingen+4

AI Agent Evaluation AI Agent Benchmarks LLM Agent Orchestration

Open paper AskPDF arXiv

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

Abstract

Web Agents

Explore similar work

MASEval: Extending Multi-Agent Evaluation from Models to Systems

AI Planning Framework for LLM-Based Web Agents

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities