MASEval: Extending Multi-Agent Evaluation from Models to Systems

Cornelius Emde; Alexander Rubinstein; Anmol Goel; Ahmed Heakl; Sangdoo Yun; Seong Joon Oh; Martin Gubri

cs.AIMar 9, 2026

MASEval: Extending Multi-Agent Evaluation from Models to Systems

Authors: Cornelius Emde, Alexander Rubinstein, Anmol Goel, Ahmed Heakl, Sangdoo Yun, Seong Joon Oh, Martin Gubri

Organizations: Parameter Lab · University of Oxford · University of Tübingen · TU Darmstadt ·

Abstract

The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval.

Figures & tables

Panel of Figure 1 — Figure 1: MASEval provides a unified evaluation layer that enables framework-agnostic, system-level comparison across any agent framework and benchmark.

Library	Multi-Agent	System Eval	Agent-Agnostic	Benchmarks	Flexible Interaction	BYO	Trace-First	Mature
MASEval (Ours)	✓	✓	✓	✓	✓	✓	✓	✓
AnyAgent ( Mozilla AI, 2025 )	$\circ$	✓	✓	✗	$\circ$	✓	$\circ$	✓
MLflow GenAI ( Zaharia et al., 2018 )	$\circ$	$\circ$	✓	✗	$\circ$	✓	✓	✓
HAL Harness ( Kapoor et al., 2026 )	$\circ$	✓	✓	✓	$\circ$	$\circ$	$\circ$	$\circ$
Inspect-AI ( UK AI Safety Institute, 2024 )	$\circ$	✓	$\circ$	✓	$\circ$	$\circ$	$\circ$	✓
OpenCompass ( Contributors, 2023 )	✗	$\circ$	✗	✓	$\circ$	$\circ$	$\circ$	✓

Table 1: Comparison with related libraries. ✓ = built-in support or explicitly designed to be extensible,

\circ

= partial support, ✗ = not supported/impractical. Multi-Agent : native orchestration with per-agent tracing and independent message histories. System-Level : compare framework implementations, not just LLMs. Agent-Agnostic : evaluate any framework via thin adapters without code recreation. Benchmarks : ships complete, ready-to-run benchmarks. Flexible Interaction : Flexible Agent-Environment-User Interaction. First-class user simulation with personas and tool access. BYO : bring your own logging, agents, environments, and tools. Open-source, works offline, no mandatory cloud services. Trace-First : evaluate intermediate steps across environment and agents via first-class traces, not post-hoc fixes. Mature : published on PyPI, CI/CD, good test coverage, active maintenance.

Panel of Figure 2 — Figure 2: MASEval adopts a “Bring Your Own” (BYO) philosophy. Users implement custom components by extending MASEval’s abstract base classes ; the core runtime orchestrates execution and collects traces. This enables maximum flexibility while minimizing boilerplate code.

Panel of Figure 3 — Figure 3: Benchmark task lifecycle with flexible execution. The outer loop iterates over tasks; the inner loop handles repetitions. The Execute phase shows agent-user interaction as a flexible bidirectional loop.

		MACS $\uparrow$		ConVerse $\uparrow$		MultiAgentBench $\uparrow$
Framework	Model	Travel	Mortgage	Travel Planning	Real Estate	Research	Bargaining
smolagents	Gemini-3.0-Flash	84.0	94.4	84.8	82.4	99.0	89.0
	GPT-5-mini	59.8	85.8	90.1	84.2	98.1	91.7
	Haiku 4.5	90.4	85.6	86.2	86.0	100.0	90.2
LangGraph	Gemini-3.0-Flash	85.8	89.4	60.0	72.2	98.0	82.7
	GPT-5-mini	60.8	73.7	73.3	77.0	95.5	93.6

Table 2: Performance across frameworks, models, benchmarks, and domains. ConVerse columns shows

(1-\text{ASR})

on security split. MultiAgentBench Research reports Completion Rate and Bargaining reports Task Score. Bold marks the best result per task column. Cross-Model statistics measure variability across models within each framework, averaged over frameworks. Cross-Framework statistics measure the reverse.

				Change
Benchmark	Component	Original	MASEval	LoC	%
$\tau^{2}$ -Bench	Definition	6,822	3,450	$-$ 3,372	$-$ 49.4
	Interface	1,982	343	$-$ 1,639	$-$ 82.7
	Total	8,804	3,793	$-$ 5,011	$-$ 56.9
ConVerse	Definition	1,320	1,283	$-$ 37	$-$ 2.8
	Interface	778	71	$-$ 707	$-$ 90.9

Table 3: Lines of code comparison for benchmark evaluation logic. Definition covers task specifications, environments, and evaluators. Interface covers the CLI and entry points that wire components into a runnable benchmark. MASEval replaces framework- and benchmark-specific orchestration with shared abstractions.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Type	Domain
GAIA-2 ( Mialon et al., 2024 )	SA	Capability
$\tau^{2}$ -bench ( Barres et al., 2025 )	SA	Capability
MMLU ( Hendrycks et al., 2021 )	SA	Capability
MACS ( Shu et al., 2024 )	MA	Collaboration
MultiAgentBench ( Zhu et al., 2025 )	MA	Coordination & Competition
ConVerse ( Gomaa et al., 2025 )	MA	Safety & Security

Appendix

Table 4: Currently supported benchmarks. SA

=

Single-Agent, MA

=

Multi-agent.

Framework	Architecture
smolagents ( Roucher et al., 2025 )	Code-based tool calling
LangGraph ( LangChain, 2024a )	Stateful graph execution
LlamaIndex ( Liu, 2022 )	Async-first workflows
CAMEL ( Li et al., 2023 )	Role-playing multi-agent

Appendix

Table 5: Currently supported agentic frameworks. Each framework is integrated via a thin adapter that exposes a unified interface for execution, message history, tracing, and configuration capture.

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 519 papers

Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance
Feb 3, 2026Abdelghny Orogat, Ana Rostam, Essam MansourMulti-Agent LLM Systems Multi-Agent Orchestration
Concordia University
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration
Aug 9, 2026Guo Chen, Ziwen Li, Reed Li +4Multi-Agent LLM Systems AI Agent Evaluation
Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China

Feb 3, 2026J/K move · Enter open · S save

Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance

Abdelghny Orogat, Ana Rostam, Essam Mansour

Concordia University

Multi-Agent LLM Systems Multi-Agent Orchestration LLM Agent Evaluation AI Agent Benchmarks Software Architecture Design

Open paper AskPDF arXiv

Multi-agent LLM frameworks are data-intensive systems that govern how agents orchestrate tasks, manage state, and coordinate decisions. These architectural choices control execution overhead, memory behavior, planning effectiveness, and coordination scalability. Their impact on system performance remains poorly understood. Existing benchmarks evaluate individual agent capabilities in isolation and lack standardized framework-level comparison. We make four contributions. We introduce an architectural taxonomy that decomposes multi-agent LLM frameworks along five dimensions: orchestration, memory, planning interfaces, specialization, and communication topology. We develop MAFBench, a unified evaluation suite that integrates existing benchmarks within a standardized execution pipeline. We conduct a controlled empirical study across nine frameworks, fixing the underlying LLM and varying only architectural design choices. We distill the results into six evidence-based design principles. Architectural design, not only model intelligence, governs performance. Orchestration alone increases latency by over 60x, and a minimal implementation of the same paradigm isolates that cost as implementation rather than paradigm. Schema-constrained planning interfaces reduce accuracy by up to 32 points through formatting failures, not reasoning errors. Communication topology drops coordination success from above 90% to below 30% under mismatched structure. Memory architecture controls recall and scalability independent of context window size, and no evaluated framework natively supports controlled knowledge revision.

Abstract

AI Agent Evaluation

Explore similar work

Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration