cs.SEApr 1, 2026

Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering

Authors: Jingyue Li, André Storhaug

Organizations: Norwegian University of Science and Technology, Trondheim, Norway

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design description frequently renders the reproduction of results infeasible. To synthesize current evaluation practices for Agentic AI in SE, this study analyzes 18 papers on the topic, published or accepted by ICSE 2026, ICSE 2025, FSE 2025, ASE 2025, and ISSTA 2025. The analysis identifies prevailing approaches and their limitations in evaluating Agentic AI for SE, both in current research and potential future studies. To address these shortcomings, this position paper proposes a set of guidelines and recommendations designed to empower reproducible, explainable, and effective evaluations of Agentic AI in software engineering. In particular, we recommend that Agentic AI researchers make their Thought-Action-Result (TAR) trajectories and LLM interaction data, or summarized versions of these artifacts, publicly accessible. Doing so will enable subsequent studies to more effectively analyze the strengths and weaknesses of different Agentic AI approaches. To demonstrate the feasibility of such comparisons, we present a proof-of-concept case study that illustrates how TAR trajectories can support systematic analysis across approaches.

Figures & tables

Figure 1. The process to analyze TAR trajectories
Figure 2. Prompt template for summarizing individual run from iteration 0 to n.
Figure 3. Prompt template for comparative analyses between the agents on the same runs.
Figure 4. Prompt template for aggregate analysis of comparisons 1 to n.
Figure 5. Final results of the TAR trajectory analysis
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 516 papers

Explore similar work

CardsList
Jul 7, 2026

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contributors deeply embedded into collaborative development environments. Despite their accelerated adoption, existing evaluation techniques are limited due to their fragmented nature and distorted projection of true model capabilities, often obtained from hypothetical syntactic scenarios. This research aims to bridge this gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice. Our evaluation approach focuses on contamination-awareness, in-the-wild agentic behavior assessment, and trajectory-aware benchmarks and metrics capturing realistic coding contexts, human-aligned behavior, and model failure modes.

Razvan Mihai Popescu

Delft University of Technology Delft, The Netherlands

Software Engineering AgentsLLM Agent EvaluationAgent EvaluationAI Agent BenchmarksLLM Agent Reliability
AskPDF
Oct 2, 2026

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

Large language models (LLMs) are increasingly used in software engineering, including agentic systems that coordinate multiple agents, but impose higher computational and environmental costs. In this paper, we present a comprehensive empirical study of agentic LLM systems across five software engineering tasks: code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis. For each task, we compare LLM configurations that range from a non-agentic single-query baseline to multi-agent workflows, using six open-weight LLMs, two prompt strategies, and three hardware platforms. We assess each configuration in terms of accuracy, inference latency, and energy consumption. Our results reveal substantial trade-offs between agentic complexity and energy efficiency: multi-agent designs consume on average 6.36×\times as much energy and run 6.07×\times as long as the non-agentic baseline, with worst-case slowdowns of up to 160×\times for individual task--hardware pairs. Accuracy gains from additional agents are limited and task-specific: multi-agent improves average vulnerability-detection accuracy, but lightweight non-agentic and single-agent configurations still dominate the Pareto front, accounting for 59 of 66 Pareto-optimal configurations. Model and prompt choice act as task-specific levers whose effective direction varies between tasks rather than as global defaults. We translate these findings into design guidelines for sustainable, task-aware LLM-based development tools.

Merve Astekin, Yan Naing Tun, Arda Goknil +5

SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

Software Engineering AgentsLLM Agent EvaluationMulti-Agent LLM SystemsLLM Inference EfficiencyEnergy-Efficient ML
AskPDF
May 12, 2026

A Research Agenda on Agents and Software Engineering: Outcomes from the Rio A2SE Seminar

The rise of agentic AI is reshaping software engineering in two intertwined directions: agents are increasingly applied to support software engineering tasks, and Agentic AI systems themselves are complex systems that require re-thinking currently established software engineering practices. To chart a coherent research agenda covering the two directions, we organized the A2SE seminar in Rio de Janeiro, bringing together 18 experts from academia and industry. Through structured presentations, collaborative topic clustering, and focused group discussions, participants identified six thematic areas: Governance, Software Engineering for Agents, Agents for Software Architecture, Quality and Evaluation, Sustainability, and Code, and they prioritized short-term and long-term research directions for each. This paper presents the resulting community-driven, opinionated research agenda, offering the SE community a structured foundation for coordinating efforts at this critical juncture.

Davide Taibi, Henry Muccini, Karthik Vaidhyanathan +15

University of Southern Denmark, Vejle, Denmark · Software Engineering Research Center, IIIT Hyderabad, India · Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

Software Engineering AgentsAI Agent Governance
AskPDF