Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

Merve Astekin; Yan Naing Tun; Arda Goknil; Erik Johannes Husom; Lwin Khin Shar; Hasan Sözer; Ratnadira Widyasari; Hui Song

cs.SEOct 2, 2026

Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows

Authors: Merve Astekin, Yan Naing Tun, Arda Goknil, Erik Johannes Husom, Lwin Khin Shar, Hasan Sözer, Ratnadira Widyasari, Hui Song

Organizations: SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye

Abstract

Large language models (LLMs) are increasingly used in software engineering, including agentic systems that coordinate multiple agents, but impose higher computational and environmental costs. In this paper, we present a comprehensive empirical study of agentic LLM systems across five software engineering tasks: code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis. For each task, we compare LLM configurations that range from a non-agentic single-query baseline to multi-agent workflows, using six open-weight LLMs, two prompt strategies, and three hardware platforms. We assess each configuration in terms of accuracy, inference latency, and energy consumption. Our results reveal substantial trade-offs between agentic complexity and energy efficiency: multi-agent designs consume on average 6.36 $\times$ as much energy and run 6.07 $\times$ as long as the non-agentic baseline, with worst-case slowdowns of up to 160 $\times$ for individual task--hardware pairs. Accuracy gains from additional agents are limited and task-specific: multi-agent improves average vulnerability-detection accuracy, but lightweight non-agentic and single-agent configurations still dominate the Pareto front, accounting for 59 of 66 Pareto-optimal configurations. Model and prompt choice act as task-specific levers whose effective direction varies between tasks rather than as global defaults. We translate these findings into design guidelines for sustainable, task-aware LLM-based development tools.

Figures & tables

Dimension	Experimental Setup
Software engineering tasks	Code generation, technical debt identification, code vulnerability detection, log parsing, and log analysis.
Datasets	HumanEval (code generation), MLCQ (technical debt detection), PrimeVul (code vulnerability detection), and LogHub’s HDFS (log parsing and log analysis).
System configurations	Non-agentic single-query models, single-agent setups, dual-agent configurations, and multi-agent workflows.
Hardware platforms	Server-SMU (8× NVIDIA H100), Workstation-SMU (RTX A2000), and Server-STF (2× RTX 6000 Ada).
Models	gemma3:4b, codegemma:7b-instruct, qwen3:4b-instruct, qwen2.5-coder:7b-instruct, deepseek-coder:6.7b-instruct, and gpt-oss:20b (Server-SMU only).
Prompting strategies	Zero-shot and few-shot prompting (3 task-specific examples per prompt).

Table 1. Overview of the empirical study design, including the software engineering tasks and datasets, agentic configurations, LLMs, hardware platforms, prompting strategies, measurement and execution protocols, statistical analyses, and their mapping to the four RQs. The experimental campaign comprises 1,800 runs from the core factorial design and 40 additional GPT-OSS runs on Server-SMU, for 1,840 runs in total.

Smell Type	Function Level	Class Level	Total
None	67	67	134
Blob	-	51	51
Data Class	-	67	67
Long Method	67	-	67
Feature Envy	66	-	66
Total			385

Table 2. Composition of the cleaned MLCQ evaluation subset by code-smell class and analysis granularity. Function-level samples include Long Method, Feature Envy, and smell-free functions; class-level samples include Blob, Data Class, and smell-free classes. “None” denotes samples without a target smell, and dashes indicate that a smell category is not applicable at the corresponding granularity. After duplicate removal and ambiguity filtering, the subset contains 385 samples in total.

Task	Dataset	Samples	Source
Code Generation	HumanEval	164	( Chen et al., 2021 )
Technical Debt	MLCQ	385	( Madeyski and Lewowski, 2020 )
Vulnerability Detection	PrimeVul Pair	386	( Ding et al., 2025 )
Log Parsing	HDFS-v1	385	( Zhu et al., 2023 )
Log Analysis	HDFS-v1	385	( Zhu et al., 2023 )

Table 3. Datasets and evaluation-set sizes for the five software engineering tasks. Sample counts denote HumanEval programming problems, cleaned MLCQ code snippets, PrimeVul Pair functions (193 vulnerable and 193 benign), HDFS-v1 log messages for log parsing, and HDFS-v1 block-ID sessions for log analysis. Although both HDFS tasks contain 385 samples, they use different units of analysis; the log-analysis subset was stratified to preserve the original normal-to-anomalous session distribution.

Figure 1. Overview of the four LLM system configurations evaluated in this study, ordered by increasing agentic complexity: non-agentic (NA), a direct stateless LLM query; single-agent (SA), one autonomous agent; dual-agent (DA), two collaborating task-specific agents; and multi-agent (MA), a coordinated four-agent workflow.

Figure 2. Sequence diagrams of the evaluated LLM system configurations. The non-agentic baseline issues a direct stateless query, whereas the single-agent configuration delegates the task to one autonomous agent. The dual-agent configuration adds a second agent for critique, refinement, or complementary analysis, and the multi-agent configuration coordinates four task-specific roles through a staged workflow that produces the final decision. Agent roles are instantiated separately for each software engineering task.

Configuration	# Agents	Agentic	Key Characteristics
Non-Agentic (NA)	0	No	Direct LLM query, no state
Single Agent (SA)	1	Yes	Autonomous task completion
Dual Agent (DA)	2	Yes	Task-specific two-agent collaboration
Multi-Agent (MA)	4	Yes	Task-specific coordinated workflow

Table 4. LLM system configurations evaluated in the study, ordered by increasing agentic complexity. The “# Agents” column counts active LLM-based agents, while the concrete DA and MA roles vary by software engineering task as described in Section 3.3 .

Machine	CPU	Memory (GB)	GPU	GPU memory (GB)
Server-SMU	Intel Xeon Platinum 8480C	2048	8 $\times$ NVIDIA H100	80 (HBM3) $\times 8$
Workstation-SMU	Intel Xeon W-2223 8-core @ 3.60 GHz	128	NVIDIA RTX A2000	12
Server-STF	Intel Xeon w9-3475X 36-core @ 4.8GHz	503	2 $\times$ NVIDIA RTX 6000 Ada	48 $\times 2$

Table 5. Hardware platforms used for experimental data collection, spanning two server-grade systems and one workstation-class system. The table reports the installed CPU, system memory, GPU configuration, and accelerator memory for each platform. GPU-memory values are given per accelerator, and the multiplicity indicates the number of installed GPUs rather than necessarily the number used concurrently by each experiment.

Measurement Level	Data Source / Tool	What It Captures	Reporting Unit
CPU Energy	CodeCarbon (Intel RAPL)	Power draw of CPU package(s) during execution	kWh (aggregated per run)
GPU Energy	CodeCarbon (NVML interface)	Power draw of GPU(s) at 1-second intervals	kWh (aggregated per run)
Total Pipeline Energy	CodeCarbon (combined metrics)	Sum of CPU, GPU, and memory energy consumption over the run	kWh

Table 6. Energy-measurement configuration used in the experiments. CodeCarbon attributes CPU energy through Intel RAPL, GPU energy through NVIDIA NVML sampled at one-second intervals, and total pipeline energy as the combined CPU, GPU, and memory consumption recorded during each run. All values are reported in kWh using tracking_mode="process" to estimate the energy attributable to the experimental process rather than the full machine.

Task	Dataset	Accuracy Metric(s)
Code Generation	HumanEval ( Chen et al., 2021 )	Pass@k
Technical Debt Identification	MLCQ Dataset ( Madeyski and Lewowski, 2020 )	Accuracy, Macro F1, Per-Class F1
Code Vulnerability Detection	PrimeVul ( Ding et al., 2025 )	Accuracy, Precision, Recall, F1-score
Log Parsing	LogHub ( Zhu et al., 2023 )	Parsing Accuracy (Exact Match), Edit Similarity, Longest Common Subsequence Similarity
Log Analysis	LogHub ( Zhu et al., 2023 )	Accuracy, Precision, Recall, F1-score

Table 7. Task-specific effectiveness metrics and corresponding evaluation datasets. Higher values indicate better performance for all reported metrics.

Figure 3. Overview of experimental workflow showing model initialization, execution, and measurement phases

Figure 4. RQ1 task-specific effectiveness across LLM system configurations. Bars and error bars report the mean

\pm

one standard deviation across model–prompt combinations after first averaging over the available hardware platforms within each combination. Performance is measured using Pass@1 for code generation, Macro F1 for technical debt detection, F1-score for vulnerability detection and log analysis, and exact-match parsing accuracy for log parsing. NA denotes the non-agentic direct-query baseline, SA the single-agent configuration, DA the dual-agent configuration, and MA the four-agent multi-agent configuration.

Configuration 1	Configuration 2	$p$ -value	Significant	Rank-biserial $r$
DA	MA	0.0122	Yes	-0.795
DA	NA	0.0122	Yes	-0.795
DA	SA	0.0161	Yes	-0.769
MA	NA	0.339	No	+0.333
MA	SA	0.339	No	+0.333
NA	SA	0.677	No	+0.154

Table 8. RQ1 pairwise Wilcoxon signed-rank comparisons among the non-agentic (NA), single-agent (SA), dual-agent (DA), and four-agent multi-agent (MA) configurations for vulnerability-detection F1-score, following the significant omnibus Friedman test. Repeated runs are first averaged within each configuration, and hardware is then averaged within each task–agent–model–prompt combination, yielding paired hardware-balanced model–prompt observations. Statistical significance is assessed at

p<0.05

. Matched-pairs rank-biserial correlation (

r

) reports effect size and direction: positive values favor Configuration 1, negative values favor Configuration 2, and larger absolute values indicate stronger differences.

Configuration 1	Configuration 2	$p$ -value	Significant	Rank-biserial $r$
DA	MA	0.0049	Yes	-0.872
DA	NA	0.850	No	+0.077
DA	SA	0.910	No	+0.051
MA	NA	0.0210	Yes	+0.744
MA	SA	0.0771	No	+0.590
NA	SA	0.424	No	+0.282

Table 9. RQ2 pairwise Wilcoxon signed-rank comparisons of total pipeline energy consumption among the non-agentic (NA), single-agent (SA), dual-agent (DA), and four-agent multi-agent (MA) configurations for vulnerability detection, following the significant omnibus Friedman test. Repeated runs are first averaged within each configuration, and hardware is then averaged within each agent–model–prompt combination, yielding paired hardware-balanced model–prompt observations. Statistical significance is assessed at

p<0.05

. Matched-pairs rank-biserial correlation (

r

) reports effect size and direction: positive values indicate higher energy consumption for Configuration 1, negative values indicate higher consumption for Configuration 2, and larger absolute values indicate stronger differences.

Task	Metric	Factor	Test	$p$ -value
Code Generation	Pass@1	Agent Design	Friedman	0.830
	Pass@1	Model	Friedman	0.0018
	Pass@1	Prompt Type	Wilcoxon	$8.65\times 10^{-4}$
	Pass@1	Hardware	Friedman	0.0387
	Energy	Agent Design	Friedman	$3.50\times 10^{-5}$
	Energy	Model	Friedman	0.0077

Table 10. Factors showing paired-test differences across tasks. Multi-level factors use Friedman tests, while two-level prompt comparisons use Wilcoxon signed-rank tests. Agent, model, and prompt tests use hardware-balanced observations; hardware tests use configuration means because hardware cannot be tested after it has been averaged out. Significant differences (

p<0.05

) are highlighted in bold.

Task	Server-SMU		Server-STF		Workstation-SMU
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Code Generation (Pass@1)	96.4 $\pm$	4.7	94.1 $\pm$	15.9	93.9 $\pm$	10.9
Technical Debt Detection (Macro F1)	38.4 $\pm$	11.7	38.0 $\pm$	11.6	38.1 $\pm$	11.5
Vulnerability Detection (F1-score)	46.5 $\pm$	17.1	44.5 $\pm$	18.8	45.9 $\pm$	17.2
Log Parsing (Parsing Accuracy)	27.9 $\pm$	31.2	28.6 $\pm$	31.9	28.2 $\pm$	31.6
Log Analysis (F1-score)	6.8 $\pm$	3.1	6.6 $\pm$	3.0	6.1 $\pm$	2.5

Table 11. RQ1 task-specific effectiveness across hardware platforms on the hardware-balanced subset. Values report mean

\pm

standard deviation over the agent configurations, models, and prompting strategies available on all three platforms; GPT-OSS is excluded.

Figure 5. RQ1 hardware effects on task-specific performance using the hardware-balanced subset of configurations. Each cell reports the deviation, in percentage points, of a hardware platform’s mean score from the mean across the three platforms for the corresponding task; positive and negative values indicate performance above and below the task mean, respectively. Srv-SMU, Srv-STF, and WS-SMU denote Server-SMU, Server-STF, and Workstation-SMU. Aggregates include only the agent configurations, models, and prompting strategies available on all three platforms.

Task	Zero-shot		Few-shot
	mean $\pm$	std	mean $\pm$	std
Code Generation (Pass@1)	97.3 $\pm$	6.4	92.7 $\pm$	14.0
Technical Debt Detection (Macro F1)	40.2 $\pm$	12.2	38.7 $\pm$	12.4
Vulnerability Detection (F1-score)	45.8 $\pm$	18.0	46.8 $\pm$	16.7
Log Parsing (Parsing Accuracy)	3.4 $\pm$	6.9	54.3 $\pm$	24.1
Log Analysis (F1-score)	7.2 $\pm$	3.2	5.7 $\pm$	2.2

Table 12. RQ1 effect of prompting strategy on task-specific effectiveness. Values report mean

\pm

standard deviation for zero-shot and three-example few-shot prompting, averaged over agent configurations, models, and hardware platforms.

Task	CG-7B		DS-6.7B		G3-4B		GPT-OSS		Q2.5C-7B		Q3-4B
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Code Generation (Pass@1)	95.6 $\pm$	4.7	88.8 $\pm$	19.8	99.8 $\pm$	0.5	97.9 $\pm$	1.9	98.0 $\pm$	3.0	91.8 $\pm$	12.8
Technical Debt Detection (Macro F1)	29.7 $\pm$	2.9	23.3 $\pm$	3.2	52.6 $\pm$	2.6	58.6 $\pm$	3.7	47.2 $\pm$	3.8	37.9 $\pm$	6.2
Vulnerability Detection (F1-score)	35.1 $\pm$	19.9	40.4 $\pm$	18.2	51.1 $\pm$	9.2	56.5 $\pm$	5.4	43.9 $\pm$	19.1	57.8 $\pm$	9.8
Log Parsing (Parsing Accuracy)	24.9 $\pm$	30.8	15.1 $\pm$	28.1	28.0 $\pm$	29.1	38.1 $\pm$	27.5	42.2 $\pm$	37.3	30.9 $\pm$	26.4
Log Analysis (F1-score)	7.0 $\pm$	3.7	5.2 $\pm$	1.3	5.6 $\pm$	0.6	5.1 $\pm$	1.5	8.5 $\pm$	3.4	6.4 $\pm$	3.0

Table 13. RQ1 task-specific effectiveness by LLM model. Values report mean

\pm

standard deviation across agent configurations, prompting strategies, and available hardware platforms. Performance is measured using Pass@1 for code generation, Macro F1 for technical debt detection, F1-score for vulnerability detection and log analysis, and exact-match parsing accuracy for log parsing. GPT-OSS results include only Server-SMU.

Figure 6. RQ1 task-specific effectiveness by LLM model and system configuration. Rows denote CodeGemma-7B (CG-7B), DeepSeek-Coder-6.7B (DS-6.7B), Gemma-3-4B (G3-4B), GPT-OSS-20B (GPT-OSS), Qwen2.5-Coder-7B (Q2.5C-7B), and Qwen3-4B (Q3-4B). Within each task panel, columns denote the non-agentic direct-query baseline (NA), single-agent (SA), dual-agent (DA), and four-agent multi-agent (MA) configurations. Each cell reports the corresponding task-specific performance metric as a percentage. Values are averaged over prompting strategies and the hardware platforms available for the corresponding model; GPT-OSS includes only Server-SMU measurements. Higher values indicate better performance.

Smell Class	Mean F1	Std	Best Agent	Best Model	Best Prompt
No Smell	55.8	14.7	NA (57.4)	G3-4B (69.6)	Zero-shot (56.9)
Blob	30.0	20.8	NA (32.8)	GPT-OSS (61.2)	Zero-shot (32.2)
Data Class	40.6	26.5	MA (41.7)	GPT-OSS (75.3)	Few-shot (41.3)
Feature Envy	8.1	7.2	DA (10.1)	CG-7B (13.1)	Zero-shot (10.1)
Long Method	62.7	22.1	SA (66.1)	GPT-OSS (88.7)	Few-shot (63.8)

Table 14. RQ1 technical-debt-specific per-class results: overall F1 score for each smell class, together with the best agent configuration, model, and prompting strategy. NA denotes the non-agentic direct-query baseline, SA the single-agent configuration, DA the dual-agent configuration, and MA the four-agent multi-agent configuration. Values in parentheses report the corresponding per-class F1 score (%).

Metric	NA		SA		DA		MA		Zero-shot		Few-shot
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Parsing Accuracy	34.5 $\pm$	36.6	31.2 $\pm$	34.1	27.6 $\pm$	28.6	26.1 $\pm$	26.6	3.4 $\pm$	6.9	54.3 $\pm$	24.1
Edit Similarity	69.6 $\pm$	25.4	64.0 $\pm$	27.1	63.3 $\pm$	28.7	65.0 $\pm$	25.5	46.7 $\pm$	16.1	80.8 $\pm$	22.4
LCS Similarity	90.4 $\pm$	9.3	89.2 $\pm$	10.0	90.5 $\pm$	7.3	90.6 $\pm$	7.9	84.8 $\pm$	9.7	94.7 $\pm$	3.8

Table 15. RQ1 log parsing metric sensitivity: exact-match parsing accuracy and partial template similarity metrics (mean

\pm

std in %). The first four columns compare agent configurations using the main RQ1 agent aggregation; the final two columns compare prompt strategies using the main RQ1 prompt aggregation.

Figure 7. RQ2 mean energy consumption across software engineering tasks, agentic configurations, and hardware platforms. Rows represent the non-agentic direct-query baseline (NA), single-agent (SA), dual-agent (DA), and four-agent multi-agent (MA) configurations; columns represent code generation (CodeGen), technical debt detection (TechDebt), vulnerability detection (Vuln), log parsing (LogParse), and log analysis (LogAnal). Srv-SMU, Srv-STF, and WS-SMU denote Server-SMU, Server-STF, and Workstation-SMU, respectively. Each cell reports mean total pipeline energy consumption in kWh, aggregated over the available models, prompting strategies, and repetitions for that task–configuration–hardware combination. Color intensity is mapped logarithmically to accommodate the large range of energy values, whereas the cell annotations report the untransformed means.

Hardware	NA		SA		DA		MA		Avg.
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Server-SMU	0.162 $\pm$	0.006	0.166 $\pm$	0.012	0.315 $\pm$	0.031	0.588 $\pm$	0.090	0.308 $\pm$	0.048
Server-STF	0.132 $\pm$	0.010	0.173 $\pm$	0.007	0.499 $\pm$	0.021	1.210 $\pm$	1.714	0.504 $\pm$	0.857
Workstation-SMU	0.082 $\pm$	0.014	0.120 $\pm$	0.022	0.380 $\pm$	0.020	0.686 $\pm$	0.114	0.317 $\pm$	0.059
Overall	0.128 $\pm$	0.011	0.154 $\pm$	0.015	0.392 $\pm$	0.025	0.813 $\pm$	0.993	0.372 $\pm$	0.497

Table 16. RQ2: Average energy consumption (kWh) by hardware platform and agent configuration (mean

\pm

std). Hardware rows average the task-level means shown in Figure 7 ; the Overall row averages over all task, model, prompt, and hardware configurations. Lowest agent mean per row is bold.

Figure 8. RQ2 mean process-attributed energy per run, decomposed into GPU, CPU, and RAM components for the three hardware platforms using the hardware-balanced subset of configurations (GPT-OSS excluded). Server-SMU is equipped with eight NVIDIA H100 GPUs with 80 GB HBM3 each, Server-STF with two NVIDIA RTX 6000 Ada GPUs with 48 GB each, and Workstation-SMU with one NVIDIA RTX A2000 GPU with 12 GB. Bar height reports mean total pipeline energy in kWh, while segment labels report each component’s share of the measured total. These percentages represent CodeCarbon energy attribution and should not be interpreted as device-utilization rates.

Task	Hardware	Zero-shot		Few-shot
		mean $\pm$	std	mean $\pm$	std
Code Generation	Server-SMU	0.157 $\pm$	0.001	0.164 $\pm$	0.002
	Server-STF	0.149 $\pm$	0.003	0.485 $\pm$	1.691
	Workstation-SMU	0.089 $\pm$	0.044	0.196 $\pm$	0.047
Technical Debt Detection	Server-SMU	0.071 $\pm$	0.007	0.046 $\pm$	0.007
	Server-STF	0.141 $\pm$	0.008	0.094 $\pm$	0.011

Table 17. RQ2 effect of prompting strategy on total pipeline energy consumption by task and hardware platform. Values report mean

\pm

standard deviation in kWh per run for zero-shot and three-example few-shot prompting, aggregated over the available models and agent configurations.

Task	Hardware	CG-7B		DS-6.7B		G3-4B		GPT-OSS		Q2.5C-7B		Q3-4B
		mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
CodeGen	Server-SMU	0.045 $\pm$	0.001	0.230 $\pm$	0.001	0.062 $\pm$	0.001	0.438 $\pm$	0.000	0.061 $\pm$	0.000	0.127 $\pm$	0.002
	Server-STF	0.090 $\pm$	0.002	0.449 $\pm$	0.002	0.129 $\pm$	0.008	— $\pm$	—	0.126 $\pm$	0.006	0.792 $\pm$	2.673
	Workstation-SMU	0.071 $\pm$	0.000	0.420 $\pm$	0.002	0.018 $\pm$	0.005	— $\pm$	—	0.100 $\pm$	0.000	0.104 $\pm$	0.102
TechDebt	Server-SMU	0.016 $\pm$	0.000	0.210 $\pm$	0.001	0.018 $\pm$	0.010	0.070 $\pm$	0.000	0.020 $\pm$	0.000	0.017 $\pm$	0.011
	Server-STF	0.034 $\pm$	0.000	0.443 $\pm$	0.013	0.047 $\pm$	0.015	— $\pm$	—	0.032 $\pm$	0.001	0.032 $\pm$	0.007

Table 18. RQ2 total pipeline energy consumption by task, LLM model, and hardware platform. Values report mean

\pm

standard deviation in kWh per run, averaged over agent configurations, prompting strategies, and available repetitions. Dashes indicate unsupported model–hardware combinations. GPT-OSS was evaluated only once per condition on Server-SMU; therefore, its reported zero standard deviation does not indicate zero variability.

Figure 9. RQ2 relationship between model size and mean total pipeline energy consumption on Server-SMU, the only platform on which all six models were evaluated. Each point represents a model’s mean energy per run, averaged across the five tasks, four agentic configurations, and two prompting strategies; the vertical axis is logarithmic, while the plotted values remain in kWh. Energy does not scale monotonically with parameter count: CG-7B is the lowest-energy model, whereas the similarly sized Q2.5C-7B consumes 6.4

\times

more energy on average. GPT-OSS was evaluated once per condition, while the other models used three repetitions.

Task	Hardware	NA		SA		DA		MA
		mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Code Generation	Server-SMU	726.7 $\pm$	15.8	712.0 $\pm$	9.8	1403.3 $\pm$	19.9	3935.5 $\pm$	36.5
	Server-STF	955.6 $\pm$	6.5	1023.7 $\pm$	3.2	1655.8 $\pm$	55.3	2220.4 $\pm$	570.4
	Workstation-SMU	2672.6 $\pm$	142.8	2307.0 $\pm$	1519.3	4978.8 $\pm$	561.3	7538.7 $\pm$	2300.5
Technical Debt Detection	Server-SMU	199.6 $\pm$	2.1	289.5 $\pm$	9.0	713.6 $\pm$	6.7	1175.9 $\pm$	11.1
	Server-STF	222.2 $\pm$	2.4	367.9 $\pm$	1.8	951.4 $\pm$	94.5	1785.8 $\pm$	72.9

Table 19. RQ3 end-to-end latency by task, hardware platform, and LLM system configuration. Values report mean

\pm

standard deviation in seconds, aggregated over available models, prompting strategies, and repetitions. NA, SA, DA, and MA denote the non-agentic, single-agent, dual-agent, and four-agent configurations, respectively; the lowest mean in each task–hardware row is shown in bold.

Hardware	NA		SA		DA		MA		Avg.
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
Server-SMU	1826.1 $\pm$	27.2	1821.8 $\pm$	110.5	3354.3 $\pm$	211.7	6289.3 $\pm$	1581.7	3322.9 $\pm$	799.9
Server-STF	949.9 $\pm$	69.6	1182.3 $\pm$	32.6	3288.5 $\pm$	70.9	6220.0 $\pm$	487.9	2910.2 $\pm$	249.5
Workstation-SMU	2585.0 $\pm$	707.6	3690.4 $\pm$	690.4	11694.3 $\pm$	547.5	21002.2 $\pm$	3977.7	9743.0 $\pm$	2067.6
Overall	1789.5 $\pm$	410.8	2205.9 $\pm$	404.1	5940.0 $\pm$	341.4	10865.5 $\pm$	2487.4	5200.2 $\pm$	1288.0

Table 20. RQ3: Average inference latency (seconds) by hardware platform and agent configuration (mean

\pm

std). Hardware rows average the task-level means shown in Figure 10 ; the Overall row averages over all task, model, prompt, and hardware configurations. Lowest agent mean per row is bold.

Figure 10. RQ3 mean end-to-end latency across software engineering tasks, agentic configurations, and hardware platforms. Rows represent the non-agentic direct-query baseline (NA), single-agent (SA), dual-agent (DA), and four-agent multi-agent (MA) configurations; columns represent code generation (CodeGen), technical debt detection (TechDebt), vulnerability detection (Vuln), log parsing (LogParse), and log analysis (LogAnal). Srv-SMU, Srv-STF, and WS-SMU denote Server-SMU, Server-STF, and Workstation-SMU, respectively. Each cell reports mean latency in seconds, averaged over the available models, prompting strategies, and repetitions for the corresponding task–configuration–hardware combination. Latency covers model loading, inference, and agent orchestration. Color intensity is mapped logarithmically to accommodate the wide range of values, whereas cell annotations report the untransformed arithmetic means.

Task	Hardware	Zero-shot		Few-shot
		mean $\pm$	std	mean $\pm$	std
Code Generation	Server-SMU	1736.2 $\pm$	18.5	1652.6 $\pm$	26.4
	Server-STF	1056.2 $\pm$	16.3	1871.5 $\pm$	404.9
	Workstation-SMU	2730.7 $\pm$	1358.3	6017.9 $\pm$	1457.0
Technical Debt Detection	Server-SMU	707.9 $\pm$	9.1	481.4 $\pm$	6.7
	Server-STF	968.2 $\pm$	3.8	695.5 $\pm$	84.3

Table 21. RQ3 effect of prompting strategy on end-to-end latency by task and hardware platform. Values report mean

\pm

standard deviation in seconds per run for zero-shot and three-example few-shot prompting, aggregated over the available models and agent configurations.

Task	Hardware	CG-7B		DS-6.7B		G3-4B		GPT-OSS		Q2.5C-7B		Q3-4B
		mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
CodeGen	Server-SMU	407.3 $\pm$	10.0	1952.5 $\pm$	20.4	878.4 $\pm$	43.1	4982.7 $\pm$	0.0	574.3 $\pm$	11.5	1371.0 $\pm$	9.5
	Server-STF	627.3 $\pm$	5.7	2760.9 $\pm$	1.6	1085.9 $\pm$	68.5	— $\pm$	—	859.9 $\pm$	15.0	1985.3 $\pm$	636.9
	Workstation-SMU	2179.7 $\pm$	7.8	12862.7 $\pm$	66.7	545.6 $\pm$	159.3	— $\pm$	—	3074.8 $\pm$	12.8	3208.5 $\pm$	3144.8
TechDebt	Server-SMU	183.9 $\pm$	3.2	1889.6 $\pm$	10.6	282.5 $\pm$	10.9	824.4 $\pm$	0.0	225.2 $\pm$	2.9	162.4 $\pm$	8.3
	Server-STF	318.0 $\pm$	1.4	2765.0 $\pm$	5.4	495.7 $\pm$	121.5	— $\pm$	—	298.1 $\pm$	2.4	282.4 $\pm$	54.8

Table 22. RQ3 end-to-end latency by task, LLM model, and hardware platform. Values report mean

\pm

standard deviation in seconds per run, averaged over agent configurations, prompting strategies, and available repetitions. Dashes indicate unsupported model–hardware combinations. GPT-OSS was evaluated once per condition on Server-SMU; therefore, its zero standard deviation does not imply zero variability.

Figure 11. RQ3 mean end-to-end latency by LLM model and hardware platform, shown separately for code generation (CodeGen), technical debt detection (TechDebt), vulnerability detection (Vuln), log parsing (LogParse), and log analysis (LogAnal). Rows denote Server-SMU (Srv-SMU), Server-STF (Srv-STF), and Workstation-SMU (WS-SMU); columns denote CodeGemma-7B (CG-7B), DeepSeek-Coder-6.7B (DS-6.7B), Gemma-3-4B (G3-4B), GPT-OSS-20B (GPT-OSS), Qwen2.5-Coder-7B (Q2.5C-7B), and Qwen3-4B (Q3-4B). Each cell reports mean latency in seconds, averaged over agent configurations, prompting strategies, and available repetitions. Latency includes model loading, inference, and agent orchestration. Color intensity is mapped logarithmically, whereas cell annotations report the untransformed arithmetic means. Blank cells indicate unsupported model–hardware combinations; GPT-OSS was evaluated only on Server-SMU because of its memory requirements.

Figure 12. RQ3 relationship between total pipeline energy consumption and end-to-end latency across the hardware-balanced configurations. Each point represents one task–model–prompt–agent configuration, with energy measured in kWh and latency in seconds. Point color denotes the non-agentic baseline (NA), single-agent (SA), dual-agent (DA), or four-agent multi-agent (MA) configuration. Both axes use logarithmic scales, and the reported Pearson correlation is computed on the log-transformed values (

r=0.987

p<10^{-100}

). The strong positive association indicates that configurations with longer execution times generally also consume more energy; it does not by itself establish a causal relationship.

Figure 13. RQ4 task-level trade-off between task-specific performance and total pipeline energy consumption. Each point represents one model–agent–prompt configuration in the two-dimensional performance–energy projection; point color denotes the non-agentic direct-query baseline (NA), single-agent (SA), dual-agent (DA), or four-agent multi-agent (MA) configuration. Energy is reported in kWh on a logarithmic x-axis. The y-axis reports Pass@1 for code generation, Macro F1 for technical debt detection, F1-score for vulnerability detection and log analysis, and exact-match parsing accuracy for log parsing. Black circles identify Pareto-optimal configurations, for which no alternative achieves equal or higher performance with equal or lower energy, and dashed lines trace the resulting Pareto fronts. Latency is not represented in this projection but is included in the three-objective analysis reported in Table 23 .

Task	Hardware	Agent	Model	Prompt	Accuracy	Energy (kWh)	Latency (s)	# Pareto Configs
CodeGen	Server-SMU	SA	DS-6.7B	Zero-shot	100.0	0.0208	194.8	1
	Server-STF	SA	CG-7B	Zero-shot	100.0	0.0448	333.2	4
	Workstation-SMU	SA	G3-4B	Few-shot	100.0	0.0020	63.6	1
LogAnal	Server-SMU	NA	Q3-4B	Few-shot	14.9	0.0249	299.2	5
	Server-STF	NA	CG-7B	Zero-shot	14.1	0.0222	201.2	6
	Workstation-SMU	NA	G3-4B	Zero-shot	4.9	0.0156	494.8	6

Table 23. RQ4 balanced Pareto-optimal recommendation for each task–hardware pair, selected from the three-dimensional frontier using equally weighted normalized task performance, energy, and latency. Performance uses each task’s primary effectiveness metric, and “# Pareto Configs” reports the size of the corresponding frontier.

Task	Best Accuracy	Lowest Energy	Lowest Latency	Best Overall
Code Generation	NA (96.8)	NA (0.0924)	SA (1307.8)	NA
Technical Debt Detection	NA (42.8)	NA (0.0181)	NA (252.3)	NA
Vulnerability Detection	MA (55.4)	NA (0.4271)	NA (5748.7)	NA
Log Parsing	NA (34.5)	NA (0.0266)	NA (359.3)	NA
Log Analysis	NA (7.5)	SA (0.0712)	SA (1063.9)	SA

Table 24. RQ4 task-specific comparison of agentic configurations across effectiveness, energy consumption, and end-to-end latency. Each cell reports the best configuration, with its corresponding mean value in parentheses; “Best Overall” identifies the strongest combined trade-off across the three objectives.

Figure 14. Cross-task comparison of LLM effectiveness and energy cost. Rows denote CodeGemma-7B (CG-7B), DeepSeek-Coder-6.7B (DS-6.7B), Gemma-3-4B (G3-4B), GPT-OSS-20B (GPT-OSS), Qwen2.5-Coder-7B (Q2.5C-7B), and Qwen3-4B (Q3-4B); columns denote code generation (CodeGen), technical debt detection (TechDebt), vulnerability detection (Vuln), log parsing (LogParse), and log analysis (LogAnal). The left heatmap reports mean task-specific performance as a proportion—Pass@1, Macro F1, F1-score, or parsing accuracy, depending on the task—and the right heatmap reports mean total pipeline energy per run in kWh. Hardware is first averaged within each model–prompt combination, after which values are averaged over agent configurations and prompting strategies. The energy color scale is logarithmic, whereas cell annotations report untransformed mean values. GPT-OSS values are based on Server-SMU because that model was not evaluated on the other platforms. The differing performance and energy leaders across tasks show no model is uniformly optimal.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Smell Class	NA		SA		DA		MA
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
No Smell	57.4 $\pm$	10.6	57.3 $\pm$	13.2	57.1 $\pm$	15.0	51.3 $\pm$	18.7
Blob	32.8 $\pm$	24.6	31.3 $\pm$	21.3	28.6 $\pm$	18.7	27.4 $\pm$	18.7
Data Class	41.1 $\pm$	26.2	39.2 $\pm$	25.9	40.2 $\pm$	28.0	41.7 $\pm$	27.1
Feature Envy	4.9 $\pm$	5.8	7.5 $\pm$	6.1	10.1 $\pm$	7.1	9.8 $\pm$	8.7
Long Method	65.0 $\pm$	22.6	66.1 $\pm$	19.5	61.0 $\pm$	23.3	58.9 $\pm$	23.1

Appendix

Table 25. RQ1 TD-specific per-class results: F1 by smell class and agent configuration (mean

\pm

std, averaged over models, hardware, and prompts).

Smell Class	CG-7B		DS-6.7B		G3-4B		GPT-OSS		Q2.5C-7B		Q3-4B
	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std	mean $\pm$	std
No Smell	63.7 $\pm$	11.7	55.8 $\pm$	2.2	69.6 $\pm$	6.6	67.7 $\pm$	2.1	47.5 $\pm$	9.7	38.3 $\pm$	14.6
Blob	2.7 $\pm$	3.9	17.3 $\pm$	6.8	53.7 $\pm$	10.3	61.2 $\pm$	12.1	38.5 $\pm$	7.0	27.7 $\pm$	13.2
Data Class	2.9 $\pm$	7.9	14.3 $\pm$	6.5	57.0 $\pm$	3.8	75.3 $\pm$	3.9	64.1 $\pm$	9.4	53.0 $\pm$	5.3
Feature Envy	13.1 $\pm$	9.2	8.0 $\pm$	4.8	2.5 $\pm$	1.7	0.0 $\pm$	0.0	10.7 $\pm$	6.0	8.8 $\pm$	7.4
Long Method	66.3 $\pm$	5.1	21.3 $\pm$	7.4	80.4 $\pm$	2.4	88.7 $\pm$	2.6	75.5 $\pm$	5.0	61.6 $\pm$	6.8

Appendix

Table 26. RQ1 TD-specific per-class results: F1 by smell class and LLM model (mean

\pm

std, averaged over agent configurations, hardware, and prompts).

Smell Class	Zero-shot		Few-shot
	mean $\pm$	std	mean $\pm$	std
No Smell	56.9 $\pm$	13.8	54.6 $\pm$	15.6
Blob	32.2 $\pm$	21.7	27.9 $\pm$	19.8
Data Class	39.9 $\pm$	27.8	41.3 $\pm$	25.3
Feature Envy	10.1 $\pm$	8.3	6.0 $\pm$	5.3
Long Method	61.7 $\pm$	22.7	63.8 $\pm$	21.6

Appendix

Table 27. RQ1 TD-specific per-class results: F1 by smell class and prompt strategy (mean

\pm

std, averaged over agent configurations, models, and hardware).

Software Engineering Agents

Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.

See all 323 papers

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering
Jul 7, 2026Razvan Mihai PopescuSoftware Engineering Agents LLM Agent Evaluation
Delft University of Technology Delft, The Netherlands
AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
Apr 13, 2026Rajesh Kumar, Waqar Ali, Junaid Ahmed +2Software Engineering Agents Multi-Agent Coding
International Research Center for Complexity Sciences, Hangzhou International Innovation Institute, Beihang University, Hangzhou, 311115, China · Department of Computer Science, College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou 325060, China · Fakulti Teknologi Maklumat dan Komunikasi, Universiti Teknikal Malaysia Melaka, Melaka 76100, Malaysia +2
EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design
May 19, 2026Gioele Molinari, Florian Felten, Soheyl Massoudi +1Multi-Agent LLM Systems LLM Agent Evaluation
IDEAL Chair of Artificial Intelligence in Engineering Design ETH Zurich Zurich, Switzerland · IDEAL, ETH Zurich, Zurich, Switzerland · Autom8.build, Brussels, Belgium

Jul 7, 2026J/K move · Enter open · S save

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

Razvan Mihai Popescu

Delft University of Technology Delft, The Netherlands

Software Engineering Agents LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks LLM Agent Reliability

Open paper AskPDF arXiv

Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contributors deeply embedded into collaborative development environments. Despite their accelerated adoption, existing evaluation techniques are limited due to their fragmented nature and distorted projection of true model capabilities, often obtained from hypothetical syntactic scenarios. This research aims to bridge this gap by providing a comprehensive evaluation methodology for LLM-powered agents that is grounded in real-world software development practice. Our evaluation approach focuses on contamination-awareness, in-the-wild agentic behavior assessment, and trajectory-aware benchmarks and metrics capturing realistic coding contexts, human-aligned behavior, and model failure modes.

Abstract

Software Engineering Agents

Explore similar work

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design