Complexity-Aware Evaluation of LLM Comprehension

Ali Mohammadi Esfahani; Nafiseh Kahani; Samuel A. Ajila

cs.SESep 29, 2026

Complexity-Aware Evaluation of LLM Comprehension

Authors: Ali Mohammadi Esfahani, Nafiseh Kahani, Samuel A. Ajila

Organizations: Systems and computer Engineering Carleton University Ottawa, Canada

Abstract

Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior prediction, function explanation, debugging, and code review. However, aggregate benchmark accuracy can conceal how model reliability changes as source code becomes structurally more complex. This paper presents a complexity-aware framework for evaluating LLM code comprehension using cyclomatic complexity, nesting depth, branching factor, and Halstead volume. We evaluate DeepSeek-Coder-V2 and Llama through two complementary tasks: automatic input-output prediction over 300 Python functions and manually assessed semantic comprehension over a balanced subset of 60 functions. The functions are grouped into Low-, Medium-, and High-complexity bands. DeepSeek-Coder-V2 achieves an overall automatic accuracy of 78.33%, compared with 70.33% for Llama. However, accuracy decreases substantially from Low to High complexity, from 93.52% to 52.78% for DeepSeek-Coder-V2 and from 87.04% to 47.22% for Llama. Incorrect predictions are consistently associated with higher values of all four complexity metrics, and correlation and logistic-regression analyses confirm broadly comparable negative associations between structural complexity and correctness. Manual semantic comprehension shows the same degradation pattern, with accuracy decreasing from 100.00% to 75.00% for DeepSeek-Coder-V2 and from 90.00% to 60.00% for Llama. These findings demonstrate that complexity-aware evaluation provides a more diagnostic assessment of LLM code-comprehension reliability than aggregate accuracy alone.

Figures & tables

Dataset	Low	Medium	High	Total
HumanEval	32	48	0	80
MBPP-sanitized	50	30	0	80
CRUXEval	26	20	0	46
LeetCode	0	12	48	60
BigCodeBench-Hard	0	10	24	34
Total	108	120	72	300

TABLE I: Automatic benchmark composition by dataset source and complexity band.

Band	Cyclomatic Complexity	Nesting Depth	Branching Factor	Halstead Volume
Low	1–3	0–1	0–2	$V<100$
Medium	4–6	2–3	3–5	$100\leq V<300$
High	$\geq 7$	$\geq 4$	$\geq 6$	$V\geq 300$

TABLE II: Complexity band definitions used for structural code analysis.

Model	Band	Correct	Total	Accuracy
DeepSeek-Coder-V2	Low	101	108	93.52%
	Medium	96	120	80.00%
	High	38	72	52.78%
CodeLlama-7b-Instruct-hf	Low	94	108	87.04%
	Medium	83	120	69.17%
	High	34	72	47.22%

TABLE III: Automatic comprehension accuracy by complexity band.

Model	Dataset	Correct	Total	Accuracy
DeepSeek-Coder-V2	HumanEval	70	80	87.50%
	MBPP-sanitized	71	80	88.75%
	CRUXEval	39	46	84.78%
	LeetCode	39	60	65.00%
	BigCodeBench-Hard	16	34	47.06%
CodeLlama-7b-Instruct-hf	HumanEval	65	80	81.25%

TABLE IV: Automatic comprehension accuracy by dataset source.

Model	Outcome	CC	ND	BF	HV
DeepSeek-Coder-V2	Correct	3.84	1.72	3.11	142.60
	Incorrect	7.42	3.46	6.89	318.75
CodeLlama-7b-Instruct-hf	Correct	3.51	1.58	2.94	131.40
	Incorrect	7.24	3.32	6.27	297.80

TABLE V: Average complexity metrics for correct and incorrect predictions.

Model	CC	ND	BF	HV
DeepSeek-Coder-V2	$-0.41$	$-0.35$	$-0.43$	$-0.37$
CodeLlama-7b-Instruct-hf	$-0.39$	$-0.33$	$-0.40$	$-0.35$

TABLE VI: Spearman correlations between structural complexity and automatic prediction correctness. All correlations are statistically significant at

p<0.001

Question Type	Low	Medium	High	Total
Purpose	7	7	6	20
Variable Role	7	6	7	20
Edge Case	6	7	7	20
Total	20	20	20	60

TABLE VII: Distribution of manual question types across complexity bands.

Model	Question Type	Correct	Total	Acc.
DeepSeek-Coder-V2	Purpose	19	20	95.00%
	Variable Role	18	20	90.00%
	Edge Case	16	20	80.00%
CodeLlama-7b-Instruct-hf	Purpose	17	20	85.00%
	Variable Role	16	20	80.00%
	Edge Case	13	20	65.00%

TABLE VIII: Manual comprehension accuracy by question type.

Model	Band	Correct	Total	Acc.
DeepSeek-Coder-V2	Low	20	20	100.00%
	Medium	18	20	90.00%
	High	15	20	75.00%
CodeLlama-7b-Instruct-hf	Low	18	20	90.00%
	Medium	16	20	80.00%
	High	12	20	60.00%

TABLE IX: Manual comprehension accuracy by complexity band.

Model	Task	Low	Medium	High	Low–High Drop
DeepSeek-Coder-V2	Automatic	93.52%	80.00%	52.78%	40.74 pp
	Manual	100.00%	90.00%	75.00%	25.00 pp
CodeLlama-7b-Instruct-hf	Automatic	87.04%	69.17%	47.22%	39.82 pp
	Manual	90.00%	80.00%	60.00%	30.00 pp

TABLE X: Comparison of automatic and manual comprehension accuracy across complexity bands.

Model	Metric	Coef.	OR	95% CI	$p$ -value
DeepSeek-Coder-V2	CC	$-0.41$	0.66	[0.57, 0.76]	$<0.001$
	ND	$-0.34$	0.71	[0.60, 0.84]	$<0.001$
	BF	$-0.38$	0.68	[0.58, 0.79]	$<0.001$
	HV/100	$-0.27$	0.76	[0.66, 0.88]	0.002
CodeLlama-7b-Instruct-hf	CC	$-0.36$	0.70	[0.61, 0.80]	$<0.001$
	ND	$-0.31$	0.73	[0.62, 0.86]	$<0.001$

TABLE XI: Multivariate logistic regression results for automatic prediction correctness.

Model	$\boldsymbol{\chi^{2}}$	df	$p$ -value	Cramér’s $V$
DeepSeek-Coder-V2	6.15	2	0.046	0.32
CodeLlama-7b-Instruct-hf	5.22	2	0.074	0.29

TABLE XII: Association between complexity band and manual comprehension correctness.

Explore similar work

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval
May 29, 2026Nazmus AshrafiCode Generation Multi-Agent Large Language Model Systems
Independent Researcher
Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code
Jul 2, 2026Zihao Xu, Yuekang Li, Gelei Deng +2Performance Metric High-Level Source Code
University of New South Wales, Australia · Nanyang Technological University, Singapore · Griffith University, Australia +2
From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs
Jun 16, 2026Siyue Chen, Yifu Guo, Yuquan Lu +9LLM Reasoning Strategies Reasoning Skills
1South China University of Technology · 2Sun Yat-sen University · 4Shanghai Jiao Tong University +5

May 29, 2026, cs.SEJ/K move · Enter open · S save

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

Nazmus Ashrafi

Independent Researcher

Code Generation Multi-Agent Large Language Model Systems Multi-Agent Orchestration

Open paper AskPDF arXiv

Large-language-model code generation has shifted from single-shot prompting to multi-agent orchestrations - analyst, coder, tester, and debugger pipelines - and is evaluated almost exclusively on functional correctness. Whether these architectures also affect the structural complexity of the code they produce, and which orchestration layers carry the cost, remains largely unexamined: prior work has documented prompt-level effects on code complexity, but the architecture-level question is open. We compare six widely-used multi-agent configurations (Basic, AC, ACT, Debugger, AC+Debugger, ACT+Debugger) under two models from the GPT-4o family across all 164 HumanEval tasks - 1,968 paired observations - using the five RADON complexity metrics (SLOC, cyclomatic complexity, and Halstead Volume, Difficulty, and Effort). We apply a paired non-parametric statistical pipeline (Friedman omnibus, Wilcoxon signed-rank post-hoc with Holm correction, Kendall's

W

and matched-pairs rank-biserial effect sizes) in both all-completions and passing-only conditions. The six architectures collapse into two indistinguishable complexity clusters separated by a 50-130% gap, the same partition in both models and under both conditions; among the architectural layers, the analyst-coder split inflates complexity, the runtime debugger does not - and on the analyst-coder background actively deflates it - and the tester re-inflates it. The heavy cluster's additional complexity buys no pass@1 advantage: the leanest architectures match or beat the heaviest on accuracy. Architectural elaboration in LLM code generation should therefore be justified by measured benefit on the dimensions that matter, not assumed.

Abstract

Explore similar work

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code

From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs