Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

Yuangang Li; Justin Tian Jin Chen; Ethan Yu; David Hong; Iftekhar Ahmed

cs.SEApr 14, 2026

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

Authors: Yuangang Li, Justin Tian Jin Chen, Ethan Yu, David Hong, Iftekhar Ahmed

Organizations: University of California, Irvine

Abstract

Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.

Figures & tables

Figure 1: Output-based vs. Reasoning-based evaluation in a code generation task.

Output Modality	Dataset	Size	# Correct / Incorrect Reasoning
Code Generation	CoderEval-RE	230	70 / 160
Code Generation	SWEBench-RE	111	21 / 90
Code Summarization	ClassEval-RE	139	112 / 27
Bug Detection	DebugBench-RE	252	153 / 99

Table 1: CodeRQ-Bench Dataset Distribution

Limitation	Definition	Evaluator ( REC. / SOCR. / CaSE )	Error Type ( Missed / FA )	Representative Case
Evidence-Ungrounded Assessment	The evaluator does not ground its assessment in the evidence needed to verify reasoning correctness in coding tasks, where many claims must be searched and checked against repository context, API documents, or task specifications.	14.4% / 41.6% / 44.0%	85.4% / 14.6%	Appx. B.1.1
Ambiguity Assessment Deficit	The evaluator cannot determine whether a task is ambiguous or underspecified, nor assess whether the reasoning handles such ambiguity.	30.2% / 36.5% / 33.3%	77.4% / 22.6%	Appx. B.1.2
Score Aggregation Failure	The evaluator produces an unreliable final score because its scoring rule combines step-level judgments in a way that either lets one misjudged step override an otherwise correct trace or allows fatal correctness errors to be diluted by other steps.	70.5% / – / 29.5%	24.7% / 75.3%	Appx. B.1.3
Self-Generated Reference Bias	The evaluator judges reasoning against self-generated references or standards, creating circular validation in which shared errors or biases are mistaken for correctness.	– / 100% / –	97.3% / 2.7%	Appx. B.1.4
Code-Unaware Assessment	The evaluator fails to reliably assess code-related reasoning because it cannot correctly represent or judge code-relevant semantics in the reasoning process.	100% / – / –	9.0% / 91.0%	Appx. B.1.5

Table 2: Taxonomy of existing reasoning-quality evaluator limitations on coding tasks. For each limitation, we report its definition, the distribution of annotated mismatch cases across evaluators, and the breakdown by error type. REC. , SOCR. , and CaSE denote ReCEval, SocREval, and CaSE, respectively; Missed and FA denote missed errors and false alarms.

Figure 2: Distribution of five limitation categories across 1,069 mismatch cases from three evaluation methods. Background bars show aggregated totals (percentages relative to all cases; categories are non-exclusive). Foreground bars decompose each category by method.

Figure 3: Overview of VERA. The left panel shows an input sample with the task description, the LLM-generated reasoning chain, and the generated code output; the right panel shows the two-stage evaluation process. Orange text marks Stage 1 evidence-verified claims, producing a base score $p$ , and green text marks Stage 2 ambiguity-related content, producing a correction term $\delta$ . The final score combines $p$ and $\delta$ .

Method	CoderEval-RE (Generation)				SWEbench-RE (Generation)
Method	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$
RECEVAl	0.1255	0.1000 (p=0.1308)	0.5627	0.3516	0.0878	0.0596 (p=0.5344)	0.5439	0.2125
SOCREVAl	0.1399	0.1248 (p=0.0588)	0.5700	0.3410	0.1556	0.1830 (p=0.0545)	0.5778	0.2165
CaSE	0.0368	0.0392 (p=0.5543)	0.5184	0.3131	-0.0243	-0.0588 (p=0.5400)	0.4878	0.1855
MAD ∗	-0.0408	-0.0480 (p=0.4684)	0.4796	0.2963	-0.0746	-0.0644 (p=0.5019)	0.4627	0.1785
LLM-as-Judge	0.0864	0.1077 (p=0.1032)	0.5432	0.3252	0.0222	0.0280 (p=0.7707)	0.5111	0.1924

Table 3: Performance of VERA and six baselines on the four CodeRQ-Bench datasets, evaluated using Somers’ D, Spearman’s

\rho

, AUCROC, and AUPRC (

\uparrow

higher is better). Best results highlighted in bold.

Method	CoderEval-RE				DebugBench-RE
Method	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$
RECEVAL	0.1255	0.1000 (p=0.1308)	0.5627	0.3516	-0.0843	-0.0539 (p=0.3945)	0.4579	0.8176
SOCREVAL	0.1399	0.1248 (p=0.0588)	0.5700	0.3410	0.1233	0.1004 (p=0.1120)	0.5617	0.8566
CaSE	0.0368	0.0392 (p=0.5543)	0.5184	0.3131	0.0822	0.0597 (p=0.3450)	0.5411	0.8628
MAD ∗	-0.0408	-0.0480 (p=0.4684)	0.4796	0.2963	-0.0342	-0.0307 (p=0.6273)	0.4829	0.8393
LLM-as-Judge	0.0864	0.1077 (p=0.1032)	0.5432	0.3252	0.0504	0.0338 (p=0.5931)	0.5252	0.8441

Table 4: Association between reasoning-evaluator scores and final-output correctness on CoderEval-RE and DebugBench-RE, evaluated using Somers’ D, Spearman’s

\rho

, AUCROC, and AUPRC (

\uparrow

higher is better). Best results are highlighted in bold.

Dataset	Method	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$
CoderEval-RE (Generation)	VERA (Ours)	0.3811	0.3076 (p<0.0001)	0.6905	0.4615
	w/o verif.	0.2945	0.2369 (p=0.0003)	0.6472	0.4356
	w/o ambig.	0.3216	0.2658 (p<0.0001)	0.6608	0.4364
SWEbench-RE (Generation)	VERA (Ours)	0.2799	0.1947 (p=0.0406)	0.6399	0.3058
	w/o verif.	-0.0376	-0.0261 (p=0.7855)	0.4812	0.1842
	w/o ambig.	0.2571	0.1796 (p=0.0593)	0.6286	0.3012

Table 5: Ablation study of VERA on four CodeRQ-Bench datasets, including two ablated variants: w/o verif. (without evidence verification) and w/o ambig. (without ambiguity-aware correction).

Method	Somers’ D $\uparrow$	Spearman’s $\rho$ $\uparrow$	AUCROC $\uparrow$	AUPRC $\uparrow$
Best baseline	0.1613	0.1713	0.5807	0.5279
VERA (full)	0.3785	0.2988	0.6893	0.6120
No correction	0.3526	0.2807	0.6763	0.6032
No threshold	0.3712	0.2928	0.6856	0.6084
No ambiguity scaling	0.3784	0.2988	0.6892	0.6111

Table 6: Formula ablation averaged across the four CodeRQ-Bench datasets. “Best baseline” is the best of the six baselines for each metric.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension: Comprehension
R1	The reasoning must correctly identify the input parameters and return values of the code.
R2	The reasoning must accurately trace the control flow (loops, conditionals, branches) of the code.
R3	The reasoning must correctly interpret the operations performed on data structures.
R4	The reasoning must not attribute functionality that does not exist in the code.
Dimension: Analysis
R1	The reasoning must correctly map variable names and function calls to their semantic roles.

Appendix

Table A1: Annotation rules for ClassEval-RE (Code Summarization) across four evaluation dimensions.

Dimension: Comprehension
R1	The reasoning must correctly identify parameters, their types (if inferable), and their roles where bugs are identified.
R2	The reasoning must correctly identify the return value(s), including affected conditionals where bugs are identified.
R3	The reasoning must correctly describe how bug-related data structures (e.g., lists, maps, trees) are read, modified, or traversed.
Dimension: Analysis
R1	All bugs that affect observable behavior must be acknowledged.
R2	The reasoning must correctly describe the execution order of conditionals, loops, and function calls.

Appendix

Table A2: Annotation rules for DebugBench-RE (Bug Detection) across four evaluation dimensions.

Panel of Table 12 — Table A3: Prompt template used to generate diagnostic annotations for missed-error cases, where the evaluator assigns a high score to reasoning that is actually incorrect. Here, {case_content} includes the original task, the LLM-generated reasoning chain and final output, and, the execution log containing the evaluator’s intermediate assessment outputs. Text in blue braces denotes dynamic variables.

Panel of Table 13 — Table A4: Prompt template used to generate diagnostic annotations for false-alarm cases, where the evaluator assigns a low score to reasoning that is actually correct. Here, {case_content} includes the original task, the LLM-generated reasoning chain and final output, and, when available, the execution log containing the evaluator’s intermediate assessment outputs. Text in blue braces denotes dynamic variables.

Panel of Table 14 — Table A5: Unified prompt template used to extract open-coded limitation labels from diagnostic annotations in the second stage of the pipeline. Here, {method_name} denotes the evaluation method being analyzed; {mismatch_type} denotes the mismatch type of the case, whose value is either missed errors or false alarms ; {mismatch_description} provides the corresponding textual description of that mismatch type; and {annotation} denotes the diagnostic annotation generated in the previous stage for the case. The prompt returns 2–4 limitation labels in JSON format. Text in blue braces denotes dynamic variables.

Panel of Table 15 — Table A6: Prompt template used for GPT-based taxonomy consolidation in the third stage of the pipeline. Here, {num_pcs} denotes the number of preclusters produced after embedding-based pre-clustering, and {formatted_preclusters} denotes the formatted list of those preclusters together with their frequency, size, and task-distribution information. The prompt asks the model to consolidate these preclusters into a final taxonomy and return category names, definitions, boundary rules, assigned preclusters, and consistency checks in JSON format. Text in blue braces denotes dynamic variables.

Panel of Table 16 — Table A7: Prompt template used in the verified reasoning evaluation stage of VERA. The placeholders {question}, {chain}, and {output} are instantiated with the task description, the candidate reasoning chain, and the corresponding generated output, respectively.

Panel of Table 17 — Table A8: Prompt template used in the ambiguity-aware score correction stage of VERA. The placeholders {question}, {chain}, and {output} are instantiated with the task description, the candidate reasoning chain, and the corresponding generated output, respectively.

Figure A1: Threshold sensitivity analysis for the ambiguity threshold $\tau$ in VERA.

Language Model Reasoning Evaluation

See all 36 papers

Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured Reasoning Code Generation Evaluation
University of California, Berkeley
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Sep 23, 2026Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband +3LLM Evaluation Software Engineering Benchmarks
York University Toronto, Canada · The University of Texas at Dallas Texas, USA
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent

Apr 18, 2026J/K move · Enter open · S save

Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

Jiaxin Fang, Runyuan He, Sahil Bhatia, Neel Gajare+1

University of California, Berkeley

Structured Reasoning Code Generation Evaluation Code Generation Benchmarks Large Reasoning Models Reasoning Trace Analysis

Open paper AskPDF arXiv

Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models