Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.
Figures & tables
Figure 1: Data collection pipeline and instance structure of RepoCod .
Benchmarks
Instance Statistics
Repository Scale
#Instances
#Tokens
Cyclo.
#Tests
#Lines
#Files
CrossCodeEval
2,665
13.2
1.0
0
-
-
RepoBench
23,561
13.7
1.0
0
-
-
LCA
934
12.0
1.0
0
-
-
CoderEval 1 1 1 CoderEval only releases the task instances along with ground truths but omits the test cases.
230
108.2
4.7
-
48,821
152
DevEval 2 2 2 For data statistics not available in DevEval’s paper, we recompute the data from their released dataset available in their GitHub repository.
1,825
86.3
3.5
2.1
36,640
164
Table 1: Benchmarks Comparison. ‘-’ indicates the data is not publicly available. #Instances : number of instances in each benchmark; #Tokens : average number of tokens of the canonical solution; Cyclo. average cyclomatic complexity of the canonical solution; #Tests : average number of test cases per instance; #Lines : average number of lines per repositories;
Dataset
Repository-Level
File-Level
Self-Contained
CoderEval
10.0
53.5
36.5
DevEval
31.3
41.2
27.5
RepoCod
50.8
18.1
31.1
Table 2: Data context complexity distribution (%) of code generation benchmarks.
Models
RAG BM25
RAG Dense
Current-File
CodeLlama-7B
10.7
10.4
5.7
CodeLlama-34B
12.4
12.8
9.6
DeepSeekCoder-6.7B
14.0
14.1
10.9
DeepSeekCoder-33B
16.7
17.1
14.9
OpenCodeInterpreter-6.7B
12.1
12.5
13.2
OpenCodeInterpreter-33B
15.3
16.3
18.3
Table 3: Pass@1(%) of SOTA LLMs on RepoCod .
Figure 2: GPT-4o’s passed tasks in different retrievals.
Figure 3: Commercial LLMs’ passed tasks.
Figure 4: Pass@1 under context complexity and length complexity. Let n represent the number of instances in each bin. Length complexities distribution:: [0, 66) : n =201, [66, 128) : n =194, [128, 232) : n =194, [232, 435) : n =196, [435, 4308] : n =195. Cyclomatic complexity distribution ( M ): Simple ( M≤2 ): n =269, Moderate ( 3≤M≤5 ): n =211 , Complex ( 6≤M≤10 ): n =215, and Very Complex ( M≥11 ): n =285.
Model
Method
Recall Range
0
(0, 0.50]
(0.50, 1]
DeepSeek-v2.5
RAG BM25
10.4
10.9
28.8
RAG Dense
12.3
11.0
27.2
Claude-3.5
RAG BM25
7.1
4.5
24.7
RAG Dense
7.2
8.7
28.3
GPT-4o
RAG BM25
20.3
16.4
41.1
Table 4: Pass@1 by Recall for RAG BM25 and RAG Dense . Distribution: (BM25: 492/110/73, Dense: 456/127/92).
Model
Method
Self
File
Repo.
Overall
GPT-4o-Mini
Baseline
11.5
7.3
2.6
6.2
RAG BM25
22.6
22.0
8.0
15.1
RAG Dense
24.3
23.7
6.2
15.0
Current-File
30.5
23.7
9.6
18.7
Callees
27.9
23.7
7.0
16.5
RAG Dense-oracle
28.2
24.3
5.2
15.8
Table 5: Pass@1 (%) of GPT-4o across context complexities. Self, File, and Repo. refer to Self-Contained, File-Level and Repository-Level.
Figure 5: Models’ pass@1 against syntax error rate in different retrievals. Each pair of model and retrieval approach is represented as a dot.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Repository-level
File-level
Self-contained
Total
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
astropy
360.6
447.9
10.5
51
404.1
195.7
4.9
14
227.6
238.2
5.9
20
336.5
357.0
8.5
85
datasets
499.7
348.7
9.6
19
292.3
168.8
5.1
18
312.5
128.9
3.1
22
366.6
211.9
5.8
59
flask
277.9
171.9
5.2
8
356.1
160.4
5.2
10
233.0
88.6
3.5
25
270.0
120.8
4.2
43
more-itertools
224.6
70.2
2.6
8
293.4
127.0
4.0
18
243.2
105.2
5.1
60
252.0
106.5
4.6
86
plotly.py
1,357.3
1373.9
37.0
15
1,337.3
356.0
7.9
7
1,687.3
590.5
24.3
54
1,589.9
723.5
25.3
76
Appendix
Table 6: Basic statistics of RepoCod , with details broken down by each collected repository.
Figure 6: Two examples from RepoBench showing misleading metrics results. The yellow and blue highlights indicate the difference between ground truth (GT) and the LLM-generated code.
Figure 7: Illustrative example of prompting LLMs and evaluating the LLM-generated code.
Setting
pylint
sympy
sphinx
seaborn
flask
more-itertools
scikit-learn
xarray
datasets
plotly.py
astropy
Total
CodeLlama-7B
7.69
7.22
9.09
23.08
37.21
13.95
5.10
6.02
22.03
6.58
5.88
10.41
CodeLlama-34B
0.00
4.12
18.18
21.79
32.56
24.42
5.73
16.87
18.64
14.47
10.59
12.76
DeepSeekCoder-6.7B
7.69
6.19
15.15
26.92
41.86
12.79
6.37
16.87
16.95
25.00
14.12
14.08
DeepSeekCoder-33B
3.85
7.22
27.27
28.21
37.21
11.63
7.32
15.66
33.90
43.42
16.47
17.14
OpenCodeInterpreter-6.7B
3.85
4.12
12.12
16.67
41.86
18.60
6.37
10.84
16.95
21.05
12.94
12.45
OpenCodeInterpreter-33B
11.54
7.22
21.21
20.51
41.86
29.07
8.28
13.25
25.42
23.68
16.47
16.33
Appendix
Table 7: Pass@1(%) of LLMs on RepoCod under RAG Dense retrieval, with details shown for each repository.
Repository
RAG BM25
RAG Dense
pylint
0.14
0.18
sympy
0.16
0.14
sphinx
0.09
0.11
seaborn
0.17
0.21
flask
0.11
0.20
more-itertools
0.05
0.10
Appendix
Table 8: Recall rate of the top three and top ten retrieved contents across retrieval settings for repository-level and file-level contexts.
Models
RAG BM25
RAG Dense
Current-File
Pass
Fail
Pass
Fail
Pass
Fail
CodeLlama-7B
18.2
11.6
9.1
11.9
30.4
22.3
CodeLlama-34B
1.8
15.2
1.9
11.5
22.5
34.5
DeepSeek-6.7B
2.2
9.6
7.0
5.7
20.0
13.1
DeepSeek-33B
8.2
7.4
8.4
3.9
17.9
26.5
OCI-6.7B
2.3
2.5
3.5
2.8
2.5
1.8
Appendix
Table 9: Ratio of generation length over canonical solution length
Models
RAG BM25
RAG Dense
Current-File
Pass
Fail
Pass
Fail
Pass
Fail
CodeLlama-7B
0.9
1.7
1.0
1.7
1.0
1.8
CodeLlama-34B
0.9
1.5
0.8
1.4
0.9
1.5
DeepSeek-6.7B
1.0
1.6
0.9
1.4
0.9
1.8
DeepSeek-33B
0.8
1.8
1.0
1.7
0.9
1.9
OCI-6.7B
1.6
2.1
2.8
2.4
1.7
1.5
Appendix
Table 10: Ratio of filtered generation length over canonical solution length.
Figure 8: Prompt Example for KMeans.fit in the file “ sklearn/cluster/_kmeans.py ” from RepoCod .
Model
Retrieval
# of Dependencies
0
1
2
[3,4]
DeepSeek-v2.5
RAG BM25
31.8
22.1
14.2
7.1
5.7
RAG Dense
35.4
28.4
14.2
6.0
6.2
Current-File
42.6
31.1
22.0
17.3
9.1
Claude-3.5
RAG BM25
27.2
18.9
9.9
2.4
2.3
RAG Dense
33.1
22.6
12.8
4.8
0.6
Appendix
Table 11: Pass Rate by Number of dependencies, binned by the number of unique dependencies. The bins are distributed as follows: 0 (305 instances), 1 (190 instances), 2 (141 instances), [3,4] (168 instances), and [5,35] (176 instances).
Figure 9: An example of the correct output generated by LLMs, for function “ false_alarm_probability ” in the file “ astropy/timeseries/periodograms/lombscargle/core.py ” from RepoCod .
Figure 10: A failed case study, for function ‘ _set_order ’ in the file ‘ sklearn/linear_model/_coordinate_descent.py ’ from RepoCod .
Figure 11: An example of the wrong output generated by LLMs, for function “ transform ” in the file “ sklearn/decomposition/_factor_analysis.py ” from RepoCod .
Figure 12: Correct generation result’s relationship under different retrieval settings.
Figure 13: Pass@1 under different context complexity.
Models
RAG BM25
RAG Dense
Current-File
CodeLlama-7B
1.0
1.5
1.0
CodeLlama-34B
2.5
2.0
1.0
DeepSeek-6.7B
2.5
2.0
0.5
DeepSeek-33B
4.0
3.5
2.5
OpenCodeInterpreter-6.7B
1.0
5.0
1.5
OpenCodeInterpreter-33B
4.0
6.0
5.5
Appendix
Table 12: Pass@1(%) of SOTA LLMs on RepoCod -LITE.
Dataset
Repository-level
File-level
Self-contained
Total
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
#NL
#GT
Cyclo.
#Funcs.
astropy
321.7
1,104.0
25.3
3
678.8
1,107.2
20.0
4
485.3
618.8
14.0
9
503.0
831.9
17.6
16
datasets
1,432.8
832.8
24.0
4
1,331.0
526.0
14.0
1
1211.0
318.0
8.0
1
1,378.8
695.8
19.7
6
flask
661.0
424.0
12.0
1
595.0
515.0
12.0
1
452.0
691.0
16.0
1
569.3
543.3
13.3
3
more-itertools
-
-
-
0
503.0
815.0
11.0
1
-
-
-
0
503.0
815.0
11.0
1
plotly.py
1,806.0
3,393.0
132.0
1
1,574.4
1,191.8
30.2
14
1,933.0
1464.2
56.9
23
1,797.5
1,414.6
49.0
38
Appendix
Table 13: Basic statistics of RepoCod _LITE, with details broken down by each collected repository.
Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.
Jiseung Hong, Benjamin G. Ascoli, Jinho D. Choi
Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University
Large language models (LLMs) increasingly rely on explicit reasoning to solve coding tasks, yet evaluating the quality of this reasoning remains challenging. Existing reasoning evaluators are not designed for coding, and current benchmarks focus primarily on code generation, leaving other coding tasks largely unexplored. We introduce CodeRQ-Bench, the first benchmark for evaluating LLM reasoning quality across three coding task categories: generation, summarization, and classification. Using this benchmark, we analyze 1,069 mismatch cases from existing evaluators, identify five recurring limitations, and derive four design insights for reasoning evaluation in coding tasks. Guided by these insights, we propose VERA, a two-stage evaluator that combines evidence-grounded verification with ambiguity-aware score correction. Experiments on CodeRQ-Bench show that VERA consistently outperforms strong baselines across four datasets, improving AUCROC by up to 0.26 and AUPRC by up to 0.21. We release CodeRQ-Bench at https://github.com/MrLYG/CodeRQ-Bench, supporting future investigations.
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.