Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

Shanchao Liang; Yiran Hu; Nan Jiang; Lin Tan

cs.SEOct 29, 2024

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

Authors: Shanchao Liang, Yiran Hu, Nan Jiang, Lin Tan

Organizations: Purdue University

Abstract

Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.

Figures & tables

Figure 1: Data collection pipeline and instance structure of RepoCod .

Benchmarks	Instance Statistics				Repository Scale
Benchmarks	#Instances	#Tokens	Cyclo.	#Tests	#Lines	#Files
CrossCodeEval	2,665	13.2	1.0	0	-	-
RepoBench	23,561	13.7	1.0	0	-	-
LCA	934	12.0	1.0	0	-	-
CoderEval 1 1 1 CoderEval only releases the task instances along with ground truths but omits the test cases.	230	108.2	4.7	-	48,821	152
DevEval 2 2 2 For data statistics not available in DevEval’s paper, we recompute the data from their released dataset available in their GitHub repository.	1,825	86.3	3.5	2.1	36,640	164

Table 1: Benchmarks Comparison. ‘-’ indicates the data is not publicly available. #Instances : number of instances in each benchmark; #Tokens : average number of tokens of the canonical solution; Cyclo. average cyclomatic complexity of the canonical solution; #Tests : average number of test cases per instance; #Lines : average number of lines per repositories;

Dataset	Repository-Level	File-Level	Self-Contained
CoderEval	10.0	53.5	36.5
DevEval	31.3	41.2	27.5
RepoCod	50.8	18.1	31.1

Table 2: Data context complexity distribution (%) of code generation benchmarks.

Models	RAG BM25	RAG Dense	Current-File
CodeLlama-7B	10.7	10.4	5.7
CodeLlama-34B	12.4	12.8	9.6
DeepSeekCoder-6.7B	14.0	14.1	10.9
DeepSeekCoder-33B	16.7	17.1	14.9
OpenCodeInterpreter-6.7B	12.1	12.5	13.2
OpenCodeInterpreter-33B	15.3	16.3	18.3

Table 3: Pass@1(%) of SOTA LLMs on RepoCod .

Figure 2: GPT-4o’s passed tasks in different retrievals.

Figure 3: Commercial LLMs’ passed tasks.

Panel of Figure 4 — Figure 4: Pass@1 under context complexity and length complexity. Let $n$ represent the number of instances in each bin. Length complexities distribution:: [0, 66) : $n$ =201, [66, 128) : $n$ =194, [128, 232) : $n$ =194, [232, 435) : $n$ =196, [435, 4308] : $n$ =195. Cyclomatic complexity distribution ( $M$ ): Simple ( $M\leq 2$ ): $n$ =269, Moderate ( $3\leq M\leq 5$ ): $n$ =211 , Complex ( $6\leq M\leq 10$ ): $n$ =215, and Very Complex ( $M\geq 11$ ): $n$ =285.

Model	Method	Recall Range
Model	Method	0	(0, 0.50]	(0.50, 1]
DeepSeek-v2.5	RAG BM25	10.4	10.9	28.8
DeepSeek-v2.5	RAG Dense	12.3	11.0	27.2
Claude-3.5	RAG BM25	7.1	4.5	24.7
Claude-3.5	RAG Dense	7.2	8.7	28.3
GPT-4o	RAG BM25	20.3	16.4	41.1

Table 4: Pass@1 by Recall for RAG BM25 and RAG Dense . Distribution: (BM25: 492/110/73, Dense: 456/127/92).

Model	Method	Self	File	Repo.	Overall
GPT-4o-Mini	Baseline	11.5	7.3	2.6	6.2
	RAG BM25	22.6	22.0	8.0	15.1
	RAG Dense	24.3	23.7	6.2	15.0
	Current-File	30.5	23.7	9.6	18.7
	Callees	27.9	23.7	7.0	16.5
	RAG Dense-oracle	28.2	24.3	5.2	15.8

Table 5: Pass@1 (%) of GPT-4o across context complexities. Self, File, and Repo. refer to Self-Contained, File-Level and Repository-Level.

Figure 5: Models’ pass@1 against syntax error rate in different retrievals. Each pair of model and retrieval approach is represented as a dot.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Repository-level				File-level				Self-contained				Total
Dataset	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.
astropy	360.6	447.9	10.5	51	404.1	195.7	4.9	14	227.6	238.2	5.9	20	336.5	357.0	8.5	85
datasets	499.7	348.7	9.6	19	292.3	168.8	5.1	18	312.5	128.9	3.1	22	366.6	211.9	5.8	59
flask	277.9	171.9	5.2	8	356.1	160.4	5.2	10	233.0	88.6	3.5	25	270.0	120.8	4.2	43
more-itertools	224.6	70.2	2.6	8	293.4	127.0	4.0	18	243.2	105.2	5.1	60	252.0	106.5	4.6	86
plotly.py	1,357.3	1373.9	37.0	15	1,337.3	356.0	7.9	7	1,687.3	590.5	24.3	54	1,589.9	723.5	25.3	76

Appendix

Table 6: Basic statistics of RepoCod , with details broken down by each collected repository.

Figure 6: Two examples from RepoBench showing misleading metrics results. The yellow and blue highlights indicate the difference between ground truth (GT) and the LLM-generated code.

Figure 7: Illustrative example of prompting LLMs and evaluating the LLM-generated code.

Setting	pylint	sympy	sphinx	seaborn	flask	more-itertools	scikit-learn	xarray	datasets	plotly.py	astropy	Total
CodeLlama-7B	7.69	7.22	9.09	23.08	37.21	13.95	5.10	6.02	22.03	6.58	5.88	10.41
CodeLlama-34B	0.00	4.12	18.18	21.79	32.56	24.42	5.73	16.87	18.64	14.47	10.59	12.76
DeepSeekCoder-6.7B	7.69	6.19	15.15	26.92	41.86	12.79	6.37	16.87	16.95	25.00	14.12	14.08
DeepSeekCoder-33B	3.85	7.22	27.27	28.21	37.21	11.63	7.32	15.66	33.90	43.42	16.47	17.14
OpenCodeInterpreter-6.7B	3.85	4.12	12.12	16.67	41.86	18.60	6.37	10.84	16.95	21.05	12.94	12.45
OpenCodeInterpreter-33B	11.54	7.22	21.21	20.51	41.86	29.07	8.28	13.25	25.42	23.68	16.47	16.33

Appendix

Table 7: Pass@1(%) of LLMs on RepoCod under RAG Dense retrieval, with details shown for each repository.

Repository	RAG BM25	RAG Dense
pylint	0.14	0.18
sympy	0.16	0.14
sphinx	0.09	0.11
seaborn	0.17	0.21
flask	0.11	0.20
more-itertools	0.05	0.10

Appendix

Table 8: Recall rate of the top three and top ten retrieved contents across retrieval settings for repository-level and file-level contexts.

Models	RAG BM25		RAG Dense		Current-File
Models	Pass	Fail	Pass	Fail	Pass	Fail
CodeLlama-7B	18.2	11.6	9.1	11.9	30.4	22.3
CodeLlama-34B	1.8	15.2	1.9	11.5	22.5	34.5
DeepSeek-6.7B	2.2	9.6	7.0	5.7	20.0	13.1
DeepSeek-33B	8.2	7.4	8.4	3.9	17.9	26.5
OCI-6.7B	2.3	2.5	3.5	2.8	2.5	1.8

Appendix

Table 9: Ratio of generation length over canonical solution length

Models	RAG BM25		RAG Dense		Current-File
Models	Pass	Fail	Pass	Fail	Pass	Fail
CodeLlama-7B	0.9	1.7	1.0	1.7	1.0	1.8
CodeLlama-34B	0.9	1.5	0.8	1.4	0.9	1.5
DeepSeek-6.7B	1.0	1.6	0.9	1.4	0.9	1.8
DeepSeek-33B	0.8	1.8	1.0	1.7	0.9	1.9
OCI-6.7B	1.6	2.1	2.8	2.4	1.7	1.5

Appendix

Table 10: Ratio of filtered generation length over canonical solution length.

Figure 8: Prompt Example for KMeans.fit in the file “ sklearn/cluster/_kmeans.py ” from RepoCod .

Model	Retrieval	# of Dependencies
Model	Retrieval	0	1	2	[3,4]
DeepSeek-v2.5	RAG BM25	31.8	22.1	14.2	7.1	5.7
	RAG Dense	35.4	28.4	14.2	6.0	6.2
	Current-File	42.6	31.1	22.0	17.3	9.1
Claude-3.5	RAG BM25	27.2	18.9	9.9	2.4	2.3
	RAG Dense	33.1	22.6	12.8	4.8	0.6

Appendix

Table 11: Pass Rate by Number of dependencies, binned by the number of unique dependencies. The bins are distributed as follows: 0 (305 instances), 1 (190 instances), 2 (141 instances), [3,4] (168 instances), and [5,35] (176 instances).

Figure 9: An example of the correct output generated by LLMs, for function “ false_alarm_probability ” in the file “ astropy/timeseries/periodograms/lombscargle/core.py ” from RepoCod .

Figure 10: A failed case study, for function ‘ _set_order ’ in the file ‘ sklearn/linear_model/_coordinate_descent.py ’ from RepoCod .

Figure 11: An example of the wrong output generated by LLMs, for function “ transform ” in the file “ sklearn/decomposition/_factor_analysis.py ” from RepoCod .

Panel of Figure 12 — Figure 12: Correct generation result’s relationship under different retrieval settings.

Panel of Figure 13 — Figure 13: Pass@1 under different context complexity.

Models	RAG BM25	RAG Dense	Current-File
CodeLlama-7B	1.0	1.5	1.0
CodeLlama-34B	2.5	2.0	1.0
DeepSeek-6.7B	2.5	2.0	0.5
DeepSeek-33B	4.0	3.5	2.5
OpenCodeInterpreter-6.7B	1.0	5.0	1.5
OpenCodeInterpreter-33B	4.0	6.0	5.5

Appendix

Table 12: Pass@1(%) of SOTA LLMs on RepoCod -LITE.

Dataset	Repository-level				File-level				Self-contained				Total
Dataset	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.	#NL	#GT	Cyclo.	#Funcs.
astropy	321.7	1,104.0	25.3	3	678.8	1,107.2	20.0	4	485.3	618.8	14.0	9	503.0	831.9	17.6	16
datasets	1,432.8	832.8	24.0	4	1,331.0	526.0	14.0	1	1211.0	318.0	8.0	1	1,378.8	695.8	19.7	6
flask	661.0	424.0	12.0	1	595.0	515.0	12.0	1	452.0	691.0	16.0	1	569.3	543.3	13.3	3
more-itertools	-	-	-	0	503.0	815.0	11.0	1	-	-	-	0	503.0	815.0	11.0	1
plotly.py	1,806.0	3,393.0	132.0	1	1,574.4	1,191.8	30.2	14	1,933.0	1464.2	56.9	23	1,797.5	1,414.6	49.0	38

Appendix

Table 13: Basic statistics of RepoCod _LITE, with details broken down by each collected repository.

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 311 papers

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation
Mar 26, 2026Jiseung Hong, Benjamin G. Ascoli, Jinho D. ChoiRepository-Level Code Generation Code Generation Evaluation
Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
Sep 29, 2026Hantian Ding, Chloe Bi, Jiacheng Zhu +5Software Engineering Agents Repository-Level Code Generation
Meta Superintelligence Labs

Mar 26, 2026J/K move · Enter open · S save

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

Jiseung Hong, Benjamin G. Ascoli, Jinho D. Choi

Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University

Repository-Level Code Generation Code Generation Evaluation Code Generation Benchmarks Agentic Code Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.

Abstract

LLM Code Generation

Explore similar work

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch