ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

Jiseung Hong; Benjamin G. Ascoli; Jinho D. Choi

cs.SEMar 26, 2026

ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation

Authors: Jiseung Hong, Benjamin G. Ascoli, Jinho D. Choi

Organizations: Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University

Abstract

Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.

Figures & tables

Benchmark	Repository Level	Prompt-Free Evaluation	Contamination Free	Maximum Context	Manually Curated	Agent Support
LiveCodeBench Jain et al. (2024)	✗	✗	✓	$\sim$ 3K	✓	✗
$\infty$ Bench Zhang et al. (2024)	✗	✗	✗	256K	✗	✗
SWE-Bench Jimenez et al. (2024)	✓	✗	✗	50K	✓	✓
Commit0 Zhao et al. (2024)	✓	✗	✗	$-$	✓	✓
RepoBench-C Liu et al. (2023)	✓	✓	✗	8K	✓	✗
LocBench Chen et al. (2025)	✓	✗	✗	$-$	✓	✓

Table 1: Comparison of Representative Code Benchmarks. We compare unsaturated code generation benchmarks across six dimensions reflecting key considerations for rigorous LLM code evaluation. Contamination Free indicates that instances are synthetically generated, collected after known LLM training cutoffs, or otherwise explicitly filtered to exclude instances present in pretraining data.

Figure 1: ReCUBE is constructed through the following steps: (1) Repository Collection from recent, high-quality GitHub repositories. (2) Functional Subset Creation by decomposing the repositories into file clusters that have self-contained functionality. (3) Repository-level Context Extraction from the functional subsets, which captures all the information needed to understand the entire subset. (4) Subset Masking on each functional subset, yielding a total of 366 instances.

Metric	Total	External	Internal
Test Count	10,785	6,334	4,451

Table 2: Test case statistics across 366 target files.

Panel of Figure 2 — Figure 2: Distribution of benchmark variations across functional domains.

Panel of Figure 3 — Figure 3: Distribution of documentation completeness.

Dataset	Average Context	Maximum Context	# Instances
ReCUBE	71K	111K	366
ReCUBE-large	146K	338K	138

Table 3: Statistics for ReCUBE and ReCUBE-large .

	GPT-5 Mini		Qwen3-Coder 30B
Difficulty	$\tau$	$\rho$	$\tau$	$\rho$
Hardest	$-$ 0.184	$-$ 0.259	$+$ 0.020	$+$ 0.026
Hard	$+$ 0.280*	$+$ 0.404*	$+$ 0.304*	$+$ 0.408*
Medium	$+$ 0.125	$+$ 0.172	$-$ 0.171	$-$ 0.217
Easiest	$-$ 0.063	$-$ 0.086	$-$ 0.135	$-$ 0.200
Overall	$+$ 0.009	$+$ 0.013	$-$ 0.073	$-$ 0.106

Table 4: Correlations between caller coverage and Average Pass Rate by task difficulty.

		Overall (%)		External TCs (%)		Internal TCs (%)
Setting	Model	SPR	APR	SPR	APR	SPR	APR
Full-Context	GPT-OSS 20B	25.07	36.83	22.19	32.61	30.63	39.11
	Devstral Small 2507	21.76	30.20	17.36	25.57	27.93	34.99
	Qwen3-Coder 30B	25.34	38.54	21.86	34.44	31.23	40.88
	Claude 3 Haiku	18.66	23.17	14.79	18.85	21.02	25.74
	Gemini 2.5 Flash	25.69	40.28	23.15	37.26	33.03	43.00

Table 5: Experimental Results on ReCUBE . Results show overall Strict Pass Rate (SPR) and Average Pass Rate (APR) along with External and Internal Test Cases (TCs) pass rates. Subscripts denote change from the full-context baseline.

Panel of Figure 4 — Figure 4: Performance Across Functional Domains. We report Strict Pass Rate (%) across six functional domains for eight models in the full-context setting.

Panel of Figure 5 — Figure 5: Absolute Caller Coverage (top) and SPR (bottom) per difficulty quartile for the bash-only Agent vs. Agent +CCE across GPT-5 Mini and Qwen3-Coder 30B.

Setting / Model	GPT-5 mini	Qwen3-Coder-30B	Devstral
Full-Context	55.86 +1.02	38.36 +1.07	13.72 $-$ 16.20
Full-Context + CoT	55.28 $-$ 0.80	34.04 $-$ 3.03	14.20 $-$ 11.34
Agent	54.47 +5.08	35.48 $-$ 5.58	33.74 +0.25
Agent + CCE toolkit	57.72 +1.58	42.60 $-$ 0.11	36.84 $-$ 3.28

Table 6: APR (%) on large-scale benchmark instances. Subscripts denote change in percentage points from ReCUBE .

Scale	Method	Cost	Turns	SPR/Cost	SPR/Turn
Original	Agent	$ 0.1226	21.35	254.6	1.46
Original	Agent (Ours)	$ 0.1241	20.30	312.5	1.91
Large	Agent	$ 0.1410	25.66	221.4	1.22
Large	Agent (Ours)	$ 0.1166	19.81	332.6	1.96

Table 7: Cost-efficiency comparison for GPT-5-mini across two different agent settings and benchmark scales.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Repository	Functionality	Key Components
OpenManus	Agent orchestration with sandbox execution	Base agent, ReAct pattern, tool execution
nanochat	GPT training infrastructure	SFT/RL training, data loading, checkpointing
DeepSeek-OCR	Vision-based OCR system	CLIP/SAM encoders, image/PDF processing
deer-flow	Workflow-based agent with RAG	Graph orchestration, web crawling, retrieval
openai-agents	Core runtime with guardrails	Agent execution, function schemas, prompts
openai-agents	Computer use & file editing	Editor interface, computer control

Appendix

Table 8: Detailed breakdown of the 40 functional subsets in our benchmark. Each entry represents a specific functional subset of a larger repository, selected to target distinct development capabilities.

Parameter	Value
Temperature	0.0
Max Tokens	8,192 (Full-Context)
Max Tokens	16,384 (Full-Context + CoT)
Timeout (s)	300
Inference Backend	vLLM == 0.17.0
enable_auto_tool_choice	False (Default)

Appendix

Table 9: VLLM configuration for Qwen/Qwen3-Coder-30B-A3B-Instruct.

Parameter	Value
Temperature	0.0
Max Tokens	8,192 (Full-Context)
Max Tokens	16,384 (Full-Context + CoT)
Timeout (s)	300
Inference Backend	vLLM == 0.17.0
config-format	mistral

Appendix

Table 10: VLLM configuration for mistralai/Devstral-Small-2507.

Parameter	Value
Temperature	0.0
Max Tokens	8,192 (Full-Context)
Max Tokens	16,384 (Full-Context + CoT)
Timeout (s)	300
Inference Backend	vLLM == 0.17.0
max-model-len	131,072 (Default)

Appendix

Table 11: VLLM configuration for openai/gpt-oss-20b.

Parameter	Value
Max Completion Tokens	128,000
Step Limit	75
Environment	Docker
Timeout (s)	300

Appendix

Table 12: mini-SWE-agent configuration for proprietary models (GPT-5, GPT-5 Mini).

Parameter	Value
Temperature	0.0
Max Completion Tokens	8,192
Step Limit	75
Environment	Apptainer
Timeout (s)	300
Inference Backend	vLLM == 0.17.0

Appendix

Table 13: mini-SWE-agent configuration for open models (Qwen3-Coder 30B, Devstral Small 2507).

	Overall		Ext. TCs		Int. TCs
Setting	SPR	APR	SPR	APR	SPR	APR
Agent	25.06	36.83	22.19	32.61	30.63	39.11
Agent + CCE	21.01	30.49	16.40	23.95	24.32	33.20

Appendix

Table 14: GPT-OSS 20B agent results reported separately from Table 5 due to its incompatibility with mini-SWE-agent, which stems from its exclusive training on OpenAI’s Harmony response format OpenAI (2025) .

Model	Input	Output
gpt-5-mini	0.25	2.00
gemini-2.5-flash	0.30	2.50
gpt-5	1.25	10.00
gemini-2.5-pro	1.25	10.00
claude-3-haiku	0.25	1.25

Appendix

Table 15: API pricing per 1M tokens (USD).

Method	Model	Scale	Cost ($)
Full-Context	claude-3-haiku	Original	0.0202
	gemini-2.5-flash	Original	0.0302
	gemini-2.5-pro	Original	0.1158
	gpt-5	Original	0.1207
	gpt-5-mini	Original	0.0271
Full-Context +CoT	gpt-5	Original	0.1245

Appendix

Table 16: Average cost per instance ($) by method and model.

Repository-Level Code Generation

Repository-level code generation builds or modifies software across files using project context, dependencies, and requirements.

See all 43 papers

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
Oct 29, 2024Shanchao Liang, Yiran Hu, Nan Jiang +1LLM Code Generation Code Generation Evaluation
Purdue University
Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
Jul 27, 2026Bowen Qin, Yi XieSoftware Engineering Agents AI Agent Benchmarks
1National University of Singapore (NUS) · 2Peking University
Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP
Mar 28, 2026Martin Vogel, Falk Meyer-Eschenbach, Severin Kohler +2AI Coding Agents Graph-Based Agent Memory
Independent Researcher, Berlin, Germany · Institute of Medical Informatics, Charité – Universitätsmedizin Berlin, Berlin, Germany · Clinical Study Center, Berlin Institute of Health, Berlin, Germany +3

Oct 29, 2024J/K move · Enter open · S save

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

Shanchao Liang, Yiran Hu, Nan Jiang, Lin Tan

Purdue University

LLM Code Generation Code Generation Evaluation Code Generation Benchmarks

Open paper AskPDF arXiv

Recently, a number of repository-level code generation benchmarks-such as CoderEval, DevEval, RepoEval, RepoBench, and LongCodeArena-have emerged to evaluate the capabilities of large language models (LLMs) beyond standalone benchmarks like HumanEval and MBPP. Thus, a natural question is, would LLMs have similar performance in real world coding tasks as their performance in these benchmarks? Unfortunately, one cannot answer this question, since these benchmarks consist of short completions, synthetic examples, or focus on limited scale repositories, failing to represent real-world coding tasks. To address these challenges, we create REPOCOD, a Python code-generation benchmark containing complex tasks with realistic dependencies in real-world large projects and appropriate metrics for evaluating source code. It includes 980 whole-function generation tasks from 11 popular projects, 50.8% of which require repository-level context. REPOCOD includes 314 developer-written test cases per instance for better evaluation. We evaluate ten LLMs on REPOCOD and find that none achieves more than 30% pass@1 on REPOCOD, indicating the necessity of building stronger LLMs that can help developers in real-world software development. In addition, we found that retrieval-augmented generation achieves better results than using target function dependencies as context.

Abstract

Repository-Level Code Generation

Explore similar work

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'

Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents

Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP