The rapid development of large language models (LLMs) for code, which closely aligns with the practical needs of real-world software development, has sparked growing attention in repository-level code completion. However, existing Code LLMs present challenges in focusing on suitable contexts from the repository and in deeply reasoning about cross-file dependencies. To address these challenges, we propose a novel reinforcement learning framework for repository-level code completion. To better understand both in-file and cross-file contexts, we employ identifier-driven intent recognition to capture completion intent, thereby improving the model's performance in real-world scenarios. To enhance cross-file reasoning, we propose a reward-driven completion learning to effectively reward in complex repository completion scenarios. To guide LLMs in completing code that aligns with intended functionality and repository context, we introduce a selective-exploration strategy that directs the model to focus on low-confidence, high-reward tokens, promoting the exploration of valuable, underexplored completion patterns. Experimental results show that our approach significantly improves the performance of Code LLMs. The code and the dataset are available at https://github.com/zhenlongDai/RepoCompletion.
Figures & tables
Figure 1: Situations of repo-level code completion.
Figure 2: Overview of IntentCoder. (a) Illustration of the Completion-Focused RL process. (b) Illustration of the Selective-Exploration Strategy Optimization.
Language
Sets
Num.
Completion position
IF
CF-F
CF-R
Java&Python
train
32,000
10,100
11,300
10,600
val
1,000
300
400
300
test
6,000
2,000
2,000
2,000
Table 1: Dataset Statistics of ICRepo.
Model
RepoBench-Java
RepoBench-Python
Code Match
Identifier Match
Code Match
Identifier Match
EM
ES
EM
Precision
Recall
F1
EM
ES
EM
Precision
Recall
F1
StarCoder2-7B
12.20
29.78
15.37
24.64
28.19
25.09
1.80
14.47
3.83
7.72
13.25
8.40
CodeLlama-7B
12.40
48.50
15.90
35.60
41.60
36.46
4.47
25.56
6.37
18.26
26.68
18.84
aixcoder-7B-v2
33.07
68.97
38.70
60.57
65.97
61.07
23.20
59.94
29.63
54.70
62.84
55.41
Starcoder2-15B
12.20
28.59
15.67
24.51
27.77
24.92
1.83
13.70
4.07
7.66
13.05
8.34
Table 2: Evaluation results on the RepoBench-Java/Python dataset. All results in the table are reported in percentage ( % ). The best method is in boldface, and the second best method is underlined for each metric.
Model
Code Match
Identifier Match
EM
ES
EM
Precision
Recall
F1
IntentCoder
35.93
71.58
42.70
65.57
63.80
63.37
w/o IDIR
34.60
69.98
41.27
63.74
63.10
62.05
w/o RDCL
34.30
68.97
40.20
62.24
60.87
60.25
w/o CFCL
34.47
67.73
40.70
62.06
59.61
59.45
w/o SESO
34.47
70.43
40.97
64.58
62.99
62.39
Table 3: Results of Ablation study on the RepoBench-Python dataset in Qwen2.5-Coder-7b.
Figure 3: Performance of IntentCoder in Various Completion Scenarios (CF-F,CF-R,IF).
Method
Java
Python
fail
pass
overall
fail
pass
overall
DeepSeek-Coder-6.7B
-131
+351
+220
-130
+189
+59
fine-tuning
-117
+599
+482
-61
+609
+548
GRPO
-78
+456
+378
-40
+691
+651
CoLT
-91
+462
+371
-156
+672
+516
IntentCoder deepseek
-83
+645
+562
-41
+709
+668
Table 4: Result of Exact Match (EM) count.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
model
Java
Python
In-file
→
Both
In-file
→
Both
CodeLlama-7B
290
-143 | +227
374
134
-63 | +100
174
StarCoder2-7B
31
-143 | +192
80
55
-30 | +29
54
DeepSeek-Coder-6.7B
686
-131 |+ 351
906
390
-130 | +189
449
Qwen2.5-Coder-7B
631
-128 | +301
804
489
-138 | +225
576
DeepSeek-Coder-33B
541
-150 | +410
801
373
-166 | +214
421
Appendix
Table 5: Results of Exact Match (EM) Count Code Completion.
Model
BM25
OpenAI text-embedding-ada
Code
Identifier
Code
Identifier
EM
ES
EM
EM
ES
EM
Qwen2.5-Coder-7B
10.33
57.93
18.75
15.71
64.66
26.13
Qwen2.5-Coder-7B fine-tuning
11.87
58.09
19.35
20.35
67.80
27.21
Qwen2.5-Coder-7B GRPO
13.18
60.36
20.06
22.02
68.33
29.41
Qwen2.5-Coder-7B CoLT
12.90
59.72
19.54
23.18
69.72
30.67
Appendix
Table 6: Evaluation results on CrossCodeEval-Java.
Figure 5: The case study for IntentCodercompares it with other Code LLMs.
Model
RepoBench-Java
RepoBench-Python
Code Match
Identifier Match
Code Match
Identifier Match
EM
ES
EM
Precision
Recall
F1
EM
ES
EM
Precision
Recall
F1
IntentCoder Qwen
42.47
77.52
48.57
69.95
69.60
68.75
35.93
71.58
42.70
65.57
63.80
63.37
w/o Identifier-Driven Intent Recognition
40.27
76.77
46.47
68.73
68.62
67.67
34.60
69.98
41.27
63.74
63.10
62.05
w/o Reward-driven Completion Learning
40.80
76.94
47.67
69.56
68.49
68.03
34.30
68.97
40.20
62.24
60.87
60.25
w/o Completion-focused RL
40.90
76.13
47.07
68.49
67.62
66.95
34.47
67.73
40.70
62.06
59.61
59.45
Appendix
Table 7: Results of Ablation study on the RepoBench-java and RepoBench-Python dataset. All results in the table are reported in percentage ( % ).
Understanding large, complex, and rapidly evolving code bases remains a core challenge for software engineering at scale, hindering onboarding, maintenance, refactoring, and impact analysis. Providing this information to a Large-Language Model (LLM) is another hurdle, as the LLM must understand not only the required code method but also its associated methods in the grand scheme of the repository. We propose Code Comprehensor, a hybrid retrieval-reasoning framework combining (i) CodeGraph, a code dependency callgraph, (ii) CodeScribe, which uses code-aware LLMs to generate hierarchical documentation across repository call graphs, and (iii) CodeContrast, a CLIP-style dual-encoder model trained on CodeScribe output to learn aligned text-code embeddings. CodeContrast enables fast, semantically precise retrieval of relevant code-context neighborhoods, which serve as grounded evidence for an MCP-based LLM orchestrator answering domain-specific queries. Evaluation on OpenCV and a proprietary 5M-LOC repository shows that CodeScribe-derived supervision significantly improves cross-modal retrieval in text-to-code accuracy over baseline approaches. Richer textual supervision (hierarchical documentation, parent/sibling context, and LLM-generated synthetic questions) consistently enhances performance across both repositories. LLM-as-a-judge evaluation using the popular RAGAS of the Code Comprehensor output yields above 90% faithfulness scores.
Sujoy Roychowdhury, Joseph H. R. Isaac, Jiangning Gao +2
While Large Language Models (LLMs) have demonstrated exceptional proficiency in code completion, they typically adhere to a Hard Completion (HC) paradigm, compelling the generation of fully concrete code even amidst insufficient context. Our analysis of 3 million real-world interactions exposes the limitations of this strategy: 61% of the generated suggestions were either edited after acceptance or rejected despite exhibiting over 80% similarity to the user's subsequent code, suggesting that models frequently make erroneous predictions at specific token positions. Motivated by this observation, we propose Adaptive Placeholder Completion (APC), a collaborative framework that extends HC by strategically outputting explicit placeholders at high-entropy positions, allowing users to fill directly via IDE navigation. Theoretically, we formulate code completion as a cost-minimization problem under uncertainty. Premised on the observation that filling placeholders incurs lower cost than correcting errors, we prove the existence of a critical entropy threshold above which APC achieves strictly lower expected cost than HC. We instantiate this framework by constructing training data from filtered real-world edit logs and design a cost-based reward function for reinforcement learning. Extensive evaluations across 1.5B--14B parameter models demonstrate that APC reduces expected editing costs from 19% to 50% while preserving standard HC performance. Our work provides both a theoretical foundation and a practical training framework for uncertainty-aware code completion, demonstrating that adaptive abstention can be learned end-to-end without sacrificing conventional completion quality.
Large Language Models (LLMs) have recently emerged as capable coding assistants that operate over large codebases through either agentic exploration or full-context generation. Existing benchmarks capture a broad range of coding capabilities, such as resolving GitHub issues, but none of them directly isolate and measure how effectively LLMs leverage repository-level context during code generation. To address this, we introduce ReCUBE, a benchmark in which LLMs reconstruct a masked file within a real-world repository, using all remaining source files, dependency specifications, and documentation as their only source of context. ReCUBE evaluates reconstructed code with usage-aware test cases that simulate both internal module logic and external cross-file integration, reflecting real-world software usage patterns. We further propose the Caller-Centric Exploration (CCE) toolkit, a set of dependency graph-based tools that can be integrated into agentic frameworks to guide agents toward the most relevant caller files during repository exploration. Experiments across eight models in four settings show that repository-level context utilization remains highly challenging even for state-of-the-art models, with GPT-5 achieving only 37.57% strict pass rate in the full-context setting. Agents augmented with our CCE toolkit consistently outperform all baselines across all evaluated models, with improvements of up to 7.56% in strict pass rate. We release our benchmark, code, and evaluation framework as open source for the NLP research community.
Jiseung Hong, Benjamin G. Ascoli, Jinho D. Choi
Language Technologies Institute, Carnegie Mellon University · Computer Science, Emory University