To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation

Yitong Zhang; Chengze Li; Ruize Chen; Guowei Yang; Xiaoran Jia; Yijie Ren; Jia Li

cs.SEMar 16, 2026

To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation

Authors: Yitong Zhang, Chengze Li, Ruize Chen, Guowei Yang, Xiaoran Jia, Yijie Ren, Jia Li

Organizations: College of AI, Tsinghua University, Beijing, China · Fitten Tech Co., Ltd., Beijing, China · School of Computer Science, Nanjing University, Nanjing, China · Software Institute, Nanjing University, Nanjing, China ·

Abstract

Large Language Models (LLMs) have shown strong potential for code generation, yet they remain limited in private-library-oriented code generation, where the goal is to generate code using APIs from private libraries. Existing approaches mainly rely on retrieving private-library API documentation and injecting relevant knowledge into the context at inference time. However, our study shows that this is insufficient: even given accurate required knowledge, LLMs still struggle to invoke private-library APIs effectively. To address this limitation, we propose PriCoder, an approach that teaches LLMs to invoke private-library APIs through automatically synthesized data. Specifically, PriCoder models private-library data synthesis as the construction of a graph, and alternates between two graph operators: (1) Progressive Graph Evolution, which improves data diversity by progressively synthesizing more diverse training samples from basic ones, and (2) Multidimensional Graph Pruning, which improves data quality through a rigorous filtering pipeline. To support rigorous evaluation, we construct two new benchmarks based on recently released libraries that are unfamiliar to the tested models. Experiments on three mainstream LLMs show that PriCoder substantially improves private-library-oriented code generation, yielding gains of over 20% in pass@1 in many settings, while causing negligible impact on general code generation capability.

Figures & tables

Method	NumbaEval			NdonnxEval
Method	pass@1	pass@3	pass@5	pass@1	pass@3	pass@5
LLaMA3.1-8B-Instruct
Vanilla	8.13	16.97	21.73	15.86	23.50	26.55
Oracle	13.10	26.73	33.66	30.36	42.79	47.45
Gain ( $\uparrow$ )	+4.97	+9.76	+11.93	+14.50	+19.29	+20.90
DeepSeek-Coder-6.7B-Instruct

Table 1. pass@k (%) on NumbaEval and NdonnxEval .

Figure 1. Two cases on NdonnxEval with DeepSeek-Coder-6.7B-Instruct. Despite having access to oracle required API knowledge, the model fails to invoke these APIs effectively.

Figure 2. Overview of PriCoder . Starting from API nodes, Progressive Graph Evolution constructs increasingly diverse sample nodes, while Multidimensional Graph Pruning removes low-quality nodes through syntactic verification, execution verification, and overall functionality verification.

Benchmark	#Instances	Avg. APIs	Avg. Tests	Library	First Release	Used Version Date	#APIs
NdonnxEval	169	4.56	9.00	ndonnx	2024.06	2025.12	179
NumbaEval	187	6.73	9.10	numba-cuda	2024.06	2026.02	725

Table 2. Overview of the constructed benchmarks. #Instances denotes the number of instances in each benchmark; Avg. APIs denotes the average number of required APIs per instance; Avg. Tests denotes the average number of test cases per instance; Library denotes the target private library; First Release denotes the initial release time of the target library; Used Version Date denotes the release time of the library version used in this paper; and #APIs denotes the total number of APIs in the target library.

Model	Method	NdonnxEval						NumbaEval
Model	Method	pass@1	pass@3	pass@5	exec@1	exec@3	exec@5	pass@1	pass@3	pass@5	exec@1	exec@3	exec@5
DeepSeek -6.7B	Vanilla	20.36	29.16	33.26	25.74	37.09	42.20	10.59	21.26	27.82	22.25	42.34	52.23
	+ PriCoder	48.17	60.36	64.92	61.72	74.19	78.58	24.12	38.47	45.04	52.73	74.26	82.21
	Naive RAG	14.97	25.16	29.87	25.15	38.75	43.85	8.07	18.05	24.94	17.38	38.04	50.71
	+ PriCoder	46.80	60.60	65.60	62.49	76.03	80.58	19.30	32.66	39.01	43.48	66.74	75.76
	EpiGen	16.86	25.78	30.27	27.04	40.32	46.31	11.18	23.61	30.97	22.83	45.91	57.40

Table 3. pass@k and exec@k (%) on NdonnxEval and NumbaEval .

Benchmark	Method	Recall
NdonnxEval	Naive RAG	22.94
	CAPIR	18.76
	EpiGen	19.41
NumbaEval	Naive RAG	3.87
	CAPIR	10.36
	EpiGen	4.80

Table 4. Recall (%) of the APIs required by each task for the three non-oracle RAG methods.

Model	Method	NdonnxEval				NumbaEval
Model	Method	Inv. $\uparrow$	Rec. $\uparrow$	N.E. $\downarrow$	S.E. $\downarrow$	Inv. $\uparrow$	Rec. $\uparrow$	N.E. $\downarrow$	S.E. $\downarrow$
DeepSeek -6.7B	Vanilla	92.96	20.25	22.73	2.78	92.25	21.98	6.76	1.02
	Oracle	86.98	48.85	0.48	2.60	93.64	45.55	2.20	3.42
	PriCoder	98.11	66.38	3.40	1.54	95.24	49.64	4.08	0.91
Qwen -7B	Vanilla	95.86	22.55	25.58	4.73	91.93	27.98	7.81	1.04
	Oracle	99.88	47.38	2.02	1.24	90.21	54.29	2.87	2.62

Table 5. Direct measures of private-library API use on NdonnxEval and NumbaEval . Inv. , Rec. , N.E. , and S.E. denote the library invocation rate, required API recall, non-existent-call rate, and signature-error rate, respectively. All values are percentages.

Table 6. pass@k and exec@k (%) on HumanEval under two private-library settings. For retrieval-based baselines, the retrieved corpus is the API documentation of the corresponding library; PriCoder is trained on synthesized data constructed for the same library.

Table 7. pass@k and exec@k (%) on MBPP under two private-library settings. For retrieval-based baselines, the retrieved corpus is the API documentation of the corresponding library. PriCoder is trained on synthesized data constructed for the same library.

Figure 3. Case study on HumanEval with LLaMA-8B under the numba-cuda setting.

Setting	NdonnxEval				NumbaEval
Setting	pass@1	pass@5	exec@1	exec@5	pass@1	pass@5	exec@1	exec@5
Vanilla	23.37 (+0.00)	35.42 (+0.00)	28.52 (+0.00)	43.40 (+0.00)	16.04 (+0.00)	39.51 (+0.00)	26.74 (+0.00)	63.08 (+0.00)
Direct Synthesis	32.96 (+9.59)	40.99 (+5.57)	39.11 (+10.59)	46.83 (+3.43)	17.91 (+1.87)	40.69 (+1.18)	31.55 (+4.81)	66.67 (+3.59)
w/o Graph Pruning	43.85 (+20.48)	57.61 (+22.19)	52.07 (+23.55)	67.36 (+23.96)	28.45 (+12.41)	46.92 (+7.41)	49.25 (+22.51)	76.17 (+13.09)
w/o Graph Evolution	44.85 (+21.48)	57.66 (+22.24)	52.60 (+24.08)	67.02 (+23.62)	32.57 (+16.53)	52.74 (+13.23)	53.96 (+27.22)	79.16 (+16.08)
PriCoder	47.99 (+24.62)	62.76 (+27.34)	55.86 (+27.34)	71.68 (+28.28)	35.61 (+19.57)	58.29 (+18.78)	56.47 (+29.73)	83.60 (+20.52)

Table 8. Ablation study of PriCoder on NdonnxEval and NumbaEval .

Setting	pass@1	pass@5	exec@1	exec@5
Vanilla	85.43 (-0.00)	92.85 (-0.00)	98.29 (-0.00)	99.95 (-0.00)
Direct Synthesis	82.50 (-2.93)	88.25 (-4.60)	97.87 (-0.42)	98.78 (-1.17)
w/o Graph Pruning	82.13 (-3.30)	89.29 (-3.56)	97.74 (-0.55)	98.64 (-1.31)
w/o Graph Evolution	84.63 (-0.80)	90.45 (-2.40)	98.48 (+0.19)	99.91 (-0.04)
PriCoder	84.02 (-1.41)	90.99 (-1.86)	98.23 (-0.06)	99.78 (-0.17)

Table 9. Ablation study of PriCoder on HumanEval under numba-cuda setting .

Figure 4. Distribution of the number of distinct ndonnx APIs invoked per synthesized sample. The annotations report the mean number of APIs per sample, the proportion of samples invoking at least four APIs, and API coverage.

Figure 5. pass@5 and exec@5 on NdonnxEval with different synthesized training data scale.

Panel of Figure 6 — Figure 6. Impact of the model used for synthesizing training data on the effectiveness of PriCoder .

Model	Method	pass@1	pass@5	exec@1	exec@5
Qwen2.5-Coder-3B-Instruct	Vanilla	15.92	29.45	21.42	37.27
Qwen2.5-Coder-3B-Instruct	PriCoder	41.66	58.39	52.37	73.50
Qwen2.5-Coder-7B-Instruct	Vanilla	23.37	35.42	28.52	43.40
Qwen2.5-Coder-7B-Instruct	PriCoder	47.99	62.76	55.86	71.68
Qwen2.5-Coder-14B-Instruct	Vanilla	24.62	36.80	29.59	44.45
Qwen2.5-Coder-14B-Instruct	PriCoder	53.20	66.60	60.95	73.76

Table 10. pass@k and exec@k (%) on NdonnxEval for Qwen models of different scales. For PriCoder , all models are trained on synthetic data synthesized by Qwen-7B for NdonnxEval .

Panel of Figure 7 — Figure 7. Impact of the allocation between Graph Initial Evolution and Graph Iterative Evolution.

Figure 8. pass@5 and exec@5 of DeepSeek-6.7B on NdonnxEval with different API sampling bias strengths

\alpha

Model	Method	NdonnxEval				NumbaEval
Model	Method	pass@1	pass@5	exec@1	exec@5	pass@1	pass@5	exec@1	exec@5
	Vanilla	20.36	33.26	25.74	42.20	10.59	27.82	22.25	52.23
DeepSeek-6.7B	TreeSynth	35.86	50.64	40.30	59.62	14.28	33.09	30.91	65.39
	PriCoder	48.17	64.92	61.72	78.58	24.12	45.04	52.73	82.21
	Vanilla	23.37	35.42	28.52	43.40	16.04	39.51	26.74	63.08
Qwen-7B	TreeSynth	37.99	51.75	47.16	60.99	18.45	41.14	30.27	67.38

Table 11. Comparison with TreeSynth on NdonnxEval and NumbaEval .

Model	Method	NdonnxEval				NumbaEval
Model		pass@1	pass@5	exec@1	exec@5	pass@1	pass@5	exec@1	exec@5
Qwen-7B	Vanilla	23.37	35.42	28.52	43.40	16.04	39.51	26.74	63.08
	Oracle	52.60	66.16	60.24	73.87	24.97	50.54	35.78	71.36
	Oracle with Source Code	53.31	64.82	61.18	73.51	5.08	17.23	10.37	32.93
	API Usage Retrieval	41.89	52.76	49.82	62.52	30.96	46.77	53.10	74.82
	PriCoder	47.99	62.76	55.86	71.68	35.61	58.29	56.47	83.60

Table 12. Comparison of PriCoder with methods that provide source code or retrieved API-usage examples at inference time on NdonnxEval and NumbaEval .

Model	Method	NdonnxEval				NumbaEval
Model	Method	pass@1	pass@5	exec@1	exec@5	pass@1	pass@5	exec@1	exec@5
Qwen-7B	Vanilla	23.37	35.42	28.52	43.40	16.04	39.51	26.74	63.08
	Source-SFT	19.76	27.71	21.95	30.34	2.41	9.11	5.19	18.16
	PriCoder	47.99	62.76	55.86	71.68	35.61	58.29	56.47	83.60
LLaMA-8B	Vanilla	15.86	26.55	23.14	38.65	8.13	21.73	22.78	55.25
	Source-SFT	5.56	7.76	6.27	10.27	1.34	3.60	2.83	9.46

Table 13. Comparison of fine-tuning on library source code with Vanilla and PriCoder on NdonnxEval and NumbaEval . All values are percentages.

Model	Method	NdonnxEval			NumbaEval
Model	Method	pass@1	pass@3	pass@5	pass@1	pass@3	pass@5
Qwen-7B	Vanilla	23.37	31.69	35.42	16.04	31.82	39.51
	Cross-Library Transfer	23.55	31.90	34.71	12.57	25.38	33.21
	PriCoder	47.99	59.25	62.76	35.61	51.83	58.29
LLaMA-8B	Vanilla	15.86	23.50	26.55	8.13	16.97	21.73
	Cross-Library Transfer	12.66	20.79	24.29	6.26	14.51	20.19

Table 14. Cross-library transfer results in pass@k (%) on NdonnxEval and NumbaEval . For Cross-Library Transfer , each model is fine-tuned on the synthesized data constructed for the other library.

Benchmark	Synthesis model	Requirement	Solution	API exact match (%)
NdonnxEval	Qwen-7B	0.608	0.680	0.57
	LLaMA-8B	0.609	0.666	0.36
	DeepSeek-6.7B	0.608	0.675	1.15
NumbaEval	Qwen-7B	0.718	0.785	0.25
	LLaMA-8B	0.718	0.736	0.25
	DeepSeek-6.7B	0.702	0.744	0.03

Table 15. Mean train–test similarity for each synthesized dataset. Requirement and solution columns report mean maximum cosine similarity; API exact match reports the percentage of training samples whose API set matches at least one benchmark instance.

Figure 9. Train–test similarity distributions on NdonnxEval (top) and NumbaEval (bottom). Columns show requirement (left) and solution (right). Four panels compare three synthesis models on two libraries. Rows show Ndonnx and Numba; columns show maximum cosine similarity histograms for requirements and solutions.

Model	Condition	Retained	pass@1	pass@5	exec@1	exec@5
Qwen-7B	Similarity filtering	18,346	46.63	57.82	54.32	65.84
Qwen-7B	Random deletion	18,346	46.75	58.44	54.67	65.10
LLaMA-8B	Similarity filtering	18,432	32.25	49.52	48.93	70.50
LLaMA-8B	Random deletion	18,432	32.43	49.58	47.87	67.97

Table 16. Performance (%) on NdonnxEval after similarity filtering and random deletion.

Figure 10. Effect of synthesized training-data scale after similarity filtering on NdonnxEval with DeepSeek-6.7B . Random deletion retains the same number of samples at each size. Two line charts compare pass at 5 and exec at 5 after similarity filtering and size-matched random deletion. Both metrics increase consistently with the number of retained training samples under both conditions.

Model	Library	recall@1	recall@5
DeepSeek-6.7B	ndonnx	0.22	1.12
DeepSeek-6.7B	numba-cuda	2.99	6.12
Qwen-7B	ndonnx	0.22	1.12
Qwen-7B	numba-cuda	7.32	11.55
LLaMA-8B	ndonnx	0.00	0.00
LLaMA-8B	numba-cuda	0.17	0.57

Table 17. Zero-shot exact-signature recall (%) on the target libraries.

Figure 11. Performance (%) of PriCoder on NdonnxEval across four runs of data synthesis and fine-tuning per model. Points show means, and error bars indicate standard deviations.

Model	Synthesis (8 GPUs)	Fine-tuning (2 GPUs)	Total time	GPU-hours
Qwen-7B	15:26:23	00:33:17	15:59:40	124.63
DeepSeek-6.7B	22:57:22	00:34:44	23:32:06	184.81
LLaMA-8B	27:09:15	00:41:26	27:50:41	218.61

Table 18. Cost of PriCoder for synthesizing 20K ndonnx training samples and fine-tuning each model. Times are reported as hh:mm:ss. GPU-hours are calculated as eight times the synthesis duration plus twice the fine-tuning duration, in hours.

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 311 papers

Learning from Execution: Self-Evolving Memory for Private-Library Code Generation
Apr 27, 2026Mofei Li, Taozhi Chen, Guowei Yang +1Memory-Augmented Language Models Knowledge Augmentation for Language Models
College of AI Tsinghua University Beijing, China · Fitten Tech Co., Ltd. Beijing, China · Emory University Atlanta, GA, USA
NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs
Jun 2, 2026Jinnuo Liu, Yue Peng, Jinhan Niu +1Knowledge Augmentation for Language Models LLM Fine-Tuning
NYU Shanghai
Embarrassingly Simple Self-Distillation Improves Code Generation
Apr 1, 2026Ruixiang Zhang, Richard He Bai, Huangjie Zheng +3Language Model Decoding Language Model Distillation
Apple

Apr 27, 2026J/K move · Enter open · S save

Learning from Execution: Self-Evolving Memory for Private-Library Code Generation

Mofei Li, Taozhi Chen, Guowei Yang, Jia Li

College of AI Tsinghua University Beijing, China · Fitten Tech Co., Ltd. Beijing, China · Emory University Atlanta, GA, USA

Memory-Augmented Language Models Knowledge Augmentation for Language Models Execution-Guided Code Generation

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved strong performance on general code generation, but their effectiveness drops sharply in enterprise settings where software development relies on internal private libraries absent from public pre-training corpora. Existing Retrieval-Augmented Generation (RAG) methods provide a training-free solution by retrieving static API documentation, but our analysis shows that documentation mainly helps models identify what APIs to use and remains insufficient for teaching how to use them correctly. Even with oracle API-document retrieval, LLMs still make recurring errors at the API, cross-API, and task levels, including API misuse or hallucination, flawed API composition, and incorrect solution strategies. To address this limitation, we propose MEMCoder, a training-free self-evolving memory framework for private-library code generation. MEMCoder augments existing RAG pipelines with a Multi-level Evolving Memory that continuously accumulates and reuses execution-derived Usage Guidelines at the API, cross-API, and task levels. During generation, MEMCoder retrieves both static API documentation and relevant historical memories to guide code generation; after execution, it analyzes feedback to refine memory through a closed loop of generation, execution, reflection, and update. Extensive experiments on NdonnxEval and NumbaEval show that MEMCoder consistently enhances different RAG backbones across LLMs of different scales, yielding an average absolute pass@1 improvement of 18.41 percentage points. Moreover, MEMCoder outperforms existing self-evolving memory methods and validates the effectiveness of organizing execution feedback into multi-level usage memories.

Abstract

LLM Code Generation

Explore similar work

Learning from Execution: Self-Evolving Memory for Private-Library Code Generation

NovelAPIBench: Diagnosing How A Code LLM Learns to Use Novel APIs

Embarrassingly Simple Self-Distillation Improves Code Generation