Automated program comprehension underpins many software engineering tasks, from code summarisation to clone detection. While recent deep learning models have achieved strong results, they typically rely on source code alone, overlooking contextual signals such as version history or structural relationships. We conduct an empirical study of how enriching code representations with such signals affects model performance on four tasks: code clone detection, code classification, code summarisation, and vulnerability detection, using three datasets (SeSaMe, CodeSearchNet-Java, Vul4J). We evaluate five foundational code models (CodeBERT, GraphCodeBERT, CodeT5, PLBART, ASTNN) and seven code-specialised LLMs (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder), comparing prompt-based context injection against fine-tuned use of context. Context improves classification for all five foundational models and, after fine-tuning, for most LLMs. For summarisation, the effect depends on the model family: version history improves all four foundational models on every metric, whereas prompt-level context helps two of the seven LLMs and hurts the other five. Combining context types reaches up to +21.48% macro-F1 in code classification and +15.04% F1 in clone detection. A human evaluation of CodeT5 summaries on 100 Java snippets ranks context-augmented summaries significantly higher for Accuracy and Content Adequacy. An exploratory vulnerability detection evaluation provides initial evidence of transfer to security-oriented tasks.
Figures & tables
Figure 1. A motivating example of using Version History to detect code clones. Example of using version history to support code clone detection.
Figure 4. Research Framework. Icons designed by Magnific, Vectors Tank, and Afian Rochmah Afif from Flaticon (flaticon.com). A three-step research framework showing the encoding of source code and additional context, the aggregation of their embeddings, and the final modelling for downstream tasks.
Dataset Size
Source Statistics
Version History
Call Graph
Method Age (days)
Partition
Clone Pairs a
Unique Methods
Projects
Files
Total
Max
Min
Total
Max
Min
Max
Min
Train
677
1,296
11
932
8,074
727
1
8,104
383
1
6,334
34
Valid
84
164
11
151
855
245
1
1,341
205
1
6,334
34
Test
86
170
11
156
995
243
1
863
166
1
6,334
125
Table 1. SeSaMe - Dataset Partitions and Additional Context Statistics
Version History
Call Graph
Method Age (days)
Partition
Proj.
File
Method
Total
Max
Min
Total
Max
Min
Max
Min a
Train
1,905
22,190
58,339
179,471
131
1
309,546
2,622
1
7,368
0
Valid
97
528
1,271
3,918
29
1
6,882
382
1
4,742
0
Test
115
1,412
3,649
10,377
45
1
17,145
1,189
1
4,143
0
Table 2. CodeSearchNet - Dataset Partitions and Additional Context Statistics
Figure 5. Concatenation Scenario. Concatenation.
Figure 6. Max-Pooling Scenario. Maxpool.
Figure 7. Concatenation of Absolute Difference Scenario. Concatenation of Absolute Difference.
Table 3. Comprehensive performance of foundational models and LLMs (1.5B–34B) on Code Clone Detection and Code Classification. For LLMs, results are reported in zero-shot (Z) and fine-tuned (F) modes. Columns headed % give the relative change of each context condition over the code-only baseline (W/O Ctx) of the same model and, for LLMs, the same mode; the table does not compare zero-shot against fine-tuned results. Accuracy has been omitted for brevity.
Table 4. Master performance of foundational models and LLMs on Code Summarisation (BLEU-4, BERTScore-F1, ROUGE-L, and METEOR). For LLMs, results are reported in zero-shot (Z) and fine-tuned (F) modes. Columns headed % give the relative change of each context condition over the code-only baseline (W/O Ctx) of the same model and, for LLMs, the same mode; the table does not compare zero-shot against fine-tuned results.
Table 5. Human evaluation of CodeT5-generated summaries with additional context (Win/Tie/Loss analysis by dimension).
Dimension
Context(s)
# Rank 1
# Rank 2
# Rank 3
# Rank ≥ 4
Mean Rank
Accuracy
Without Context
59
33
6
2
1.51
Code + CallGraph
73 ( ↑ 23.73%)
22
5
0
1.32
Code + VersionHistory
67 ( ↑ 13.56%)
28
5
0
1.38
Conciseness
Without Context
55
31
14
0
1.59
Code + CallGraph
69 ( ↑ 25.45%)
25
4
2
1.39
Code + VersionHistory
53 ( ↓ 3.64%)
33
13
1
1.62
Table 6. Ranking distribution and mean ranks of CodeT5-generated summaries with additional context (human evaluation).
Table 7. Model performance on Code Clone Detection with combined context types (automated evaluation). Columns headed % give the relative change of each context condition over the code-only baseline (W/O Ctx) of the same model and, for LLMs, the same mode (Z = zero-shot, F = fine-tuned); the table does not compare zero-shot against fine-tuned results.
Table 8. Model performance on Code Classification with combined context types (automated evaluation). Columns headed % give the relative change of each context condition over the code-only baseline (W/O Ctx) of the same model and, for LLMs, the same mode (Z = zero-shot, F = fine-tuned); the table does not compare zero-shot against fine-tuned results.
Models
Metrics
W/O Ctx
Vers. Hist. + CG
%
Vers. Hist. + M. Age
%
Vers. Hist. + CG + M. Age
%
CG + Vers. Hist.
%
CG + Vers. Hist. + M. Age
%
CodeBERT
BLEU-4
17.12
17.26
0.82%
17.34
1.29%
17.42
1.75%
17.23
0.64%
17.29
0.99%
BERTScore-F1
0.2570
0.2537
-1.28%
0.2516
-2.11%
0.2504
-2.56%
0.2532
-1.47%
0.2523
-1.82%
ROUGE-L
0.3058
0.3035
-0.76%
0.3047
-0.38%
0.3059
0.02%
0.3020
-1.27%
0.3024
-1.11%
METEOR
0.2626
0.2636
0.40%
0.2610
-0.61%
0.2614
-0.46%
0.2626
0.00%
0.2614
-0.45%
GraphCodeBERT
BLEU-4
17.85
18.18
1.85%
17.74
-0.62%
18.09
1.34%
17.74
-0.62%
17.97
0.67%
BERTScore-F1
0.3993
0.4045
1.30%
0.4022
0.73%
0.4072
1.98%
0.3966
-0.68%
0.4053
1.50%
Table 9. Automated evaluation of Code Summarisation with combined context types (BLEU-4, METEOR, ROUGE-L, and BERTScore-F1). Columns headed % give the relative change of each context condition over the code-only baseline (W/O Ctx) of the same model and, for LLMs, the same mode (Z = zero-shot, F = fine-tuned); the table does not compare zero-shot against fine-tuned results.
Table 10. Human evaluation of CodeT5-generated summaries with combined context types (Win/Tie/Loss analysis by dimension).
Dimension
Context(s)
# Rank 1
# Rank 2
# Rank 3
# Rank ≥ 4
Mean Rank
Accuracy
Without Context
59
33
6
2
1.51
Code + CallGraph
73 ( ↑ 23.73%)
22
5
0
1.32
Code + VersionHistory
67 ( ↑ 13.56%)
28
5
0
1.38
Code + CallGraph + VersionHistory
72 ( ↑ 22.03%)
21
6
1
1.36
Code + VersionHistory + CallGraph
72 ( ↑ 22.03%)
22
6
0
1.34
Code + CallGraph + VersionHistory + MethodAge
65 ( ↑ 10.17%)
33
2
0
1.37
Table 11. Ranking distribution and mean ranks of CodeT5-generated summaries with combined context types (human evaluation).
Table 12. Code Clone Detection Results with 3 Different Aggregation Techniques.
Table 13. Code Classification Results with 3 Different Aggregation Techniques.
Task
Context
No-Context
Concat
Projection
Δ vs. Baseline
Clone Det.
Code + VH
0.7180
0.7442
0.6786
− 5.5%
Code + CG
0.7180
0.7907
0.7000
− 2.5%
Code + VH + CG
0.7180
0.7500
0.7308
+ 1.8%
Classification
Code + VH
0.6698
0.6320
0.5852
− 12.6%
Code + CG
0.6698
0.7345
0.6551
− 2.2%
Code + VH + CG
0.6698
0.7304
0.5816
− 13.2%
Table 14. Fixed-dimension projection control experiment (CodeBERT). A linear projection layer maps the concatenated representation back to 768 dim before the task head, holding classifier capacity constant regardless of context count. F1 is reported for clone detection; macro-F1 for classification. Δ denotes change relative to the no-context baseline.
Task
Condition
No-Context
With VH
Δ vs. Baseline
Clone Det.
code_vh_1 ( N =1)
0.7180
0.7273
+ 1.29%
code_vh_3 ( N =3)
0.7180
0.7241
+ 0.85%
code_vh_all (default)
0.7180
0.7273
+ 1.29%
Classification
code_vh_1 ( N =1)
0.6698
0.6027
− 10.02%
code_vh_3 ( N =3)
0.6698
0.6081
− 9.22%
code_vh_all (default)
0.6698
0.6089
− 9.10%
Table 15. History depth ablation (CodeBERT, SeSaMe dataset). All three conditions use the same architecture; only the number of historical versions in code_vh varies. F1 is reported for clone detection; macro-F1 for classification. Δ denotes change relative to the no-context baseline.
Model
Metric
Baseline
code_cg
code_vh
code_cg_vh
code_vh_nod
code_cg_vh_nod
code_vh_cg
code_vh_cg_nod
Val.
%
Val.
%
Val.
%
Val.
%
Val.
%
Val.
%
Val.
%
CodeLlama-34B
F1
0.1053
0.1311
+24.6%
0.1017
-3.4%
0.1519
+44.3%
0.1111
+5.6%
0.1348
+28.1%
0.1035
-1.7%
0.1600
+52.0%
Prec.
0.1250
0.1026
-18.0%
0.0811
-35.1%
0.1053
-15.8%
0.0735
-41.2%
0.0896
-28.4%
0.0833
-33.3%
0.1132
-9.4%
Rec.
0.0909
0.1818
+100.0%
0.1364
+50.0%
0.2727
+200.0%
0.2273
+150.0%
0.2727
+200.0%
0.1364
+50.0%
0.2727
+200.0%
Acc.
0.8090
0.7023
-13.2%
0.7023
-13.2%
0.6236
-22.9%
0.5506
-31.9%
0.5674
-29.9%
0.7079
-12.5%
0.6461
-20.1%
CodeLlama-13B
F1
0.1667
0.1304
-21.7%
0.2041
+22.4%
0.1277
-23.4%
0.1538
-7.7%
0.0454
-72.7%
0.1333
-20.0%
0.1053
-36.8%
Table 16. Zero-shot vulnerability detection on the Vul4J benchmark. Results show F1, Precision (Prec.), Recall (Rec.), and Accuracy (Acc.) under seven context conditions. Fine-tuning is not applicable due to the small dataset size.
Dataset Size
Version History
Call Graph
Method Age (days)
Date Range
Partition
Proj.
Files
Methods
Max
Min
Avg
Max
Min
Avg
Max
Min
Avg
Oldest
Newest
Vulnerable
21
22
22
33
1
4.9
295
1
78.2
3,695.9
0.0
483.4
2009-02-24
2019-05-07
Clean a
21
22
156
34
1
3.1
643
0
46.3
4,033.7
0.0
436.5
2009-02-24
2020-11-04
Entire Dataset
21
22
178
34
1
3.3
643
0
50.3
4,033.7
0.0
442.3
2009-02-24
2020-11-04
Table 17. Vul4J – Dataset Partitions and Additional Context Statistics
Bucket (Jaccard range)
Clone pairs
Baseline correct
Context correct
A: 0.00–0.25 (low surface similarity)
10
7
9
B: 0.25–0.50 (moderate)
12
10
9
C: 0.50–0.75 (high)
1
0
0
D: 0.75–1.00 (very high)
2
1
2
Table 18. Stratification of SeSaMe clone pairs (label =1 , N=25 ) by token-level Jaccard similarity between the two methods. Per-bucket counts of pairs correctly classified by the code-only baseline and by the context-augmented CodeT5 ( code+VH+CG+age , concat aggregation).
Change category
Count
Callee semantic injection (context introduces a concept that appears only in the callee body)
9
Action-verb correction (baseline picks a generic verb; context picks a task-specific verb)
7
Identifier accuracy (baseline hallucinates or generalises; context selects the right noun)
5
Caller semantic alignment (context aligns terminology with the caller’s parameter naming)
3
Conciseness improvement (context summary is shorter while preserving meaning)
1
Conciseness regression (context absorbs caller/callee terminology and grows the summary)
1
Table 19. Manual classification of the dominant change between the baseline and the best context-augmented summary, across the 26 CodeSearchNet snippets with context wins in all three human-evaluation dimensions.
Variant
Generated summary
Rank (Acc / Cont / Conc)
codet5_baseline
“Notifies all listeners that a Throwable has been thrown”
3 / 2 / 3
codet5_code_cg (best)
“Fires a throwable to all listeners”
2 / 1 / 1
Table 25
Variant
Generated summary
Rank (Acc / Cont / Conc)
codet5_baseline
“Optimize the regular expression”
4 / 3 / 3
codet5_code_cg (best)
“Optimize the NFA”
2 / 1 / 1
Table 26
Variant
Generated summary
Rank (Acc / Cont / Conc)
codet5_baseline
“Returns an AsyncDatastore that wraps the underlying Datastore”
3 / 3 / 3
codet5_code_cg_vh (best)
“Creates an async datastore with caching enabled or disabled”
1 / 1 / 2
Table 27
Variable
Retained
Discarded
Cliff’s δ
Methods per repository
9 [3, 26]
11 [4, 35]
0.07 (negligible)
Method length (code tokens)
66 [43, 116]
70 [44, 124]
0.04 (negligible)
Prior versions per method
1 [1, 2]
1 [1, 2]
0.03 (negligible)
Method age (days)
903 [358, 1,548]
794 [390, 1,416]
0.03 (negligible)
Build system
Retained ( n )
Discarded ( n )
Retention rate
Maven
2,090
781
72.8%
Table 20. Retained-versus-discarded comparison for call-graph mining on CodeSearchNet-Java. Retained: 2,269 repositories (79,730 methods) that built successfully from their pinned commit. Discarded: 1,460 repositories (68,109 methods) with a Maven or Gradle build file that failed to build. Values are medians with interquartile ranges in brackets; Cliff’s δ is the effect size (negligible below 0.147).
Model
Condition
P
R
macro-F1
Δ vs. Baseline
CodeBERT
Baseline (no context)
0.7103
0.6498
0.6698
(reference)
Original context
0.8504
0.7522
0.7813
+ 16.65%
Shuffled context, seed 42
0.5739
0.5352
0.5411
− 19.21%
Shuffled context, seed 1234
0.4580
0.4360
0.4195
− 37.37%
Shuffled context, mean of seeds
0.5160
0.4856
0.4803
− 28.29%
CodeT5
Baseline (no context)
0.7005
0.6364
0.6569
(reference)
Table 21. Across-project shuffled-context control experiment on Code Classification (SeSaMe). For each test method with project label Pi , the version history and call-graph payload is replaced with that of a uniformly randomly chosen donor with a different label; the method’s own source code and label are preserved. We report two random seeds (42 and 1234) and their mean. P and R denote macro-averaged precision and recall; macro-F1 is the primary metric for this multi-class task. Δ denotes percentage change in macro-F1 relative to the no-context baseline.
Split
Condition
Median
p25
p75
Train
Original
108
54
259
Shuffled
111
54
291
Dev
Original
102
53
255
Shuffled
102
59
317
Test
Original
103
60
239
Shuffled
108
65
338
Table 22. Context-token-length distribution for the original and shuffled conditions of the leakage control experiment (Table 21 ), by split. The shuffled context is drawn from a different project per row by uniform random sampling. Medians and interquartile ranges are comparable between conditions across all three splits, confirming that no systematic length confound was introduced by shuffling.
Step
CodeSearchNet
SeSaMe
Vul4J
Repo clone
40.6 h
0.46 h
0.70 h
Compile (Maven/Gradle)
5.5 h
0.37 h
0.35 h
Call graph extraction
4.1 h
0.05 h
0.06 h
Version history mining
8.0 h
0.46 h
0.03 h
Method age
≈ 0 (derived from VH first-commit timestamp)
Total
∼ 58 h
∼ 1.3 h
∼ 1.1 h
Table 23. Preprocessing pipeline wall-clock time (5 servers, 4 vCPU / 16 GB RAM each). CodeSearchNet figures are measured; SeSaMe and Vul4J are estimated using the same per-repository and per-method rates.
Semantic code search and clone detection are essential for software development, maintenance, and reuse. This paper evaluates the effectiveness, efficiency, and scalability of contemporary deep learning models for first-stage recall in large-scale code-to-code search engines. Benchmarking across multiple programming languages and datasets reveals critical limits in the precision and scalability of these models on Terabyte-scale source-code collections. We present LLM-based code normalisation and query-rewriting schemes that yield significant gains in precision for lower-performing models. Our results question the sustainability of resource-constrained deployment and the assumed robustness of current code-specialised LLMs across datasets. We conclude with actionable insights for building scalable, efficient code-retrieval systems.
Leonardo Venuta, Francesco Tosoni, Paolo Ferragina
Code summarization is the task of generating natural language descriptions of source code, which is critical for software comprehension and maintenance. While large language models (LLMs) have achieved remarkable progress on this task, an open question remains: can human expertise in code understanding further guide and enhance these models? We propose EyeLayer, a lightweight attention-augmentation module that incorporates human eye-gaze patterns, as a proxy of human expertise, into LLM-based code summarization. EyeLayer models human attention during code reading via a Multimodal Gaussian Mixture, redistributing token embeddings based on learned parameters (μ_i, σ_i^2) that capture where and how intensively developers focus. This design enables learning generalizable attention priors from eye-tracking data and incorporating them into LLMs seamlessly, without disturbing existing representations. We evaluate EyeLayer across diverse model families (i.e., LLaMA-3.2, Qwen3, and CodeBERT) covering different scales and architectures. EyeLayer consistently outperforms strong fine-tuning baselines across standard metrics, achieving gains of up to 13.17% on BLEU-4. These results demonstrate that human gaze patterns encode complementary attention signals that enhance the semantic focus of LLMs and transfer effectively across diverse models for code summarization.
Understanding large, complex, and rapidly evolving code bases remains a core challenge for software engineering at scale, hindering onboarding, maintenance, refactoring, and impact analysis. Providing this information to a Large-Language Model (LLM) is another hurdle, as the LLM must understand not only the required code method but also its associated methods in the grand scheme of the repository. We propose Code Comprehensor, a hybrid retrieval-reasoning framework combining (i) CodeGraph, a code dependency callgraph, (ii) CodeScribe, which uses code-aware LLMs to generate hierarchical documentation across repository call graphs, and (iii) CodeContrast, a CLIP-style dual-encoder model trained on CodeScribe output to learn aligned text-code embeddings. CodeContrast enables fast, semantically precise retrieval of relevant code-context neighborhoods, which serve as grounded evidence for an MCP-based LLM orchestrator answering domain-specific queries. Evaluation on OpenCV and a proprietary 5M-LOC repository shows that CodeScribe-derived supervision significantly improves cross-modal retrieval in text-to-code accuracy over baseline approaches. Richer textual supervision (hierarchical documentation, parent/sibling context, and LLM-generated synthetic questions) consistently enhances performance across both repositories. LLM-as-a-judge evaluation using the popular RAGAS of the Code Comprehensor output yields above 90% faithfulness scores.
Sujoy Roychowdhury, Joseph H. R. Isaac, Jiangning Gao +2