Enhancing Neural Code Representation with Additional Context
Organizations: The University of Melbourne, Australia · Singapore Management University, Singapore
Abstract
Automated program comprehension underpins many software engineering tasks, from code summarisation to clone detection. While recent deep learning models have achieved strong results, they typically rely on source code alone, overlooking contextual signals such as version history or structural relationships. We conduct an empirical study of how enriching code representations with such signals affects model performance on four tasks: code clone detection, code classification, code summarisation, and vulnerability detection, using three datasets (SeSaMe, CodeSearchNet-Java, Vul4J). We evaluate five foundational code models (CodeBERT, GraphCodeBERT, CodeT5, PLBART, ASTNN) and seven code-specialised LLMs (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder), comparing prompt-based context injection against fine-tuned use of context. Context improves classification for all five foundational models and, after fine-tuning, for most LLMs. For summarisation, the effect depends on the model family: version history improves all four foundational models on every metric, whereas prompt-level context helps two of the seven LLMs and hurts the other five. Combining context types reaches up to +21.48% macro-F1 in code classification and +15.04% F1 in clone detection. A human evaluation of CodeT5 summaries on 100 Java snippets ranks context-augmented summaries significantly higher for Accuracy and Content Adequacy. An exploratory vulnerability detection evaluation provides initial evidence of transfer to security-oriented tasks.
Figures & tables
| Dataset Size | Source Statistics | Version History | Call Graph | Method Age (days) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Partition | Clone Pairs a | Unique Methods | Projects | Files | Total | Max | Min | Total | Max | Min | Max | Min |
| Train | 677 | 1,296 | 11 | 932 | 8,074 | 727 | 1 | 8,104 | 383 | 1 | 6,334 | 34 |
| Valid | 84 | 164 | 11 | 151 | 855 | 245 | 1 | 1,341 | 205 | 1 | 6,334 | 34 |
| Test | 86 | 170 | 11 | 156 | 995 | 243 | 1 | 863 | 166 | 1 | 6,334 | 125 |
| Version History | Call Graph | Method Age (days) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Partition | Proj. | File | Method | Total | Max | Min | Total | Max | Min | Max | Min a |
| Train | 1,905 | 22,190 | 58,339 | 179,471 | 131 | 1 | 309,546 | 2,622 | 1 | 7,368 | 0 |
| Valid | 97 | 528 | 1,271 | 3,918 | 29 | 1 | 6,882 | 382 | 1 | 4,742 | 0 |
| Test | 115 | 1,412 | 3,649 | 10,377 | 45 | 1 | 17,145 | 1,189 | 1 | 4,143 | 0 |
| Dimension | Context(s) | # Rank 1 | # Rank 2 | # Rank 3 | # Rank 4 | Mean Rank |
|---|---|---|---|---|---|---|
| Accuracy | Without Context | 59 | 33 | 6 | 2 | 1.51 |
| Code + CallGraph | 73 ( 23.73%) | 22 | 5 | 0 | 1.32 | |
| Code + VersionHistory | 67 ( 13.56%) | 28 | 5 | 0 | 1.38 | |
| Conciseness | Without Context | 55 | 31 | 14 | 0 | 1.59 |
| Code + CallGraph | 69 ( 25.45%) | 25 | 4 | 2 | 1.39 | |
| Code + VersionHistory | 53 ( 3.64%) | 33 | 13 | 1 | 1.62 |
| Models | Metrics | W/O Ctx | Vers. Hist. + CG | % | Vers. Hist. + M. Age | % | Vers. Hist. + CG + M. Age | % | CG + Vers. Hist. | % | CG + Vers. Hist. + M. Age | % | ||||||
| CodeBERT | BLEU-4 | 17.12 | 17.26 | 0.82% | 17.34 | 1.29% | 17.42 | 1.75% | 17.23 | 0.64% | 17.29 | 0.99% | ||||||
| BERTScore-F1 | 0.2570 | 0.2537 | -1.28% | 0.2516 | -2.11% | 0.2504 | -2.56% | 0.2532 | -1.47% | 0.2523 | -1.82% | |||||||
| ROUGE-L | 0.3058 | 0.3035 | -0.76% | 0.3047 | -0.38% | 0.3059 | 0.02% | 0.3020 | -1.27% | 0.3024 | -1.11% | |||||||
| METEOR | 0.2626 | 0.2636 | 0.40% | 0.2610 | -0.61% | 0.2614 | -0.46% | 0.2626 | 0.00% | 0.2614 | -0.45% | |||||||
| GraphCodeBERT | BLEU-4 | 17.85 | 18.18 | 1.85% | 17.74 | -0.62% | 18.09 | 1.34% | 17.74 | -0.62% | 17.97 | 0.67% | ||||||
| BERTScore-F1 | 0.3993 | 0.4045 | 1.30% | 0.4022 | 0.73% | 0.4072 | 1.98% | 0.3966 | -0.68% | 0.4053 | 1.50% | |||||||
| Dimension | Context(s) | # Rank 1 | # Rank 2 | # Rank 3 | # Rank 4 | Mean Rank |
|---|---|---|---|---|---|---|
| Accuracy | Without Context | 59 | 33 | 6 | 2 | 1.51 |
| Code + CallGraph | 73 ( 23.73%) | 22 | 5 | 0 | 1.32 | |
| Code + VersionHistory | 67 ( 13.56%) | 28 | 5 | 0 | 1.38 | |
| Code + CallGraph + VersionHistory | 72 ( 22.03%) | 21 | 6 | 1 | 1.36 | |
| Code + VersionHistory + CallGraph | 72 ( 22.03%) | 22 | 6 | 0 | 1.34 | |
| Code + CallGraph + VersionHistory + MethodAge | 65 ( 10.17%) | 33 | 2 | 0 | 1.37 |
| Task | Context | No-Context | Concat | Projection | vs. Baseline |
|---|---|---|---|---|---|
| Clone Det. | Code + VH | 0.7180 | 0.7442 | 0.6786 | 5.5% |
| Code + CG | 0.7180 | 0.7907 | 0.7000 | 2.5% | |
| Code + VH + CG | 0.7180 | 0.7500 | 0.7308 | 1.8% | |
| Classification | Code + VH | 0.6698 | 0.6320 | 0.5852 | 12.6% |
| Code + CG | 0.6698 | 0.7345 | 0.6551 | 2.2% | |
| Code + VH + CG | 0.6698 | 0.7304 | 0.5816 | 13.2% |
| Task | Condition | No-Context | With VH | vs. Baseline |
|---|---|---|---|---|
| Clone Det. | code_vh_1 ( =1) | 0.7180 | 0.7273 | 1.29% |
| code_vh_3 ( =3) | 0.7180 | 0.7241 | 0.85% | |
| code_vh_all (default) | 0.7180 | 0.7273 | 1.29% | |
| Classification | code_vh_1 ( =1) | 0.6698 | 0.6027 | 10.02% |
| code_vh_3 ( =3) | 0.6698 | 0.6081 | 9.22% | |
| code_vh_all (default) | 0.6698 | 0.6089 | 9.10% |
| Model | Metric | Baseline | code_cg | code_vh | code_cg_vh | code_vh_nod | code_cg_vh_nod | code_vh_cg | code_vh_cg_nod | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Val. | % | Val. | % | Val. | % | Val. | % | Val. | % | Val. | % | Val. | % | |||
| CodeLlama-34B | F1 | 0.1053 | 0.1311 | +24.6% | 0.1017 | -3.4% | 0.1519 | +44.3% | 0.1111 | +5.6% | 0.1348 | +28.1% | 0.1035 | -1.7% | 0.1600 | +52.0% |
| Prec. | 0.1250 | 0.1026 | -18.0% | 0.0811 | -35.1% | 0.1053 | -15.8% | 0.0735 | -41.2% | 0.0896 | -28.4% | 0.0833 | -33.3% | 0.1132 | -9.4% | |
| Rec. | 0.0909 | 0.1818 | +100.0% | 0.1364 | +50.0% | 0.2727 | +200.0% | 0.2273 | +150.0% | 0.2727 | +200.0% | 0.1364 | +50.0% | 0.2727 | +200.0% | |
| Acc. | 0.8090 | 0.7023 | -13.2% | 0.7023 | -13.2% | 0.6236 | -22.9% | 0.5506 | -31.9% | 0.5674 | -29.9% | 0.7079 | -12.5% | 0.6461 | -20.1% | |
| CodeLlama-13B | F1 | 0.1667 | 0.1304 | -21.7% | 0.2041 | +22.4% | 0.1277 | -23.4% | 0.1538 | -7.7% | 0.0454 | -72.7% | 0.1333 | -20.0% | 0.1053 | -36.8% |
| Dataset Size | Version History | Call Graph | Method Age (days) | Date Range | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Partition | Proj. | Files | Methods | Max | Min | Avg | Max | Min | Avg | Max | Min | Avg | Oldest | Newest |
| Vulnerable | 21 | 22 | 22 | 33 | 1 | 4.9 | 295 | 1 | 78.2 | 3,695.9 | 0.0 | 483.4 | 2009-02-24 | 2019-05-07 |
| Clean a | 21 | 22 | 156 | 34 | 1 | 3.1 | 643 | 0 | 46.3 | 4,033.7 | 0.0 | 436.5 | 2009-02-24 | 2020-11-04 |
| Entire Dataset | 21 | 22 | 178 | 34 | 1 | 3.3 | 643 | 0 | 50.3 | 4,033.7 | 0.0 | 442.3 | 2009-02-24 | 2020-11-04 |
| Bucket (Jaccard range) | Clone pairs | Baseline correct | Context correct |
|---|---|---|---|
| A: 0.00–0.25 (low surface similarity) | 10 | 7 | 9 |
| B: 0.25–0.50 (moderate) | 12 | 10 | 9 |
| C: 0.50–0.75 (high) | 1 | 0 | 0 |
| D: 0.75–1.00 (very high) | 2 | 1 | 2 |
| Change category | Count |
|---|---|
| Callee semantic injection (context introduces a concept that appears only in the callee body) | 9 |
| Action-verb correction (baseline picks a generic verb; context picks a task-specific verb) | 7 |
| Identifier accuracy (baseline hallucinates or generalises; context selects the right noun) | 5 |
| Caller semantic alignment (context aligns terminology with the caller’s parameter naming) | 3 |
| Conciseness improvement (context summary is shorter while preserving meaning) | 1 |
| Conciseness regression (context absorbs caller/callee terminology and grows the summary) | 1 |
| Variant | Generated summary | Rank (Acc / Cont / Conc) |
|---|---|---|
| codet5_baseline | “Notifies all listeners that a Throwable has been thrown” | 3 / 2 / 3 |
| codet5_code_cg (best) | “Fires a throwable to all listeners” | 2 / 1 / 1 |
| Variant | Generated summary | Rank (Acc / Cont / Conc) |
|---|---|---|
| codet5_baseline | “Optimize the regular expression” | 4 / 3 / 3 |
| codet5_code_cg (best) | “Optimize the NFA” | 2 / 1 / 1 |
| Variant | Generated summary | Rank (Acc / Cont / Conc) |
|---|---|---|
| codet5_baseline | “Returns an AsyncDatastore that wraps the underlying Datastore” | 3 / 3 / 3 |
| codet5_code_cg_vh (best) | “Creates an async datastore with caching enabled or disabled” | 1 / 1 / 2 |
| Variable | Retained | Discarded | Cliff’s |
|---|---|---|---|
| Methods per repository | 9 [3, 26] | 11 [4, 35] | 0.07 (negligible) |
| Method length (code tokens) | 66 [43, 116] | 70 [44, 124] | 0.04 (negligible) |
| Prior versions per method | 1 [1, 2] | 1 [1, 2] | 0.03 (negligible) |
| Method age (days) | 903 [358, 1,548] | 794 [390, 1,416] | 0.03 (negligible) |
| Build system | Retained ( ) | Discarded ( ) | Retention rate |
| Maven | 2,090 | 781 | 72.8% |
| Model | Condition | P | R | macro-F1 | vs. Baseline |
|---|---|---|---|---|---|
| CodeBERT | Baseline (no context) | 0.7103 | 0.6498 | 0.6698 | (reference) |
| Original context | 0.8504 | 0.7522 | 0.7813 | 16.65% | |
| Shuffled context, seed 42 | 0.5739 | 0.5352 | 0.5411 | 19.21% | |
| Shuffled context, seed 1234 | 0.4580 | 0.4360 | 0.4195 | 37.37% | |
| Shuffled context, mean of seeds | 0.5160 | 0.4856 | 0.4803 | 28.29% | |
| CodeT5 | Baseline (no context) | 0.7005 | 0.6364 | 0.6569 | (reference) |
| Split | Condition | Median | p25 | p75 |
|---|---|---|---|---|
| Train | Original | 108 | 54 | 259 |
| Shuffled | 111 | 54 | 291 | |
| Dev | Original | 102 | 53 | 255 |
| Shuffled | 102 | 59 | 317 | |
| Test | Original | 103 | 60 | 239 |
| Shuffled | 108 | 65 | 338 |
| Step | CodeSearchNet | SeSaMe | Vul4J |
|---|---|---|---|
| Repo clone | 40.6 h | 0.46 h | 0.70 h |
| Compile (Maven/Gradle) | 5.5 h | 0.37 h | 0.35 h |
| Call graph extraction | 4.1 h | 0.05 h | 0.06 h |
| Version history mining | 8.0 h | 0.46 h | 0.03 h |
| Method age | 0 (derived from VH first-commit timestamp) | ||
| Total | 58 h | 1.3 h | 1.1 h |
Code Language Models
Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.
See all 122 papers