Context Matters: Improving LLM-Based Code Summarization with Hierarchical Information
Organizations: Instituto de Computação Universidade Estadual de Campinas (UNICAMP) Campinas, São Paulo, Brazil
Abstract
Source code documentation plays an important role in software quality activities, such as code review, program comprehension, maintenance, and bug fixing. However, it is often fragmented across individual code elements and focuses on local implementation details, providing little information about their surrounding context. This limits both developers' understanding of software systems and the effectiveness of Large Language Models (LLMs), whose reasoning is constrained by finite context windows. Enriching documentation with hierarchical context has the potential to improve software comprehension and support downstream, quality-related tasks for both developers and LLM-based tools. In this paper, we present an in silico study that showcases the impact of augmenting context on the generation of code-level documentation, using a variety of LLMs. By varying the function, class, and module hierarchy visible to the LLMs, we assess whether this variation produces a significant difference in the resulting code documentation compared with the existing documentation, which serves as a proxy for developer intent. Our results show that Documentation-Aware prompts with class or module-level context consistently yield summaries that are closest to the original documentation. Adding surrounding documentation yields statistically significant improvements over the baseline across models, with effect sizes ranging from negligible (DeepSeek) through weak (GPT-4o-mini) to medium (Gemini). We restrict the study to this immediate hierarchical context; call-graph traversal and cross-module dependency context fall outside our scope and are left to future work. These findings provide a clear path to improve automated documentation tools and enhance LLMs' ability to reason about codebases for downstream tasks, such as code retrieval.
Figures & tables
| Repository | SHA Commit | Modules | Classes | Functions | Docstrings | After Filter |
|---|---|---|---|---|---|---|
| astropy | d16bfe05 | 227 | 978 | 2,328 | 3,533 | 1,205 |
| django | e7fd69d0 | 153 | 606 | 2,153 | 2,912 | 480 |
| flask | d8c37f43 | 1 | 42 | 154 | 197 | 10 |
| matplotlib | b7d05919 | 104 | 543 | 2,398 | 3,045 | 1,798 |
| pylint | 182cc539 | 66 | 162 | 577 | 805 | 343 |
| pytest | 6995257c | 13 | 19 | 286 | 318 | 1 |
| Provider | Model ID | Knowledge Cutoff |
|---|---|---|
| OpenAI | gpt-4o-mini-2024-07-18 | Oct 2023 |
| gemini-3-flash-preview | Jan 2025 | |
| DeepSeek | Deepseek v3.2 | Mar 2025 |
| Condition | Median | Q1 | Q3 | IQR |
|---|---|---|---|---|
| Function-only (Baseline) | 586 | 334 | 960 | 626 |
| Class DI | 641 | 365 | 1009 | 644 |
| Class DA | 622 | 334 | 988 | 654 |
| Module DI | 662 | 367 | 1042 | 675 |
| Module DA | 674 | 362 | 1046 | 684 |
| Metric | Function-only (Baseline) | Class DI | Class DA | Module DI | Module DA |
|---|---|---|---|---|---|
| BERTScore | 0.5566 0.1157 | 0.5683 0.1288 | 0.6182 0.1669 | 0.5745 0.1353 | 0.6124 0.1651 |
| BLEU | 0.0611 0.1393 | 0.0810 0.1681 | 0.1738 0.2771 | 0.0926 0.1848 | 0.1664 0.2696 |
| ROUGE | 0.2482 0.1897 | 0.2649 0.2111 | 0.3400 0.2758 | 0.2731 0.2211 | 0.3286 0.2714 |
| METEOR | 0.2831 0.1814 | 0.3058 0.1993 | 0.4028 0.2631 | 0.3176 0.2098 | 0.3955 0.2592 |
| Model | Metric | |||
|---|---|---|---|---|
| deepseek v3.2 | BERTScore | 226.354 | 0.006 | .001 |
| deepseek v3.2 | BLEU | 677.710 | 0.019 | .001 |
| deepseek v3.2 | ROUGE | 215.258 | 0.006 | .001 |
| deepseek v3.2 | METEOR | 443.108 | 0.013 | .001 |
| gemini-3-flash-preview | BERTScore | 2501.200 | 0.072 | .001 |
| gemini-3-flash-preview | BLEU | 2587.016 | 0.075 | .001 |
| Comparison | Metric | r | Effect |
|---|---|---|---|
| DeepSeek v3.2 | |||
| Baseline vs. DI Class | BERTScore | 0.005 | Neg. |
| BLEU | 0.041 | Neg. | |
| ROUGE | 0.012 | Neg. | |
| METEOR | 0.018 | Neg. | |
| Baseline vs. DI Module | BERTScore | 0.008 | Neg. |
| Comparison | Metric | r | Effect |
|---|---|---|---|
| DeepSeek v3.2 | |||
| Baseline vs. DA Class | BERTScore | 0.100 | Weak |
| BLEU | 0.178 | Weak | |
| ROUGE | 0.086 | Neg. | |
| METEOR | 0.142 | Weak | |
| Baseline vs. DA Module | BERTScore | 0.069 | Neg. |
| Comparison | Metric | r | Effect |
|---|---|---|---|
| DeepSeek v3.2 | |||
| DI Class vs. DA Class | BERTScore | 0.095 | Neg. |
| BLEU | 0.137 | Weak | |
| ROUGE | 0.098 | Neg. | |
| METEOR | 0.124 | Weak | |
| DI Module vs. DA Module | BERTScore | 0.061 | Neg. |
| Function ( requests , Response.apparent_encoding ) |
| @property |
| def apparent_encoding(self): |
| return chardet.detect(self.content)[’encoding’] |
| Reference (ranked 2nd): "The apparent encoding, provided by the chardet library" |
| DA, module (ranked 1st): "The apparent encoding, guessed by using the chardet library. The encoding is detected by analyzing the response content (bytes). This is used as a fallback when the ’encoding’ attribute is None. Returns: str or None […]" |
| DI, module (ranked 3rd): "The apparent encoding of the response content, as detected by chardet. […] It is particularly useful when the server does not specify an encoding in the Content-Type header […] For large responses, consider alternative approaches to avoid loading the entire content into memory. […]" |
| Model | Context | Ref | DA | DI | r | |
|---|---|---|---|---|---|---|
| DeepSeek v3.2 | Class | 42.4 | 31.0 | 26.6 | <.0001 | 0.159 |
| DeepSeek v3.2 | Module | 47.1 | 27.5 | 25.4 | .0162 | 0.077 |
| gemini-3-flash-preview | Class | 8.1 | 47.3 | 44.6 | .0004 | 0.113 |
| gemini-3-flash-preview | Module | 9.6 | 47.6 | 42.8 | .0006 | 0.109 |
| gpt-4o-mini | Class | 51.5 | 28.3 | 20.2 | <.0001 | 0.134 |
| gpt-4o-mini | Module | 53.4 | 27.0 | 19.6 | .0006 | 0.109 |
| Model | Context | Ref Rank1 (median std) | r | |
|---|---|---|---|---|
| GPT-4o-mini | Class | <.0001 | -0.284 | |
| GPT-4o-mini | Module | <.0001 | -0.288 | |
| gemini-3-flash-preview | Class | <.0001 | -0.207 | |
| gemini-3-flash-preview | Module | <.0001 | -0.194 | |
| DeepSeek v3.2 | Class | <.0001 | -0.250 | |
| DeepSeek v3.2 | Module | <.0001 | -0.281 |
Text Summarization
Text summarization develops methods for condensing documents while preserving the information, perspective, and evidence readers need.
See all 151 papers