Software documentation is crucial for repository comprehension. While Large Language Models (LLMs) advance documentation generation from code snippets to entire repositories, existing benchmarks have two key limitations: (1) they lack a holistic, repository-level assessment, and (2) they rely on unreliable evaluation strategies, such as LLM-as-a-judge, which suffers from vague criteria and limited repository-level knowledge. To address these issues, we introduce SWD-Bench, a novel benchmark for evaluating repository-level software documentation. Inspired by documentation-driven development, our strategy evaluates documentation quality by assessing an LLM's ability to understand and implement functionalities using the documentation, rather than by directly scoring it. This is measured through function-driven Question Answering (QA) tasks. SWD-Bench comprises three interconnected QA tasks: (1) Functionality Detection, to determine if a functionality is described; (2) Functionality Localization, to evaluate the accuracy of locating related files; and (3) Functionality Completion, to measure the comprehensiveness of implementation details. We construct the benchmark, containing 4,170 entries, by mining high-quality Pull Requests and enriching them with repository-level context. Experiments reveal limitations in current documentation generation methods and show that source code provides complementary value. Notably, documentation from the best-performing method improves the issue-solving rate of SWE-Agent by 20.00%, which demonstrates the practical value of high-quality documentation in supporting documentation-driven development.
Figures & tables
Figure 1. A typical workflow of documentation-driven development.
Figure 2. The overview of SWD-Bench’s data construction pipeline.
# Entry
# Func. Desc.
% Detect. Ratio
# Loc. File
# Comp. Detail
Min
Max
Avg.
Min
Max
Avg.
4,170
771.45
50.65
1
62
2.01
3
23
7.48
Table 1. Statistics of the SWD-Bench. # Func. Desc. denotes the average character length of functionality descriptions. % Detect. Ratio is the percentage of positive entries for the detection task. # Loc. File and # Comp. Detail represent the number of files to locate and details to complete, respectively.
Model
GPT-4.1
Gemini-2.5-pro
Top 1024
Top 2048
Top 4096
Top 1024
Top 2048
Top 4096
Functionality Detection
Metric (%)
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
No Doc
46.72
-8.03
–
–
–
–
50.63
1.18
–
–
–
–
Baseline Methods
H-Written
53.59
11.74
54.06
13.70
55.16
15.38
58.13
15.71
61.56
22.07
62.97
24.66
Table 2. Experimental results for the functionality-driven tasks. Top XX indicates the token size of the retrieved documentation context. No Doc refers to the setting of not inquiring about documentation. The largest and second-largest values in each column are highlighted with an underline , and the largest value is also bolded .
Figure 3. A case study on the functionality detail task (Entry-ID: “ pylint-dev/pylint/8824 ”). (A) The task question and answer. (B) The related source code to implement the described functionality. (C) The retrieved documentation from Chat and RepoAgent. (D) A comparison of results between the current evaluation method and our evaluation strategy.
Figure 4. Performance of SWE-Agent on issue solving when provided with different software documentation.
Model
GPT-4.1
Gemini-2.5-pro
Top 1024
Top 2048
Top 4096
Top 1024
Top 2048
Top 4096
Functionality Detection
Metric (%)
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
B-ACC
MCC
RepoAgent
62.19
25.92
62.97
26.74
63.75
27.77
63.28
25.04
67.66
33.38
69.53
37.23
RepoAgent+Code
67.81
34.80
70.47
39.33
72.66
43.27
68.13
34.32
74.53
43.52
76.25
51.89
Functionality Localization
Table 3. Performance comparison on functionality-driven tasks between the standalone RepoAgent and RepoAgent augmented with source code (RepoAgent + Code). Top XX indicates the token size of the retrieved documentation and source code context, which is evenly allocated to each source. The largest value in each column is marked in bold and underlined .
The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.
Yuexi Yang, Alyssa Wu, Ji Luo +4
Zhejiang University · Hunyuan, Tencent · Ningbo Global Innovation Center, Zhejiang University
We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation recall and pretraining memorization. The framework makes two methodological contributions: (1) an answer-first generation pipeline where a tool-equipped agent explores source code to produce verified gold answers before deriving questions, ensuring every task is grounded in real code structure; and (2) a three-condition experimental design evaluating agents under closed-book (no repository), code-only (documentation removed), and documented (full repository) conditions, with deltas directly quantifying documentation utility and memorization. We generate 528 code-derivable and 100 doc-dependent tasks across 10 Python repositories from SWE-Bench, scored by an LLM judge on accuracy, completeness, and specificity. Experiments on four frontier models reveal that code access is the dominant factor (+0.23 mean gain over closed-book), documentation provides modest additional benefit (+0.071 on doc-dependent tasks), and code-only ≈ documented on code-derivable tasks, validating the design. The framework is open-source and applicable to any well-documented Python repository.
Large Language Models (LLMs) have shown impressive capabilities across software engineering tasks, including question answering (QA). However, most studies and benchmarks focus on isolated functions or single-file snippets, overlooking the challenges of real-world program comprehension, which often spans multiple files and system-level dependencies. In this work, we introduce StackRepoQA, the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects. Using this dataset, we systematically evaluate two widely used LLMs (Claude 3.5 Sonnet and GPT-4o) under both direct prompting and agentic configurations. We compare baseline performance with retrieval-augmented generation methods that leverage file-level retrieval and graph-based representations of structural dependencies. Our results show that LLMs achieve moderate accuracy at baseline, with performance improving when structural signals are incorporated. Nonetheless, overall accuracy remains limited for repository-scale comprehension. The analysis reveals that high scores often result from verbatim reproduction of Stack Overflow answers rather than genuine reasoning. To our knowledge, this is the first empirical study to provide such evidence in repository-level QA. We release StackRepoQA to encourage further research into benchmarks, evaluation protocols, and augmentation strategies that disentangle memorization from reasoning, advancing LLMs as reliable tool for repository-scale program comprehension.