Large Language Models (LLMs) have shown impressive capabilities across software engineering tasks, including question answering (QA). However, most studies and benchmarks focus on isolated functions or single-file snippets, overlooking the challenges of real-world program comprehension, which often spans multiple files and system-level dependencies. In this work, we introduce StackRepoQA, the first multi-project, repository-level question answering dataset constructed from 1,318 real developer questions and accepted answers across 134 open-source Java projects. Using this dataset, we systematically evaluate two widely used LLMs (Claude 3.5 Sonnet and GPT-4o) under both direct prompting and agentic configurations. We compare baseline performance with retrieval-augmented generation methods that leverage file-level retrieval and graph-based representations of structural dependencies. Our results show that LLMs achieve moderate accuracy at baseline, with performance improving when structural signals are incorporated. Nonetheless, overall accuracy remains limited for repository-scale comprehension. The analysis reveals that high scores often result from verbatim reproduction of Stack Overflow answers rather than genuine reasoning. To our knowledge, this is the first empirical study to provide such evidence in repository-level QA. We release StackRepoQA to encourage further research into benchmarks, evaluation protocols, and augmentation strategies that disentangle memorization from reasoning, advancing LLMs as reliable tool for repository-scale program comprehension.
Figures & tables
Figure 1. Overview of the data collection and preprocessing pipeline. Block diagram illustrating the StackRepoQA data collection pipeline, including GitHub repository selection, Stack Overflow question and answer extraction, filtering, and mapping questions to source code artifacts.
No. Forks
Size (MB)
Stars
Contributors
Age In Years
Average
1954.51
264.79
7220.50
176.58
11.62
Max
13581.00
5777.54
57793.00
411.00
16.65
Median
984.00
110.37
3741.00
148.00
11.78
Min
63.00
10.71
958.00
21.00
2.218
std( σ )
2350.26
562.66
8751.16
110.08
2.96
Total
259950
35217.55
960327
23485
1545.21
Table 1. Basic statistics on dataset repositories after filtering
Figure 2. Overview of the multi-agent architecture used in our evaluation Block diagram showing the multi-agent architecture employed in the experimental setup, including file-level and graph-level retrieval agents, summarization agent, and supervisor agent.
Related Questions
Unrelated Questions
All
LLM
Claude
GPT
Average
Claude
GPT
Average
Average
Claude
5.88
7.21
6.54
6.97
7.99
7.48
6.67
GPT-4o
6.77
7.08
6.93
6.96
7.13
7.06
6.94
Average
6.73
7.27
6.81
Table 2. Average scores from different LLM run without any augmentation and both LLMs as judges
Configuration
Score (related)
Score (unrelated)
File Agent only
5.66
5.46
Graph Agent only
6.12
5.49
All agents (Without LLM inbuilt knowledge)
6.11
5.60
All agents (With LLM inbuilt knowledge)
6.95
5.43
Table 3. Results from different configurations for ablation study
Related Questions
Unrelated Questions
LLM
Claude
GPT
Claude
GPT
Claude
6.45
7.45
6.18
6.22
GPT-4o
4.78
5.61
3.85
4.49
Table 4. Results of Multi-agent system on QA task using different LLMs
The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.
Yuexi Yang, Alyssa Wu, Ji Luo +4
Zhejiang University · Hunyuan, Tencent · Ningbo Global Innovation Center, Zhejiang University
Answering developer questions about a software repository is a critical yet under-explored problem in software engineering. While existing repository understanding methods have advanced the field, they predominantly rely on surface-level code retrieval and lack the ability for deep reasoning over multiple files, complex software architectures, and grounding answers in long-range code dependencies. To address these limitations, we propose DeepRepoQA, a novel question answering (QA) framework for repository-level code understanding. DeepRepoQA builds on an agentic framework where LLM agents find answers through a systematic tree search over the repository structure. A Monte-Carlo Tree Search (MCTS) mechanism is employed to empower agents to dynamically search, navigate, and inspect code, enabling effective multi-hop reasoning over long-range code dependencies. Comprehensive experiments on the SWE-QA benchmark demonstrate substantial performance gains over strong baselines, validating the effectiveness of systematic MCTS-guided exploration for multi-hop repository reasoning.
Weihan Peng, Yuling Shi, Yingwei Ma +3
Shanghai Jiao Tong University · The Hong Kong University of Science and Technology · University of California San Diego
Agentic repository-level code understanding is essential for automating complex software engineering tasks, yet the field lacks reliable benchmarks. Existing evaluations often overlook the long tail topics and rely on popular repositories where Large Language Models (LLMs) can cheat via memorized knowledge. To address this, we introduce SWE-QA-Pro, a benchmark constructed from diverse, long-tail repositories with executable environments. We enforce topical balance via issue-driven clustering to cover under-represented task types and apply a rigorous difficulty calibration process: questions solvable by direct-answer baselines are filtered out. This results in a dataset where agentic workflows significantly outperform direct answering (e.g., a ~13-point gap for Claude Sonnet 4.5), confirming the necessity of agentic codebase exploration. Furthermore, to tackle the scarcity of training data for such complex behaviors, we propose a scalable synthetic data pipeline that powers a two-stage training recipe: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from AI Feedback (RLAIF). This approach allows small open models to learn efficient tool usage and reasoning. Empirically, a Qwen3-8B model trained with our recipe surpasses GPT-4o by 2.3 points on SWE-QA-Pro and substantially narrows the gap to state-of-the-art proprietary models, demonstrating both the validity of our evaluation and the effectiveness of our agentic training workflow.
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni +13
University of Waterloo · University of Toronto · The Hong Kong University of Science and Technology +2