Current AI-powered code assistance tools often struggle with poorly-defined problem statements that lack sufficient task context and requirements specification. Recent analysis of software engineering agents reveals that failures on such underspecified requests are highly correlated with longer trajectories involving either over-exploration or repeated attempts at applying the same fix without proper evolution or testing, leading to suboptimal outcomes across software development tasks. We introduce CodeScout, a contextual query refinement approach that systematically converts underspecified user requests into comprehensive, actionable problem statements through lightweight pre-exploration of the target codebase. Our key innovation is demonstrating that structured analysis before task execution can supplement existing agentic capabilities without requiring any modifications to their underlying scaffolds. CodeScout performs targeted context scoping, conducts multi-perspective analysis examining potential fixes and exploration opportunities, then synthesizes these insights into enhanced problem statements with reproduction steps, expected behaviors, and targeted exploration hints. This pre-exploration directly addresses the identified failure patterns by reducing non-converging agent trajectories while clarifying user intent in natural language space. We evaluate CodeScout using state-of-the-art agentic scaffolds and language models on SWEBench-Verified, demonstrating a 20% improvement in resolution rates with up to 27 additional issues resolved compared to the default baseline method. Our results suggest that systematic query refinement through contextual analysis represents a promising direction for enhancing AI code assistance capabilities.
Figures & tables
Figure 1: The original SWEBench problem statement for Instance django__django-11790 lack relevant initial context. As a result, the downstream agent is not able to fix the issue, despite spending 21 steps exploring the repository, analyzing code, iterating the fix patch. In contrast, the enhanced problem statement generated by our approach resolves the issue in 6 agentic steps, as it includes relevant insights that can be used off-the-bat by the agent.
Figure 2: CodeScout: The pre-exploration with Repository Knowledge Graph Construction, which represents code structure and relationships. Building on this, three main stages follow: 1) High Level Scoping , where an LLM agent identifies relevant exploration targets, 2) Fine-grained Context Analysis , which extracts structured insights for each target, and 3) Problem Synthesis , where the original problem statement is combined with filtered insights to generate the augmented specification.
Figure 3: Main comparison between default (no augmentation) and our contextual augmentation across three LLMs (GPT-5-mini, Qwen3 Coder, DeepSeek R1) and three scaffolds (SWE-Agent, OpenHands, Mini-SWE-Agent). Augmentation yields consistent improvements in resolution rate; gains are largest when the runtime LLM has weaker agentic abilities.
Figure 4: Localization comparison (file-level and function-level) using SWE-Agent as the scaffold.
Ablations/Baselines
DeepSeek R1
GPT-5-mini
Qwen3 Coder
Default
114
194
183
CodeScout (Ours)
125
209
207
Agentic Intra-trajectory Augmentation
109
177
158
CodeScout (No Filtering)
116
190
190
CodeScout (BM25 Entity Selection)
119
195
198
Table 1: Ablation results (SWE-Agent). Numbers are counts of resolved issues on SWEBench-Verified.
Figure 5: Cumulative issues resolved as a function of total tokens consumed (input+output). (a) Agent-only token accounting; (b) token accounting including augmentation overhead.
Figure 6: Augmentation metrics: number of LLM calls, tokens and dollar cost per instance.
Figure 7: Distribution of problem-statement lengths (token count) for Default vs Augmented.
Figure 8: Evolution of tool-call proportions across a trajectory (Default vs CodeScout) for SWE-Agent.
Agent Base LLM
Default
Augmented by
DeepSeek R1
Qwen3 Coder
GPT5-mini
DeepSeek R1
108
125
164
132
Qwen3 Coder
183
194
209
190
GPT5-mini
194
196
207
209
Table 2: Cross-synthesis results: rows are the base (runtime) model used by the agent; columns show resolved counts when problem-statements are augmented by the indicated model.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: CodeScout visualization for matplotlib__matplotlib-13989 with DeepSeek-R1.
Figure 10: CodeScout visualization for matplotlib__matplotlib-13989 with Qwen3-Coder.
Figure 11: CodeScout visualization for matplotlib__matplotlib-13989 with GPT-5-mini.
Figure 12: CodeScout visualization for pytest-dev__pytest-10051 with DeepSeek-R1.
Figure 13: CodeScout visualization for pytest-dev__pytest-10051 with Qwen3-Coder.
Figure 14: CodeScout visualization for pytest-dev__pytest-10051 with GPT-5-mini.
Figure 15: CodeScout visualization for scikit-learn__scikit-learn-10297 with DeepSeek-R1.
Figure 16: CodeScout visualization for scikit-learn__scikit-learn-10297 with Qwen3-Coder.
Figure 17: CodeScout visualization for scikit-learn__scikit-learn-10297 with GPT-5-mini.
Figure 18: CodeScout visualization for sphinx-doc__sphinx-10323 with DeepSeek-R1.
Figure 19: CodeScout visualization for sphinx-doc__sphinx-10323 with Qwen3-Coder.
Figure 20: CodeScout visualization for sphinx-doc__sphinx-10323 with GPT-5-mini.
Figure 21: CodeScout visualization for sympy__sympy-11618 with DeepSeek-R1.
Figure 22: CodeScout visualization for sympy__sympy-11618 with Qwen3-Coder.
Figure 23: CodeScout visualization for sympy__sympy-11618 with GPT-5-mini.
Figure 24: CodeScout relevance score distribution across all three methods.
Figure 25: CodeScout maximum relevance score distribution per instance.
Figure 26: CodeScout number of targets distribution per instance.
As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that decouples underspecification detection from code execution. Across both proprietary and open-weight frontier LLMs, our scaffold achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated information-seeking behavior, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.
Nicholas Edwards, Sebastian Schuster
Faculty of Computer Science, University of Vienna, Vienna, Austria · UniVie Doctoral School Computer Science, University of Vienna, Vienna, Austria
Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified that adopts perturbation as a diagnostic tool to increase the demand for context reasoning by transforming how the repository is exposed. First, RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access. RepoMirage-Extend further turns perturbation-targeted structural bottlenecks into explicit tasks beyond issue resolution, where the average performance declines from 66.8% in the original setting to 25.3%, indicating a significant deficiency in repository context reasoning. Further trajectory analysis reveals an exploration drift, where agents access broader repository context but fail to turn it into effective structure information. Motivated by this observation, we propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving, and show that explicit structural scaffolding yields notable gains. These results uncover an previously overlooked gap in repository context reasoning for code agents and suggest that stronger structure-aware methods are potential to improve them.
Hanyu Li, Yichi Zhang, Speed Zhu +3
Beijing University of Posts and Telecommunications · Tsinghua University · Tencent
Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.