cs.SEOct 29, 2026

When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

Authors: Yishuo Cai, Minzheng Wang, Liang Zhao, Linghao Zhang, Lei Li, Shicheng Li, Xu Sun, Fuli Luo

Organizations: LLM Core, Xiaomi · Peking University · University of Hong Kong

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

To succeed in real-world long-horizon software development, coding agents must satisfy multi-dimensional task specifications, such as interface compatibility and error handling, and do so without breaking existing dependencies. However, current works lack specification in both quantity and diversity, often emphasizing primary goals while ignoring complex constraints. Synthesizing tasks to bridge this gap presents two challenges: naively adding specifications risks reducing tasks to trivial checklists rather than compelling joint reasoning, and verifying these subtle interactions risks overlooking hidden flaws or rejecting valid alternatives. To address these challenges, we introduce CodeSpecWeave, a framework for synthesizing tasks with rich, interdependent specifications. Grounded in real developer seeds, a synthesis agent iteratively develops both coupled task requirements and executable tests, while a review agent audits solution trajectories from multiple models to guarantee test reliability. Using this pipeline, we construct 4,670 training tasks across 2,865 repositories. Applying Reinforcement learning on these tasks significantly improves performance over the base model on DeepSWE-v1.1, largely outperforming existing baselines. Furthermore, we curate CodeSpecWeave-Bench, a rigorous benchmark comprising 100 tasks and 1,044 specifications, which clearly separates frontier models and reveals significant headroom in handling interdependent engineering constraints.

Figures & tables

Figure 1: Comparison of task specifications across datasets. Top: Mean extracted specifications and distinct behavioral categories per task across existing SWE training datasets and CodeSpecWeave , with 95% bootstrap confidence intervals. Category counts follow the taxonomy in Table 5 . Bottom: Example task structures. Prior datasets typically contain few independent rules (left), whereas CodeSpecWeave tasks feature multiple interdependent specifications tied together by shared execution contexts and lifecycle constraints (right).
Figure 2: Overview of CodeSpecWeave . The synthesis agent repeatedly consults specification seeds, explores the repository, and runs generated tests to refine a task and its verifier. Schematic links between specifications and test cases illustrate many-to-many coverage. Solver trials and trajectory review then assess difficulty and query–test alignment. Accepted task–verifier pairs are retained; selected candidates with false positives or false negatives are repaired and revalidated.
Figure 3: Specification distribution in CodeSpecWeave - Bench . (a) The five behavioral categories in Table 5 , with functional semantics expanded into three blue subcategories. Each of the 1,044 specifications contributes once to one of the seven segments. (b) Specifications per task across 100 tasks (mean: 10.44).
IssueTasks%
False positives only1,54850.5
False negatives only42613.9
Both2939.6
None observed80026.1
Total with issues2,26773.9
Table 1: Test alignment breakdown on 3,067 candidates.
Figure 4: Pass rates and mean reported output tokens on CodeSpecWeave - Bench . Hatched bars show pass rates (left axis); the line shows mean output tokens (right axis). Colors identify agent frameworks.
Training SettingsCodeSpecWeave - BenchDeepSWE- v1.1SWE-bench VerifiedSWE-bench ProAvg
MiMo-V2.5
w/ Vanilla47.0012.3969.2750.1144.69
w/ SWE-PI44.006.7868.0049.4342.05
w/ MiMo Coding50.0018.6273.6353.8149.01
w/ CodeSpecWeave52.0025.3072.0051.3050.15
MiMo-V2.6-Flash-SFT
Table 2: Task success rates (%, higher is better). Avg is the unweighted mean across the four benchmarks. Within each model, the best results are in bold and the second-best are underlined .
Figure 5: Learning curves for CodeSpecWeave , SWE-PI, and MiMo Coding, trained from MiMo-V2.5 (left) and MiMo-V2.6-Flash-SFT (right).
MeasureMiMo-V2.5MiMo-V2.6-Flash-SFT
 VanillaCodeSpecWeaveVanillaCodeSpecWeave
Thinking tokens (k)29.7141.622.869.5
Modified files4.479.794.096.17
Changed lines4513,632350667
Edit–test loops3.484.342.964.90
Trajectories with ≥2\geq 2 loops (%)63.072.769.078.3
Table 3: Reasoning, code changes, and edit–test loops on CodeSpecWeave - Bench . The edit–test loop definition is given in Appendix E .
Figure 6: Task success rates on CodeSpecWeave - Bench for MiMo-V2.6-Flash-SFT before and after reinforcement learning on the 4,670-task CodeSpecWeave training set. Results cover all 100 benchmark tasks. (a) Tasks grouped by specifications per task, with annotations showing percentage-point gains over Vanilla. (b) Tasks grouped by specification category (each point denotes the success rate on tasks requiring that category).
Figure 7: Training dynamics through RL update for the two CodeSpecWeave runs. From left to right: mean task-group score, policy entropy, response length, and main-agent turns.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
Training batch size128
Mini-batch size128
Samples per task16
Sampling temperature1.0
Top- pp0.97
Prompt token limit8,192
Appendix
Table 4: RL training hyperparameters.
Specification TypeScopeCount
Functional SemanticsEnd-to-end data transformation and algorithmic logic470
Parsing & RepresentationSyntax matching, encoding, and format serialization182
Computation & AlgorithmsMathematical calculations and core algorithmic logic145
Domain & Protocol RulesBusiness constraints and protocol-specific semantics143
Interface & I/O ContractsPublic APIs, argument types and defaults, input/output schemas108
Control Flow & OrderingConditional branching, execution order, and concurrency98
Appendix
Table 5: Taxonomy and distribution of the 1,044 benchmark specifications. Functional specifications ( n=470n=470 ) are partitioned into three fine-grained subcategories.
Figure 8: Query length distributions of CodeSpecWeave , Scale-SWE, SWE-rebench V2, SWE-Lego, and R2E-Gym.
RequirementExpected behavior
ProportionalityWithout change, allocate post-fee hours proportionally and resolve rounding ties by request order.
Change allocationSplit post-fee hours 50/50 between recipients and change, assigning the odd hour to change.
Minimum allocationGive each recipient at least one hour whenever the recipient share permits it.
Output orderKeep recipients in request order and append non-zero-coin change last.
CompatibilityPreserve the single-recipient cap and its existing output values.
Appendix
Table 7: Requirements in the example task.
ModelVanillaCodeSpecWeave
MiMo-V2.577.186.0
Flash-SFT77.787.1
Appendix
Table 8: Task-averaged rates of editing and retesting after a failed test (%).
Figure 9: Failure pattern (1): a specification is targeted but incompletely enforced. The patch includes link recognition and punctuation handling, and the agent’s probe recognizes a parenthesized URL. Yet a URL following the text prefix remains plain text in the final evaluation. The targeted requirement is satisfied in one boundary setting but violated in another. URL and email strings are abbreviated; underlining denotes a URI span.
Figure 10: Failure pattern (2): an update satisfies one requirement while violating another. The patch addresses both signature union and state isolation. After a whole-state copy, however, both states share a transaction. The update adds the required signature to the destination transaction while also changing the original, violating isolation. Thus the same update fails to satisfy both requirements together. The original-state count is measured by the verifier.
More in this topic

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 322 papers

Explore similar work

CardsList
May 15, 2026

RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades

Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.

Xinbo Xu, Ruihan Yang, Haiyang Shen +13

UniPat AI · Peking University · Fudan University +4

AI Coding AgentsSoftware Engineering BenchmarksLong-Horizon Agent EvaluationLong-Horizon Agent TasksOpen-Source Software Development
AskPDF
Jun 29, 2026

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.

Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal +1

1Scale AI

Coding AgentsSoftware Engineering BenchmarksAI Agent EvaluationUser Simulation
AskPDF
Apr 3, 2026

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.

KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara +2

Fujitsu Research

Code Generation EvaluationSoftware EngineeringLong-Horizon Agent EvaluationSoftware Engineering Benchmarks
AskPDF