TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble

Pengfei He; Jiayuan Zhou; Shaowei Wang; Ruiqi Pan

cs.SEOct 7, 2026

TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble

Authors: Pengfei He, Jiayuan Zhou, Shaowei Wang, Ruiqi Pan

Organizations: University of Manitoba, Winnipeg, Canada · Huawei Canada, Markham, Canada · Huawei Technologies, Hangzhou, China

Abstract

SWE-agent ensembles improve issue resolution by combining candidate patches from different agents with complementary strengths. The central problem is therefore test-based selection: generate tests, execute candidate patches, and identify the best patch. We formulate this process as test-space optimization: evolving an executable repository test suite until it distinguishes competing patches. Existing test-generation methods are limited optimizers. They usually lack an explicit loss for ensemble selection, optimize through incomplete directions that mostly create new tests or delete old ones, and perform one-off generation without feedback from repeated failures. Inspired by gradient descent with momentum, we introduce TestGRAD, a framework for automatic test optimization. TestGRAD centers on three concepts. Differential loss gives the optimizer an explicit execution-defined target: useful tests should separate candidate patches by behavior. Full CRUD gradients expand the update direction from merely creating or deleting tests to reading existing test infrastructure, creating new tests, updating stale assertions, and deleting only obsolete tests. Failure Pattern Momentum mines frequent failure sequences from memory, allowing the optimizer to avoid repeated non-discriminative directions while compressing the failure-history context. On SWE-bench Verified, TestGRAD achieves 84.2% Pass@1 with a 4-agent ensemble, outperforming the strongest baseline (80.6%) by an absolute improvement of 3.6 percentage points, while compressing failure-history context by over $100\times$ .

Figures & tables

Gradient Descent with Momentum	TestGRAD
Parameter state. Current parameter vector $\theta_{t}$ .	Test suite state. Current candidate test suite $y_{t}$ .
Objective. Loss function $\mathcal{L}(\theta_{t})$ .	Objective. Differential loss $\mathcal{L}(y_{t})$ measuring whether the current test suite has produced a differential outcome.
Retention strength. $\beta\in[0,1]$ controls how much of the previous velocity is retained.	Support-based retention. The $\displaystyle\beta=\operatorname{MinSupport}$ threshold defines the minimum frequency a sequence must satisfy. Lowering this threshold retains more sequences and gives momentum broader influence.
Momentum. $\displaystyle\beta\nabla_{t-1}$ . The accumulated direction from previous steps.	Momentum analogue. $\displaystyle m_{t}=\operatorname{Mine}_{\beta}(M)$ . A persistent test update distilled from recurring failure sequences in memory $M$ , after the latest failed update has been appended.
Update with Momentum. $\displaystyle\nabla_{t}=\beta\nabla_{t-1}+(1-\beta)\frac{\partial\mathcal{L}}{\partial\theta_{t}}$ . Combines instantaneous gradient with momentum to suppress noisy one-step gradients and reinforce consistent directions.	Gradient analogue. $\displaystyle\nabla_{\text{TestGrad}}=\operatorname{Optimizer}(\mathcal{L},y_{t},m_{t})$ . Combines current feedback with momentum to suppress one-off noisy edits and reinforce consistent test-improvement directions.
State update. $\displaystyle\theta_{t+1}=\theta_{t}-\eta\nabla_{t}$ .	Test update. $\displaystyle y_{t+1}=y_{t}+\nabla_{\text{TestGrad}}$ , where $\nabla_{\text{TestGrad}}$ is a combination of CRUD operations on the original test suite.

Table 1: Role-level analogy between gradient descent with momentum and test-space optimization. A one-step CRUD edit plays the role of an instantaneous gradient, whereas a high-support recurring failed edit sequence plays the role of momentum carried across attempts.

Figure 1: TestGRAD framework overview. Given an issue and candidate patches from multiple SWE agents, TestGRAD iteratively evolves the repository test suite: the forward pass executes tests to obtain differential loss; failed non-differential updates are stored in failure memory and mined into Failure Pattern Momentum; the backward pass uses the loss and momentum to produce the next CRUD test update.

Panel of Figure 2 — Figure 2: TestGRAD gradient computation in the backward pass. The derivative step is implemented by an LLM optimizer that converts execution-defined loss and Failure Pattern Momentum into a CRUD update over the repository test suite.

Panel of Figure 3 — Figure 3: TestGRAD momentum-gradient transition from $y_{t}$ to $y_{t+1}$ , illustrated on sphinx-doc__sphinx-10673 . (a) The current suite $y_{t}$ checks warning text and yields uniform outcomes ( $\mathcal{L}(y_{t})=1$ ). (b) Failure memory is mined into $m_{t}$ ; the highlighted statements form the frequent sequence shared by all three failed suites (support $3\geq\beta$ ), marking this repeated failed direction as a dead end and steering the optimizer toward AST inspection. (c) The updated suite $y_{t+1}$ probes toctree structure with assert_node() , producing a discriminative signal for the checker. Colors: test momentum .

Base LLM	Augment		Agentless		TRAE		Ours
Base LLM	Pass@1 (%)	Gain (%)	Pass@1 (%)	Gain (%)	Pass@1 (%)	Gain (%)	Pass@1 (%)	Gain (%)
Minimax-m2.7	79.0	- 0.2	80.4	+ 1.2	80.6	+ 1.4	84.2	+ 5.0
GLM-5.1	78.6	- 0.6	80.2	+ 1.0	80.2	+ 1.0	83.6	+ 4.4
DeepSeek-v3.2	79.2	+ 0.0	79.8	+ 0.6	80.6	+ 1.4	84.0	+ 4.8

Table 2: Performance comparison. Gain is computed relative to the baseline Pass@1 (79.2%). Positive gains are shown in green, while negative gains are shown in red.

Figure 4: Pass@1 vs. ensemble size for TestGRAD (green) and TRAE (orange), together with Oracle (blue dashed), Adversary (red dashed), and Expected (purple dotted) bounds. Results shown for Minimax, DeepSeek, and GLM (left to right).

Agent	N	Adv.	Exp.	Oracle	P@1
Livesweagent	1	79.2	79.2	79.2	79.2
+Sonar	2	76.8	79.8	82.8	81.2
+OpenHands	3	74.6	79.3	83.4	81.2
+Trae_doubao	4	70.2	79.2	87.0	84.2
+Gemini	5	68.4	79.1	88.0	83.8
+Atlassian	6	66.8	78.7	88.0	83.4

Table 3: Pass@1 across ensemble sizes ( TestGRAD , Minimax-m2.7).

Model	Setting	P@1	$\Delta$
TestGRAD	Complete	84.2	–
$\textsc{TestGRAD}_{\text{w/o Read}}$	w/o suite understanding	81.2	-3.0
$\textsc{TestGRAD}_{\text{w/o Update}}$	w/o updating tests	81.8	-2.4
$\textsc{TestGRAD}_{\text{w/o R+U}}$	w/o Read and Update	79.4	-4.8
$\textsc{TestGRAD}_{\text{w/o Diff Loss}}$	pass-count selection	80.8	-3.4
$\textsc{TestGRAD}_{\text{w/o Momentum}}$	w/o failure momentum	82.6	-1.6

Table 4: Ablation study on SWE-bench Verified (Minimax-m2.7).

Figure 5: Average token consumption of Raw Memory (dark gray) vs. Momentum (coral) across optimization iterations, shown on a log scale for DeepSeek, GLM, and Minimax. Italic labels denote representative compression ratios (Raw / Momentum); ratios above

100{\times}

are capped as

{>}100x

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: (A) CD-only generation produces a reproduction script that must hand-roll all setup boilerplate and tests the old exception-raising behavior, making it non-discriminative; tests are deleted by name heuristic, discarding valid ones. (B) TestGRAD applies a CRUD gradient: it reads the test suite first to extract infrastructure ( @isolate_apps , SimpleTestCase , .check() convention), creates a grounded test reusing that infrastructure ( blue = reused from Read), updates the expected model state in test_render , and semantically deletes test_missing_parent_link whose expected exception is no longer raised after the fix. Instance: django__django-12325 .

Aspect	Typical evidence	Freq. (%)	Cohen’s $\kappa$
Project APIs / Imports	from django.core.checks import Error	96	0.91
Assertion Conventions	self.assertQuerysetEqual(...) , assert_array_equal	91	0.88
setUp / Fixtures	setUp() , setUpTestData() , conftest.py	83	0.84
Test Base Classes	class MyTest(SimpleTestCase):	79	0.86
Expected Values	literals in assertEqual , docstring examples	71	0.79
Helper Functions	make_response() , assert_called_with(...)	64	0.76

Appendix

Table 1: Frequency of the six retained test-suite understanding aspects across 200 sampled SWE-bench instances (SWE-bench Verified excluded). Frequency is the fraction of instances where the aspect is present in the target test file. The six discarded aspects each fell below 50%.

Metric	Method	Minimax-m2.7	DeepSeek-v3.2	GLM-5.1
Avg. Time (min)	Augment	1.3	1.8	1.4
	Agentless	7.2	8.6	7.5
	TRAE	19.3	26.7	20.8
	TestGRAD	12.2	23.0	20.7
Avg. Tokens (K)	Augment	2.8	3.2	2.9
	Agentless	14.8	16.2	15.3

Appendix

Table 2: Average per-instance overhead across methods and base LLMs. Time is in minutes; tokens are in thousands (K). TestGRAD values are measured from run logs; baseline values are estimated from conversation logs and API call traces.

Explore similar work

FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents
Oct 6, 2026Jia Liufu, Bin Hu, Linglin Jing +5Swe-Bench Verified Retrying
NVIDIA · Tsinghua University
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
May 21, 2026Yuxuan Sun, Yuze Zhao, Yufeng Wang +6Test Generation Swe-Bench Verified
1State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · 2Independent Researcher. · 3Beihang University. +3
Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks
Aug 4, 2026Chenyu Wang, Yunbo Lyu, Junda He +4Retrying Swe-Bench Verified
Singapore Management University · University of Alberta · Nanjing University of Science and Technology +1

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

Jia Liufu, Bin Hu, Linglin Jing, Terry Kong+4

NVIDIA · Tsinghua University

Swe-Bench Verified Retrying

Open paper AskPDF arXiv

Repository-level software engineering (SWE) is a challenging long-horizon setting: agents must reason over extended interactions, use tools, and adapt to stateful environments. Recent work trains SWE agents with reinforcement learning methods such as Group Relative Policy Optimization (GRPO), which independently sample multiple trajectories per issue, test the resulting patches, and compare terminal rewards within a fixed group. However, this training setup does not reuse verifier feedback from failed patches as context for subsequent attempts, even though this feedback contains valuable diagnostic information about what went wrong. Training on recovery trajectories is challenging because the preceding outcome determines whether the next trajectory is generated, while the failed execution determines its conditioning context. We introduce FC-SWE, a failure-conditioned RL framework that incorporates recovery attempts into policy training. After a patch fails verification, FC-SWE restores the repository to its original task state and uses the failed patch and verifier feedback as context for a recovery trajectory. FC-SWE adapts GRPO to these chains of complete, multi-turn tool-use trajectories through two mechanisms. Trajectory-local rewards preserve each attempt's verifier outcome, preventing recovery success from rewarding an earlier failed patch. Active-set advantage estimation forms a comparison group from all initial and recovery trajectories actually executed for the same issue, so failed attempts remain in the group while unexecuted attempts are excluded. On all 500 SWE-bench Verified tasks under a verifier-assisted protocol, FC-SWE with Qwen3.5-4B and SWE-agent achieves 41.7% Resolved@1 and 52.8% Resolved@2, compared with 38.9% and 48.5% for GRPO. Although trained with at most two attempts per chain, FC-SWE reaches 70.7% Resolved@11 under an eleven-attempt test-time budget.

Abstract

Explore similar work

FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks