Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

Boyi Liu; Yite Wang; Xinyu Lian; Fan Shu; Sida Li; Feng Yan; Mingjia Zhang; Yuxiong He

cs.LGOct 17, 2026

Arctic-SWE: Revisiting the Long Horizon Agent Training by Modeling-System Co-Design

Authors: Boyi Liu, Yite Wang, Xinyu Lian, Fan Shu, Sida Li, Feng Yan, Mingjia Zhang, Yuxiong He,

Organizations: Snowflake · University of Illinois at Urbana-Champaign · University of Houston · University of Chicago

Abstract

Long-horizon agent training, including software engineering, is advancing rapidly, but the recipe and infrastructure needed for a strong reinforcement-learning baseline remain difficult to reproduce. This makes gains from new algorithms or data hard to distinguish from improvements to an under-tuned baseline. We present Arctic-SWE, a fully asynchronous RL framework that establishes such a baseline through model-system co-design, using overfitting on a small task pool as the testbed. This testbed exposes failures that early reward gains conceal and guides the integration of established components: gated reward, importance-weight control, aligned training and inference engines, and consistent state across policy updates. No single component is new, yet together they form the missing foundation. On raw R2E-Gym data, without new algorithms or data work, Arctic-SWE trains Qwen3.5-4B to 62.0% pass@1 on SWE-bench Verified. This is comparable to concurrent FrogNano (61.5%), which relies on online task synthesis, after about one-sixth as many training rollouts. Building on this foundation, we further filter and refine R2E-Gym, SWE-rebench V2, and Scale-SWE. With this curated subset, the 4B model improves from 47.4%/25.0%/37.9% to 65.0%/38.3%/59.7% on SWE-bench Verified/Pro-v1/Pro-v2, and the recipe also improves Qwen3.5-9B from 56.2%/31.9%/48.8% to 65.7%/39.6%/70.2% on Verified/Pro-v1/Pro-v2. Controlled comparisons show that both curation and test strengthening improve held-out performance, although stronger tests reduce apparent training-task mastery. Arctic-SWE provides a documented starting point for studying better algorithms, data and environments for asynchronous agent RL.

Figures & tables

Figure 1: Early gains can conceal instability. The async Prime-RL and sync SkyRL recipes both show instability after climbing to about 70%. The Gated reward and + raw format check runs show 21-update means over faint per-update values; the Prime-RL and SkyRL lines are per-update. Epochs use a square-root axis.

Figure 2: System layer corrections change the experience the optimizer sees. (a,b) Numerical alignment lowers mismatch KL without a throughput penalty (dark curves are averages). (c,d) One run resumed from update 200 (dashed) with per-rank placement: rollouts dropped at the off-policy cap and wall-clock time per update.

Figure 3: With the corrected system layer, the full recipe fits the pool. The raw-format-check run of Figure 1 against the same recipe with the system fixes of Section 3.3 , our full recipe . Faint gray lines are rerun of Figure 1 setting with system fixes in place, which tell us that system corrections alone are not enough to stable async RL.

Figure 4: The foundation alone nears FrogNano. Verified pass@1 on unchanged R2E-Gym, first 500 of 800 updates.

Outcome	Relation among (issue, tests, gold patch)	$n$	%
Pipeline 1: execution only, no model		over $21{,}099$
Source candidates	—	21,099	100
Reject	tests and gold patch disagree, or not executable	$-3{,}238$	15.3
$=$ Validated pool	tests and gold patch weakly agree	17,861	84.7
Pipeline 2: privileged LLM judge		over $17{,}861$
Reject	tests disagree w. the issue: too loose/over-restrictive/off target	$-9{,}379$	52.5

Table 1: Three artifacts disagree. Pipeline 1 establishes only weak agreement that the tests separate the buggy tree from gold. Pipeline 2 may repair only the issue, so a task survives here only if its tests and gold patch are sound on their own; Section 4.2 returns to the too_loose rows, broken out above. Verdict-level and per-corpus breakdowns in Appendix E .

Figure 5: Both curation steps pay off on held-out evaluation. Each panel is a single-knob comparison in which only the training deck changes; every row in every arm passes standard fail-to-pass validation. (a, c) SWE-bench Verified pass@1 (483 problems

\times

4 rollouts), with margins annotated at matched steps: the curated deck against a size- and mixture-matched unfiltered control (a), and strengthened against raw suites over the identical 4,889 instance IDs (c). (b, d) Fraction of the deck retired as mastered : solved by all eight rollouts in its group, then never sampled again. Panels (a, b) accompany Section 4.1 and (c, d) Section 4.2 .

Table 8

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Remedy	Evaluation	Training
Two native tools, one call per turn	execute_bash and edit_via_str_replace	Same
Format feedback	Malformed or prose-bearing turn is not executed; the model is told why and continues	Rollout ends
Duplicate tool call	In case of duplicated tool call without successful code edit, a short note suggests changing strategy	Optional; terminate if configured allowed number of duplicate tool calls reaches
Workflow hint	Prompt asks to run a relevant public test before submitting	Same
Submission	Optional; unit tests execute regardless	Exact echo MINI_SWE_AGENT_FINAL_OUTPUT ; a non-exact command ends the rollout with zero score
Output and time limits	10k-character head and tail, 60 s command timeout, as in the public harness	Same

Appendix

Table 5: Harness behavior relative to the public harnesses. Evaluation is the setting behind every reported SWE-bench score; training uses the same prompt and tools but ends a rollout where evaluation gives feedback, with reward 0 (Section 3 ) or

-1

(Section 5 ; Appendix I.3 ).

Reasoning block
	Server parser		RL framework, train time				Arctic-SWE
Flaw	vLLM	SGLang	Prime-RL	SkyRL	slime	miles	Gated	+ raw
commit	9bf44c4	1f6ce4b	c28afbb	b04e9c5	8ee9c1e	3d01116
No </think>	R	R	$\times$	R	R	R	$\times$	$\times$
Second </think>	P	P	P	R	P	P	$\times$	$\times$
Extra <think>	R	R	R	R	R	R	R	$\times$

Appendix

Table 6: Public training stacks let most format flaws through. Outcome of 13 malformed Qwen3.5 turns: P, executed as written; R, silently repaired and executed; D, executed with part of the action dropped or replaced;

\times

, no call executed. Shaded cells execute a call. RL frameworks run the train-time rollout path of their SWE example at the commit shown; the SkyRL column uses bash-block analogues, and – marks flaws without one. Arctic-SWE runs the gated reward without and with the raw token check.

Cache	$D$ ( $10^{-4}$ )	Mismatch KL ( $10^{-4}$ )
Stale attention KV only	3.78 [3.43, 4.12]	3.94
Stale GatedDeltaNet state only	3.46 [3.15, 3.81]	3.47
Both stale	7.21 [6.66, 7.76]	7.24

Appendix

Table 7: Stale-cache error at a 32k-token prefix with caches 10 updates stale, averaged over three checkpoint pairs (95% bootstrap intervals over trajectories). Mismatch KL applies Equation 11 to the recorded tokens.

Data and sampling
Setting	Value
Task pool	3,606 R2E-Gym tasks with their original issues and tests
Sampling	Without replacement from a shuffled deck; retired tasks are skipped
Groups and batch	8 rollouts per task; each update trains 128 rollouts from at least 16 groups with mixed rewards
Uniform groups	Groups whose rewards are all 0 or all 1 stay out of the batch but still count toward eviction
Task eviction

Appendix

Table 8: Recipe of the unchanged R2E-Gym run, beyond Appendix A .

Recipe choice	Alternative	Base	Checkpoints	Recipe	Alternative
Mask ratios above 2	Clip at 2	E	10	54.0	52.6
Off-policy cap 8	Cap 4	R	10	53.2	53.8
Token mean over the batch	Mean per rollout within each prompt	E	10	52.6	49.2
No std. normalization	Std. normalization	R ′	10	51.9	50.2
No length penalty	Input and turn penalty	R ′	10	51.9	45.4
No length penalty	Input, output and turn penalty	R ′	10	51.9	42.6

Appendix

Table 9: Full-data ablations: mean SWE-bench Verified pass@1 (%) over matched checkpoints.

Figure 7: Full-data ablations behind the recipe. SWE-bench Verified pass@1 against update; the blue line is the recipe’s choice. Panels (a) and (c) vary the earlier configuration, whose clip and token-mean curves average two of its runs; (b) varies the recipe with BF16

A_{\log}

, (d) and (e) that recipe with

\beta_{2}=0.95

, and (f) steps from

\beta_{2}=0.95

to the final recipe.

Pipeline 1: measurement (execution only, no model)
	R2E-Gym	SWE-rebench	Scale-SWE	Total
Source instances	3,606	6,584	10,909	21,099
no container image	–	–	$-310$	$-310$
could not be measured	$-69$	$-317$	$-197$	$-583$
failed validation	$-22$	$-959$	$-1{,}364$	$-2{,}345$
$=$ Validated pool	3,515	5,308	9,038	17,861

Appendix

Table 10: Corpus composition through the two curation pipelines. Pipeline 1 applies the same fail-to-pass gate prior corpora use and involves no model; pipeline 2 is the privileged LLM judge introduced here. Within pipeline 1, could not be measured covers sandbox and exec failures, suites that did not run, and gold trees failing their own suite: these are recorded as failures to measure, never as quality verdicts, so they are kept separate from failed validation , where the task itself is defective. The acceptance rows give each pipeline in isolation and the two composed.

	Verdict	R2E-Gym	SWE-rebench	Scale-SWE	Total	%
Ship	issue unchanged	1,756	1,303	3,208	6,267	35.1
Ship	issue rewritten	330	473	430	1,233	6.9
Reject: test quality	too_loose	986	1,155	3,210	5,351	30.0
	over_restrictive	282	1,227	1,634	3,143	17.6
	off_target	28	642	215	885	5.0
Reject: gold patch	partially fixes issue	60	425	329	814	4.6

Appendix

Table 11: Pipeline-2 LLM-judge verdicts over the 17,861 instances that clear pipeline 1. The two test-quality verdicts that motivate this work, too_loose and over_restrictive , together account for 47.6% of all F2P-validated candidates. The rewritten share is 6.9% of everything judged, which is 16.4% of the 7,500 tasks actually shipped.

	When	Source tree	Tests	Required
1	baseline	gold applied	original	$1$
2	baseline	buggy	original	$0$
3	agent check	gold applied	strengthened	$1$
4	agent check	buggy	strengthened	$0$
5	later shortcut gate	shortcut applied	original	$1$
6	later shortcut gate	shortcut applied	strengthened	$0$

Appendix

Table 12: Grading configurations. Rows 1–4 and 7–8 were used for the trained suites; rows 5–6 belong to the later shortcut gate. Agent checks may be repeated, and the gold–buggy separation is required at submission and final verification.

	R2E-Gym	SWE-rebench	Scale-SWE	Total	%
Input tasks	986	1,155	3,210	5,351	100
Strengthened	857	1,096	2,956	4,909	91.7
Kept original ( not_weak )	117	48	116	281	5.3
Reverted to original	6	10	138	154	2.9
baseline failure	3	6	127	136	2.5
infrastructure failure	2	3	8	13	0.2

Appendix

Table 13: Outcomes for all too_loose candidates. The four main dispositions partition the input; indented rows break down reverted tasks. Accepted suites passed final gold–buggy grading, not necessarily the later shortcut gate.

	R2E-Gym	SWE-rebench	Scale-SWE
Accepted strengthened tasks	857	1,096	2,956
Graded nodes added	3,808	4,117	13,531
Mean added nodes per task	4.4	3.8	4.6
Median added lines of test code	27	31	35
Added nodes failing on buggy	51%	84%	84%
Tasks: every added node fails on buggy	53%	69%	71%

Appendix

Table 14: Test changes across accepted suites. Buggy-tree failure rates do not measure rejection of all incorrect fixes. Tasks with no added nodes may still strengthen existing graded assertions. The three every / some / no added node rows are computed over the tasks that added at least one graded node (639, 932 and 2,590 respectively), not over all accepted tasks.

Reason	$n$	Instances
Network-dependent gold test	9	matplotlib-20488 , requests-1724 , requests-1766 , requests-1921 , requests-2317 , sphinx-7985 , sphinx-8269 , sphinx-8475 , sphinx-10435
Recorded test names do not match the log	3	astropy-7606 , sphinx-8595 , sphinx-9711
Image no longer collects	2	astropy-8872 , pylint-4661
Gold patch fails its own tests	2	astropy-8707 , django-10097
Already passes before any edit	1	django-7530
Total excluded	17	$500-17=\mathbf{483}$ retained

Appendix

Table 15: The seventeen SWE-bench Verified instances excluded from the scored subset. Only the first group is a consequence of the isolation policy; the rest are defects in the task or its image.

Checkpoint	Temperature	Top- $p$	Top- $k$
Qwen3.5-4B (initialization)	1.0	off	off
Qwen3.5-9B (initialization)	1.0	off	off
FrogNano-Qwen3.5-4B (our setting)	0.6	0.95	20
Arctic-SWE-4B, R2E-Gym only	1.0	0.95	20
Arctic-SWE-4B + curated RL	1.0	0.95	off
Arctic-SWE-9B + curated RL	1.0	off	off

Appendix

Table 16: Sampling parameters behind each self-evaluated row of Table 4 . “off” means the filter is disabled (top-

p=1.0

, no top-

k

truncation). All rows use 4 rollouts per problem and a 100-turn limit.

Benchmark Role & Size
Configuration / Parameter	SWE-bench Verified	SWE-bench Pro
Role	Validation Set	Test Set
Evaluated Instances	483 (17 excluded)	731 (all retained)
Rollouts per Problem	4	4
Agent Interaction Budgets
Max Turns	100	150

Appendix

Table 17: Detailed evaluation settings across benchmarks. Both settings enable thinking mode and share identical decoding parameters, while SWE-bench Pro expands interaction limits to handle longer trajectory horizons.

Stop condition	Meaning	Reward
Invalid: the trajectory was never a well-formed attempt
format_invalid	malformed tool call or empty reasoning envelope	$-1$
submission_invalid	submitted without the exact required command	$-1$
truncation	a single model response cut off mid-generation	$-1$
repetition	degenerate looping (detectors off in this run)	$-1$
Exhausted: a genuine attempt that ran out of budget

Appendix

Table 18: Stop conditions and the reward they receive. The two overrides fire only for the conditions listed under invalid ; every other terminal condition keeps whatever the test suite awarded, which for an unfinished trajectory is

0

Cell	Tasks	Rollouts	Mean reward	Solved $4/4$
r2e_evicted	300	1,200	0.952	0.867
rebench_evicted	300	1,200	0.946	0.867
scaleswe_evicted	300	1,200	0.968	0.917

Appendix

Table 19: The 4B’s retired tasks re-scored at step 350. Every task here scored 1.0 on the group that retired it, so 1.0 is the reference each row is read against.

Cell	Mean reward	$\Delta$ vs. step 350	Solved $4/4$	Early exits
r2e_evicted	0.851	$-0.101$	0.693	28/1,200
rebench_evicted	0.845	$-0.101$	0.697	32/1,200
scaleswe_evicted	0.852	$-0.116$	0.750	40/1,200

Appendix

Table 20: The identical retired decks re-scored at step 500. Only the weights differ from Table 19 .

Cell	Step 350	Step 500	$\Delta$	Solved $4/4$	Early exits
r2e_evicted	0.973	0.968	$-0.005$	$0.907\to 0.917$	$1\to 1$
rebench_evicted	0.963	0.951	$-0.013$	$0.890\to 0.880$	$2\to 4$
scaleswe_evicted	0.970	0.967	$-0.003$	$0.923\to 0.920$	$3\to 2$

Appendix

Table 21: The 9B’s retired tasks, re-scored at both checkpoints. Decks are cut from the 9B’s own step-350 eviction state, so 1.0 is again a value this run measured on exactly these rows. Columns are directly comparable to Tables 19 and 20 .

Corpus	9B $\Delta$	s.e.	4B $\Delta$	Separation
R2E-Gym	$-0.0050$	0.0077	$-0.101$	12.4 s.e.
SWE-rebench V2	$-0.0125$	0.0092	$-0.101$	0 9.6 s.e.
Scale-SWE	$-0.0033$	0.0085	$-0.117$	13.4 s.e.

Appendix

Table 22: Per-task paired change, steps 350 to 500. Standard errors are over tasks. Every 9B decline is within 1.4 standard errors of zero, and between 9.6 and 13.4 standard errors from the 4B’s decline on the same corpus.

	Arctic-SWE on raw R2E-Gym	FrogNano
Base model	Qwen3.5-4B	Qwen3.5-4B
Harness / objective	Mini-SWE-Agent-Plus / CISPO	Leaf / DPPO
Verified evaluation	483 tasks, 4 rollouts, 100 max turns, temperature 1.0, top- $p$ 0.95, top- $k$ 20	500 tasks, 3 seeds, 150 max turns, temperature 0.6 top- $p$ 0.95, top- $k$ 20
Best checkpoint (%)	61.96 (update 340)	61.5
Mean, updates 300–500 (%)	58.44	Not reported
Trainer / inference GPUs	2 / 6	2 / 6

Appendix

Table 23: Descriptive same-base-model comparison. Our scores use 483 valid-gold tasks; the external headline is the reported SWE-bench Verified result. Harnesses, task inclusion, training distributions and evaluation budgets are not matched. Rollout totals are nominal update budgets, not an audit of executed work or compute equivalence.

Figure 8: Resource usage during a ten-minute local k3s run, sampled every ten seconds. Left: aggregate CPU usage of the k3s cgroup subtree, expressed in core equivalents. Right: memory charged to the enclosing GPU-pod cgroup, including sandboxes, training and inference processes, and page cache.

Setup	Per-sandbox resources	$/slot-h	Total ($)
Ours, EC2 pool (on-demand)	4 vCPU / 16 GiB limits	0.0970	1,709
Northflank	4 vCPU / 16 GB	0.2000	3,524
Morph Devboxes	4 vCPU / 16 GiB	0.2000	3,525
Prime Intellect	4 vCPU / 16 GB	0.3600	6,344
Daytona †	4 vCPU / 16 GiB	0.4608	8,121
E2B †	4 vCPU / 16 GiB	0.4608	8,121

Appendix

Table 24: Standalone sandbox rental budgets for 252 concurrent capacity slots over 69.93 hours (251,753 seconds), using the reference run’s cap and reported runtime. Our row charges the entire EC2 pool and lists per-sandbox burst limits; commercial rows reserve the listed resources throughout. These are pricing scenarios, with performance unmeasured across providers. Sources and assumptions are in Appendix M .

Long-Horizon RL

See all 32 papers

QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents
Sep 27, 2026Weiqi Wang, Yuxin Zhou, Mouxiang Chen +9Reinforcement Learning Long-Horizon Agent Tasks
Alibaba Token Hub, Alibaba Group · University of Science and Technology of China · Tsinghua University
Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
Cross-Benchmark Generalization in Long-Horizon Agents
Jul 31, 2026Sushant Mehta, Logan Ritchie, Liudas Panavas +1Long-Horizon Agent Evaluation RL Post-Training
Surge AI

Sep 27, 2026J/K move · Enter open · S save

QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents

Weiqi Wang, Yuxin Zhou, Mouxiang Chen, Siyuan Zhang+8

Alibaba Token Hub, Alibaba Group · University of Science and Technology of China · Tsinghua University

Reinforcement Learning Long-Horizon Agent Tasks Long-Horizon LLM Agents RL for LLM Agents Long-Horizon RL GPU-Accelerated RL

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly undertake extreme-long (xlong) horizon tasks, where a single execution can span hours, hundreds of model--environment interactions, and nearly 1M tokens per rollout. Applying online reinforcement learning (RL) to such executions poses two fundamental challenges: (1) severe execution variance and prolonged rollout delays cause massive GPU idling; and (2) complex non-linear branching generates massive trajectory redundancy, crippling training efficiency. To address these, we presents QwenGyre, an end-to-end framework for xlong-horizon online RL. QwenGyre elastically reallocates GPUs between rollout and training without interrupting live executions, while its trajectory processor reconstructs branching histories, scores partial progress, and deduplicates redundant paths to bound training costs. Scaled to our flagship model, Qwen~3.8 2.4T, with 700K tokens per rollout, QwenGyre yields a 6.0% absolute gain on NL2RepoBench (52.5%

\to

58.5%) in 48 steps. Across our evaluations on diverse domains of training datasets, QwenGyre delivers up to

1.85\times

and

1.78\times

speedups over Colocate and Async, respectively.

Abstract

Long-Horizon RL

Explore similar work

QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agents

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

Cross-Benchmark Generalization in Long-Horizon Agents