When assessing the quality of coding agents, predominant benchmarks focus on solving single issues on GitHub, such as SWE-Bench. In contrast, in real use, these agents solve more various and complex tasks that involve other skills such as exploring codebases, testing software, and designing architecture. In this paper, we first characterize some transferable skills that are shared across diverse tasks by decomposing trajectories into fine-grained components, and derive a set of principles for designing auxiliary training tasks to teach language models these skills. Guided by these principles, we propose a training environment, Hybrid-Gym, consisting of a set of scalable synthetic tasks, such as function localization and dependency search. Experiments show that agents trained on our synthetic tasks effectively generalize to diverse real-world tasks that are not present in training, improving a base model by 25.4% absolute gain on SWE-Bench Verified, 7.9% on SWT-Bench Verified, and 5.1% on Commit-0 Lite. Hybrid-Gym also complements datasets built for the downstream tasks (e.g., improving SWE-Play by 4.9% on SWT-Bench Verified). Code available at: https://github.com/yiqingxyq/Hybrid-Gym.
Figures & tables
Figure 1: ( Left ) We decompose general coding agent tasks into a set of intermediate components and compute the percentage of agent actions spent on each component. Our training tasks partially cover verification and fully cover reasoning, repository exploration, and implementation, which consist of around 68% of the actions. ( Right ) Example actions for each component. Compared to a baseline training method, SWE-Gym ( Pan et al., 2024 ) , training with our Hybrid-Gym significantly reduces failures due to insufficient reasoning, insufficient exploration, and failed file editing, improving the final resolved rate on SWE-Bench Verified from 20.6% to 32.4%
Command
Example
Func-Localize
Issue-Localize
Dep-Search
Func-Gen
SWE-Bench
SWT-Bench
Commit-0
grep
grep -i "pipeline"
31.3%
36.1%
48.4%
89.9%
35.4%
37.4%
5.3%
find
find . -type f -name "least_angle.py"
17.5%
11.1%
21.4%
8.7%
16.2%
21.5%
31.6%
cd
cd /workspace/scikit-learn
21.9%
31.5%
2.0%
0.0%
26.7%
0.0%
5.3%
ls
ls -la
6.6%
2.3%
3.7%
0.0%
2.4%
12.4%
36.8%
Table 1: The use of Linux commands for repository exploration in our proposed training tasks (left) and downstream tasks (right). We show the percentage of each command used in the agent steps targeting repository exploration.
Model
Func-Localize
Issue-Localize
Dep-Search
Func-Gen
Success %
Non-Empty %
Success %
Non-Empty %
Success %
Non-Empty %
Success %
Non-Empty %
Qwen2.5Coder-7B
0.0%
50.0%
3.3%
16.7%
6.7%
60.0%
5.0%
70.0%
Qwen2.5Coder-32B
40.0%
95.0%
33.3%
83.3%
56.7%
86.7%
25.0%
95.0%
Claude-Sonnet-4.5
70.0%
100.0%
63.3%
100.0%
80.0%
100.0%
50.0%
100.0%
Table 2: Performance of the student and teacher models on a subset of instances sampled from each training task.
Figure 4
Training Tasks
Training Dataset
Issue Solving (swe)
Test Generation (swt)
Library Build (cmt)
SWE-bench Verified
SWT-Bench Lite & Verified
Commit-0 Lite
swe
swt
cmt
syn
Resolved %
Localized %
Non-Loop %
Lite Resolved
Verified Resolved
Resolved %
Base Model: Qwen2.5Coder-7B
( base ) Qwen2.5Coder-7B
1.8
4.4
60.4
0.72
0.23
4.80
✓
SWE-Play-general
8.6 +6.8
49.2 +44.8
93.4 +33.0
2.54 +1.82
1.62 +1.39
9.25 +4.45
✓
SWE-Gym
10.6 +8.8
48.2 +43.8
79.0 +18.6
1.45 +0.73
0.69 +0.46
8.61 +3.81
Table 4: Results on three real-world coding tasks. We show whether the datasets contain swe, swt, cmt, or synthetic tasks (syn) in training. Results with a white background are task transfer results. Results with a light grey background are trained with in-domain tasks. We highlight the best result for each dataset and show the relative gain and loss compared to the base model, Qwen2.5Coder. Hybrid-Gym achieves strong task transfer performances across all three benchmarks and further improves in-domain datasets (e.g., SWE-Play).
Training Tasks
SWE-Bench
SWT-Bench
Commit-0
(# Instances)
Resolved %
Resolved %
Resolved %
Qwen2.5Coder-7B
1.8
0.23
4.80
+ Issue-Solving (491)
10.6 +8.8
0.69 +0.46
8.61 +3.81
+ Func-Localize (500)
12.8 +11.0
3.00 +2.77
8.74 +3.94
+ Issue-Localize (500)
9.6 +7.8
2.31 +2.08
9.09 +4.29
+ Dep-Search (500)
6.4 +4.6
2.31 +2.08
9.25 +4.45
Table 5: Performance of training Qwen2.5Coder-7B on each of the Hybrid-Gym tasks and the issue-solving task. We sample 500 instances from each of our tasks. Even without finetuning on issue-solving, function localization achieves a larger performance gain than SWE-Gym, which has 491 issue-solving instances. We report the resolved rates on SWE-Bench Verified, SWT-Bench Verified, and Commit-0 Lite.
Figure 3: Controlled experiments on training data characteristics. (a) Output Format: Removing the file editing actions ( str_replace ) from function localization trajectories causes a large drop in SWE-bench resolution rate. (b) Repo-Exploration: script-level code generation (LCB) does not effectively transfer to repo-level issue-solving and even underperforms documentation generation, a simple repo-level task. (c) Task Complexity: Transfer improves as training tasks become more complex. (d) Trajectory Complexity: With a fixed data size, training on longer (more agent steps) trajectories substantially improves downstream performance.
Figure 4: Effect of teacher model and data selection. (a) Effect of sampling strategy at fixed training budget. Repository diversity improves training but using the same repositories as in evaluation does not. (b) training on issue localization trajectories with different teacher models . o3-mini (edited) indicates the same set of trajectories as o3-mini, but with text and action steps combined.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Func-Localize
Issue-Localize
Dep-Search
Func-Gen
SWE-Bench
SWT-Bench
Commit-0
Analysis and reasoning
1.6%
9.3%
6.3%
0.9%
6.0%
11.4%
1.3%
Repository exploration
77.9%
66.2%
55.5%
44.6%
49.1%
40.3%
27.3%
Execution of existing code
0.1%
0.3%
0.1%
0.2%
4.7%
6.3%
3.7%
Solution implementation
17.4%
23.9%
35.3%
45.3%
12.7%
16.2%
53.8%
Solution verification
3.0%
0.3%
2.8%
9.0%
27.6%
25.9%
10.9%
Appendix
Table 6: We categorize the agent’s actions based on the intermediate components of the task, where the categories are pre-defined.
Figure 5: The error analysis of different methods on 50 instances sampled from SWE-Bench Verified.
Dataset
execute-bash
view
str-replace
Func-Localize
5.55
4.43
1.16
Issue-Localize
9.37
7.76
5.15
Dep-Search
1.78
3.12
1.89
Func-Gen
0.53
1.75
1.47
SWE-Bench
10.65
6.88
3.65
SWT-Bench
16.54
6.09
5.16
Appendix
Table 7: The average number of OpenHands tools called in each trajectory.