Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.
Figures & tables
Figure 1: Overview of MiRL. (a) A few generalizable misbehaviors account for a large share of error steps. (b) MiRL starts rollouts from misbehavior prefixes and rewards rollouts that pass the tests and avoid the misbehavior. (c) MiRL matches or outperforms TMax-4B using 35% of its rollouts. TB-Lite and TB-2.1 denote Terminal-Bench Lite ( OpenThoughts-Agent team et al., 2026 ) and Terminal-Bench 2.1 ( Merrill et al., 2026 ) .
Figure 2: An example terminal environment. The agent must move duplicate photos from backup/ into dupes/ and list them in report.csv; the tests check the result.
Figure 3: Overview. (a) The error localizer flags wrong policy steps, and the teacher corrects them; (b) the annotator groups error steps into types, where a few misbehaviors dominate; (c) prefixes that rerun exactly are kept for live RL rollouts; (d) rollouts start from them and gain reward only if they pass the tests and avoid the misbehavior.
Figure 4: Error steps concentrate in a few misbehaviors, followed by a long tail of environment-specific types; the top four appear across many environments.
Figure 5: Terminal task performance: mean (%) and standard error over 3 runs. Dashed boxes are the scores reported in the TMax paper ( Ivison et al., 2026 ) .
Figure 6: Ablations of MiRL. Starting from TMax-189, each bar adds one part: environment selection, prefix start, and the behavior reward re .
Figure 7: Misbehavior rate (%): the share of 16 samples that repeat the misbehavior on unseen prefixes. Judges: Gemini 3.8 Flash (used in training) and Qwen3.8-27B.
Figure 8: Training resources of MiRL relative to TMax-4B, up to the best checkpoint rather than the full training run. Checkered segments are error profiling and solid segments are RL training. GPU time is in A100 hours.
Figure 9: Training resources of the four recipes on 189 environments, up to the best checkpoint rather than the full training run. Checkered segments are preprocessing and solid segments are RL training.
Figure 10: MiRL trained on different numbers of environments selected from error profiling (0.25 × , 0.5 × , 1 × , and 2 × ). Mean (%) and standard error over 3 runs; horizontal lines show TMax-4B and Qwen3.5-4B.
Figure 11: TB-2.1 scores of TMax-4B, RaR+, and MiRL with 16 extra seeds. (a) Mean standard error over k seeds, averaged over all k-seed subsets of the 16 seeds and over the three recipes. (b) Mean and standard error over the 3 runs in Figure 5 and over the 16 extra seeds. (c) Distribution of the 16 seeds: whiskers show the minimum and maximum, boxes the 25th and 75th percentiles, and circles the means.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
RL training (all recipes)
Algorithm
DPPO ( Qi et al., 2026 )
Policy model
Qwen3.5-4B
Optimizer
AdamW
Learning rate
10−6 , constant
Group size
16 (TMax: 32)
Appendix
Table 1: Hyperparameters. The RL settings follow TMax ( Ivison et al., 2026 ) except where noted. All calls to Gemini 3.8 Flash use the medium thinking level.
Figure 12: Examples of misbehavior steps (1/2). Each example shows the definition of the misbehavior, an excerpt of the task, the last step of the prefix, the error step of the policy, and the explanation of the error localizer. The percentage is the share of error steps with the misbehavior. Long text is cut with an ellipsis.
Figure 13: Examples of misbehavior steps (2/2). Each example shows the definition of the misbehavior, an excerpt of the task, the last step of the prefix, the error step of the policy, and the explanation of the error localizer. The percentage is the share of error steps with the misbehavior. Long text is cut with an ellipsis.
#
Misbehavior
Steps
Envs.
1
Makes false claims about tests
15.4%
46.7%
2
Repeats unnecessary checks
11.5%
42.2%
3
Retries without fixing the cause
11.2%
36.5%
4
Fails to produce runnable commands
5.1%
19.4%
5
Kills processes without checking their identities
4.7%
14.0%
6
Treats guesses as facts
2.5%
11.0%
Appendix
Table 2: Share of the 12,746 error steps (Steps) and of the 2,017 profiled environments (Envs.) for each misbehavior. A step or an environment can have more than one misbehavior.
#
Long-tail error step
Environment detail
1
Uses a greedy regex that treats an entire log line as one key-value pair.
the format of the audit log
2
Starts frame counting at 1 instead of 0, shifting every 10-frame bucket.
Extracts metric names instead of values with an incorrect awk field index.
the NAME=...VALUE=... line format
5
Reads little-endian WAV samples as big-endian integers.
the audio file
6
Rejects version 1.5.0 despite the ^1.4.2 requirement allowing it.
the version rule
Appendix
Table 3: Examples of long-tail error steps. Each comes from a detail of one environment and rarely carries over to other environments.
Model
Input
Cached
Output
Qwen3.5-4B
$0.33
$0.066
$1.005
Qwen3.8-27B
¥3.646
¥0.729
¥21.875
Gemini 3.8 Flash
$0.75
$0.075
$3.75
Claude Opus 5
$5
$0.50
$25
Appendix
Table 4: Prices per million tokens at the time of our experiments: Qwen3.5-4B (policy) on Tinker, Qwen3.8-27B (error localizer and teacher) on Alibaba Cloud, Gemini 3.8 Flash (judges) from Google, and Claude Opus 5 (annotator) from Anthropic.
Package
Version
Use
open-instruct
7387d2f9
RL training ( grpo_fast )
DeepSpeed
0.18.4
RL training
Ray
2.53.0
RL training
PyTorch
2.10.0
RL training
Transformers
5.4.0
RL training
vLLM
0.19.1
Inference (bf16)
Appendix
Table 5: Software and model versions. open-instruct and the TMax agent come from the same TMax commit. vLLM uses prefix caching and the qwen3_xml tool-call parser; Harbor allows at most 64 format errors; PivotRL matches steps with SequenceMatcher at a threshold of 0.95; Qwen3.8-27B as the misbehavior-rate judge uses thinking with low effort.
Terminal agents have shown strong abilities in complex coding tasks. However, they often struggle to recover from errors during long-running execution (e.g., command errors, faulty implementations), eventually leading to task failures. To this end, we present ReTerm, a self-improving terminal agent trained by learning how to recover from its own failed terminal tasks. To support its training, we construct a customized data pipeline and produce ReTerm-10K, a high-quality dataset that provides effective recovery demonstrations for SFT cold start. Furthermore, we propose ReTerm-GRPO, an online RL algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts. By learning from whether these attempts successfully complete the task, the agent self-improves through its own recovery experience. Built on Qwen3.5-9B, our ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0, improving over the base model by 14.6 points. Further analysis shows that our method achieves consistently higher recovery rates across diverse error categories compared with standard GRPO training. All code, data and model are released at https://github.com/twothsnd/ReTerm.
Wei Xie, Kaituo Feng, Shiming Fu +3
Fudan University · MMLab, The Chinese University of Hong Kong · Shanghai Jiao Tong University
Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.
Junyao Yang, Yucheng Shi, Zhongzhi Li +4
Tencent Hy Foundation Model · National University of Singapore · University of Georgia +2
Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.
Yihang Yao, Bo Pang, Xuan Phi Nguyen +3
Salesforce AI Research · Carnegie Mellon University