MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors

Runchu Tian; Zaid Khan; Zun Wang; Justin Chih-Yao Chen; Mohit Bansal

cs.LGOct 26, 2026

MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors

Authors: Runchu Tian, Zaid Khan, Zun Wang, Justin Chih-Yao Chen, Mohit Bansal

Organizations: Department of Computer Science, University of North Carolina at Chapel Hill

Abstract

Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.

Figures & tables

Figure 1: Overview of MiRL. (a) A few generalizable misbehaviors account for a large share of error steps. (b) MiRL starts rollouts from misbehavior prefixes and rewards rollouts that pass the tests and avoid the misbehavior. (c) MiRL matches or outperforms TMax-4B using 35% of its rollouts. TB-Lite and TB-2.1 denote Terminal-Bench Lite ( OpenThoughts-Agent team et al., 2026 ) and Terminal-Bench 2.1 ( Merrill et al., 2026 ) .

Figure 2: An example terminal environment. The agent must move duplicate photos from backup/ into dupes/ and list them in report.csv; the tests check the result.

Figure 3: Overview. (a) The error localizer flags wrong policy steps, and the teacher corrects them; (b) the annotator groups error steps into types, where a few misbehaviors dominate; (c) prefixes that rerun exactly are kept for live RL rollouts; (d) rollouts start from them and gain reward only if they pass the tests and avoid the misbehavior.

Figure 4: Error steps concentrate in a few misbehaviors, followed by a long tail of environment-specific types; the top four appear across many environments.

Figure 5: Terminal task performance: mean (%) and standard error over 3 runs. Dashed boxes are the scores reported in the TMax paper ( Ivison et al., 2026 ) .

Figure 6: Ablations of MiRL. Starting from TMax-189, each bar adds one part: environment selection, prefix start, and the behavior reward

r_{e}

Figure 7: Misbehavior rate (%): the share of 16 samples that repeat the misbehavior on unseen prefixes. Judges: Gemini 3.8 Flash (used in training) and Qwen3.8-27B.

Figure 8: Training resources of MiRL relative to TMax-4B, up to the best checkpoint rather than the full training run. Checkered segments are error profiling and solid segments are RL training. GPU time is in A100 hours.

Figure 9: Training resources of the four recipes on 189 environments, up to the best checkpoint rather than the full training run. Checkered segments are preprocessing and solid segments are RL training.

Figure 10: MiRL trained on different numbers of environments selected from error profiling (0.25

\times

, 0.5

\times

, 1

\times

, and 2

\times

). Mean (%) and standard error over 3 runs; horizontal lines show TMax-4B and Qwen3.5-4B.

Figure 11: TB-2.1 scores of TMax-4B, RaR+, and MiRL with 16 extra seeds. (a) Mean standard error over k seeds, averaged over all k-seed subsets of the 16 seeds and over the three recipes. (b) Mean and standard error over the 3 runs in Figure 5 and over the 16 extra seeds. (c) Distribution of the 16 seeds: whiskers show the minimum and maximum, boxes the 25th and 75th percentiles, and circles the means.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
RL training (all recipes)
Algorithm	DPPO ( Qi et al., 2026 )
Policy model	Qwen3.5-4B
Optimizer	AdamW
Learning rate	$10^{-6}$ , constant
Group size	16 (TMax: 32)

Appendix

Table 1: Hyperparameters. The RL settings follow TMax ( Ivison et al., 2026 ) except where noted. All calls to Gemini 3.8 Flash use the medium thinking level.

Panel of Figure 12 — Figure 12: Examples of misbehavior steps (1/2). Each example shows the definition of the misbehavior, an excerpt of the task, the last step of the prefix, the error step of the policy, and the explanation of the error localizer. The percentage is the share of error steps with the misbehavior. Long text is cut with an ellipsis.

Panel of Figure 13 — Figure 13: Examples of misbehavior steps (2/2). Each example shows the definition of the misbehavior, an excerpt of the task, the last step of the prefix, the error step of the policy, and the explanation of the error localizer. The percentage is the share of error steps with the misbehavior. Long text is cut with an ellipsis.

#	Misbehavior	Steps	Envs.
1	Makes false claims about tests	15.4%	46.7%
2	Repeats unnecessary checks	11.5%	42.2%
3	Retries without fixing the cause	11.2%	36.5%
4	Fails to produce runnable commands	5.1%	19.4%
5	Kills processes without checking their identities	4.7%	14.0%
6	Treats guesses as facts	2.5%	11.0%

Appendix

Table 2: Share of the 12,746 error steps (Steps) and of the 2,017 profiled environments (Envs.) for each misbehavior. A step or an environment can have more than one misbehavior.

#	Long-tail error step	Environment detail
1	Uses a greedy regex that treats an entire log line as one key-value pair.	the format of the audit log
2	Starts frame counting at 1 instead of 0, shifting every 10-frame bucket.	the bucket rule
3	Replaces udb inside existing udbuf, producing udbufbuf.	the variable names in the C file
4	Extracts metric names instead of values with an incorrect awk field index.	the NAME=...VALUE=... line format
5	Reads little-endian WAV samples as big-endian integers.	the audio file
6	Rejects version 1.5.0 despite the ^1.4.2 requirement allowing it.	the version rule

Appendix

Table 3: Examples of long-tail error steps. Each comes from a detail of one environment and rarely carries over to other environments.

Model	Input	Cached	Output
Qwen3.5-4B	$0.33	$0.066	$1.005
Qwen3.8-27B	¥3.646	¥0.729	¥21.875
Gemini 3.8 Flash	$0.75	$0.075	$3.75
Claude Opus 5	$5	$0.50	$25

Appendix

Table 4: Prices per million tokens at the time of our experiments: Qwen3.5-4B (policy) on Tinker, Qwen3.8-27B (error localizer and teacher) on Alibaba Cloud, Gemini 3.8 Flash (judges) from Google, and Claude Opus 5 (annotator) from Anthropic.

Package	Version	Use
open-instruct	7387d2f9	RL training ( grpo_fast )
DeepSpeed	0.18.4	RL training
Ray	2.53.0	RL training
PyTorch	2.10.0	RL training
Transformers	5.4.0	RL training
vLLM	0.19.1	Inference (bf16)

Appendix

Table 5: Software and model versions. open-instruct and the TMax agent come from the same TMax commit. vLLM uses prefix caching and the qwen3_xml tool-call parser; Harbor allows at most 64 format errors; PivotRL matches steps with SequenceMatcher at a threshold of 0.95; Qwen3.8-27B as the misbehavior-rate judge uses thinking with low effort.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 603 papers

ReTerm: Recovery Training for Self-Improving Terminal Agents
Oct 13, 2026Wei Xie, Kaituo Feng, Shiming Fu +3LLM Agent Training Self-Improving Agents
Fudan University · MMLab, The Chinese University of Hong Kong · Shanghai Jiao Tong University
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
Sep 10, 2026Junyao Yang, Yucheng Shi, Zhongzhi Li +4RL for Tool Use Long-Horizon Agent Evaluation
Tencent Hy Foundation Model · National University of Singapore · University of Georgia +2
Learning Generalizable Behaviors for Terminal Agents
Aug 23, 2026Yihang Yao, Bo Pang, Xuan Phi Nguyen +3Terminal Agents Generalization in RL
Salesforce AI Research · Carnegie Mellon University

Oct 13, 2026J/K move · Enter open · S save

ReTerm: Recovery Training for Self-Improving Terminal Agents

Wei Xie, Kaituo Feng, Shiming Fu, Yong Liu+2

Fudan University · MMLab, The Chinese University of Hong Kong · Shanghai Jiao Tong University

LLM Agent Training Self-Improving Agents RL for LLM Agents

Open paper AskPDF arXiv

Terminal agents have shown strong abilities in complex coding tasks. However, they often struggle to recover from errors during long-running execution (e.g., command errors, faulty implementations), eventually leading to task failures. To this end, we present ReTerm, a self-improving terminal agent trained by learning how to recover from its own failed terminal tasks. To support its training, we construct a customized data pipeline and produce ReTerm-10K, a high-quality dataset that provides effective recovery demonstrations for SFT cold start. Furthermore, we propose ReTerm-GRPO, an online RL algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts. By learning from whether these attempts successfully complete the task, the agent self-improves through its own recovery experience. Built on Qwen3.5-9B, our ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0, improving over the base model by 14.6 points. Further analysis shows that our method achieves consistently higher recovery rates across diverse error categories compared with standard GRPO training. All code, data and model are released at https://github.com/twothsnd/ReTerm.

Abstract

RL for LLM Agents

Explore similar work

ReTerm: Recovery Training for Self-Improving Terminal Agents

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

Learning Generalizable Behaviors for Terminal Agents