Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.
Figures & tables
Figure 1: Learning to fail gracefully. (a) GracefulRL augments standard RLVR with an explicit fallback behavior, ordered as \textscSuccess≻\textscFallback≻\textscError . (b) Standard policy gradient (PG) can lose optimization signal near saturated policies. In contrast, the GracefulRL objective retains a non-vanishing gradient. (c) Using models trained with GracefulRL on the AIME benchmark, the most cautious model ( c=0.8 ) reduces the error rate from 79.1% to 8.9% , an 88.8% relative reduction, while success only drops by 3.1% compared to RL with binary rewards ( c=0 ).
Figure 2: A simple 3-class softmax policy. We use logits zS,zF and zE to create a policy π=(pS,pF,pE)=softmax(zS,zF,zE) . (a) For initializations with low initial success probabilities, ∇θJpg leads to traps near the fallback and error vertices ( pF=1,pE=1 ), (b) however, ∇θJ\textscGracefulRL is more robust to initialization leading to higher success and (c) higher reward.
Figure 3: (a) The SafeMaze task requires the agent to reach the Goal , but if it can’t, it should backtrack to Safe . (b) GRPO and RLOO can collapse to a degenerate policy pF=1 that is, always going to the Safe square. (c) GracefulRL is robust and Pareto-dominates reward-tuning and DPG (across 3 seeds), achieving higher a Success@1 rate at a lower Error@1 rate.
Figure 4: (a) Training for abstention using standard policy gradient (RLOO) can converge to a policy that always abstains, while our method learns a meaningful policy. (b), (c) Reliability frontiers for the trained Qwen3-1.7B-Base and Qwen3-4B-Base models, comparing RLCR+Thresholding and GracefulRL . GracefulRL achieves a higher success rate at a lower error rate.
Figure 5: Different behaviors under uncertainty. A model trained with RLVR has no incentive to acknowledge limitations, and guesses when uncertain. But a model trained with GracefulRL verbalizes and acknowledges its lack in capabilities. Full traces appear in Appendix D.9 .
Figure 7: Minesweeper results. (a) Success@1 and Error@1 rates after training the Qwen3-4B-Instruct-2507 model. In this setting, GracefulRL achieves a lower rate and higher performance compared to RL with no fallback option. (b) Probability of an error on each turn for MaxRL ( c=0 ) and GracefulRL ( c=0.5 ). Number of errors significantly increase at the end of the game for MaxRL since the incentive to ‘play it safe’ reduces and risk taking moves offer better expected return.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: RLOO with fixed fallback rewards on SafeMaze. We measure pS,pF,pE during training for c=0.1 , c=0.2 , and c≥0.3 . Empirically, we observe that when c≪pS , the policy can lead to no coverage on fallback trajectories, that is, pF→0 . Similarly, when c≫pS , the policy can lead to a degenerate policy where it always goes to the Safe square.
Figure 9: SafeMaze rollout and smoothing sweeps. (a) Reliability frontiers after 5,000 steps for 16 , 32 , and 64 rollouts. (b) Training reward with 32 rollouts and c=0.4 : larger ϵ helps initially, whereas ϵ=1 performs best later. The colored bands mark the highest-reward curve in each training interval pointing to the general principle: when bounded by compute, larger ϵ helps in achieveing higher reward.
Figure 10: Entropy-bonus sweep for GRPO (top) and RLOO (bottom) on SafeMaze . Columns show the probabilities of success ( pS ), fallback ( pF ), and error ( pE ) during training. None of the tested entropy coefficients preserves task success: GRPO and most RLOO runs collapse to fallback, while RLOO with coefficient 1 collapses to error.
Hyperparameter
RLCR
Ours
Gradient estimator
RLOO
GracefulRL
Training data
DAPO-Math-17K
DAPO-Math-17K
Fallback utilities c
–
{0,0.2,0.4,0.6,0.8}
Smoothing ϵ
–
1
Optimizer
AdamW
AdamW
Learning rate / schedule
10−6 / constant
10−6 / constant
Appendix
Table 1: Training hyperparameters for the Qwen3-4B-Base RLCR and GracefulRL runs. A dash denotes a method-specific parameter that does not apply.
Hyperparameter
RLCR
Ours
Gradient estimator
RLOO
GracefulRL
Training data
DAPO-Math-17K
DAPO-Math-17K
Fallback utilities c
–
{0,0.2,0.4,0.6,0.8}
Smoothing ϵ
–
1
Optimizer
AdamW
AdamW
Learning rate / schedule
10−6 / constant
10−6 / constant
Appendix
Table 2: Training hyperparameters for the Qwen3-1.7B-Base RLCR and GracefulRL runs. A dash denotes a method-specific parameter that does not apply.
c
Conditioning suffix
0.0
“Be very optimistic and always attempt an answer, even if you are unsure.”
0.2
“Be optimistic but stay mindful of potential mistakes.”
0.4
“Be balanced, leaning towards attempting an answer.”
0.6
“Be moderately careful and verify your approach before committing to an answer.”
0.8
“Be quite careful and avoid guessing.”
Appendix
Table 3: Conditioning suffixes for the five GracefulRL fallback utilities.
Figure 11: MATH500 validation reward across training checkpoints for RLCR (left) and our conditioned GracefulRL method (right), at the 1.7B (top) and 4B (bottom) model scales. Stars mark the checkpoints selected for the final evaluation.
Figure 12: Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-1.7B-Base. The final panel reports the aggregate frontier across datasets.
Figure 13: Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-4B-Base. The final panel reports the aggregate frontier across datasets.
Figure 14: Effect of ϵ in language-model RL. Training reward for Qwen3-1.7B-Base with fallback reward c=0.4 over the first 400 optimization steps. Pale lines show raw reward and dark lines show the smoothed reward.
Figure 15: Training dynamics of the Qwen3-4B-Base RLCR policy on AIME. Although the Brier score generally improves (decreases) throughout training, answer accuracy peaks early and then declines. The RLCR reward combines these two quantities and therefore need not track accuracy.
Figure 16: Success and error dynamics on AIME. Qwen3-4B-Base trained with GracefulRL , evaluated at c∈{0,0.2,0.4,0.6,0.8} , and MaxRL trained with binary rewards (dark dashed curve). The left and right panels show pS and pE , respectively, as percentages. Curves use checkpoints evaluated every 50 steps and plot a 3-point moving average.
Figure 17: Effect of training compute on the AIME reliability frontier. Qwen3-4B-Base evaluated after 200 ( 1× ), 400 ( 2× ), and 800 ( 4× ) training steps. Each marker is for a different c value and the dashed curves fit the resulting frontiers.
Figure 18: Online reward tuning loses fallback behavior. Success, fallback, and error rates (moving average) on on-policy training rollouts for Qwen3-1.7B-Base and Qwen3-4B-Base, aggregated across the five conditioning levels.
Figure 19: Per-dataset abstention performance on AbstentionBench . Abstention F1 across twenty datasets for GracefulRL at c∈{0.2,0.4,0.6,0.8} (orange) and Qwen3-4B-Instruct-2507 (blue). The methods use matching per-dataset sample counts and class balance; the effect of conditioning varies by dataset.
Setting
Value
Fresh rollout batch
32 puzzle prompts ×16 episodes =512 episodes; one optimizer update per batch
Sampling during training
Temperature 1.0 , top- p1.0 ; at most 4,096 generated tokens per turn and 32,000 tokens of context
One eight-H100 node: four actor GPUs (tensor parallelism 2 , data parallelism 2 ) and four rollout GPUs (two tensor-parallel- 2 SGLang engines); CPU optimizer offload
Appendix
Table 4: Shared training settings for the four Minesweeper c -sweep runs.
Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.
Shuning Shang, Hubert Strauss, Stanley Wei +2
Princeton Language and Intelligence, Princeton University
Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.
Ziyi Wang, Yuxuan Lu, Yimeng Zhang +8
Northeastern University · Independent Researcher · Northwestern University
Post-training of frontier language models is conducted on curated task suites, and inevitably leaves a distribution shift between training and deployment environments. This exposes developers to generalization failures, which are relatively poorly understood. To better understand such generalization failures, we believe the community should construct clean demonstrations under simplified conditions. To facilitate this, we propose a simple and flexible way to construct language models which fail to generalize in controllable ways when subsequently trained with Reinforcement Learning (RL) on a given distribution of training tasks. Our construction uses Supervised Fine-Tuning on a dataset of a mixture of transcripts corresponding to a collection of 'conditional policies', which can each independently be assigned certain behaviors on each different task distribution, to obtain a model that is then well approximated as a 'mixture of conditional policies.' We observe that RL training then selects for policies that obtain the highest reward on the training distribution. This can produce striking behaviors: in a controlled setting with two distributions containing identical questions prepended with two different 'trigger strings', RL training on either distribution actively degrades performance on the other to zero, even though the underlying task is identical. We also use our construction to illustrate two novel ways in which generalization may fail in future language models, corresponding to distribution shifts of task coverage and temporal context respectively. While our construction is deliberately simple and may not closely resemble 'natural' generalization failures, the resulting 'model organisms' are of interest for alignment stress-testing and generalization science, and can be used as existence proofs that training success and generalization can come apart in structured ways.