Training Language Models to Fail Gracefully

Daman Arora; Xuecheng Liu; Himanshu Gaurav Singh; Fahim Tajwar; Abitha Thankaraj; Ruslan Salakhutdinov; Haiwen Feng; Yuda Song

cs.LGOct 14, 2026

Training Language Models to Fail Gracefully

Authors: Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh, Fahim Tajwar, Abitha Thankaraj, Ruslan Salakhutdinov, Haiwen Feng, Yuda Song,

Organizations: Carnegie Mellon University · UC Berkeley

Abstract

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

Figures & tables

Figure 1: Learning to fail gracefully. (a) GracefulRL augments standard RLVR with an explicit fallback behavior, ordered as

\textsc{Success}\succ\textsc{Fallback}\succ\textsc{Error}

. (b) Standard policy gradient (PG) can lose optimization signal near saturated policies. In contrast, the GracefulRL objective retains a non-vanishing gradient. (c) Using models trained with GracefulRL on the AIME benchmark, the most cautious model (

c=0.8

) reduces the error rate from

79.1\%

8.9\%

, an

88.8\%

relative reduction, while success only drops by 3.1% compared to RL with binary rewards (

c=0

Figure 2: A simple 3-class softmax policy. We use logits

z_{S},z_{F}

and

z_{E}

to create a policy

\pi=(p_{S},p_{F},p_{E})=\mathrm{softmax}(z_{S},z_{F},z_{E})

. (a) For initializations with low initial success probabilities,

\nabla_{\theta}\mathcal{J}_{pg}

leads to traps near the fallback and error vertices (

p_{F}=1,p_{E}=1

), (b) however,

\nabla_{\theta}\mathcal{J}_{\textsc{GracefulRL}}

is more robust to initialization leading to higher success and (c) higher reward.

Figure 3: (a) The SafeMaze task requires the agent to reach the Goal , but if it can’t, it should backtrack to Safe . (b) GRPO and RLOO can collapse to a degenerate policy

p_{F}=1

that is, always going to the Safe square. (c) GracefulRL is robust and Pareto-dominates reward-tuning and DPG (across 3 seeds), achieving higher a

\operatorname{Success@}1

rate at a lower

\operatorname{Error@}1

rate.

Figure 4: (a) Training for abstention using standard policy gradient (RLOO) can converge to a policy that always abstains, while our method learns a meaningful policy. (b), (c) Reliability frontiers for the trained Qwen3-1.7B-Base and Qwen3-4B-Base models, comparing RLCR+Thresholding and GracefulRL . GracefulRL achieves a higher success rate at a lower error rate.

Panel of Figure 5 — Figure 5: Different behaviors under uncertainty. A model trained with RLVR has no incentive to acknowledge limitations, and guesses when uncertain. But a model trained with GracefulRL verbalizes and acknowledges its lack in capabilities. Full traces appear in Appendix D.9 .

Figure 6: Abstention

F_{1}

: GracefulRL (orange), Instruct (blue).

Figure 7: Minesweeper results. (a)

\operatorname{Success@}1

and

\operatorname{Error@}1

rates after training the Qwen3-4B-Instruct-2507 model. In this setting, GracefulRL achieves a lower rate and higher performance compared to RL with no fallback option. (b) Probability of an error on each turn for MaxRL (

c=0

) and GracefulRL (

c=0.5

). Number of errors significantly increase at the end of the game for MaxRL since the incentive to ‘play it safe’ reduces and risk taking moves offer better expected return.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: RLOO with fixed fallback rewards on SafeMaze. We measure

p_{S},p_{F},p_{E}

during training for

c=0.1

c=0.2

, and

c\geq 0.3

. Empirically, we observe that when

c\ll p_{S}

, the policy can lead to no coverage on fallback trajectories, that is,

p_{F}\rightarrow 0

. Similarly, when

c\gg p_{S}

, the policy can lead to a degenerate policy where it always goes to the Safe square.

Figure 9: SafeMaze rollout and smoothing sweeps. (a) Reliability frontiers after

5{,}000

steps for

16

32

, and

64

rollouts. (b) Training reward with

32

rollouts and

c=0.4

: larger

\epsilon

helps initially, whereas

\epsilon=1

performs best later. The colored bands mark the highest-reward curve in each training interval pointing to the general principle: when bounded by compute, larger

{\epsilon}

helps in achieveing higher reward.

Figure 10: Entropy-bonus sweep for GRPO (top) and RLOO (bottom) on SafeMaze . Columns show the probabilities of success (

p_{S}

), fallback (

p_{F}

), and error (

p_{E}

) during training. None of the tested entropy coefficients preserves task success: GRPO and most RLOO runs collapse to fallback, while RLOO with coefficient

1

collapses to error.

Hyperparameter	RLCR	Ours
Gradient estimator	RLOO	GracefulRL
Training data	DAPO-Math-17K	DAPO-Math-17K
Fallback utilities $c$	–	$\{0,0.2,0.4,0.6,0.8\}$
Smoothing $\epsilon$	–	$1$
Optimizer	AdamW	AdamW
Learning rate / schedule	$10^{-6}$ / constant	$10^{-6}$ / constant

Appendix

Table 1: Training hyperparameters for the Qwen3-4B-Base RLCR and GracefulRL runs. A dash denotes a method-specific parameter that does not apply.

Hyperparameter	RLCR	Ours
Gradient estimator	RLOO	GracefulRL
Training data	DAPO-Math-17K	DAPO-Math-17K
Fallback utilities $c$	–	$\{0,0.2,0.4,0.6,0.8\}$
Smoothing $\epsilon$	–	$1$
Optimizer	AdamW	AdamW
Learning rate / schedule	$10^{-6}$ / constant	$10^{-6}$ / constant

Appendix

Table 2: Training hyperparameters for the Qwen3-1.7B-Base RLCR and GracefulRL runs. A dash denotes a method-specific parameter that does not apply.

$c$	Conditioning suffix
$0.0$	“Be very optimistic and always attempt an answer, even if you are unsure.”
$0.2$	“Be optimistic but stay mindful of potential mistakes.”
$0.4$	“Be balanced, leaning towards attempting an answer.”
$0.6$	“Be moderately careful and verify your approach before committing to an answer.”
$0.8$	“Be quite careful and avoid guessing.”

Appendix

Table 3: Conditioning suffixes for the five GracefulRL fallback utilities.

Figure 11: MATH500 validation reward across training checkpoints for RLCR (left) and our conditioned GracefulRL method (right), at the 1.7B (top) and 4B (bottom) model scales. Stars mark the checkpoints selected for the final evaluation.

Figure 12: Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-1.7B-Base. The final panel reports the aggregate frontier across datasets.

Figure 13: Per-dataset reliability frontiers for RLCR and GracefulRL (Ours) with Qwen3-4B-Base. The final panel reports the aggregate frontier across datasets.

Figure 14: Effect of

\epsilon

in language-model RL. Training reward for Qwen3-1.7B-Base with fallback reward

c=0.4

over the first

400

optimization steps. Pale lines show raw reward and dark lines show the smoothed reward.

Figure 15: Training dynamics of the Qwen3-4B-Base RLCR policy on AIME. Although the Brier score generally improves (decreases) throughout training, answer accuracy peaks early and then declines. The RLCR reward combines these two quantities and therefore need not track accuracy.

Figure 16: Success and error dynamics on AIME. Qwen3-4B-Base trained with GracefulRL , evaluated at

c\in\{0,0.2,0.4,0.6,0.8\}

, and MaxRL trained with binary rewards (dark dashed curve). The left and right panels show

p_{S}

and

p_{E}

, respectively, as percentages. Curves use checkpoints evaluated every

50

steps and plot a 3-point moving average.

Figure 17: Effect of training compute on the AIME reliability frontier. Qwen3-4B-Base evaluated after

200

(

1\times

400

(

2\times

), and

800

(

4\times

) training steps. Each marker is for a different

c

value and the dashed curves fit the resulting frontiers.

Figure 18: Online reward tuning loses fallback behavior. Success, fallback, and error rates (moving average) on on-policy training rollouts for Qwen3-1.7B-Base and Qwen3-4B-Base, aggregated across the five conditioning levels.

Figure 19: Per-dataset abstention performance on AbstentionBench . Abstention

F_{1}

across twenty datasets for GracefulRL at

c\in\{0.2,0.4,0.6,0.8\}

(orange) and Qwen3-4B-Instruct-2507 (blue). The methods use matching per-dataset sample counts and class balance; the effect of conditioning varies by dataset.

Setting	Value
Fresh rollout batch	$32$ puzzle prompts $\times$ $16$ episodes $=512$ episodes; one optimizer update per batch
Sampling during training	Temperature $1.0$ , top- $p$ $1.0$ ; at most $4{,}096$ generated tokens per turn and $32{,}000$ tokens of context
Policy loss	Per-token loss; lower/upper clipping ratios $0.20/0.28$ ; entropy coefficient $0$
Optimizer	Adam; constant learning rate $10^{-6}$ ; weight decay $0.1$ ; $(\beta_{1},\beta_{2})=(0.9,0.98)$
Compute layout	One eight-H100 node: four actor GPUs (tensor parallelism $2$ , data parallelism $2$ ) and four rollout GPUs (two tensor-parallel- $2$ SGLang engines); CPU optimizer offload

Appendix

Table 4: Shared training settings for the four Minesweeper

c

-sweep runs.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 410 papers

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Apr 28, 2026Shuning Shang, Hubert Strauss, Stanley Wei +2Reward Modeling Policy Gradient
Princeton Language and Intelligence, Princeton University
SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
Jun 11, 2026Ziyi Wang, Yuxuan Lu, Yimeng Zhang +8AI Agent Reliability Reinforcement Learning
Northeastern University · Independent Researcher · Northwestern University
Demonstrating Generalization Failures via Mixtures of Conditional Policies
Jul 3, 2026Jou Barzdukas, Jack Peck, Julian Schulz +3OOD Generalization
University of Virginia · ETH Zurich · MATS +3

Apr 28, 2026J/K move · Enter open · S save

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

Shuning Shang, Hubert Strauss, Stanley Wei, Sanjeev Arora+1

Princeton Language and Intelligence, Princeton University

Reward Modeling Policy Gradient Policy Learning Reward Design for RL Reward Model Evaluation RL from Human Feedback

Open paper AskPDF arXiv

Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.

Abstract

RL for Language Models

Explore similar work

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents

Demonstrating Generalization Failures via Mixtures of Conditional Policies