When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning

Steven Y. Feng; Noah D. Goodman; Michael C. Frank; Evan Hubinger; Paul C. Bogdan; Andrew Lampinen

cs.LGSep 27, 2026

When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning

Authors: Steven Y. Feng, Noah D. Goodman, Michael C. Frank, Evan Hubinger, Paul C. Bogdan, Andrew Lampinen

Organizations: Stanford University · Anthropic

Abstract

Outcome-based reinforcement learning can produce models with similar task performance but very different ways of communicating about their mistakes. We study failure disclosure: whether a model admits that an attempted solution failed rather than staying silent or presenting it as successful. Across repeated outcome-only GRPO training runs, failure disclosure varies far more than task accuracy. The pattern extends to a second reasoning task and stabilized PPO, persists at 7B, and also appears in an instruction-conditioned 32B setting. We also find that small floating-point and sampling differences during training can redirect reporting behavior even when the task objective and earlier training history are held fixed. Additional tests show that failure disclosure is not a single decision: Checking the answer, entering a report, and completing the admission can separate, and the weak point depends on the task and response format. Further, experiments with neutral controls show more broadly that behaviors left weakly constrained by training are especially likely to vary across runs, of which failure disclosure is an example. We can reduce variability in failure disclosure by discouraging the model from drifting from its starting policy on failed, well-formed responses. This makes reporting substantially more consistent, though its effect on task performance depends on the setting. Stable task accuracy therefore does not guarantee stable safety-relevant behavior: Researchers should measure these behaviors directly across runs and design training methods that keep them reliable.

Figures & tables

Figure 1: Failure disclosure is substantially less reproducible across retrainings than task success. (A) In the primary set of 20 Qwen2.5-1.5B retrainings, failure disclosure has 3.9

\times

larger cross-run SD than solve rate. (B) From one mid-RL state at step 150, ten continuations differing only in future rollout randomness end with failure-disclosure rates from 0.11 to 0.88. This is the widest of eight mid-RL branch-and-continue replication tests; across its eight ten-continuation cells, the median within-cell rate SD is 0.10 (Appendix Table 19 ). (C) In 20 matched Qwen2.5-1.5B pairs, a reference-policy penalty applied to failed, well-formed responses substantially reduces the run-to-run spread of disclosure. The unanchored points in C use the same underlying checkpoints as A, but an independent probe sampling draw used for the anchoring comparison; the measured rates are therefore similar but not identical. Across the distribution figures, dots show retraining runs, short horizontal lines show means, and faint vertical lines show observed min–max ranges.

Figure 2: Failure-disclosure variability appears across models, tasks, and reporting setups. (A) On Countdown, failure disclosure varies substantially more across retrainings than solve for Qwen2.5-1.5B & OLMo-2-1B. Qwen-7B is larger-scale evidence; instruction-conditioned Qwen2.5-32B tests reporting without report SFT; capability is also broader. (B) This pattern also appears across tasks with Qwen2.5-1.5B: Failure disclosure varies substantially more than task performance on both Countdown and shortest path, using each task’s primary capability evaluation.

Figure 3: Training differences and behavioral cues shape reporting variability. (A) Each row uses a different execution configuration, with solve and disclosure measured from the same run. Changing only the configuration moves failure disclosure from 0.186 to 0.848, while solve remains 0.327–0.463. (B) Starting from the same mid-RL training state, changing only future rollout randomness produces widely different disclosure outcomes. (C) Models nearly always reach a point where a report could begin, but differ in whether they admit failure after. (D) Inserting the paragraph break from the learned report template makes later admission more frequent and consistent in the primary recipe; adding a verifier verdict does not help further. (E) The same neutral phrase varies widely when weakly constrained but is nearly reproducible when tied to an explicit cue. The primary disclosure spread, a failure-conditioned neutral codeword, and success-side pleasantries are shown for comparison. Appendices E and F report the additional localization and control results.

Figure 4: Reference anchoring reduces retraining variability and future divergence from the same mid-RL state. (A) Anchoring makes disclosure more consistent across retrainings in 1.5B and 7B Countdown and shortest-path. In 1.5B Countdown, disclosure SD falls from 0.189 to 0.089; at matched mean solve, the unanchored SD is 0.139. At 7B, disclosure SD falls from 0.281 to 0.090, and probe solve is 0.069 higher under anchoring. Changes in average disclosure differ across settings. (B) After training with anchoring, keeping the anchor on leads to less variation across future continuations. The left plot shows four anchor-off and four anchor-on continuations from one shared mid-RL state. The right plot summarizes all ten mid-RL states. Token-level localization results are described in the text and Appendix I ; the 7B sensitivity analysis is in Appendix H.4 .

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Size	Reporting setup	Role
Qwen2.5-1.5B	1.5B	Failure-report behavior installed through SFT	primary phenomenon, mechanism, and anchoring experiments
Qwen2.5-1.5B (second reference)	1.5B	Failure report + success pleasantry installed through SFT	independent second run set; success-side pleasantry control
Qwen2.5-1.5B (shortest-path reference)	1.5B	Failure-report behavior installed through SFT	second reasoning task; shortest-path anchoring
Qwen2.5-7B	7B	Failure-report behavior installed through SFT	larger-scale persistence experiment; 7B anchoring
Qwen2.5-14B	14B	Failure-report behavior installed through SFT	appendix execution-configuration boundary
Qwen2.5-32B-Instruct	32B	Instruction-conditioned; no report SFT	report-SFT-dependence boundary; not a matched scale-up

Appendix

Table 1: Models used in the reported results. Boundary-only families are included only where they help define the scope of our claims.

Task	What the model does	Automatic success check	Role in paper
Countdown	Use four numbers exactly once to build an arithmetic expression that reaches a target.	AST-parsed expression using +, –, $\times$ , and $\div$ only; exact rational evaluation; multiset number-use check	main setting for the primary runs, execution-configuration tests, mid-RL continuation tests, routing, and anchoring
Shortest path	Find a valid shortest path in a small generated graph or grid.	Reasoning Gym scorer, binarized at score = 1.0	second verifiable reasoning task; 20 retraining runs, anchoring, and shelter experiments
4 $\times$ 4 grid / constraint puzzles	Solve or assess small Latin-square / Shidoku-style constraint grids.	Exact solver oracle; every row, column, and when applicable box must be a permutation of 1–4	targeted pipeline tests of failure checking and report routing; not a co-equal replication
Code	Write Python functions for MBPP+ tasks and optionally report whether tests pass or fail.	True test-bank result from executed tests; self-report is ignored	boundary tests for direct reward, reward $\times$ reference anchoring, and surface/function separation

Appendix

Table 2: Task domains and their role in the paper. Countdown and shortest path support the main claims; grid and code settings are targeted mechanism and boundary tests.

Setting	Post-training	Batch	Reward / reference term
Qwen-1.5B / Countdown (primary)	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences = 4 prompts $\times$ 8 generations	Task reward; $\beta=0$ (no reference KL)
Qwen-1.5B / Countdown (second run set)	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences	Task reward; $\beta=0$
OLMo-1B / Countdown	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences	Task reward; $\beta=0$
Qwen-1.5B / shortest-path	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences	Task reward; $\beta=0$
Qwen-7B / Countdown	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences	Task reward; $\beta=0$
Qwen-32B / Countdown (instruction-conditioned)	Dr. GRPO; 300 steps; $G=8$ ; LR $10^{-6}$	32 sequences (4 GPUs)	Task reward; $\beta=0$

Appendix

Table 3: Main post-training recipes. The main outcome-RL runs use the same optimization scale; intervention rows change the reference term or algorithm as described in the text.

Setting	Frozen stochastic probe	Capability read
Primary Qwen2.5-1.5B / Countdown	20 pinned prompts $\times$ 8 samples $\times$ 3 probe seeds = 480 rows; vLLM ( Kwon et al., 2023 )	Correct / 480 on the same probe rows
OLMo-2-1B / Countdown	20 $\times$ 8 $\times$ 3 = 480 rows; HF decoder	Same probe rows; greedy dev-100 reported beside
Qwen2.5-1.5B / shortest-path	9 pinned prompts $\times$ 8 $\times$ 3 = 216 rows	Same probe rows; greedy solve on the 100 shortest-path dev prompts reported beside and used for headline selectivity
Qwen2.5-7B / Countdown	17 pinned prompts $\times$ 8 $\times$ 3 = 408 rows	Same probe rows; greedy dev-100 reported beside
Qwen2.5-32B-Instruct / Countdown	20 $\times$ 8 $\times$ 3 = 480 rows per prompt condition	Correct / 480 on the same probe rows

Appendix

Table 4: Frozen evaluation surfaces for the main experiments. Stochastic probes use temperature 1.0 and a 1,024-token completion budget; greedy companions use temperature 0.

Component	What is held fixed or restored	Check
Model weights	Restored from the parent checkpoint.	Same-key continuations match endpoint and model artifacts.
Optimizer state	Adam moments are restored; the fork stops if state is missing.	Restoration is checked before continuation.
Scheduler / trainer step	Learning-rate scheduler and global training step are restored.	Continuation resumes from the parent step.
RNG state	CPU, CUDA, NumPy, and Python RNG states are restored; the continuation key then changes future rollout randomness.	The same key reproduces; different keys branch.
Data schedule	Parent prompt schedule and iterator position are held fixed.	All keyed continuations see the same training order.
Cross-host replay	The same saved state and continuation key are replayed on separate pods.	Bit-exact replay check gives endpoint and per-run-record agreement.

Appendix

Table 5: What is restored in the exact-state mid-RL continuation experiments. These checks isolate future stochasticity without changing the earlier training history.

Condition	Unit	Calibrated disclosure [95% bootstrap interval]	Solve
Instruction prompt, before RL	1 parent	0.641 [0.593, 0.688]	0.288
No instruction, before RL	1 parent	0.000	0.215
Instruction-conditioned retrainings @300	8 draws	0.173–1.000	0.302–0.619

Appendix

Table 6: Calibration for the instruction-conditioned Qwen2.5-32B boundary experiment, which elicits failure reporting through the prompt without report SFT. The frozen-detector lower bound and the blind-read calibrated estimate are both stated in the main text.

Readout	What counts	Denominator	Use
Countdown disclosure (primary)	Explicit acknowledgment that the attempted Countdown solution failed.	Terminated task failures	main 1.5B Countdown measure; anchoring and mid-RL continuation experiments
Countdown disclosure (broad)	Broader semantic admissions, including paraphrases and answer-channel acknowledgments.	Terminated task failures	7B main readout and detector-sensitivity checks
Shortest-path disclosure	Domain-specific statements that no valid or shortest route was found.	Terminated shortest-path failures	shortest-path phenomenon and anchoring experiments
Route-entry disclosure	Broad semantic admission detector after removing the injected rider or prefill.	Failed continuations in the routing probe	P-B2 route-entry analysis
Auxiliary behavior detectors	Simple lexical or structured detectors for pleasantries and neutral codewords.	Setting-specific probe rows	P-D/P-E2/P-A/Kettle and formulation controls

Appendix

Table 7: Reader-facing failure-disclosure measures. Internal detector version names and code paths are omitted here for readability.

Audit	Setting	Sample	Main result
Human semantic-label audit	Seven analysis strata	560 sampled; 552 unique rows; 97/98 validation rows scored	Detector–human agreement 80/97 (0.825); 40/42 in concession strata, with no human false positives. Model–model $\kappa=0.822$ .
Human blind re-read	Primary / second run set / installed-channel runs	48 real endings from low, high, and median runs; random rows within runs	44/48 agreement; Cohen kappa 0.846 [0.678, 0.965].
Human second rater	Independent ending labels	40 endings, seeded stratified sample	39/40 agreement; Cohen kappa 0.950.
Model-reader audit	1.5B anchoring arms	50 positive + 50 negative per arm	Cohen kappa 0.941–1.000; 0/50 false positives in all three arms.
Model-reader audit	Shortest-path runs	50 + 50 per horizon	Cohen kappa 1.000 @150 and 0.961 @300; no misses or false positives.
Model-reader audit	7B common-path runs	50 + 50 per horizon	Cohen kappa about 0.90; small miss and false-positive counts motivate broader semantic checks.

Appendix

Table 8: Summary of the human and independent model-reader audits. The two audit types are reported separately.

Setting	$n$	False success	Silent close	Admission	False-success share of nondisclosure
1.5B, no anchor	20	0.18 (0.09–0.28)	0.14 (0.00–0.51)	0.69 (0.37–0.91)	0.67
1.5B, failure-conditional anchor	20	0.18 (0.08–0.28)	0.10 (0.00–0.32)	0.72 (0.49–0.83)	0.66
1.5B, all-token anchor	20	0.16 (0.09–0.21)	0.12 (0.04–0.29)	0.72 (0.58–0.82)	0.60
7B, no anchor	19	0.27 (0.02–0.59)	0.21 (0.00–0.46)	0.52 (0.04–0.94)	0.60
7B, failure-conditional anchor	19	0.14 (0.03–0.28)	0.02 (0.00–0.06)	0.84 (0.71–0.97)	0.89

Appendix

Table 9: How failed responses end with and without anchoring at step 300. False success means that the model presents a wrong answer as correct. Entries are the mean across runs, followed by the range across runs in parentheses, and are calculated as shares of terminated failures. Responses that do not reach a normal termination before the generation limit are reported separately and excluded from these shares. The 1.5B and 7B rows use separate fixed classifiers suited to their response formats, so comparisons involving anchoring are made within each model setting.

Setting	$n$	Disclosure $\hat{\tau}$ [95%]	Selective-dispersion statistic	Estimate [95%]
Qwen-1.5B / Countdown	20	0.864 [0.563, 1.129]	SD(disclosure) / SD(solve)	3.93 [2.64, 5.50]
Qwen-1.5B / second run set †	20	0.949 [0.613, 1.266]	$\hat{\tau}$ (disclosure) / $\hat{\tau}$ (solve)	5.67 [2.89, $\infty$ ]
OLMo-1B / Countdown	20	1.22 [0.81, 1.58]	SD(disclosure) / SD(solve)	7.51 [5.40, 10.71]
Qwen-1.5B / shortest-path ∗	20	2.81 [1.88, 3.80]	SD(disclosure) / SD(greedy solve)	7.89 [5.20, 11.55]
Qwen-7B / Countdown	19	2.28 [1.47, 3.06]	SD(disclosure) / SD(solve)	2.30 [1.55, 3.94]
Qwen-1.5B / PPO	8	–	SD(disclosure) / SD(solve)	3.55 [1.35, 7.63]

Appendix

Table 10: Headline selective-dispersion estimates and 95% uncertainty intervals. SD ratios compare across-run disclosure SD with solve SD; the second Qwen experiment instead uses the hierarchical

\hat{\tau}

ratio.

Run set	Primary SD	Broader SD	Broadest SD	Main result
Primary Qwen-1.5B	0.185	0.163	0.113	Minimum rank agreement across nested detector tiers is $\rho=0.908$ .
Second Qwen-1.5B	0.220	0.175	0.169	Minimum rank agreement across nested detector tiers is $\rho=0.917$ .
Qwen-7B	0.281	0.255	0.229	Broader semantic detectors reduce estimated SD, but substantial variation across runs remains.

Appendix

Table 11: Sensitivity to broader detector definitions. Broader semantic detectors change the estimated levels somewhat, but the large differences across retrainings remain. Broadest SD uses the union of all seven detector tiers for the 1.5B rows and the broadest single semantic detector for the 7B row.

Setting	n	Disclosure range	Solve range	SD(disclosure)	SD(solve)
Qwen-1.5B / Countdown	20	0.257–0.903	0.340–0.490	0.185	0.047
Qwen-1.5B / second run set †	20	0.171–0.895	0.530–0.740	0.220	0.063
OLMo-1B / Countdown	20	0.072–0.903	0.375–0.492	0.255	0.034
Qwen-1.5B / shortest-path ∗	20	0.000–0.986	0.460–0.610	0.349	0.044
Qwen-7B / Countdown	19	0.000–0.770	0.150–0.561	0.281	0.122
Qwen-32B / instruction-conditioned	8	0.056–0.628	0.302–0.619	0.238	0.106

Appendix

Table 12: Across-retraining ranges and raw SDs for the main phenomenon and generalization experiments. These rows are descriptive and are not pooled into a single universal effect.

Checkpoint	Mean disclosure	Across-run SD
SFT reference	0.677	–
Step 15	0.691	0.061
Step 30	0.697	0.055
Step 45	0.707	0.057
Step 60	0.708	0.095
Step 90	0.685	0.100

Appendix

Table 13: Disclosure trajectory for the primary runs on the fixed stochastic probe. The reference-model row is evaluated on the same probe; the remaining rows summarize the 20 retrainings.

Capability evaluation	Prompts per run	Solve range	Cross-run SD	Binomial floor
Primary registered stochastic probe	20 $\times$ 8 samples $\times$ 3 seeds	0.34–0.49	0.047	–
Primary post-hoc greedy dev set, H200	100	0.65–0.78	0.045	0.045
Second Qwen run set, greedy dev set	100	0.53–0.74	0.063	0.047

Appendix

Table 14: Capability dispersion on the registered probe and on the full 100-prompt dev set. The full-dev read for the primary runs is a post-hoc, inference-only robustness check; the registered 480-row probe remains primary. The simple binomial floor is a finite-sample reference, not a formal null test.

Surface	Prompts	Disclosure range	Across-run SD	Hierarchical $\hat{\tau}$
Full dev set, primary detector	100	0.13–0.93	0.235	1.03 [0.63, 1.40]
Full dev set, broadest detector	100	0.39–0.93	0.169	0.69 [0.34, 0.96]
Selected probe prompts, primary detector	20	0.09–1.00	0.251	–
Other dev prompts, primary detector	80	0.12–0.88	0.266	–

Appendix

Table 15: Post-hoc greedy disclosure on the full dev set and its selected versus unselected prompt subsets. The selected and unselected rows use smaller, run-dependent failure denominators and are shown only to test whether behavior-based probe selection inflated the spread.

Boundary	Setting	n	Main result
Group size	G=16, @150	n=3	Disclosure spread is 0.419; variability remains, but this arm also changes prompt exposure per step.
Larger batches	Batch 64/128	n=3 each	Broad disclosure variation remains, but endpoint capability is confounded; a capability-matched companion remains above the prespecified spread floor.
Gemma-2-2B	Countdown	n=3	Does not reproduce the broad primary-detector pattern at this small n; the reporting surface also differs.
Llama-3.2-3B	Countdown	n=3	Disclosure spread is 0.194; we make no broad-variability claim at this small n.
Distilled Qwen-1.5B	Countdown	n=3	Tight boundary: disclosure spread is 0.076 at step 300 and 0.129 at step 150.
Phi-4	Countdown	n=3	Disclosure stays uniformly high while capability varies substantially; capability-confounded boundary.

Appendix

Table 16: Selected training-recipe, algorithm, and model-family boundaries. These are supporting scope checks rather than headline replications.

Micro-batch / accumulation	Micro-batch	Grad. accum.	Disclosure	Solve
pd1ga32	1	32	0.186	0.463
pd2ga16	2	16	0.519	0.354
pd4ga8	4	8	0.836	0.435
pd8ga4	8	4	0.787	0.412
pd16ga2	16	2	0.848	0.365
pd32ga1	32	1	0.644	0.327

Appendix

Table 17: Six micro-batch

\times

gradient-accumulation execution configurations with the seed and global batch size fixed. The disclosure column uses the pre-specified endpoint measure shown in Figure 3 A.

Check	Comparison	Result
Same execution configuration	5 replays across two B200 hosts / GPU indices	Group logs, probe samples, and fixed evaluation windows are bit-exact in 5/5 replays.
Micro-batch re-split	Same seed; different micro-batch / accumulation split	Step-1 rollout generation is identical; later numerical updates cause the training records to diverge.
Hardware/GPU path	B200 vs H200 path	Generation differs from step 1; endpoint disclosure is 0.787 vs 0.204 at similar solve.

Appendix

Table 18: Replay and hardware/GPU checks. Repeating the same execution configuration is bit-exact in our B200 setup; changing hardware class can change generation from the first training step.

Saved state	Keys	Measured rate @300	Solve @300
admission 1, s123 @150	10	0.556–0.911	0.62–0.73
admission 1, s123 @240	10	0.605–0.871	0.57–0.71
admission 1, s124 @150	10	0.112–0.880	0.46–0.72
admission 1, s124 @240	10	0.177–0.530	0.62–0.70
codeword 1, s123 @150	10	0.642–0.965	0.65–0.76
codeword 1, s123 @240	10	0.667–0.864	0.66–0.78

Appendix

Table 19: Endpoint ranges across continuations from the shared mid-RL states. Admission and post-history rows report failure disclosure; codeword rows report codeword emission. Solve is greedy dev-100 solve for the admission and codeword rows and probe solve (correct/480) for post-history.

Figure 5: Failure-report emission across continuations from eight shared mid-RL states. Each column starts from one identical saved parent state; the dots differ only in future rollout randomness. Short horizontal ticks show means, and faint vertical lines show the observed min–max range.

Figure 6: Task solve for the same mid-RL continuations. In the cells with the widest reporting variation, task capability varies much less.

Stage	Evidence	What it supports	Main limitation
Failure evidence / checking	Grid/constraint checks	The visible report can separate from the computation that supports a failure verdict.	Grid-only evidence; not a Countdown localization result.
Report-entry template cue	NoBreak versus Break at the same seam, 1.5B	Most tightening comes from supplying the paragraph break that begins the learned failure-report template, not from re-entry alone.	Format-specific cue; does not reveal whether the model internally detected the failure.
Natural transition probability	Teacher-forced break versus report-entry comparison	The break and next report-entry step are not distinguished in the 3 step-300 comparisons; the inserted break is strongly off-policy.	Correlational over ten runs per cell; does not identify cause or internal recognition.
Inline initiation at 7B	Two low-disclosure 7B runs	Both reach the reporting point; supplying enough of the inline admission opening yields 0.97 / 1.00 acknowledgment.	Two runs; not a population-level rate.
Failure cue versus forced initiation	Verdict-only B/ORACLE; two-word and first-third prefixes C/D	Verdict-only gives no consistent benefit; forced prefixes raise admission much more.	An inserted verdict need not reproduce the model’s own recognition state.
Route decomposition	Route-decomposition probe (P-B2)	Route entry explains 70% [49, 91] of raw variation; downstream persistence explains 23% [14, 32].	Downstream variation remains; the relative-logit comparison is not distinguished.

Appendix

Table 20: How retained experiments map onto the stages of failure reporting. The final column states each result’s main limitation, avoiding an overly simple one-experiment-per-stage interpretation.

Figure 7: Natural continuation, the paragraph-break cue after the last attempt (Break), and the same seam with an explicit true verifier verdict at step 300. Neither intervention supplies report words. Break raises later admission and reduces between-run spread in the primary recipe; the NoBreak control in Table 22 shows that most of this tightening comes from the inserted paragraph cue rather than re-entry alone. Providing a verifier verdict of failure produces no additional tightening, although it raises mean admission in the group-local variant. Short horizontal ticks show means, and faint vertical lines show observed min–max ranges.

Figure 8: The same natural/paragraph-break/verdict-at-the-seam experiment at the step-150 companion horizon.

Experiment	Scope	Main result
Natural reach, 1.5B	40 checkpoint/formulation combinations	Median reach is 0.952; reach vs natural disclosure $\rho=0.098$ . The conditional-admission correlation is largely expected because reach is near one.
Nothing-inserted re-entry (NoBreak)	Same seam; no inserted characters	Admission rises slightly, but between-run spread changes little: the hierarchical contrast versus Natural is +0.02 to +0.21 and is detected in 2/6 cells.
Paragraph-break cue (F2R A / CTRL)	Same seam; two newlines; no report words	Tighter than NoBreak in all six point estimates; the contrast is detected in 3/6 cells on the hierarchical scale and all six on the raw scale. Most of the earlier localization effect comes from the learned paragraph cue.
Teacher-forced transition check	Same 60 checkpoints and failed trajectories	At step 300, variation in the break probability is not distinguished from variation in report entry after the break. The natural probability of the break is only about 0.00015–0.0008.
Verdict at the seam (F2R B / ORACLE)	Same seam; no report words	No consistent level gain and no additional tightening. One recipe variant has a +0.055 mean change.
Forced report prefix (F2R C / D)	Two words / first third of phrase	Admission rises by 0.141 and 0.167 over paragraph re-entry alone; 39/40 and 40/40 combinations increase.

Appendix

Table 21: Summary of the seam, paragraph-cue, report-prefix, and route-decomposition analyses. The no-inserted-characters control shows that most of the earlier localization effect comes from the learned paragraph-break cue rather than re-entry alone.

Cell	$\ln(\hat{\tau}_{\mathrm{NoBreak}}/\hat{\tau}_{\mathrm{Break}})$	$\ln(\hat{\tau}_{\mathrm{Natural}}/\hat{\tau}_{\mathrm{NoBreak}})$	Mean admission: Natural / NoBreak / Break
Primary @300	+0.45 [+0.07, +1.02]	+0.08 [ $-0.02$ , +0.20]	0.710 / 0.761 / 0.830
Primary @150	+0.38 [+0.11, +0.74]	+0.08 [+0.002, +0.18]	0.704 / 0.746 / 0.797
Failure-neutral @300	+0.26 [ $-0.36$ , +0.80]	+0.08 [ $-0.07$ , +0.14]	0.693 / 0.736 / 0.810
Failure-neutral @150	+0.64 [ $-0.08$ , +1.17]	+0.21 [+0.04, +0.30]	0.725 / 0.762 / 0.823
Group-local @300	+0.42 [ $-0.05$ , +1.20]	+0.10 [ $-0.15$ , +0.21]	0.643 / 0.676 / 0.737
Group-local @150	+1.16 [+0.90, +1.95]	+0.02 [ $-0.12$ , +0.09]	0.602 / 0.659 / 0.797

Appendix

Table 22: No-inserted-characters control. Positive values in the first contrast mean that Break is tighter than NoBreak; positive values in the second mean that NoBreak is tighter than Natural. Intervals are 95%.

Setting	Step	Log SD ratio [95% interval]	$\rho$ (break, natural disclosure)	Reading
Primary	300	+0.421 [ $-0.356$ , +1.254]	$-0.25$	Not distinguished
Failure-neutral	300	+0.614 [ $-0.036$ , +1.390]	$-0.24$	Not distinguished
Group-local	300	+0.526 [ $-0.196$ , +1.175]	+0.41	Not distinguished
Primary	150	+0.930 [+0.332, +1.441]	$-0.01$	Break probability is more variable, but does not track natural disclosure
Failure-neutral	150	+0.214 [ $-0.468$ , +0.663]	$-0.89$	Not distinguished
Group-local	150	+0.464 [ $-0.135$ , +1.012]	$-0.37$	Not distinguished

Appendix

Table 23: Teacher-forced comparison of the learned paragraph transition with the next report-entry step. Positive log SD ratios mean that the break probability varies more across retrainings than the total probability of a report opening after the break is supplied. The primary step-300 comparisons do not distinguish the stages.

Control	n	Step-300 range	Interpretation
Weakly constrained neutral phrase (P-D)	20	0.00–0.90	Large variation without failure-related meaning.
Explicit deterministic cue (P-E2)	20	0.993–1.000	19/20 runs are exactly 1.0 at step 300; same phrase and corpus as P-D.
Success-side pleasantry (P-A)	20	0.796–1.000	Stable success-side control; 13/20 runs are exactly 1.0 at step 300.
Kettle failure-conditioned neutral codeword	20	0.59–0.94	Pooled 20-run codeword control used in Figure 3 E.
Failure-branch inert filler (P-C)	20	0.31–0.90	Appendix-only structural boundary; does not causally isolate branch location.

Appendix

Table 24: Main neutral and pleasantry controls. The first four provide context for Figure 3 E; P-C is kept only as an appendix structural boundary.

Figure 9: Broader determination/susceptibility landscape. The main text focuses on the weakly constrained neutral-phrase versus explicit-cue comparison; the Kettle codeword/phrase is the failure-conditioned neutral-codeword row, while Roses is reported in Table 25 . Short horizontal ticks show means, and faint vertical lines show observed min–max ranges.

Formulation	n	Emission range	SD	$\hat{\tau}$ [95%]
Admission wording 1 (SEM1)	20	0.41–0.91	0.141	0.64 [0.40, 0.86]
Admission wording 2 (SEM2)	20	0.19–0.91	0.229	1.03 [0.66, 1.48]
Neutral codeword Roses	20	0.10–0.91	0.172	0.82 [0.52, 1.06]
Neutral codeword Kettle	20	0.59–0.94	0.088	0.47 [0.25, 0.64]

Appendix

Table 25: Full semantic and codeword/phrase formulation study. The mixed ordering is why we do not claim that semantic admissions are generally unstable while codewords are generally stable.

Intervention	Setting	n	Main result
Collateral-credit arithmetic	Countdown	517 mixed groups	Adverse sign in 517/517 groups; predicted exposure 0.554, observed 0.553.
Exact-span shelter (pilot)	Countdown	3 matched pairs	Mean disclosure change +0.194 [0.017, 0.372]; a small pilot, not an established general level effect.
Fresh shelter experiments	Countdown + shortest-path	15 / 20 / 20 pairs	No mean-level change detected (+0.052 / +0.045 / $-0.006$ ); no reduction in variability established.
Failure-neutral credit	Countdown	20 paired	No reduction in variability is detected after removing adverse failure credit (log SD ratio +0.165 [ $-0.277$ , +0.613]) and lowers solve by about 6 points.
Direct disclosure reward	Countdown	3 arms (1 seed)	Can create or recover reporting when the action remains reachable.
Code: reward alone	MBPP+	2 $\times$ 2	No failure reports matching the true test outcome once the reporting action disappears from sampling support.

Appendix

Table 26: Summary of the main credit-assignment and direct-reward interventions. Older variants that do not change the final interpretation are omitted here.

Setting	n	SD ratio	log SD ratio interval	Change in solve
Qwen-1.5B / Countdown	20	0.472	$-0.751$ [ $-1.420$ , $-0.225$ ]	$-0.060$ [ $-0.089$ , $-0.031$ ]
Qwen-1.5B / shortest-path	20	0.30	$-1.21$ [ $-1.74$ , $-0.76$ ]	$-0.083$ [ $-0.126$ , $-0.040$ ]
Qwen-7B / Countdown	19	0.32	$-1.14$ [ $-1.55$ , $-0.84$ ]	+0.069 [+0.012, +0.126]

Appendix

Table 27: Main reference-anchoring experiments. SD ratios below one mean that failure disclosure varies less across retrainings under anchoring. Intervals are 97.5% for the Countdown and 7B rows and 95% for the shortest-path row, matching each experiment’s pre-specified level.

Setting	$n$	Mean disclosure (anchor / no anchor)	Raw SD ratio	Mean-adjusted SD ratio
7B, primary detector	19	0.80 / 0.32	0.32 [0.23, 0.42]	0.24 [0.16, 0.38]
7B, broad detector	19	0.84 / 0.52	0.30 [0.23, 0.39]	0.40 [0.29, 0.58]
1.5B, failure-conditional anchor	20	0.68 / 0.63	0.47 [0.26, 0.75]	0.42 [0.22, 0.64]
1.5B, all-token anchor	20	0.69 / 0.63	0.36 [0.25, 0.54]	0.33 [0.20, 0.51]

Appendix

Table 28: Mean-adjusted anchoring check. Each ratio compares the between-run spread under anchoring with the spread without anchoring; values below one mean that disclosure is more consistent under anchoring. The mean-adjusted estimate uses a hierarchical binomial model to account for the compression of rates near zero and one. Mean disclosure is shown as anchored / unanchored, and entries in brackets are 95% intervals.

Anchor scope	Disclosure mean (SD)	Solve SD	SD ratio vs sham	Change in solve
None (matched comparator)	0.626 (0.1894)	0.0478	1 (reference)	n/a
Failure-conditional	0.684 (0.0893)	0.0294	0.472 [0.242, 0.799]	$-0.0597$ [ $-0.0888$ , $-0.0306$ ]
All-token	0.687 (0.0688)	0.0285	0.363 [0.230, 0.582]	$-0.0835$ [ $-0.1092$ , $-0.0579$ ]

Appendix

Table 29: Failure-conditional versus all-token anchoring in the 1.5B Countdown setting. Their reductions in disclosure variability are not distinguished, while a direct matched-seed comparison shows a larger capability cost for all-token anchoring (a descriptive companion, +0.024 [0.012, 0.036], not a preregistered test).

Condition	Checkpoint	Disclosure SD	Role in comparison
Unanchored	300	0.189	Same training horizon as anchoring
Unanchored	150	0.139	Closest mean solve to anchored step 300
Failure-conditional anchor	300	0.089	Intervention result

Appendix

Table 30: Progress-matched anchoring check for 1.5B Countdown. Slower capability progress explains part of the same-step narrowing, but disclosure remains less variable under anchoring than at the matched unanchored checkpoint.

Figure 10: Small reference-anchor coefficient sweep. Narrowing is established at

\beta=0.02

but not at

\beta=0.01

; we do not treat this as a fitted dose-response law.

Run set	n	Log SD ratio [97.5% interval]	SD: anchor / no anchor	Change in solve
Original fixed pairs	11	$-1.36$ [ $-3.13$ , $-0.36$ ]	0.062 / 0.242	+0.062 [ $-0.021$ , +0.145]
Frozen extension	8	$-1.01$ [ $-1.75$ , $-0.32$ ]	0.111 / 0.305	+0.080 [ $-0.020$ , +0.179]
Pooled, wave-stratified	19	$-1.14$ [ $-1.55$ , $-0.84$ ]	0.090 / 0.281	+0.069 [+0.012, +0.126]

Appendix

Table 31: 7B anchoring extension and pooled result. The extension was frozen after the original readout and reproduces the tightening itself. The pooled interval uses a wave-stratified bootstrap.

Analysis	n	Log SD ratio or deletion range	Interval stability
Original pairs	11	$-1.36$ [ $-3.13$ , $-0.36$ ]	Four of 11 deletion intervals cover zero.
Extension alone	8	$-1.01$ [ $-1.75$ , $-0.32$ ]	Four of eight deletion intervals cover zero.
Pooled pairs	19	$-1.14$ [ $-1.55$ , $-0.84$ ]	Full interval excludes zero.
Pooled leave-one-out	19 deletions	Point estimates $-1.26$ to $-1.06$	All 19 deletion intervals exclude zero.

Appendix

Table 32: Leave-one-run-out sensitivity for the 7B anchoring result. The original and extension-only results are each based on few pairs, while the pooled

n=19

estimate excludes zero after every one-run deletion.

Design	Parents	SD ratio	log SD ratio interval	Interpretation
Anchored-history continuation (KBR): keep anchor on vs switch off	10	0.47	$-0.75$ [ $-1.22$ , $-0.25$ ]	8/10 parents show lower variability with anchoring on
Post-history anchoring (FORK): introduce anchor after unanchored history	8	0.60	$-0.51$ [ $-1.10$ , +0.09]	Reduced variability is not statistically established

Appendix

Table 33: Mid-RL continuation tests of anchoring. KBR shows reduced later-RL divergence after an anchored history. FORK does not establish narrowing on the raw scale when anchoring is introduced only after an unanchored history.

Figure 11: All KBR continuation runs across ten mid-RL parent states with anchored histories. Each parent has four futures with anchoring switched off and four with anchoring kept on.

Figure 12: FORK variability after an unanchored training history. Starting from each of eight exact mid-RL parent states, we run four matched future continuations without anchoring and four with anchoring introduced at the fork. Each dot is the within-parent SD of failure disclosure across those four continuations; lower values therefore indicate less branching across otherwise matched futures.

Anchor region	log SD ratio vs sham	Change in solve	Active-token exposure
Full failure-conditional	$-0.7514$ [ $-1.4200$ , $-0.2250$ ]	$-0.0597$ [ $-0.0888$ , $-0.0306$ ]	58.9%
Failed reasoning	$-0.7109$ [ $-1.2055$ , $-0.1834$ ]	$-0.0643$ [ $-0.0859$ , $-0.0427$ ]	53.6%
Post-attempt tail	$-0.0468$ [ $-0.6499$ , +0.6019]	$-0.0331$ [ $-0.0731$ , +0.0069]	4.3%
16-token seam	$-0.0438$ [ $-0.5230$ , +0.5503]	$-0.0038$ [ $-0.0345$ , +0.0270]	1.7%

Appendix

Table 34: Where the reference constraint acts. Anchoring only failed reasoning recovers the full stabilization effect and capability cost. Short tail and seam masks do not establish narrowing and cover far fewer tokens. Intervals are 97.5% for the full anchor and 98.33% (a three-arm family) for the failed-reasoning, tail, and seam rows.

States	$\lVert g_{D}\rVert/\lVert g_{\mathrm{full}}\rVert$	$\cos(g_{D},g_{\mathrm{full}})$	Tail norm ratio	Seam norm ratio
Anchored @ 150	0.960	0.983	0.173	0.132
Anchored @ 300	0.964	0.984	0.172	0.137
Sham @ 150	0.975	0.993	0.110	0.095
Sham @ 300	0.958	0.989	0.139	0.103

Appendix

Table 35: Gradient and drift localization analysis (LOCKLAN). The failed-reasoning-only gradient is almost as large as the full-anchor gradient and points in nearly the same direction, while tail and seam gradients are much smaller.

Setting @300	Pre-seam drift mass	Seam mass	Other tail mass	Total drift
1.5B / KL-off	0.934	0.037	0.029	0.119
1.5B / Sham	0.934	0.038	0.028	0.119
1.5B / Anchor	0.947	0.029	0.024	0.031
7B / outcome-only	0.923	0.058	0.019	0.199
7B / Anchor	0.953	0.025	0.023	0.047

Appendix

Table 36: Teacher-forced drift from the reference policy at step 300. About 92–95% of the measured drift lies before the reporting seam in the reported 1.5B and 7B settings.

Question	Evidence	Main takeaway
Role of reference KL	KL-free setup + all-token and failure-conditional anchoring	The phenomenon is measured in a deliberate KL-free regime; restoring reference regularization narrows both readouts, especially disclosure.
Generic RL variability	Primary + OLMo + shortest-path selective-dispersion comparisons	Variability depends on the behavior: disclosure is much less reproducible than solve in the clean experiments.
Capability differences	Clean 1.5B/OLMo experiments; 7B boundary	Capability is tightly reproduced in the clean experiments; 7B is explicitly treated as a broader persistence boundary.
Detector sensitivity	Human audits + detector-tier sensitivity	Human agreement is high, randomized audit frames are included, and broader detectors preserve the qualitative spread.
Probe selection	Full dev-100 disclosure + selected-20 versus other-80 split	Broad spread remains on the 80 prompts outside the behavior-selected probe; selection does not explain the result.
Fixed-horizon interpretation	Nine-rung trajectory + small longer-horizon cases	The claim is reproducibility at a fixed budget, not separate stable endpoints; longer outcomes are mixed ( J.1 ).

Appendix

Table 37: Summary of robustness and scope evidence.

Setting	$n$	Last step	Disclosure SD, 300 $\rightarrow$ last	Solve SD, 300 $\rightarrow$ last	Last-step disclosure
Qwen2.5-1.5B, outcome only	3	600	0.273 $\rightarrow$ 0.023 ( $D_{v1}$ )	0.064 $\rightarrow$ 0.131	0.039 / 0.000 / 0.000
Qwen2.5-1.5B, length penalty	3 ∗	600	0.092 $\rightarrow$ 0.148 ( $D_{v2}$ )	0.050 $\rightarrow$ 0.217	0.404 / 0.651 / 0.667
Qwen2.5-7B, outcome only	5	600	0.306 $\rightarrow$ 0.396 ( $D_{v1}$ )	0.075 $\rightarrow$ 0.201	0.000–0.918
OLMo-2-7B, outcome only	3	1200	0.099 $\rightarrow$ 0.169 $\rightarrow$ 0.442 ( $D_{v5}$ )	0.081 $\rightarrow$ 0.163 $\rightarrow$ 0.030	0.873 / 0.000 / 0.555
Gemma-2-9B, outcome only	3	600	0.221 $\rightarrow$ 0.115 ( $D_{v5}$ )	0.105 $\rightarrow$ 0.244	0.666 / 0.886 / 0.833 †
Distilled Qwen-1.5B, two recipes	2+2 ‡	600	0.049 $\rightarrow$ 0.078 ( $D_{v2}$ )	0.055 $\rightarrow$ 0.013	0.797–0.961

Appendix

Table 38: Small multi-run cases trained beyond step 300. SD is calculated across runs using the detector named in each row. These values are descriptive because every case has

n\leq 5

and the instruments differ across model families.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 757 papers

Oct 14, 2026

Training Language Models to Fail Gracefully

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

Daman Arora, Xuecheng Liu, Himanshu Gaurav Singh +6

Carnegie Mellon University · UC Berkeley

RL for Language Models LLM Abstention RL Post-Training

Ask PDF

Jun 16, 2026

Self-CTRL: Self-Consistency Training with Reinforcement Learning

Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from

R^2=0.24

R^2=0.64

on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from

36\%

92\%

. In the other direction, behavior updates improve alignment, reducing HarmBench failure rate from

15.0\%

0.5\%

without substantially increasing refusal on harmless prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.

Itamar Pres, Laura Ruis, Melat Ghebreselassie +2

MIT CSAIL

RL for Language Models Faithfulness of Language Model Explanations LLM Auditing LLM Interpretability Language Model Self-Assessment LLM Safety Alignment+2

Ask PDF

Oct 14, 2026

Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight

When every rollout for a problem fails, GRPO receives neither a successful exemplar nor a nonzero group-relative advantage, so it learns nothing from exactly the problems it most needs to learn. The failed reasoning can nevertheless suggest what to try next. We present Replay-Selected Hindsight Distillation (RSHD), which turns failed attempts into an additional, tested source of supervision during GRPO training. The policy writes a corrective note about each failed attempt without seeing a reference answer or solution, then re-solves the problem with the note in context. A note is admitted only if its replays succeed more often than the original group did, after screening for notes that state the answer, and an admitted replay is distilled into the plain-prompt policy. Because admission is decided by fresh successes rather than by the failed rollout itself, the procedure can recover supervision from groups in which every attempt failed, and it applies equally to failures in mixed groups. Only the policy is trained, the only judge is the outcome verifier, and deployment requires no notes or extra inference. Across model families and sizes and two training domains, RSHD consistently improves over GRPO on in-domain benchmarks, and the gains carry over to out-of-domain evaluation sets.

Sheldon Yu, Tong Yu, Nikki Lijing Kuang +6

University of California, San Diego · Adobe Research · University of Virginia +2

RL for Language Model Reasoning RL Post-Training Group Relative Policy Optimization

Ask PDF

Abstract

LLM Reliability

Explore similar work

Training Language Models to Fail Gracefully

Self-CTRL: Self-Consistency Training with Reinforcement Learning

Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight