cs.LGSep 29, 2026

RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

Authors: Michael Kirchhof, Eleonora Gualdoni, Andrew Szot, Khashayar Gatmiry, Aryo Lotfi, Abbas Kazerouni, Omar Attia, Sanjoy Chowdhury, +1 more

Organizations: Apple

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.

Figures & tables

Figure 1 : RLTL;DR rolls out attempts sequentially. When an attempt fails, we let the policy generate its own feedback based on the verifier outcomes, in the form of a high-level TL;DR insight. This insight is fed into next attempts to search for solutions more efficiently, especially on very hard tasks. When the running average success rate of the task is high enough, like in the 6th and 8th rollout, we attempt the task without insights. Besides a GRPO loss on the rollouts, we backpropagate the insights (the bold words on the right) into the model via an SFT loss. Though never predicted at test time, this internalizes and generalizes these higher-level learnings.
 SAPIAppworldLeetcode
GRPO57.8%33.7%55.1%
SGE57.7%33.5%56.4%
RLTF-SD53.0%38.9%42.2%
RLTL;DR91.1%61.7%49.1%
Table 1: Pass@1 on heldout eval sets after training on the Pass@128=0 splits. RLTL;DR’s internalized insights generalize, to unseen and also easier tasks. These numbers should not be cited as benchmark scores as their train data is only a subset of tasks and includes Appworld test-normal.
 Forward TokensBackward TokensTrain Pass@1Eval Pass@1
RLTL;DR\penalty 720\penalty\ 720 M1212 M21.5%18.9%
RLTL;DR, λ=0.01\lambda=0.01\penalty 720\penalty\ 720 M1212 M9.3%14.0%
RLTL;DR, λ=0\lambda=0\penalty 720\penalty\ 720 M1111 M6.1%13.2%
RLTL;DR, no GRPO loss, only LSFT\mathcal{L}_{\text{SFT}}9.29.2 M839839 k20.0%17.0%
SFT on all 3.5k full rollouts400400 M6.66.6 M28.0%20.9%
SFT on 1k rollouts107107 M1.91.9 M27.1%21.0%
Table 2 : We find that LSFT\mathcal{L}_{\text{SFT}} drives most performance, on the SAPI frontier difficulty split separated into train and heldout eval tasks. The first four runs are RLTL;DR ablations, showing that the SFT loss on insights is the driving factor. The next are normal SFT train runs on rollouts collected in the first RLTL;DR run. Most rollouts have insight in context, but it is not backpropagated on. The last group of experiments is the simplified SFTL;DR training that only backpropagates on the insights, not the rollouts. Forward tokens and backward tokens concern the logits that need to be computed in the update phase. Rollout collection is another 1B tokens. All results are ±0.6%\pm 0.6\% standard deviation.
 Settingnfbn_{\text{fb}}Train Pass@1Eval Pass@1
 Default (TL;DR format, self-generated with thinking)1621.4%19.2%
Insight contentDiagnostic paragraph1620.3%18.6%
Summary + Diagnostic paragraph1613.8%16.0%
Summary + Diagnostic paragraph + Corrected code1615.7%17.3%
Insight generatorStudent, non-thinking1620.1%19.8%
Student, no information about failed unit tests (only success)160 3.4%10.0%
Table 3 : Pass@1 (without insight in context) after training with different forms of insight. nfbn_{\text{fb}} is the maximum number of insights placed in the context.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

 Appworld/SAPILeetcode
Rollout collection phase
Unique tasks per rollout collection phase8128
Min attempts per task88
Avg attempts per task (due to async collection)2126
Max steps per attempt501
Max think tokens per step7680
Appendix
Table 4 : Hyperparameters used for train runs.
ShareCorrectionExamples of insight
15.3%Wrong capitalization• Use the uppercase Orange color enum instead of lowercase orange when updating the list color. • Use uppercase PM instead of lowercase pm when setting the alarm’s AM/PM marker. • Use the Priority enum value instead of a string when setting the reminder priority.
11.8%Unrecorded action• You need to mark the article as read rather than just viewing it. • Use the unified search API instead of the ticker search to create a history record that contains the query text. • You need to create a download event for the mobile listing instead of using the generic task completion function.
11.2%Extra parameter• Call complete_task without passing an answer parameter for non-QA tasks.
5.5%Unsaved setting• You need to set the default chart horizon preference instead of just viewing a chart. • The list sorting preference needs to be saved to the database, not just the sort configuration updated.
2.1%Over-filtering• Remove the route_type filter to see all hiking trails instead of getting no results.
1.2%Argument position• Pass the access token as the first positional argument, not as a keyword argument.
Appendix
Table 5 : Example insight provided by the verifier. Groups of insights pointing to the same correction are identified by lexical overlap.
Failure type: answer argument passed to a non-QA task
Diagnostic paragraph“The task failed because the agent called complete_task() with an answer parameter, but non-QA tasks should not include an answer. The verifier only checks that the BPM filter was set to 125–125, the playlist search results were recorded, and the embed code exists in the database — not the submission method.”
Corrected codeapis.supervisor.complete_task()
TL;DR insight“Remove the answer parameter from the complete task call.”
Failure type: item not persisted to saved for later
Diagnostic paragraph“The rollout failed because the product was not properly persisted to the saved_for_later collection as verified by the database. The agent called save_for_later with only product_id and note , but did not pass the access_token parameter which is required for authentication. The verifier checks the database directly via show_saved_for_later(db) , not API responses.”
Corrected codeapis.shop.save_for_later(product_id=1,
Appendix
Table 6 : The three feedback components on three failure types. Rows are building blocks, not separate conditions: the Summary + diagnostic paragraph setting replaces the insight with the paragraph, and Summary + diagnostic paragraph + corrected code appends the code to it (summaries omitted for space). Specificity escalates from a transferable rule to episode-bound detail: the hint names an action, the paragraph names the verifier’s internal check, and the code carries concrete identifiers such as product_id=1 and, in the last row, an access token valid only for that episode.
MethodTrain loop∥Δθ∥1\|\Delta_{\theta}\|_{1}∥Δθ∥2\|\Delta_{\theta}\|_{2}∥Δθ∥1>\|\Delta_{\theta}\|_{1}> bf16 precisionGini( ∣Δθ∣|\Delta_{\theta}| )top-1% share of Δθ2\Delta_{\theta}^{2}
RLTL;DRRL7.4 ⋅103\cdot 10^{3}0.792.11%0.9996.88%
RLTL;DR, λ=0.01\lambda=0.01RL3.2 ⋅103\cdot 10^{3}0.441.58%0.9999.66%
RLTL;DR, λ=0\lambda=0RL2.4 ⋅103\cdot 10^{3}0.341.58%0.9999.67%
RLTL;DR, no GRPO loss, only LSFT\mathcal{L}_{\text{SFT}}RL9.3 ⋅103\cdot 10^{3}0.892.51%0.9994.07%
SFT on all 3.5k full rolloutsSFT1.3 ⋅106\cdot 10^{6}36.1230.42%0.8735.47%
SFT on 1k rolloutsSFT0.8 ⋅106\cdot 10^{6}22.7224.86%0.8836.33%
Appendix
Table 7 : Difference of trained checkpoints to original Qwen 3.5 9B checkpoint in parameter space. The first three metrics capture the magnitude of the update, the latter two the concentration (higher = more concentrated on a small set of parameters). Comparisons should only be made within RL runs and within SFT runs, since RL at its bfloat16 precision drops many small parameter updates.
SettingNo-insight success (%)
RLTF-FM (insight generator SFT, all insight)6.2
Self-play insight SFT (insight generator SFT, effective insight only) 1 1 1 This run’s training crashed at update 75, so its value is averaged over the 75 available updates rather than the full 80.7.7
RLTL;DR (ours)14.8
RLTL;DR + RLTF-FM15.0
RLTL;DR + Self-play insight SFT15.7
Appendix
Table 8 : Training success rate on rollouts without insight in context (as in Table 3 ), averaged over the first 80 gradient updates, on the SAPI-challenging split. Neither way of training the insight generator alone recovers RLTL;DR’s actor-side LSFT\mathcal{L}_{\text{SFT}} signal; added on top of RLTL;DR, RLTF-FM gives little further gain, while Self-play insight SFT gives a more noticeable one.
AblationEval Pass@1
Insert insights if ≤0.1\leq 0.1 running success rate, λ=0.1\lambda=0.114.2%
Insert insights if ≤0.33\leq 0.33 running success rate, λ=0.1\lambda=0.115.3%
Insert insights if ≤0.5\leq 0.5 running success rate, λ=0.1\lambda=0.117.1%
Insert insights if ≤\leq 3 of first 10 rollouts successful, λ=0.1\lambda=0.119.2%
Insert if ≤0.5\leq 0.5 running success rate, λ=0.5\lambda=0.519.6%
Split Advantages18.9%
Appendix
Table 9: Ablations of the RLTL;DR loss. Trained on the 386 SAPI hard tasks, evaluated on the remaining 256 SAPI hard tasks.

Explore similar work

CardsList
Jul 6, 2026cs.LG

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.

Muhammad Zain Amin, Kibele Sebnem Yildirim

Department of Software Engineering and IT, ´Ecole de Technologie Sup´erieure, Montreal, Canada · Centre int´egr´e de traumatologie, Hˆopital du Sacr´e-Cœur de Montr´eal, Universit´e de Montr´eal, Montreal, Canada · Desautels Faculty of Management, McGill University, Montreal, Canada.

Reinforcement Learning With Verifiable RewardOffline Reinforcement LearningUnsupervised On-Policy Self-DistillationSelf-EvaluationEpisodic MemoryInference-Time
AskPDF
Jul 21, 2026cs.LG

H2^2SD: Hybrid Hindsight Self-Distillation

Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but a scalar trajectory reward offers limited token-level guidance. Existing self-distillation methods add a privileged teacher but typically assign it a fixed role: direct distribution matching may destabilize successful behavior, while magnitude-only modulation offers little corrective guidance after failure. We observe that successful and failed trajectories require different forms of hindsight supervision. A successful response already contains a valid student-generated reasoning path and can therefore serve as privileged context rather than being replaced by an external rationale. A failed response, however, requires corrective reference information. We introduce Hybrid Hindsight Self-Distillation (H2SD\mathrm{H}^{2}\mathrm{SD}), which jointly adapts teacher context and update strategy to trajectory correctness. For successful trajectories, we construct the teacher context from the verified response and a rephrasing instruction, and use the teacher only to re-evaluate the original response tokens. The resulting probabilities refine token credit assignment without changing the direction determined by the reward. For failed trajectories, a verified reference hint provides corrective guidance through reverse-KL distillation. Experiments on challenging reasoning benchmarks show that H2^2SD achieves the strongest overall performance among representative RLVR and self-distillation baselines, with stable optimization and a favorable accuracy-efficiency trade-off.

Qiye Cai, Yichuan Ma, Peiji Li +6

1Shanghai Artificial Intelligence Laboratory · 2Harbin Institute of Technology · 3Fudan University +1

Reinforcement Learning With Verifiable RewardHindsightReasoning Paths
AskPDF
Apr 1, 2026cs.LG

Policy Improvement Reinforcement Learning

Reinforcement learning has become a central post-training paradigm for improving LLM and agent capabilities. Yet existing RL post-training methods share a common blind spot: they construct local learning signals from sampled trajectories, rewards, or feedback-conditioned targets, then update the policy without explicitly verifying whether the resulting policy outperforms its predecessor. Optimizing these local signals does not necessarily produce a better policy, while finite sampling, generation stochasticity and feedback noise can further widen this gap. We argue that the missing ingredient is policy improvement feedback: the ability to measure progress across policy iterations. We introduce Policy Improvement Reinforcement Learning (PIRL), which formulates inter-iteration performance gain as an explicit objective structurally aligned with final task performance. Building on PIRL, we propose Policy Improvement Policy Optimization (PIPO), a plug-in closed-loop framework that verifies the previous update against a sliding-window historical performance anchor. PIPO uses this improvement feedback to modulate the local learning signal of the base policy optimization algorithm, reinforcing updates associated with measured progress and suppressing those associated with performance drops. We provide theoretical evidence that PIPO locally aligns policy updates with the PIRL improvement objective. Experiments on mathematical reasoning, code, tool-use, and self-distillation settings show that PIPO yields consistent gains across PPO, group-relative, and self-distillation policy optimization families.

Huaiyang Wang, Xiaojie Li, Xiaohan Wang +10

Beihang University · Meituan · Peking University

Reinforcement Learning Post-TrainingDistributional Reinforcement Learning
AskPDF