cs.AIFeb 17, 2026

Improving Interactive In-Context Learning from Natural Language Feedback

Authors: Martin Klissarov, Jonathan Cook, Diego Antognini, Hao Sun, Jingling Li, Natasha Jaques, Claudiu Musat, Edward Grefenstette

Organizations: Google DeepMind

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adapting one's thought process based on corrective feedback is an essential ability in human learning, particularly in collaborative settings. In contrast, the current large language model training paradigm relies heavily on modeling vast, static corpora. While effective for knowledge acquisition, it overlooks the interactive feedback loops essential for models to adapt dynamically to their context. In this work, we propose a framework that treats this interactive in-context learning ability not as an emergent property, but as a distinct, trainable skill. We introduce a scalable method that transforms single-turn verifiable tasks into multi-turn didactic interactions driven by information asymmetry. We first show that current flagship models struggle to integrate corrective feedback on hard reasoning tasks. We then demonstrate that models trained with our approach dramatically improve the ability to interactively learn from language feedback. More specifically, the multi-turn performance of a smaller model nearly reaches that of a model an order of magnitude larger. We also observe robust out-of-distribution generalization: interactive training on math problems transfers to diverse domains like coding, puzzles and maze navigation. Our qualitative analysis suggests that this improvement is due to an enhanced in-context plasticity. Finally, we show that this paradigm offers a unified path to self-improvement. By training the model to predict the teacher's critiques, effectively modeling the feedback environment, we convert this external signal into an internal capability, allowing the model to self-correct even without a teacher.

Figures & tables

Figure 1: Trained through RL on multi-turn didactic interactions, Gemini 2.5 Flash nearly reaches the performance of Gemini 2.5 Pro on the challenging HardMath2 dataset. Single-turn RL only slightly improves on the baseline’s ability to in-context learn from language feedback across turns.
Figure 2: (Top) Didactic Interactions via Information Asymmetry: We transform single-turn problems into multi-turn didactic interactions. A teacher model, conditioned on privileged information (e.g., the ground-truth solution), provides natural language feedback to a student model without revealing the final answer, guiding it to correct its errors. (Middle) Train-time RL Fine-tuning: We train the student model to effectively incorporate language feedback using RL. The student iterates through multiple turns; if the answer is correct (checked via automatic verification), a reward ( R=+1R=+1 ) is granted and the interaction ends. If it is incorrect, the teacher provides feedback. If the interaction reaches the maximum number of turns, TmaxT_{max} turns, the reward is zero. (Bottom) Inference-time Evaluations: We assess the trained model in three settings: (1) Learning from language feedback (interacting with an external source of language feedback), (2) General Multi-turn Tasks (out-of-domain tasks like logic puzzles or games), and (3) In-context Self-Improvement , where the model plays the role of both student and teacher to self-correct.
 (5p+e)−(3p+e)=1.82−1.24⇒2p=0.58⇒p=0.29(5p+e)-(3p+e)=1.82-1.24\Rightarrow 2p=0.58\Rightarrow p=0.29 
Figure 3: Comparison of the interactive ability between two leading closed-source models. The user prompts both models to "be concise" after an initial lengthy explanation. Model B (Right) correctly interprets this as a request to summarize the previous explanation. Model A (Left) interprets this as a setting change for future turns and acknowledges the instruction without actually rewriting the content, failing to satisfy the user’s obvious intent.
Figure 4: Cumulative accuracy across four hard reasoning domains: HardMath2, ARC-AGI, BBEH, and Codeforces. The plots illustrate the limited ability of the Gemini 2.5 models and GPT-5 to incorporate language feedback over multiple turns. We observe this ability scales with model size, with GPT-5 generally exhibiting stronger performance gains compared to the Gemini 2.5 Pro model.
Figure 5: RL 2 F on multi-turn didactic interactions is key to improve the interactive in-context learning abilities of LLMs . The improved performance additionally transfers significantly better to out-of-distribution tasks than single-turn RL.
TasksRL 2 FSingle turn RLGemini Flash 2.5
Multi-turn ARC AGI23.5620.4720.47
Multi-turn Codeforces37.0332.7733.33
Multi-turn Linguini5642.3542
Average Performance38.8631.8631.93
Table 1: Gemini 2.5 Flash, trained solely on math, is evaluated on three out-of-domain multi-turn benchmarks: ARC AGI, Codeforces, and Linguini. This experiment evaluates the ability of the trained models to in-context learn from language feedback by interacting with a teacher model at inference time . RL 2 F provides the strongest gains with an average of 7%7\% increase in performance across tasks.
 ∂tu+u2∂xu=∂xxu−(x−t)−2\partial_{t}u+u^{2}\partial_{x}u=\partial_{xx}u-(x-t)^{-2} 
Figure 6: Conversation comparison between the baseline, the Multi-Turn RL fine-tuned model, and the 2.5 Flash model. The RL 2 F fine-tuned model exhibits greater in-context plasticity, exhibiting greater responsiveness to the teacher feedback and attempting to integrate it . In comparison, the baseline simply disregards the feedback, constantly repeating the same initial solution. In the last turn, the base model simply gives up on thinking and directly outputs the same answer.
Multi-turn TasksRL 2 FSingle turn RLGemini 2.5 Flash
Aider Polyglot Code Edit18.6712.1521.33
Circuit Decoding26.4231.8517.92
Eliciting User Preferences32.5027.4029.32
Maze Navigation87.5078.3575.00
Movie Recommendations69.6168.2269.34
Only Connect Wall72.0044.7553.00
Table 2: We evaluate on 10 diverse out-of-distribution general multi-turn environments, ranging from game playing to spatial reasoning . In these tasks, the trained models do not interact with a teacher: they simply act within the agentic environment defined by the tasks. Training Gemini 2.5 Flash through RL 2 F on math shows strong out-of-distribution generalization on multi-turn agentic tasks , outperforming baselines on 7 out of 10 tasks and achieving 5% average performance boost.
Figure 7: Training models to learn from multi-turn didactic interactions significantly improves their self-improvement capabilities at inference time. Key to this result is for the student to predict the teacher’s feedback. In other words, adopting a world-modeling approach within didactic interactions leads the student model to become an autodidact.
More in this topic

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Explore similar work

CardsList
Feb 18, 2026

Learning to Learn from Language Feedback with Social Meta-Learning

Large language models (LLMs) often struggle to learn from corrective feedback within a conversational context. They are rarely proactive in soliciting this feedback, even when faced with ambiguity, which can make their dialogues feel static, one-sided, and lacking the adaptive qualities of human conversation. To address these limitations, we draw inspiration from social meta-learning (SML) in humans - the process of learning how to learn from others. We formulate SML as a finetuning methodology, training LLMs to solicit and learn from language feedback in simulated pedagogical dialogues, where static tasks are converted into interactive social learning problems. SML effectively teaches models to use conversation to solve problems they are unable to solve in a single turn. This capability generalises across domains; SML on math problems produces models that better use feedback to solve coding problems and vice versa. Furthermore, despite being trained only on fully-specified problems, these models are better able to solve underspecified tasks where critical information is revealed over multiple turns. When faced with this ambiguity, SML-trained models make fewer premature answer attempts and are more likely to ask for the information they need. This work presents a scalable approach to developing AI systems that effectively learn from language feedback.

Jonathan Cook, Diego Antognini, Martin Klissarov +2

Google DeepMind

Meta-LearningLLM Fine-Tuning
AskPDF
Jun 29, 2026

What Drives Interactive Improvement from Feedback?

We study when natural-language feedback produces improvement beyond the gains obtainable from repeated attempts alone. In multi-turn language agent setting, higher final accuracy can reflect useful feedback, but it can also arise from resampling, format correction, or additional test-time computation. To separate these effects, we introduce a controlled student-teacher protocol across Omni-MATH, Codeforces, BBEH Linguini, and ARC-AGI1, evaluating thirteen open-weight models in both student and teacher roles. We compare external feedback, self-feedback, and unguided self-refinement, while varying interaction history, task difficulty, and teacher access to privileged task information. Across settings, we find that multi-turn improvement is often not evidence of feedback use: self-generated feedback adds little beyond unguided self-refinement, whereas the strongest external teachers produce substantially larger feedback-specific gains, suggesting that useful feedback must provide guidance beyond generic retry. Dense student-teacher interaction matrices further show that interactive gains are driven more by the student's ability to use feedback than by the teacher's identity, although teacher choice remains important for a fixed student. These results suggest that feedback-based agents should be evaluated against repeated-attempt baselines, and that ability to act on feedback, not merely feedback availability, is a central bottleneck for interactive improvement. We release our controlled student-teacher evaluation framework at https://j-lojek.github.io/feedback-generation-is-a-bottleneck/.

Bartłomiej Cupiał, Jan Łojek, Mikołaj Garstecki +3

University of Warsaw · AKCES NCBR · Princeton University +2

LLM Self-RefinementLLM Agent EvaluationAI Agent BenchmarksLanguage Model Self-Improvement
AskPDF
Mar 5, 2026

Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction

While LLMs demonstrate strong reasoning capabilities when provided with full information in a single turn, they exhibit substantial vulnerability in multi-turn interactions. Specifically, when information is revealed incrementally or requires updates, models frequently fail to integrate new constraints, leading to a collapse in performance compared to their single-turn baselines. We term the root cause as Contextual Inertia: a phenomenon where models rigidly adhere to previous reasoning traces. Even when users explicitly provide corrections or new data in later turns, the model ignores them, preferring to maintain consistency with its previous (incorrect) reasoning path. To address this, we introduce Reinforcement Learning with Single-Turn Anchors (RLSTA), a generalizable training approach designed to stabilize multi-turn interaction across diverse scenarios and domains. RLSTA leverages the model's superior single-turn capabilities as stable internal anchors to provide reward signals. By aligning multi-turn responses with these anchors, RLSTA empowers models to break contextual inertia and self-calibrate their reasoning based on the latest information. Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods. Notably, our method exhibits strong cross-domain generalization (e.g., math to code) and proves effective even without external verifiers, highlighting its potential for general-domain applications. Code is available at https://github.com/Tencent/RLSTA.

Xingwu Chen, Zhanqiu Zhang, Yiwen Guo +1

School of Computing and Data Science, The University of Hong Kong · LIGHTSPEED · Independent Researcher

Multi-Turn Language Model ReasoningRL for Language Model Reasoning
AskPDF