TextualRL: Textual Reinforcement Learning for Context Management

Tianyu Huai; Jie Zhou; Bo Mao; Qin Chen; Liang He; Xipeng Qiu

cs.CLOct 13, 2026

TextualRL: Textual Reinforcement Learning for Context Management

Authors: Tianyu Huai, Jie Zhou, Bo Mao, Qin Chen, Liang He, Xipeng Qiu

Organizations: School of Computer Science and Technology, East China Normal University, Shanghai · Shanghai Innovation Institute · Fudan University

Abstract

LLM agents can adapt to task feedback by revising the textual context that guides their behavior, without changing their model parameters. However, turning execution rewards into reusable context updates is challenging: individual trajectories offer limited comparative evidence about what to change, while task-specific fixes may overgeneralize when incorporated into a shared context. We introduce TextualRL, a gradient-free reinforcement learning framework that treats a group of rollouts from the same task under the same context, rather than an isolated trajectory, as the unit of policy critique. Groupwise Policy Critique contrasts successful and failed rollouts within a task; when outcomes are uniform, it draws on cross-task evidence to identify behaviors worth preserving or formulate repair hypotheses. Cross-Group Policy Update reviews the resulting proposals against evidence pooled across groups, including observations that yield no proposed edit, to refine their content and applicability. Reviewed proposals form a bounded textual patch, with adoption of the candidate context determined by held-out validation rewards. Together, these components establish a reward-driven policy optimization loop in which multiple attempts jointly inform reusable context updates. We evaluate TextualRL with five target agents on six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the base model by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B, respectively.

Figures & tables

Figure 1: Conceptual comparison of parameter-space and context-space policy optimization. TextualRL uses groups of same-task rollouts and their rewards to propose textual edits

\Delta C

to context

C

, while the target model remains frozen.

Figure 2: Overview of TextualRL. (a) Groupwise Policy Critique preserves same-task rollout groups and uses within-task contrast for mixed outcomes, cross-task preservation for all-success outcomes, and cross-task repair for all-failure outcomes. Parallel critics return edit proposals and evidence cards separately. (b) Cross-Group Policy Update reviews proposals against pooled evidence, constructs a bounded patch, and selects the candidate context through held-out validation. Within-epoch feedback informs subsequent critique calls.

Model	Skill Source	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld
GPT-5.5	Base Model	77.7	41.8	33.1	78.8	37.6	83.6
	Human skill	81.8	72.9	66.9	90.1	38.4	91.8
	LLM skill	80.9	43.2	51.7	89.6	40.0	93.3
	Trace2Skill	82.4	49.6	65.7	90.6	52.0	87.3
	TextGrad	81.4	41.1	42.0	87.2	49.2	82.8
	GEPA	84.8	73.6	63.9	89.1	43.2	85.8

Table 1: Main test results (%). Bold and underlining mark the highest and second-highest distinct scores in each column within each target model. Subscripts on TextualRL scores show percentage-point gains over the corresponding Base Model.

Groupwise Policy Critique
Setting	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld
TextualRL (full)	82.7	60.7	47.7	94.4	56.0	84.3
w/o outcome routing	81.8	58.9	46.5	92.5	53.6	83.6
Single rollout ( $K=1$ )	80.7	57.5	46.5	93.3	50.4	82.1
Cross-Group Policy Update
Intra-group cards only	82.1	59.3	47.1	93.0	53.6	83.6

Table 2: Ablations on Qwen3.8-27B (%). Full TextualRL is the shared reference for both groups.

Method	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld	Overall
SkillOpt	$79.93\!\pm\!0.47$	$56.55\!\pm\!0.54$	$45.93\!\pm\!0.59$	$92.87\!\pm\!0.41$	$47.73\!\pm\!1.22$	$81.59\!\pm\!0.86$	$67.43\!\pm\!0.33$
TextualRL	$\mathbf{82.62}\!\pm\!0.16$	$\mathbf{60.24}\!\pm\!0.55$	$\mathbf{47.09}\!\pm\!0.58$	$\mathbf{94.03}\!\pm\!0.41$	$\mathbf{56.00}\!\pm\!0.80$	$\mathbf{84.08}\!\pm\!1.14$	$\mathbf{70.68}\!\pm\!0.26$

Table 3: Mean

\pm

sample standard deviation over three independent runs on Qwen3.8-27B (%).

Method	$K$	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld	Overall
TextualRL	1	80.7	57.5	46.5	93.3	50.4	82.1	68.4
	2	81.4	58.9	47.7	93.6	53.6	82.8	69.7
	4	82.7	60.7	47.7	94.4	56.0	84.3	71.0
	8	82.9	60.0	48.8	94.1	56.0	85.1	71.1

Table 4: Effect of training rollout count

K

on Qwen3.8-27B (%).

Harness	Skill Source	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath
Codex	Base Model	81.8	27.5	38.3	87.2	35.2
	Human skill	84.1	50.7	40.0	88.8	48.8
	LLM skill	83.4	25.0	34.3	89.8	45.6
	EvoSkill	61.4	67.5	42.4	89.3	63.2
	SkillOpt	87.3	85.0	51.1	92.2	78.4
	TextualRL	88.6 +6.8	85.7 +58.2	56.4 +18.1	92.8 +5.6	76.8 +41.6

Table 5: Test results (%) with GPT-5.5 under different agent harnesses.

Target model	Skill	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld	Overall
Qwen3.6- 35B-A3B	Origin	81.2	52.1	49.4	92.8	44.8	83.6	67.3
Qwen3.6- 35B-A3B	Transferred	81.4	53.2	48.3	93.3	47.2	83.6	67.8
Qwen3.5-4B	Origin	73.9	27.1	32.0	89.3	56.0	82.1	60.1
Qwen3.5-4B	Transferred	74.3	27.9	32.0	90.1	55.2	82.1	60.3

Table 6: Skill transfer from Qwen3.8-27B to two target models (%).

Optimizer	SearchQA	Spreadsheet	OfficeQA	DocVQA	LiveMath	ALFWorld	Overall
GPT-5.5	82.7	60.7	47.7	94.4	56.0	84.3	71.0
Qwen3.8-27B	81.5	58.6	46.5	93.9	54.4	83.6	69.7

Table 7: Optimizer comparison with Qwen3.8-27B as the fixed target (%).

Figure 3: Epoch curves for TextualRL on Qwen3.8-27B.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Train	Validation	Test	Metric
SearchQA	400	200	1,400	Normalized exact match
SpreadsheetBench	80	40	280	All-case success
OfficeQA	50	24	172	Normalized exact match
DocVQA	107	53	374	Hard accuracy
LiveMath	35	17	125	Choice accuracy
ALFWorld	39	18	134	Task success

Appendix

Table 8: Benchmark splits and evaluation metrics.

Benchmark	Method	Final tokens	Recorded iteration tokens (M)			Inference text tokens / task
Benchmark	Method	Final tokens	Target	Optimizer	Total	Inference text tokens / task
SearchQA	SkillOpt	937	31.33	7.67	39.00	2,698.2
SearchQA	TextualRL	2,422	66.26	13.57	79.83	5,035.4
Spreadsheet	SkillOpt	2,908	49.62	4.99	54.60	9,044.2
Spreadsheet	TextualRL	1,397	32.99	8.35	41.34	4,145.6
OfficeQA	SkillOpt	1,589	44.18	4.70	48.88	105,784.5

Appendix

Table 9: Token usage with Qwen3.8-27B as the target model with GPT-5.5 as optimizer. Iteration usage includes candidate validation. Inference usage sums input and visible output text per task.

Audit item	Result	Measurement
Reference traceability	99.13% (573/578)	Card resolves and its quote matches the source trajectory.
LLM relation agreement	94.77% (145/153)	Agreement between two GPT-6 Astra assessments, $\kappa=0.715$ .
Reported review changes	8.48% (37/436)	Optimizer reports narrow or revise .

Appendix

Table 10: Evidence review audit on Qwen3.8-27B runs.

Benchmark	Old skill	New skill w/o review	New skill
SearchQA	81.57	82.07	82.71
OfficeQA	45.35	45.93	47.09
DocVQA	89.57	90.37	91.44

Appendix

Table 11: Unseen-test performance (%) of three context variants for selected updates.

Figure 4: Key rule changes before and after evidence review for the updates in Table 11 .

Figure 5: Epoch curves for TextualRL on SearchQA, Spreadsheet, and ALFWorld with Qwen3.8-27B. Epoch and score definitions follow Figure 3 .

Figure 6: Representative instructions from the optimized contexts for six benchmarks. Original wording is preserved, with formatting standardized for readability.

Figure 7: A complete context update example on SearchQA.

RL for LLM Agents

Reinforcement learning for large language model agents trains models to plan, use tools, and adapt through feedback from interactive tasks.

See all 616 papers

RL with Learnable Textual Feedback: A Bilevel Approach
May 23, 2026Utsav Singh, Sidhaarth Sredharan, Souradip Chakraborty +1Reinforcement Learning RL for Language Model Reasoning
University of Central Florida · 2Carnegie Mellon University · 3Universtiy of Maryland
From Agent Failures to Text Policies: What Works and What Breaks
Jul 22, 2026Jaideep Ray, Ankit GoyalPolicy Optimization LLM Agent Self-Improvement
Independent Researcher
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Jun 17, 2026Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2Training Data Curation RL for Language Model Reasoning
OpenBMB · Tsinghua University

May 23, 2026J/K move · Enter open · S save

RL with Learnable Textual Feedback: A Bilevel Approach

Utsav Singh, Sidhaarth Sredharan, Souradip Chakraborty, Amrit Singh Bedi

University of Central Florida · 2Carnegie Mellon University · 3Universtiy of Maryland

Reinforcement Learning RL for Language Model Reasoning Bilevel Optimization Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards can improve LLM reasoning, but learning remains sample-inefficient when terminal rewards are sparse. This has motivated a growing line of work on RL with textual feedback, where a critic model generates natural language feedback to guide a reasoning model (the actor), augmenting scalar rewards with richer learning signals. However, existing methods typically treat feedback as fixed or auxiliary, which misses a key property: feedback should not merely be correct, but should improve the policy (actor model) when provided in context. This motivates a paradigm of learnable textual feedback for RL. Yet the learnability and usefulness of feedback depend on the policy's ability to learn from it, making RL with learnable feedback an inherently bilevel problem. We formalize this coupling as a Stackelberg bilevel program and derive Bilevel Natural Language Actor-Critic (Bi-NAC), which jointly trains a critic to generate reward-improving feedback and an actor to exploit it. Across MATH-500, MBPP, and GPQA, Bi-NAC improves sample and parameter efficiency over RL and fixed-critic baselines: our 2B model outperforms the 3B GRPO baseline, achieving 46.6% versus 41.4% on MATH-500, while our 6B model surpasses the 7B GRPO baseline, achieving 49.3% versus 43.6% on GPQA.

Abstract

RL for LLM Agents

Explore similar work

RL with Learnable Textual Feedback: A Bilevel Approach

From Agent Failures to Text Policies: What Works and What Breaks

Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning