LLM agents can adapt to task feedback by revising the textual context that guides their behavior, without changing their model parameters. However, turning execution rewards into reusable context updates is challenging: individual trajectories offer limited comparative evidence about what to change, while task-specific fixes may overgeneralize when incorporated into a shared context. We introduce TextualRL, a gradient-free reinforcement learning framework that treats a group of rollouts from the same task under the same context, rather than an isolated trajectory, as the unit of policy critique. Groupwise Policy Critique contrasts successful and failed rollouts within a task; when outcomes are uniform, it draws on cross-task evidence to identify behaviors worth preserving or formulate repair hypotheses. Cross-Group Policy Update reviews the resulting proposals against evidence pooled across groups, including observations that yield no proposed edit, to refine their content and applicability. Reviewed proposals form a bounded textual patch, with adoption of the candidate context determined by held-out validation rewards. Together, these components establish a reward-driven policy optimization loop in which multiple attempts jointly inform reusable context updates. We evaluate TextualRL with five target agents on six benchmarks. Averaged over the six benchmarks, TextualRL improves test accuracy over the base model by 24.4 and 15.3 percentage points for GPT-5.5 and Qwen3.8-27B, respectively.
Figures & tables
Figure 1: Conceptual comparison of parameter-space and context-space policy optimization. TextualRL uses groups of same-task rollouts and their rewards to propose textual edits ΔC to context C , while the target model remains frozen.
Figure 2: Overview of TextualRL. (a) Groupwise Policy Critique preserves same-task rollout groups and uses within-task contrast for mixed outcomes, cross-task preservation for all-success outcomes, and cross-task repair for all-failure outcomes. Parallel critics return edit proposals and evidence cards separately. (b) Cross-Group Policy Update reviews proposals against pooled evidence, constructs a bounded patch, and selects the candidate context through held-out validation. Within-epoch feedback informs subsequent critique calls.
Model
Skill Source
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
GPT-5.5
Base Model
77.7
41.8
33.1
78.8
37.6
83.6
Human skill
81.8
72.9
66.9
90.1
38.4
91.8
LLM skill
80.9
43.2
51.7
89.6
40.0
93.3
Trace2Skill
82.4
49.6
65.7
90.6
52.0
87.3
TextGrad
81.4
41.1
42.0
87.2
49.2
82.8
GEPA
84.8
73.6
63.9
89.1
43.2
85.8
Table 1: Main test results (%). Bold and underlining mark the highest and second-highest distinct scores in each column within each target model. Subscripts on TextualRL scores show percentage-point gains over the corresponding Base Model.
Setting
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
TextualRL (full)
82.7
60.7
47.7
94.4
56.0
84.3
Groupwise Policy Critique
w/o outcome routing
81.8
58.9
46.5
92.5
53.6
83.6
Single rollout ( K=1 )
80.7
57.5
46.5
93.3
50.4
82.1
Cross-Group Policy Update
Intra-group cards only
82.1
59.3
47.1
93.0
53.6
83.6
Table 2: Ablations on Qwen3.8-27B (%). Full TextualRL is the shared reference for both groups.
Method
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
Overall
SkillOpt
79.93±0.47
56.55±0.54
45.93±0.59
92.87±0.41
47.73±1.22
81.59±0.86
67.43±0.33
TextualRL
82.62±0.16
60.24±0.55
47.09±0.58
94.03±0.41
56.00±0.80
84.08±1.14
70.68±0.26
Table 3: Mean ± sample standard deviation over three independent runs on Qwen3.8-27B (%).
Method
K
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
Overall
TextualRL
1
80.7
57.5
46.5
93.3
50.4
82.1
68.4
2
81.4
58.9
47.7
93.6
53.6
82.8
69.7
4
82.7
60.7
47.7
94.4
56.0
84.3
71.0
8
82.9
60.0
48.8
94.1
56.0
85.1
71.1
Table 4: Effect of training rollout count K on Qwen3.8-27B (%).
Harness
Skill Source
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
Codex
Base Model
81.8
27.5
38.3
87.2
35.2
Human skill
84.1
50.7
40.0
88.8
48.8
LLM skill
83.4
25.0
34.3
89.8
45.6
EvoSkill
61.4
67.5
42.4
89.3
63.2
SkillOpt
87.3
85.0
51.1
92.2
78.4
TextualRL
88.6 +6.8
85.7 +58.2
56.4 +18.1
92.8 +5.6
76.8 +41.6
Table 5: Test results (%) with GPT-5.5 under different agent harnesses.
Target model
Skill
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
Overall
Qwen3.6- 35B-A3B
Origin
81.2
52.1
49.4
92.8
44.8
83.6
67.3
Transferred
81.4
53.2
48.3
93.3
47.2
83.6
67.8
Qwen3.5-4B
Origin
73.9
27.1
32.0
89.3
56.0
82.1
60.1
Transferred
74.3
27.9
32.0
90.1
55.2
82.1
60.3
Table 6: Skill transfer from Qwen3.8-27B to two target models (%).
Optimizer
SearchQA
Spreadsheet
OfficeQA
DocVQA
LiveMath
ALFWorld
Overall
GPT-5.5
82.7
60.7
47.7
94.4
56.0
84.3
71.0
Qwen3.8-27B
81.5
58.6
46.5
93.9
54.4
83.6
69.7
Table 7: Optimizer comparison with Qwen3.8-27B as the fixed target (%).
Figure 3: Epoch curves for TextualRL on Qwen3.8-27B.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Train
Validation
Test
Metric
SearchQA
400
200
1,400
Normalized exact match
SpreadsheetBench
80
40
280
All-case success
OfficeQA
50
24
172
Normalized exact match
DocVQA
107
53
374
Hard accuracy
LiveMath
35
17
125
Choice accuracy
ALFWorld
39
18
134
Task success
Appendix
Table 8: Benchmark splits and evaluation metrics.
Benchmark
Method
Final tokens
Recorded iteration tokens (M)
Inference text tokens / task
Target
Optimizer
Total
SearchQA
SkillOpt
937
31.33
7.67
39.00
2,698.2
TextualRL
2,422
66.26
13.57
79.83
5,035.4
Spreadsheet
SkillOpt
2,908
49.62
4.99
54.60
9,044.2
TextualRL
1,397
32.99
8.35
41.34
4,145.6
OfficeQA
SkillOpt
1,589
44.18
4.70
48.88
105,784.5
Appendix
Table 9: Token usage with Qwen3.8-27B as the target model with GPT-5.5 as optimizer. Iteration usage includes candidate validation. Inference usage sums input and visible output text per task.
Audit item
Result
Measurement
Reference traceability
99.13% (573/578)
Card resolves and its quote matches the source trajectory.
LLM relation agreement
94.77% (145/153)
Agreement between two GPT-6 Astra assessments, κ=0.715 .
Reported review changes
8.48% (37/436)
Optimizer reports narrow or revise .
Appendix
Table 10: Evidence review audit on Qwen3.8-27B runs.
Benchmark
Old skill
New skill w/o review
New skill
SearchQA
81.57
82.07
82.71
OfficeQA
45.35
45.93
47.09
DocVQA
89.57
90.37
91.44
Appendix
Table 11: Unseen-test performance (%) of three context variants for selected updates.
Figure 4: Key rule changes before and after evidence review for the updates in Table 11 .
Figure 5: Epoch curves for TextualRL on SearchQA, Spreadsheet, and ALFWorld with Qwen3.8-27B. Epoch and score definitions follow Figure 3 .
Figure 6: Representative instructions from the optimized contexts for six benchmarks. Original wording is preserved, with formatting standardized for readability.
Figure 7: A complete context update example on SearchQA.
Reinforcement learning with verifiable rewards can improve LLM reasoning, but learning remains sample-inefficient when terminal rewards are sparse. This has motivated a growing line of work on RL with textual feedback, where a critic model generates natural language feedback to guide a reasoning model (the actor), augmenting scalar rewards with richer learning signals. However, existing methods typically treat feedback as fixed or auxiliary, which misses a key property: feedback should not merely be correct, but should improve the policy (actor model) when provided in context. This motivates a paradigm of learnable textual feedback for RL. Yet the learnability and usefulness of feedback depend on the policy's ability to learn from it, making RL with learnable feedback an inherently bilevel problem. We formalize this coupling as a Stackelberg bilevel program and derive Bilevel Natural Language Actor-Critic (Bi-NAC), which jointly trains a critic to generate reward-improving feedback and an actor to exploit it. Across MATH-500, MBPP, and GPQA, Bi-NAC improves sample and parameter efficiency over RL and fixed-critic baselines: our 2B model outperforms the 3B GRPO baseline, achieving 46.6% versus 41.4% on MATH-500, while our 6B model surpasses the 7B GRPO baseline, achieving 49.3% versus 43.6% on GPQA.
TextGrad improves language-model systems by revising text from feedback. Its core thesis is that natural-language feedback can act as a gradient for optimizing text components without changing model weights. Applying it to agents is harder because feedback arrives only after a sequence of actions, making it difficult to identify which decision caused failure. We study this problem by separating the ability to follow a useful policy from the ability to learn that policy from experience. Our main finding is a clear gap between these two abilities. Human-written policies improve two frozen 7B agents on TextWorldExpress by 5.0 success points, showing that useful policy text exists. However, policies generated from agent trajectories do not reliably outperform fixed prompting, even with richer traces, counterfactual evidence, or iterative GEPA search. The main challenge for agent-level TextGrad is therefore not executing textual policy updates, but reliably generating and selecting them from experience.
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.