CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflection

Linbo Liu; Guande Wu; Han Ding; Yawei Wang; Qiang Zhou; Yuzhe Lu; Zhichao Xu; Huan Song

cs.AIApr 8, 2026

CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflection

Authors: Linbo Liu, Guande Wu, Han Ding, Yawei Wang, Qiang Zhou, Yuzhe Lu, Zhichao Xu, Huan Song, +2 more

Organizations: AWS AI

Abstract

Large language model agents rely on effective model context to obtain task-relevant information for decision-making. Many existing context engineering approaches primarily rely on the context generated from the past experience and retrieval mechanisms that reuse these context. However, retrieved context from past tasks must be adapted by the execution agent to fit new situations, placing additional reasoning burden on the underlying LLM. To address this limitation, we propose a generative context augmentation framework using Contrastive Learning of Experience via Agentic Reflection (CLEAR). CLEAR first employs a reflection agent to perform contrastive analysis over past execution trajectories and summarize useful context for each observed task. These summaries are then used as supervised fine-tuning data to train a context augmentation model (CAM). Then we further optimize CAM using reinforcement learning, where the reward signal is obtained by running the task execution agent. By learning to generate task-specific knowledge rather than retrieve knowledge from the past, CAM produces context that is better tailored to the current task. We conduct comprehensive evaluations on the AppWorld and WebShop benchmarks. Experimental results show that CLEAR consistently outperforms strong baselines. It improves task completion rate from 72.62% to 81.15% on AppWorld test set and averaged reward from 0.68 to 0.74 on a subset of WebShop, compared with baseline agent. Our code is publicly available at https://github.com/awslabs/CLEAR.

Figures & tables

Figure 1: CLEAR training framework design. First, we execute each task $q_{i}\sim\mathcal{D}_{\text{train}}$ for $m$ times and collect groups of replay $\Gamma_{i}$ for $q_{i}$ . We employ reflection agents $\pi^{R}$ to generate instance-level instruction $c_{i}$ for each $q_{i}$ , collected into $\mathcal{D}_{\text{SFT}}$ . Then, we initialize CAM from an open-source LLM and perform SFT on $\mathcal{D}_{\text{SFT}}$ . Finally, we further perform RL on the trained CAM, which leverages the reward signal from $\pi^{E}$ for policy update of CAM.

Figure 2: During inference, a new task $q_{\text{new}}$ is sampled from $\mathcal{D}_{\text{test}}$ and is passed into $\pi^{C}_{\theta}$ to generate $c_{\text{new}}\sim\pi^{C}_{\theta}(q_{\text{new}})$ . The auxiliary context $c_{\text{new}}$ is appended to $q_{\text{new}}$ and the execution agent $\pi^{E}$ starts with $q_{\text{new}}\oplus c_{\text{new}}$ .

Method	TGC $\uparrow$		SGC $\uparrow$
Method	Avg	Pass@3	Avg	Pass@3
Baseline	72.62(2.59)	86.90	52.38(2.73)	66.07
RAG	72.02(2.15)	86.31	54.67(3.72)	71.43
ACE Zhang et al. (2025)	74.40(3.57)	85.71	58.93(6.19)	73.21
GEPA Agrawal et al. (2025)	75.99(0.74)	89.88	57.14(3.86)	78.57
MIPRO Opsahl-Ong et al. (2024)	74.40(1.46)	88.10	55.36(1.46)	80.36

Table 1: AppWorld experiments results. Task Goal Completion (TGC) and Scenario Goal Completion (SGC) on the Test-N split are reported. Results are averaged over three runs (standard deviation in parentheses) except the Pass@3 metric.

Method	Avg. Reward	Std.
Baseline	0.6799	0.0119
RAG	0.7252	0.0076
CLEAR (ours)	0.7406	0.0044

Table 2: WebShop-40k experiments results. Averaged reward on the test dataset is reported. Results are averaged over three runs with standard deviation also reported.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment ID	Method	TGC $\uparrow$		SGC $\uparrow$
Experiment ID	Method	Avg	Pass@3	Avg	Pass@3
1	Qwen/Qwen3-32B	67.40(2.95)	83.33	48.81(4.49)	66.07
2	SFT without CL	68.65 (2.68)	88.69	42.86 (4.72)	67.86
3	SFT with CL	74.21(1.72)	90.48	50.60(2.73)	71.43
4	CLEAR	81.15 (2.48)	91.67	66.67 (4.49)	82.14

Appendix

Table 3: Ablation study on AppWorld Test-N split. For the CAM, we use the following variant: (1) a Qwen/Qwen3-32B model without SFT and RL as in experiment 1. (2) a Qwen3-32B model after SFT on

\mathcal{D}_{\text{SFT\_no\_CL}}

(no RL) as in experiment 2. (3) a Qwen3-32B model after SFT with CL (no RL) as in experiment 3. (4) full CLEAR framework as in experiment 4. Results are averaged over three runs (standard deviation in parentheses) except the Pass@3 metric.

Method	Run time (s)	# of turns	CAM throughput (tokens/s)	CAM latency (s)
Baseline	63.4	17.3	NA	NA
CLEAR (ours)	77.2	18.5	69.5	1.21

Appendix

Table 4: CAM Latency study. Task run time, number of turns of the execution agent

\pi^{E}

, throughput of CAM, and latency for invoking CAM. All numbers are averaged over AppWorld Test-N split. The model for

\pi^{E}

is Claude-Sonnet-4 .

Model for $\pi^{E}$	Method	TGC $\uparrow$		SGC $\uparrow$
Model for $\pi^{E}$	Method	Avg	Pass@3	Avg	Pass@3
DeepSeek-V3.1	Baseline	41.27(3.05)	64.29	19.05(2.06)	26.79
	RAG	33.33(1.57)	53.57	10.71(4.72)	21.43
	ACE	32.54(1.50)	53.57	13.69(1.03)	28.57
	CLEAR (ours)	42.95 (2.85)	66.07	24.40 (4.49)	33.93

Appendix

Table 5: AppWorld experiments results. Task Goal Completion (TGC) and Scenario Goal Completion (SGC) on the Test-N split are reported. Results are averaged over three runs (standard deviation in parentheses) except the Pass@3 metric.

Explore similar work

May 29, 2026

Learning Agent-Compatible Context Management for Long-Horizon Tasks

LLM agents increasingly face long-horizon tasks such as web search and deep research in real-world applications, where accumulated context can cause long-context degradation and reasoning failures. Prior work mitigates this through context management with agent-side context control or fixed strategies such as summarization, which require training the agent itself for adaptation - making it impractical for closed-source agents and ignoring that different agents may require different strategies. We introduce Adaptive Context Management (AdaCoM), which trains an external LLM to manage the context of a frozen agent through flexible modification actions and end-to-end reinforcement learning. Across diverse agents on web search and deep research benchmarks, AdaCoM substantially improves performance by preserving task constraints and progress while pruning stale content. The learned strategies reveal a Fidelity-Reliability Trade-off: agents with higher vanilla ReAct performance benefit from higher-fidelity context preservation, whereas lower-performing agents require more aggressive compression to stay within a reliable reasoning regime. Transfer experiments show that AdaCoM generalizes most effectively across agents with similar capability (measured by vanilla ReAct performance), suggesting a practical path toward reusable context managers for agent systems.

Lu Yi, Runlin Lei, Liuyi Yao +6

1Renmin University of China · Work done during internship at Tongyi Lab, Alibaba Group · 2Tongyi Lab, Alibaba Group +2

Long-Horizon Agent Tasks LLM Context Management RL for LLM Agents

Ask PDF

May 6, 2026

From History to State: Constant-Context Skill Learning for LLM Agents

Large language model (LLM) agents are increasingly used to operate browsers, files, code and tools, making personal assistants a natural deployment target. Yet personal agents face a privacy-cost-capability tension: cloud models execute multi-step workflows well but expose sensitive intermediate context to external APIs, while local models preserve privacy but remain less reliable. Both settings also pay repeatedly for long skill prompts and growing histories. We propose constant-context skill learning, a context-to-weights framework for recurring agent workflows: reusable procedures are learned in lightweight task-family modules, while inference conditions only on the current observation and a compact state block. A deterministic tracker renders this state block from task progress and supplies aligned subgoal rewards, so each module can be trained with step-level SFT and refined through online RL. Across ALFWorld, WebShop, and SciWorld, our agents achieve strong performance across Qwen3-4B, Qwen3-8B and Llama-3.1-8B. With Qwen3-8B, SFT+RL reaches 89.6% unseen success on ALFWorld, 76.8% success on WebShop, and 66.4% unseen success on SciWorld. They match or exceed strong published agent-training results while reducing prompt tokens per turn by 2--7

\times

relative to controlled ReAct prompting baselines, showing that procedural context can be moved from prompts into weights.

Haoyang Xie, Xinyuan Wang, Yancheng Wang +2

School of Computing and Augmented Intelligence, Arizona State University

LLM Agent Skill Learning LLM Agent Training LLM Context Management RL for LLM Agents

Ask PDF

Sep 29, 2026

Context Language Models

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin +10

University of Washington · Meta Superintelligence Labs · MIT +1

Memory-Augmented Language Models LLM Context Management

Ask PDF