EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

Min Cai; Dan Zhang; Gao Xing; Libin Ma; Zhaohui Xu; Xi Ye

cs.AIOct 12, 2026

EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

Authors: Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye

Organizations: University of Alberta · National University of Singapore · Kuaishou Technology · Independent Researcher ·

Abstract

Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment, which jointly shape the agent's experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce EAU, a meta-agent framework for environment--agent--user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 14.7--15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.

Figures & tables

Figure 1: Illustration of how our meta-agent, EAU , manages RL training runs. EAU surpasses the prior SOTA on ColBench. Left: For the first three runs, EAU proactively restarts when it identifies issues that can only be resolved with offline patches. Middle: Run 4 and 5 displayed in blue and red are two full runs. Dashed lines indicate online, intra-run modifications. Run 4 explored individual modifications through local perturbations, whereas Run 5 used endpoint interventions followed by replication, reversal, and interaction tests.

Figure 2: Illustration of a user-centric agentic task. An agent needs to ask for clarifications about a user query.

Figure 3: Illustration of how EAU , our meta agent, explores, learns, and absorbs experience online during and across each run. Reading logs and trajectories, EAU explores different strategies online by controlling the knobs of each module, learns which strategies help and which do not. Experience is brought across different runs, serving as a solid foundation for continual exploration and learning.

Figure 4: Illustration of symbolic user simulator.

Figure 5: Illustration of environment evolution by introducing additional rewards.

Type	Method	UserGym								ColBench			Overall
		ID					OOD		Avg.	Pass	Succ.	Avg.
		Travel	Func.	Pers.	Tau	Turtle	Inten.	Tele.		Pass	Succ.
Closed-Source Models
Prompting	Gemini-3-Flash	0.574	0.423	0.695	0.167	0.153	1.718	0.829	0.651	0.515	0.382	0.449	0.606
Prompting	GPT-4.1	0.554	0.051	0.599	0.109	0.267	1.867	0.732	0.597	0.529	0.403	0.466	0.568
Prompting	GPT-4o-mini	0.596	0.089	0.683	0.091	0.117	1.277	0.512	0.481	0.463	0.342	0.403	0.463

Table 1: Comparison across UserGym and ColBench on Qwen3-4B and Qwen2.5-7B-Instruct . The best and second-best scores in each category are highlighted in bold and underlined respectively. Standard deviations across 3 repeated evaluations are shown in the subscripts.

Method	Retail	Airline	Telecom	Avg.
UserRL	16.9	7.5	13.8	13.8
InfoPO	24.4	10.0	15.6	18.0
PPO	16.3	23.8	18.1	18.5
EAU	18.1	28.8	21.3	21.5

Table 2: Results on

\tau^{2}

-bench ( Qwen3-4B ).

Variant	Pass mean	Succ. mean
LLM	$0.284\textsubscript{$ \pm $0.046}$	$0.200\textsubscript{$ \pm $0.044}$
Symbolic	$0.229\textsubscript{$ \pm $0.004}$	$0.177\textsubscript{$ \pm $0.012}$
Hybrid	$0.294\textsubscript{$ \pm $0.019}$	$0.213\textsubscript{$ \pm $0.032}$

Table 3: Ablation on the user simulator. Results are on ColBench with Qwen3-4B .

Table 9

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Knob	Range	Recorded use / scope
Environment
max_steps	$3$ – $60$	Adjusted and reverted.
min_questions	$0$ – $3$	Counter repaired; parameter adjusted.
human_response_char_limit	—	Not exercised.
User (symbolic)
facts_per_reply	$1$ – $3$	Adjusted and reverted.

Appendix

Table 6: ColBench interfaces and their use in the two analyzed runs. “Not exercised” refers to these runs only; “unavailable” denotes unsupported functionality in the documented implementation. A dash in the range column means that no range is specified or applicable.

Control / issue	Observed behavior	Reported response / status
max_steps	Reducing $10\rightarrow 6$ lowered Pass by $0.246$ ; reverting increased it by $0.177$ .	Reverted. The perturbation-and-reversal pattern concerns sensitivity to the interaction budget.
min_questions	The counter omitted turns that both asked and answered a question, deadlocking $41\%$ of episodes.	Code repair reported. Subsequent $1\rightarrow 2$ adjustment had no measured effect; $2\rightarrow 3$ was harmful.
facts_per_reply	Increasing $1\rightarrow 3$ produced reference-solution disclosure in $837$ -character replies.	Reverted. The initial reward attribution was not retained after accounting for the reported gap band.
match_threshold	The sequence $0.9/0.6/0.35/0.9$ produced no change exceeding the reported noise floor of $0.228$ .	Inconclusive at the reported noise level.
volunteer_interval	Reducing $10\rightarrow 5$ increased flow success while the question rate fell from $40\%$ to $26\%$ .	Reverted; the rise in flow success was interpreted as masking reduced questioning.
max_reveals	Re-disclosing an already revealed fact did not consume the budget, leaving repeated disclosure unrestricted by this control.	Interface limitation documented; no corresponding repair is specified in the run summary.

Appendix

Table 7: Intervention attempts and implementation diagnostics in the two ColBench runs. Reported score changes are local observations, not controlled estimates of each intervention’s contribution to final performance. A missing follow-up entry does not establish that a problem persisted throughout all subsequent runs.

Environment
Knob (range)	travel	tau	persuade	turtle	function
max_steps ( $3$ – $60$ )	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
search_failure_interval ( $1$ – $50$ )	$\checkmark$	—	—	—	—
elicitation_interval ( $1$ – $20$ )	$\checkmark$	—	—	—	—
noise_choice_number ( $0$ – $30$ )	$\checkmark$	—	—	—	—
wrong_choice_number ( $0$ – $30$ )	$\checkmark$	—	—	—	—

Appendix

Table 8: UserGym interface coverage in the documented implementation.

\checkmark

indicates support and — indicates unavailable functionality. These entries describe implementation coverage rather than intervention outcomes.

Figure 8: Experience flow from UserGym, Qwen3-4B to ColBench, Qwen2.5-7B .

Data Configuration
Parameter	UserGym	ColBench	$\tau^{2}$ -Bench
Train batch size	64	64	32
Max prompt length	1152	2048	8192
Max response length	8192	8192	16384
Actor Configuration
Learning rate	$3\times 10^{-7}$	$3\times 10^{-7}$	$1\times 10^{-6}$

Appendix

Table 9: Key hyperparameters for training on UserGym, ColBench, and

\tau^{2}

-Bench environments.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 585 papers

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution
Apr 17, 2026Shidong Yang, Ziyu Ma, Tongwen Huang +3LLM Agent Training RL for LLM Agents
AMAP, Alibaba Group
Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution
Jun 9, 2026Xucong Wang, Ziyu Ma, Shidong Yang +4LLM Agent Self-Improvement LLM Agent Training
University of Science and Technology of China · AMAP, Alibaba Group
PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems
Aug 28, 2026Hanglong Lv, Dawei Zhu, Lei Li +11User Simulation LLM Agent Training
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core, Xiaomi · The University of Hong Kong +1

Apr 17, 2026J/K move · Enter open · S save

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu+2

AMAP, Alibaba Group

LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Reinforcement learning for LLM agents is typically conducted on a static data distribution, which fails to adapt to the agent's evolving behavior and leads to poor coverage of complex environment interactions. To address these challenges, we propose CoEvolve, an agent-data mutual evolution framework that enables LLM agents to improve through closed-loop, interaction-driven training. Specifically, CoEvolve extracts feedback signals such as forgetting and uncertainty from rollout trajectories to identify failure-prone interaction patterns, and utilizes them to guide LLM-based task synthesis. The synthesized tasks are validated through environment interaction and utilized to update the data distribution, enabling joint adaptation of the agent and its data. Extensive experiments on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.

Abstract

LLM Agent Training

Explore similar work

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems