EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents
Authors: Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye
Organizations: University of Alberta · National University of Singapore · Kuaishou Technology · Independent Researcher · Alberta Machine Intelligence Institute (Amii)
Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment, which jointly shape the agent's experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce EAU, a meta-agent framework for environment--agent--user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 14.7--15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.
Figures & tables
Figure 1: Illustration of how our meta-agent, EAU , manages RL training runs. EAU surpasses the prior SOTA on ColBench. Left: For the first three runs, EAU proactively restarts when it identifies issues that can only be resolved with offline patches. Middle: Run 4 and 5 displayed in blue and red are two full runs. Dashed lines indicate online, intra-run modifications. Run 4 explored individual modifications through local perturbations, whereas Run 5 used endpoint interventions followed by replication, reversal, and interaction tests.
Figure 2: Illustration of a user-centric agentic task. An agent needs to ask for clarifications about a user query.
Figure 3: Illustration of how EAU , our meta agent, explores, learns, and absorbs experience online during and across each run. Reading logs and trajectories, EAU explores different strategies online by controlling the knobs of each module, learns which strategies help and which do not. Experience is brought across different runs, serving as a solid foundation for continual exploration and learning.
Figure 4: Illustration of symbolic user simulator.
Figure 5: Illustration of environment evolution by introducing additional rewards.
Type
Method
UserGym
ColBench
Overall
ID
OOD
Avg.
Pass
Succ.
Avg.
Travel
Func.
Pers.
Tau
Turtle
Inten.
Tele.
Closed-Source Models
Prompting
Gemini-3-Flash
0.574
0.423
0.695
0.167
0.153
1.718
0.829
0.651
0.515
0.382
0.449
0.606
Prompting
GPT-4.1
0.554
0.051
0.599
0.109
0.267
1.867
0.732
0.597
0.529
0.403
0.466
0.568
Prompting
GPT-4o-mini
0.596
0.089
0.683
0.091
0.117
1.277
0.512
0.481
0.463
0.342
0.403
0.463
Table 1: Comparison across UserGym and ColBench on Qwen3-4B and Qwen2.5-7B-Instruct . The best and second-best scores in each category are highlighted in bold and underlined respectively. Standard deviations across 3 repeated evaluations are shown in the subscripts.
Method
Retail
Airline
Telecom
Avg.
UserRL
16.9
7.5
13.8
13.8
InfoPO
24.4
10.0
15.6
18.0
PPO
16.3
23.8
18.1
18.5
EAU
18.1
28.8
21.3
21.5
Table 2: Results on τ2 -bench ( Qwen3-4B ).
Variant
Pass mean
Succ. mean
LLM
0.284\textsubscript{\pm 0.046}
0.200\textsubscript{\pm0.044}
Symbolic
0.229\textsubscript{\pm0.004}
0.177\textsubscript{\pm0.012}
Hybrid
0.294\textsubscript{\pm0.019}
0.213\textsubscript{\pm0.032}
Table 3: Ablation on the user simulator. Results are on ColBench with Qwen3-4B .
Table 9Figure 10
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Knob
Range
Recorded use / scope
Environment
max_steps
3 – 60
Adjusted and reverted.
min_questions
0 – 3
Counter repaired; parameter adjusted.
human_response_char_limit
—
Not exercised.
User (symbolic)
facts_per_reply
1 – 3
Adjusted and reverted.
Appendix
Table 6: ColBench interfaces and their use in the two analyzed runs. “Not exercised” refers to these runs only; “unavailable” denotes unsupported functionality in the documented implementation. A dash in the range column means that no range is specified or applicable.
Control / issue
Observed behavior
Reported response / status
max_steps
Reducing 10→6 lowered Pass by 0.246 ; reverting increased it by 0.177 .
Reverted. The perturbation-and-reversal pattern concerns sensitivity to the interaction budget.
min_questions
The counter omitted turns that both asked and answered a question, deadlocking 41% of episodes.
Code repair reported. Subsequent 1→2 adjustment had no measured effect; 2→3 was harmful.
facts_per_reply
Increasing 1→3 produced reference-solution disclosure in 837 -character replies.
Reverted. The initial reward attribution was not retained after accounting for the reported gap band.
match_threshold
The sequence 0.9/0.6/0.35/0.9 produced no change exceeding the reported noise floor of 0.228 .
Inconclusive at the reported noise level.
volunteer_interval
Reducing 10→5 increased flow success while the question rate fell from 40% to 26% .
Reverted; the rise in flow success was interpreted as masking reduced questioning.
max_reveals
Re-disclosing an already revealed fact did not consume the budget, leaving repeated disclosure unrestricted by this control.
Interface limitation documented; no corresponding repair is specified in the run summary.
Appendix
Table 7: Intervention attempts and implementation diagnostics in the two ColBench runs. Reported score changes are local observations, not controlled estimates of each intervention’s contribution to final performance. A missing follow-up entry does not establish that a problem persisted throughout all subsequent runs.
Knob (range)
travel
tau
persuade
turtle
function
Environment
max_steps ( 3 – 60 )
✓
✓
✓
✓
✓
search_failure_interval ( 1 – 50 )
✓
—
—
—
—
elicitation_interval ( 1 – 20 )
✓
—
—
—
—
noise_choice_number ( 0 – 30 )
✓
—
—
—
—
wrong_choice_number ( 0 – 30 )
✓
—
—
—
—
Appendix
Table 8: UserGym interface coverage in the documented implementation. ✓ indicates support and — indicates unavailable functionality. These entries describe implementation coverage rather than intervention outcomes.
Figure 8: Experience flow from UserGym, Qwen3-4B to ColBench, Qwen2.5-7B .
Parameter
UserGym
ColBench
τ2 -Bench
Data Configuration
Train batch size
64
64
32
Max prompt length
1152
2048
8192
Max response length
8192
8192
16384
Actor Configuration
Learning rate
3×10−7
3×10−7
1×10−6
Appendix
Table 9: Key hyperparameters for training on UserGym, ColBench, and τ2 -Bench environments.
Reinforcement learning for LLM agents is typically conducted on a static data distribution, which fails to adapt to the agent's evolving behavior and leads to poor coverage of complex environment interactions. To address these challenges, we propose CoEvolve, an agent-data mutual evolution framework that enables LLM agents to improve through closed-loop, interaction-driven training. Specifically, CoEvolve extracts feedback signals such as forgetting and uncertainty from rollout trajectories to identify failure-prone interaction patterns, and utilizes them to guide LLM-based task synthesis. The synthesized tasks are validated through environment interaction and utilized to update the data distribution, enabling joint adaptation of the agent and its data. Extensive experiments on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.
Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization. To address these limitations, this paper introduces Role-Agent, \textcolor{black}{a framework} that harnesses a single LLM to function concurrently as both the agent and the environment, enabling a bootstrapped co-evolution. Role-Agent comprises two synergistic components: World-In-Agent (WIA) and Agent-In-World (AIW). In WIA, the LLM acts as the agent and predicts future states after each action; the alignment between predicted and actual states is then used as a process reward, encouraging environment-aware reasoning. In AIW, the LLM analyzes failure modes from failed trajectories and retrieves tasks with similar failure patterns, thereby reshaping the training data distribution for targeted practice. Experiments on multiple benchmarks show that Role-Agent consistently improves performance, yielding an average gain of over 4% over strong baselines.
Xucong Wang, Ziyu Ma, Shidong Yang +4
University of Science and Technology of China · AMAP, Alibaba Group
Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally complete, single-turn queries. Our analysis of 16K real-world sessions shows that 75.9% of interactions are multi-turn, revealing a substantial gap between how users interact with agents and how such systems are trained and evaluated. We introduce PersonaForge, a user simulation framework for synthesizing realistic multi-turn user--agent interactions. PersonaForge combines a four-dimensional persona space, SOUL-driven behavioral control calibrated to real-user statistics, and Reverse Deep Construction grounded in authentic seed queries. Using PersonaForge, we construct a 6.3K-record training dataset and PersonaForge-Bench, a manually annotated 138-task benchmark spanning over 20 professional domains with four-dimensional scoring. Experiments on Qwen3.5-27B show that PersonaForge training improves the composite score by +4.1%, with gains across all four dimensions and the largest improvements in Task Completion (+6.0%) and Response Quality (+6.8%). Further analyses show that PersonaForge-trained agents use fewer turns and tool calls, suggesting improved interaction efficiency, while ablations confirm the contribution of SOUL components and adaptive simulation. Together, PersonaForge and PersonaForge-Bench establish a foundation for training and evaluating agents under realistic multi-turn user interaction.
Hanglong Lv, Dawei Zhu, Lei Li +11
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · LLM-Core, Xiaomi · The University of Hong Kong +1