A central challenge in building continually improving agents is that training environments are typically static or manually constructed. This restricts continual learning and generalization beyond the training distribution. We address this with COvolve, a co-evolutionary framework that leverages large language models (LLMs) to generate both environments and agent policies, expressed as executable Python code. We model the interaction between environment and policy designers as a two-player zero-sum game, ensuring adversarial co-evolution in which environments expose policy weaknesses and policies adapt in response. This process induces an automated curriculum in which environments and policies co-evolve toward increasing complexity. To guarantee robustness and prevent forgetting as the curriculum progresses, we compute the mixed-strategy Nash equilibrium (MSNE) of the zero-sum game, thereby yielding a meta-policy. This MSNE meta-policy ensures that the agent does not forget to solve previously seen environments while learning to solve previously unseen ones. Experiments in urban driving, symbolic maze-solving, and geometric navigation showcase that COvolve produces progressively more complex environments. Our results demonstrate the potential of LLM-driven co-evolution to achieve open-ended learning without predefined task distributions or manual intervention.
Figures & tables
Figure 1. A conceptual overview of the proposed COvolve , comprised of an Environment Designer and a Policy Designer that co-evolve by playing a two-player zero-sum game. The Environment Designer generates increasingly challenging environments (as code), while the Policy Designer creates policies (as code) to solve them. A mixed-strategy Nash equilibrium enables robust, open-ended learning through continual adaptation.
Figure 2. Example of co evolution in COvolve. Left: successive environment implementations, progressing from ad hoc generation to a structured, parameterized design with explicit solvability checks and controllable chokepoints. Right: successive policy implementations, progressing from basic navigation to improved handling of keys and doors, together with refinements to the A* based planner for more reliable action selection. Highlighted code blocks indicate the changes introduced.
Figure 3. A selection of evolved MiniGrid environments produced by COvolve . Complexity increases from empty grids to larger mazes with dense walls and locked doors requiring corresponding keys. The agent must reach the green goal tile, often by planning multi-step sequences of key retrieval and door unlocking.
Figure 4. A selection of evolved PyGame environments produced by COvolve . Tasks progress from open arenas to cluttered maps with dense obstacles and narrow corridors. The agent must reach the rectangular goal zone while navigating collision-free paths through increasingly constrained layouts.
Figure 5. Selected CARLA environments produced by COvolve . Tasks progress from urban driving on empty roads to crowded streets with increasingly aggressive actor behaviors. The task for the agent is to drive along the street while following traffic rules (such as stopping at red lights), and at the same time, adjust to increasingly unpredictable behaviors of fellow drivers and pedestrians.
Figure 6. Performance during environment–policy co-evolution. Left: Success rates of all discovered policies evaluated on all environments generated during evolution (policy–environment payoff matrix). Center: Comparison between the mixed-strategy Nash equilibrium (MSNE) policy mixture, the best single policy πargmax , and the latest policy πk , evaluated on the environment archive {θ0,…,θk} . Here, πargmax denotes the policy that maximizes mean performance over the entire archive. For MiniGrid, πargmax=πk . Right: Mean success over {θ0,…,θk} for three strategies: UED-Greedy (latest policy only), UED-Uniform (uniform mixture over all policies), and COvolve (MSNE mixture). As evolution progresses and the latest policy forgets earlier environments, the MSNE mixture assigns probabilities to earlier policies to preserve worst-case performance over the archive.
Environment
UED-Greedy
UED-Uniform
COvolve
MiniGrid-MultiRoom-N6-v0
1.00±0.00
0.86±0.06
1.00±0.00
DoorKey-16x16-v0 (MiniGrid)
1.00±0.00
0.62±0.24
1.00±0.00
LockedRoom-v0 (MiniGrid)
1.00±0.00
0.66±0.16
1.00±0.00
Town02 (CARLA)
0.62±0.09
0.13±0.06
0.71±0.05
Table 1. Generalization to unseen environments. Results are reported as mean ± standard deviation across evolutionary seeds, with 100 evaluation episodes per seed.
Figure 7. Result on curriculum learning. Direct training on the hardest environment (“Zero-shot”) fails, while co-evolutionary MSNE with curriculum ( right bars ) yields non-trivial performance.
Figure 8. Ablation with a weaker generative model (GPT-4.1). Comparison between UED-Greedy and COvolve (MSNE) across domains. While overall performance degrades due to weaker generation, MSNE consistently mitigates forgetting and maintains robustness across the environment archive.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
0
Unseen
1
Empty
2
Wall
3
Floor
4
Door
5
Key
6
Ball
Appendix
Table 2. MiniGrid OBJECT_IDX Mappings
0
Open (passable)
1
Closed (toggle to open)
2
Locked (requires key to unlock and toggle)
Appendix
Table 3. Door State Field
Figure 9. Example of a generated MiniGrid environment (cf. Fig. 3 ). For this environment, the agent (red arrow) must reach the green goal tile by unlocking the intermediate colored doors using the corresponding keys.
Figure 10. Example of a generated PyGame environment (cf. Fig. 4 ). In this environment, the agent (blue circle) must navigate the environment spatially to reach the goal (green rectangle).
Figure 11. Example of a generated CARLA environment (cf. Fig. 5 ). From the ego viewpoint, the car perceives pedestrians, traffic lights, and other vehicles as described in Appendix B.3 . The red car to the right of the ego vehicle (Tesla v3) was intentionally placed (from the LLM) to confuse the policy.
Figure 12. Examples of previously unseen standardized environments used to validate generalization. MiniGrid snapshots ( top ): DoorKey , LockedRoom , and the hardest goal-reaching benchmark ObstructedMaze-Full . CARLA Town02 ( bottom ).
Figure 13. Training curves of PPO ( top ) and QRDQN ( bottom ) across MiniGrid environments. The y-axis represents reward, and the x-axis represents total training steps.
Algorithm
Env 0
Env 2
Env 6
PPO
12.0±1.4 %
0.0±0.0 %
0.0±0.0 %
QRDQN
68.5±12.0 %
0.0±0.0 %
0.0±0.0 %
Appendix
Table 4. Success rates (%, mean ± std over two runs) across MiniGrid environments.
Figure 14. Training curves of PPO (top) and SAC (bottom) across PyGame environments. The y-axis represents reward, and the x-axis represents total training steps.
Algorithm
Env 0
Env 2
Env 6
PPO
61.7±2.3 %
6.2±0.7 %
0.0±0.0 %
SAC
88.2±3.0 %
22.5±5.9 %
0.0±0.0 %
Appendix
Table 5. Success rates (%, mean ± std over two runs) across PyGame environments.
Figure 15. Performance during environment–policy co-evolution for the second seed. Left: Success rates of all discovered policies evaluated on all environments generated during evolution (policy–environment payoff matrix). Center: Comparison between the mixed-strategy Nash equilibrium (MSNE) policy mixture, the best single policy ( πargmax ), and the latest policy ( πk ), evaluated on all environments {θ0,…,θk} for Pygame, πargmax coincides with πk . Right: Mean success over environments {θ0,…,θk} for three strategies: UED-Greedy (latest policy only), UED-Uniform (uniform mixture over all policies so far), and COvolve (MSNE mixture).
Reinforcement learning for LLM agents is typically conducted on a static data distribution, which fails to adapt to the agent's evolving behavior and leads to poor coverage of complex environment interactions. To address these challenges, we propose CoEvolve, an agent-data mutual evolution framework that enables LLM agents to improve through closed-loop, interaction-driven training. Specifically, CoEvolve extracts feedback signals such as forgetting and uncertainty from rollout trajectories to identify failure-prone interaction patterns, and utilizes them to guide LLM-based task synthesis. The synthesized tasks are validated through environment interaction and utilized to update the data distribution, enabling joint adaptation of the agent and its data. Extensive experiments on AppWorld and BFCL across Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B demonstrate consistent and significant improvements over strong base models, yielding absolute gains of 19.43%, 15.58%, and 18.14%, respectively.
Large language model agents increasingly operate in dynamic environments where tool interfaces, APIs, and user requirements change after deployment. Existing self-evolution methods mainly follow two paradigms: harness-based approaches, which externalize feedback into editable memories or skills for rapid adaptation, and parameter-based approaches, which internalize experience into model parameters for deeper capability improvement. However, using either mechanism alone creates a trade-off between flexibility and performance. This paper asks how an agent can coordinate both channels to achieve robust self-evolution. We present COVE, a unified agent self-evolution framework that combines harness-based and parameter-based learning through task-aware routing, stage-aware scheduling, and knowledge optimization. Through this design, COVE treats self-evolution not as indiscriminate accumulation of experience, but as a coordinated process that matches tasks and knowledge types to appropriate learning mechanisms. Experiments across multiple task categories show that COVE outperforms single-channel evolution strategies, demonstrating more robust and efficient improvement under changing environments.
Tianyun Ji, Zhenya Huang, Jiayu Liu +3
University of Science and Technology of China Hefei, Anhui, China · City University of Hong Kong Hong Kong, China · Hefei Normal University Hefei, Anhui, China +1
Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization. To address these limitations, this paper introduces Role-Agent, \textcolor{black}{a framework} that harnesses a single LLM to function concurrently as both the agent and the environment, enabling a bootstrapped co-evolution. Role-Agent comprises two synergistic components: World-In-Agent (WIA) and Agent-In-World (AIW). In WIA, the LLM acts as the agent and predicts future states after each action; the alignment between predicted and actual states is then used as a process reward, encouraging environment-aware reasoning. In AIW, the LLM analyzes failure modes from failed trajectories and retrieves tasks with similar failure patterns, thereby reshaping the training data distribution for targeted practice. Experiments on multiple benchmarks show that Role-Agent consistently improves performance, yielding an average gain of over 4% over strong baselines.
Xucong Wang, Ziyu Ma, Shidong Yang +4
University of Science and Technology of China · AMAP, Alibaba Group