Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs), but most existing RL frameworks stop at the policy update. For every new domain, the user is left with two hard systems problems: standing up an isolated environment for each of hundreds of concurrent trajectories and connecting it to training, and scheduling the rollout so that the GPU stays busy across long, multi-turn episodes that spend much of their time stalled on slow tool calls. We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both. It uses the Model Context Protocol (MCP) as the interface to the environment, so any tool already exposed as an MCP server plugs into training with no RL-specific integration code. It builds the two missing layers once and reuses them across domains: an environment-orchestration layer that provisions, isolates, and recycles the MCP environments over a pluggable container backend, and a rollout-orchestration layer whose staged pipeline overlaps trajectories to keep the GPU busy while episodes wait on tools. A backend-agnostic training layer then applies the update through an existing RL backend, with veRL and slime integrations. With one configuration, changing only the task specification, we train software-engineering, deep-research, and general tool-use agents on gpt-oss-20b and improve task reward in all three.
Figures & tables
Figure 1: Overview of our MCP-Universe RL framework. Each domain is exposed as one or several MCP servers, and all domains funnel through a single MCP interface into one shared training loop, whose environment-orchestration, rollout-orchestration, and training-engine layers are reused for every domain; only the task specification changes.
Figure 2: Per-stage concurrency in the rollout pipeline. Each row is one trajectory, flowing Acquire → Run → Eval left to right; block width is relative wall-clock. (a) Coupled: when every stage is given the same number of workers, a trajectory that finishes Acquire must wait for a free Run worker, leaving idle gaps. (b) Decoupled: giving the GPU-bound Run stage more workers than the fast Acquire stage lets trajectories flow through with less idle.
Figure 4: Software-engineering training on R2E-Gym. Left axis (solid, blue): reward; right axis (dashed, orange): average response length in thousands of tokens.
Figure 5: Deep-research training on DeepDive. Axes as in Figure 4 .
Figure 6: General-tool-use training on AgentWorldModel-1k. Axes as in Figure 4 .
Figure 7: Per-stage concurrency. Rollout throughput as the number of run workers grows, with the acquire stage and live-environment count fixed. The dotted line marks the coupled point (run = acquire); adding run workers is the decoupled setting.
GPU placement
Time / step (s)
Throughput (steps/h)
Colocated
1164
3.1
Fully asynchronous
572
6.3
Table 1: End-to-end training throughput under the two GPU placements, same engine and workload.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Policy model
gpt-oss-20b
RL algorithm
GRPO
Learning rate
2×10−6
Batch Size
16
Trajectories per task ( n )
8
Reward
binary, from evaluator
Appendix
Table 2: Hyperparameters shared across all three domains.
The Model Context Protocol provides a shared interaction substrate for the agentic web, allowing heterogeneous services to expose capabilities through a standardized interface. Learning to use this ecosystem entails a hierarchy of capabilities: tool-format skills, server-level transfer, and cross-server composition. We present MCPDisCo, an ecosystem-scale reinforcement learning framework for MCP-native agents. Using a Thought-Markov Decision Process (Thought-MDP), the framework explicitly represents internal reasoning alongside routing, execution, and termination and trains on real-server, long-horizon interactions spanning 70 MCP servers and 527 tools across six domains. We study the reasoning--tool seesaw, a joint-learning failure mode in which improvement in one objective can accompany deterioration in the other. To mitigate potential interference, MCPDisCo combines token-masked reward attribution with bidirectional gradient surgery: the former separates direct supervision for reasoning and action spans, while the latter removes conflicting components from both objective gradients in shared parameters. MCPDisCo achieves higher success with fewer chain-of-thought tokens. On the out-of-distribution MCP-Atlas and the GAIA benchmark, MCPDisCo maintains monotone gains, confirming that the learned behavior generalizes beyond the training server distribution and broader agentic tasks. Model weight and implementation are available.
Kounianhua Du, Jianxing Liu, Shuai Shao +3
Shanghai Jiao Tong University, Shanghai, China · Tencent. Shanghai, China
Reinforcement learning has become a prevalent paradigm for training tool calling agents, which typically requires online interactive environments. Existing approaches either rely on training data with ground truth annotations or require advanced proprietary language models (LMs) to synthesize environments that remain fixed once created. In this work, we propose TRUSTEE, a cost-friendly method for training tool calling agents with dynamic environments fully simulated by open-source LMs that can be as small as 8B, including task generation, user simulation, tool simulation, and trajectory evaluation, paired with an adaptive curriculum learning mechanism that controls task difficulty dynamically. Our empirical results show that TRUSTEE outperforms baselines which require extra external resources in most cases. These confirm that, with a sufficiently sophisticated design, even simulated environments with a local 8B LM as the backbone could set a strong baseline for tool learning. We hope our proposed paradigm could democratize tool learning and inspire future research on environment scaling with limited resources.
Chenming Tang, Hsiu-Yuan Huang, Weijie Liu +3
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent
Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.