Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

Yihao Hu; Zhihao Wen; Yitong Lin; Siyuan Lu; Pan Wang; Wei Wu; Jiang-Ming Yang; Tao Lin

cs.CLOct 24, 2026

Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

Authors: Yihao Hu, Zhihao Wen, Yitong Lin, Siyuan Lu, Pan Wang, Wei Wu, Jiang-Ming Yang, Tao Lin

Organizations: Westlake University (westlake.edu) · Ant International (ant-intl.com)

Abstract

Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or contain actions that are unjustified by the information visible to the agent at decision time. We identify two coupled requirements for effective trajectory supervision: compositional coverage, so that important observation-action dependencies are represented, and decision grounding, so that each supervised action is supported by the learner-visible interaction history. We introduce Trajectory2Gym, which constructs executable interactions around under-supported compositional structures and derives supervision only from grounded decisions in their realized executions. Structural-support selection allocates the resulting experience, while parent-normalized supervised fine-tuning preserves the semantic interaction as the unit of training weight. This reframes trajectory generation as the deliberate construction of learnable interaction experience, rather than the collection of successful trajectories. Across five Qwen3 scales, Trajectory2Gym improves Tau1, Tau2, and BFCL at every tested scale, increasing the equal-weight three-benchmark mean by 16.9-29.8 percentage points. The gains are strongest in stateful and multi-turn interaction, where later decisions depend on earlier observations and feedback. Our results suggest that scaling tool-use agents depends not only on collecting more successful trajectories, but on constructing interactions whose dependencies are represented and whose decisions are learnable from the agent's visible history.

Figures & tables

Figure 1: Compositional interaction requires grounded decisions. Top: Trajectory2Gym constructs linked interactions (1) and checks each training action against that history (2). Bottom: a Tau2 Retail user requests a larger bottle and a dimmer lamp, then confirms only the lamp. Both rules judge the same two candidate actions. Filtering by execution keeps both because native tool replay accepts both requests. Trajectory2Gym uses catalog evidence and the latest confirmation to keep the lamp exchange and reject the additional bottle exchange. Green checks mark correct selection decisions; the red cross marks erroneous retention. This schematic illustrates supervision selection with paraphrased dialogue and a constructed second candidate.

Closed-source references
Model	Tau1			Tau2				Mean	$\Delta$
Model	Retail	Airline	Avg.	Retail	Airline	Telecom	Avg.
GPT-5.2 (high)	75.44	63.00	69.22	81.58	83.00	89.69	84.76	76.99
Claude-4.5-Opus (high)	79.39	67.50	73.45	79.61	84.00	92.32	85.31	79.38
Re-evaluated agent-training checkpoints
Simia Tau-SFT (Qwen3-8B)	45.22	28.00	36.61	62.50	50.00	15.00	42.50	39.55

Table 1: Tau1 and Tau2 results at all five scales (Pass@1, %; higher is better). Blue rows are Trajectory2Gym-trained;

\Delta

is the paired gain in Mean, the equal average of the two benchmark averages. Tau1 Airline is a seen-task, transductive setting (Appendix A ). Agent-training checkpoints are re-evaluated on the same complete suites: Simia [ Li et al., 2025b ] , MUA-RL [ Zhao et al., 2025 ] , LoopTool [ Zhang et al., 2026a ] , EnvFactory [ Xu et al., 2026 ] , and MidTool-RL [ Jiang et al., 2026 ] (all evaluated checkpoints in Appendix F.1 ); closed-source rows are reported references.

Closed-source references
Model	Non-live				Live			Multi-turn	Overall	$\Delta$
Model	Simple	Par.	Mult.	Irrel.	Simple	Mult.	Irrel.	Base
GPT-5.5	91.50	90.50	88.50	88.75	73.64	73.79	83.82	60.00	75.63
Gemini-3.0-Pro	94.50	91.00	94.00	82.50	87.60	80.44	73.19	69.00	79.97
Agent-training references
ToolACE-2-8B	88.00	92.50	92.50	95.41	70.93	79.01	84.80	49.00	73.12

Table 2: BFCL results (%; higher is better). Overall equally averages the four-category Non-live mean, the three-category Live mean, and Multi-turn Base. Blue rows are Trajectory2Gym-trained;

\Delta

is the paired gain in Overall. Closed-source rows and ToolACE-2-8B follow Gecko Table 3 [ Zhang et al., 2026c ] without GATS test-time refinement; the other agent-training checkpoints are re-evaluated as in Table 1 . Par., Mult., and Irrel. abbreviate the Parallel, Multiple, and Irrelevance categories; a Parallel score of 0.00 is 0/100 correct, not missing data.

Compositional interaction synthesis
Setting	Tau1	Tau2	BFCL	Mean	Gain
Existing candidate pool	51.00	75.83	80.39	69.08	–
+ Targeted expansion	58.61	81.67	77.61	72.63	+3.55
Decision-grounded supervision
Loose success-derived targets	48.13	71.67	75.86	65.22	–
+ Decision grounding	51.13	74.17	78.62	67.97	+2.75

Table 3: Matched component ablations on Qwen3-8B (%). Each pair compares one Trajectory2Gym design choice with its stage-matched alternative. Mean equally averages Tau1, Tau2, and BFCL; Gain is the Trajectory2Gym choice minus its alternative, so positive values favor the proposed component. Matching conditions are given in Appendix F.3 .

Figure 3: Structural vs. random selection at a fixed 8B budget (14,819 parents, equal target tokens). (a) Canonical paths covered. (b) Score change of structural selection (points). Structural selection covers 942 more paths and improves both Tau benchmarks, with BFCL essentially unchanged.

Depth	Trajectories	Target tokens	Tau1	Tau2	BFCL	Mean
2	13,986	18.02M	55.61	82.50	78.74	72.28
3	14,819	19.42M	57.00	75.83	78.81	70.55
4	15,805	20.61M	54.00	75.00	79.31	69.44

Table 4: Sensitivity to structural-support depth at Qwen3-8B (%). Depth controls the amount of repeated support allocated to certified interaction structures. These control-specific runs use a different training view from the headline 8B checkpoint of Tables 1 and 2 .

Figure 4: Same-task comparison of evidence use and action order. Simia overlooks visible evidence; MUA-RL violates a state-dependent ordering; Trajectory2Gym reuses the evidence and preserves the required dependency (native rewards 0, 0, and 1). Interactions are independently generated and abridged for display; a step-by-step account appears in Appendix G.1 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

MUA-RL
Model	Tau1	Tau2	BFCL	Mean
8B	34.65	25.83	58.17	39.55
14B	47.57	45.00	64.05	52.21
32B	54.04	45.00	72.48	57.17
Simia (Qwen3-8B)
Tau-SFT	36.61	42.50	43.08	40.73

Appendix

Table 5: Complete public-checkpoint evaluation (%). Tau scores are domain macro averages; Mean equally averages Tau1, Tau2, and BFCL. Trajectory2Gym-8B is shown for reference.

Figure 5: Grounded supervision within a compositional interaction. A failed email lookup remains visible context without receiving imitation loss. Subsequent visible evidence supports identity recovery, cancellation, address modification, and catalog-grounded item updates. Dialogue and tool fields are abridged for display.

Decision	Assistant decision	Preceding visible evidence	Supervision
0	Ask for identity	User requests cancellations and updates	Supervised
1	Lookup by email	User-supplied email; tool returns user not found	Context only
2–4	Recover identity	Error, then user-supplied name and ZIP 80281	Supervised
5–7	Read two orders; confirm cancellations	User gives both order IDs; reads show pending status	Supervised
8–9	Cancel both orders	User explicitly confirms both cancellations	Supervised
10–14	Report results; confirm and change address	Third order is pending; user supplies and confirms the address	Supervised

Appendix

Table 6: Decision-level supervision in the multi-order Retail case. The failed lookup uses an email supplied by the user: its argument is visible, but its execution fails. The call and error remain context; user messages and tool responses receive no imitation loss.

Figure 6: Changing intent during a stateful interaction. Simia submits an exchange before a later return request, which the changed order state prevents. In the independently generated Trajectory2Gym dialogue, the user changes the request before submission and the return succeeds. The different histories make this a behavior illustration, not an identical-context comparison.

Figure 7: Delegating phone-side actions and using feedback. MUA-RL repeatedly attempts phone-side operations unavailable to the assistant. Trajectory2Gym delegates those operations to the user and adapts its continuation to the reported device state and connection speed, completing the task.

Figure 8: Recovery enables dependent operations. After a route error, Simia proposes a lookup but does not complete the booking. Trajectory2Gym retrieves the needed airport codes, books the flight, and reuses the returned booking identifier for invoice and support calls. Only the Trajectory2Gym interaction satisfies the native task checker.

Figure 9: An early return blocks later exchange attempts. LoopTool submits a return before confirmation, leaving later exchange and cancellation attempts invalid. Trajectory2Gym preserves the delivered state while obtaining confirmation, then submits the user-selected exchange.

Figure 10: A correct fare does not complete the workflow. Both agents report the correct fare, but Simia does not execute the requested booking. Trajectory2Gym books the flight and uses its returned identifier for the subsequent insurance operation, completing the dependent workflow.

Figure 11: Remaining failure against a successful external baseline. Simia completes payment, line resumption, and device repair. Trajectory2Gym transfers after sending a payment request, leaving the connectivity and overdue-bill checks unsatisfied. This case retains an external-agent success and a Trajectory2Gym failure.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

Mar 2, 2026

TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training

Training tool-use agents typically relies on outcome-based filtering: Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks. However, this paradigm ignores interaction dynamics: successful trajectories may lack error recovery or exhibit redundancy, while pass rates fail to distinguish structurally informative tasks from trivial ones. We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology. By merging equivalent action-observation states, this projection transforms scattered linear trajectories into a structured manifold that explicitly captures how tool invocations and environmental responses drive the divergence between effective strategies and failure modes. Leveraging this representation, we introduce a dual-selection mechanism: for SFT, we prioritize trajectories demonstrating reflective recovery, semantic efficiency, and strategic diversity to mitigate covariate shift and mode collapse; for RL, we select tasks with high error branch ratios and strategic heterogeneity, maximizing gradient Signal-to-Noise Ratio to address vanishing signals in sparse-reward settings. Evaluations on BFCLv3 and Tau2 Bench show that TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines. We will release the code and data soon for further investigations.

Jinluan Yang, Yuxin Liu, Zhengyu Chen +7

Zhejiang University, Hangzhou, China · Meituan, Beijing, China.

LLM Agent Training RL for Tool Use Trajectory Representation Learning

Ask PDF

Mar 2, 2026

CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification

Developing multi-turn interactive tool-use agents is challenging because real-world user needs are often complex and ambiguous, yet agents must execute deterministic actions to satisfy them. To address this gap, we introduce CoVe (Constraint-Verification), a post-training data synthesis framework designed for training interactive tool-use agents while ensuring both data complexity and correctness. CoVe begins by defining explicit task constraints, which serve a dual role: they guide the generation of complex trajectories and act as deterministic verifiers for assessing trajectory quality. This enables the creation of high-quality training trajectories for supervised fine-tuning (SFT) and the derivation of accurate reward signals for reinforcement learning (RL). Our evaluation on the challenging

τ^2

-bench benchmark demonstrates the effectiveness of the framework. Notably, our compact CoVe-4B model achieves success rates of 43.0% and 59.4% in the Airline and Retail domains, respectively; its overall performance significantly outperforms strong baselines of similar scale and remains competitive with models up to

17\times

its size. These results indicate that CoVe provides an effective and efficient pathway for synthesizing training data for state-of-the-art interactive tool-use agents. To support future research, we open-source our code, trained model, and the full set of 12K high-quality trajectories used for training.

Jinpeng Chen, Cheng Gong, Hanbo Li +9

Huawei Research · Independent Researcher

Tool-Using Agents LLM Agent Training LLM Agent Trajectory Synthesis RL for Tool Use

Ask PDF

Oct 16, 2026

Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents

Tool-using LLM agents repeatedly face decision points under uncertainty, choosing between autonomous action, additional information gathering, and human escalation. These decisions require balancing task success against interaction cost and the risk of irreversible errors. Learning to make these trade-offs is challenging because existing interaction trajectories record only the outcomes of actions taken, leaving the consequences of alternative actions unobserved. To address this gap, we construct a counterfactual dataset containing 3,115 action outcomes from 1,036 decision points in ToolSandbox. At each decision point, we execute each legal action and evaluate its downstream outcomes based on task completion, interaction cost, and irreversible errors. Using this dataset, we train SURE-Act, a controller that selects actions for a frozen backbone agent at inference time. Our experiments show that learning to rank alternative actions improves offline action-selection utility by 0.0309 over a regression-only baseline. We further show that adding SURE-Act enables end-to-end gains of up to 0.0575 in ToolSandbox's Overall Score with the Qwen3-4B agent. Moreover, SURE-Act outperforms adapted decision rules from SAGE-Agent and Calibrate-Then-Act by 0.1506 and 0.1438, respectively. This work establishes the first counterfactual dataset for ToolSandbox and provides actionable insights into decision-making under uncertainty for tool-using agents.

Qisai Liu, Qingyue Jiao, Bangwei Guo +7

Iowa State University · University of Notre Dame · Rutgers University +2

Tool-Using Agents LLM Decision-Making Tool Selection for Language Model Agents LLM Agent Reliability Counterfactual Learning

Ask PDF

Abstract

Tool-Using Agents

Explore similar work

TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training

CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification

Agents' Last Doubt: Knowing When to Act and When to Step Back in Tool-Using Agents