LOGIGEN: Logic-Driven Generation of Verifiable Agentic Tasks

Yucheng Zeng; Weipeng Lu; Linyun Liu; Shupeng Li; Zitian Qu; Chenghao Zhu; Shaofei Li; Zhengdong Tan

cs.AIFeb 28, 2026

LOGIGEN: Logic-Driven Generation of Verifiable Agentic Tasks

Authors: Yucheng Zeng, Weipeng Lu, Linyun Liu, Shupeng Li, Zitian Qu, Chenghao Zhu, Shaofei Li, Zhengdong Tan,

Organizations: Baidu Inc. · Tsinghua University

Abstract

The evolution of Large Language Models (LLMs) from static instruction-followers to autonomous agents necessitates operating within complex, stateful environments to achieve precise state-transition objectives. However, this paradigm is bottlenecked by data scarcity, as existing tool-centric reverse-synthesis pipelines fail to capture the rigorous logic of real-world applications. We introduce LOGIGEN, a logic-driven framework that synthesizes verifiable training data based on three core pillars: Hard-Compiled Policy Grounding, Logic-Driven Forward Synthesis, and Deterministic State Verification. Specifically, a Triple-Agent Orchestration is employed: the Architect compiles natural-language policy into database constraints to enforce hard rules; the Set Designer initializes boundary-adjacent states to trigger critical policy conflicts; and the Explorer searches this environment to discover causal solution paths. This framework yields a dataset of 20,000 complex tasks across 8 domains, where validity is strictly guaranteed by checking exact state equivalence. Furthermore, we propose a verification-based training protocol where Supervised Fine-Tuning (SFT) on verifiable trajectories establishes compliance with hard-compiled policy, while Reinforcement Learning (RL) guided by dense state-rewards refines long-horizon goal achievement. On $τ^2$ -Bench, LOGIGEN-32B(RL) achieves a 79.5% success rate, substantially outperforming the base model (40.7%). These results demonstrate that logic-driven synthesis combined with verification-based training effectively constructs the causally valid trajectories needed for next-generation agents.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Performance comparisons. LOGIGEN-32B (RL) achieves a 79.5% success rate on $\tau^{2}$ -Bench, significantly outperforming open-weight baselines and remaining competitive with proprietary models. It also surpasses general-purpose baselines with substantially larger parameters. (b) Performance breakdown. The gains stem from our verification-based training protocol: SFT on verifiable trajectories establishes compliance with hard-compiled policy, while RL guided by deterministic state-rewards refines long-horizon goal achievement.

Dataset	Policy Encoding	Env. Grounding	Init. Sampling	Task Synthesis	Verification
API-Bank ToolACE TOUCAN	Tool/API/MCP specs (no explicit policy)	Mocked or external tool execution (no released local stateful env)	N/A (Stateless)	Spec-conditioned LLM task synthesis (tool-first / reverse) (Training Data)	Rule / metric checks (Parsers / regex) (no state-diff)
EnvScaler AgentScaler	Programmatic logic a	Executable simulator	Programmatic (state synthesis)	Tool-graph / scenario-driven (tool-first / reverse) (Training Data)	State-based checks (validators / alignment) b
$\tau$ -Bench $\tau^{2}$ -Bench	Soft (NL guidelines + env rules)	Executable env w/ stateful backend	Curated task-provided init state	Human-authored tasks (Benchmark)	Deterministic state check
LOGIGEN (Ours)	Hard-Compiled Policy	Executable DB-backed env	Boundary-adjacent initial states	Automated logic-driven (Training Data)	Deterministic state check

Table 1: Comparison from a dataset construction perspective. Hard-compiled refers to policies enforced by database constraints. Deterministic state check ensures correctness via reproducible state comparisons.

Figure 2: LOGIGEN synthesizes verifiable agentic tasks via a Triple-Agent Orchestration : ( 1 ) the Architect expands seed domain knowledge into a Wiki Policy and compiles it into a Hard-Compiled Policy Environment; ( 2 ) the Set Designer seeds boundary-adjacent initial states (

N{-}1

) to maximize logical friction; and ( 3 ) the Explorer performs goal-conditioned exploration to discover executable multi-turn episodes, producing a spoiler-free task description and a deterministic target database snapshot for state-based verification.

Core Domain	Sub-domains & Scenarios
Travel & Hospitality	Travel Agent, Hotel Concierge, Corporate Travel Booking, Hospitality & Travel Services, Event Ticketing
Retail & Supply Chain	Order Assistant, After-Sales Support, Shopping Assistant, Membership Support, Logistics Support, Supply Chain & Logistics
Healthcare & Life Sci.	Medical Receptionist, Medical Triage & Booking, Healthcare & Life Sciences Services
Finance & Insurance	Bank Teller Bot, Insurance Claims, Insurance Services, Finance & Account Steward
Education & Knowledge	University Registrar, Course Enrollment, Librarian, Skill Development
Enterprise Ops & HR	HR Assistant, HR Team Management, Public & Admin Clerk, Collaboration Creation, Compliance Process, Planning & Strategy

Table 2: Taxonomy of Core Domains and Scenarios

Panel of Figure 3 — Figure 3: Task complexity profile of LOGIGEN. (a) Distribution of the top-10 complexity tags assigned to generated tasks, showing that most samples involve conditional logic, multi-step interaction, and attribute-based selection, with a substantial portion requiring state-based reasoning and fallback (waterfall) behaviors. (b) Difficulty-level distribution, where L1 (Simple) tasks are rare (51), while the dataset is dominated by L2 (Intermediate; 6,161) and L3 (Advanced; 3,788) tasks, indicating a strong bias toward non-trivial, policy-constrained problem solving.

Figure 4: Training and Evaluation Protocol of LOGIGEN. The protocol operates on a Data Package containing the Wiki Policy (

\mathcal{P}

), Task Description (

\mathcal{I}

), and paired database snapshots. Within the Interaction Loop , a User Simulator (driven by

\mathcal{I}

) interacts with the Agent (governed by

\mathcal{P}

and tools in

\mathcal{E}

). Each tool invocation induces a deterministic mutation of the database state (

s_{\text{origin}}\rightarrow s_{\text{1}}\rightarrow s_{\text{2}}\rightarrow\dots\rightarrow s_{\text{final}}

). The Verifier computes a binary success metric

R_{final}

by performing a canonicalized State-Diff between the resulting state

s_{\text{final}}

and the ground-truth target

s_{\text{target}}

Closed-Source Large Language Models
Model	User Simulator	$\tau^{2}$ -Bench
		Retail	Airline	Telecom	Overall
Gemini-3-pro	Gemini-3-pro	85.3	73.0	98.0	85.4
Claude-Sonnet-4.5	-	86.2	70.0	98.0	84.7
GPT-5	GPT-4.1-2025-04-14	81.6	62.5	95.8	80.0
Qwen3-Max-Thinking	GPT-4.1-2025-04-14	79.4	69.0	98.2	82.2

Table 3: Main results on

\tau^{2}

-Bench.

Model	$\tau^{2}$ -Bench
Model	Retail	Airline	Telecom	Overall
Qwen3-8B	38.6	30.5	23.3	30.8
+ Cold Start	59.1 ( $\uparrow$ 20.5)	44.0 ( $\uparrow$ 13.5)	59.1 ( $\uparrow$ 35.8)	53.5 ( $\uparrow$ 22.7)
+ TA-GRPO RL	79.5 ( $\uparrow$ 40.9)	54.7 ( $\uparrow$ 24.2)	81.3 ( $\uparrow$ 58.0)	71.8 ( $\uparrow$ 41.0)
Qwen3-32B	53.3	42.0	26.9	40.7
+ Cold Start	65.3 ( $\uparrow$ 12.0)	50.8 ( $\uparrow$ 8.8)	71.9 ( $\uparrow$ 45.0)	62.7 ( $\uparrow$ 22.0)

Table 4: Incremental performance gains of training stages on

\tau^{2}

-Bench.

Figure 5: Training reward curves for LOGIGEN-8B (left) and 32B (right) models. Compared to Vanilla GRPO, TA-GRPO exhibits higher sample efficiency and achieves superior asymptotic rewards, particularly on the 8B scale.

Figure 6: Evaluation performance on

\tau^{2}

-Bench across RL training steps. TA-GRPO consistently yields higher task completion rates than Vanilla GRPO for both 8B and 32B models.

Figure 7: Trajectory-level statistics . Distributions of (left) the number of turns (user–agent interaction rounds), (middle) the number of agent messages (total assistant replies; multiple messages may occur within a single turn), and (right) the number of tool calls made by the agent across a trajectory. Bars indicate the empirical frequency (percentage), the solid curve shows a smoothed density estimate, and dashed/dash-dotted vertical lines mark the mean and median.

Figure 8: Performance comparison of Pass @ k and Pass ˆk metrics on

\tau^{2}

-Bench. The left y-axis represents Pass @ k (potential capability), while the right y-axis represents Pass ˆk (consistency).

Figure 9: Training reward curves. The agent trained with the low-temperature simulator (T=0.1) achieves higher and faster-converging rewards, indicative of over-optimization.

Figure 10: Evaluation performance on

\tau^{2}

-Bench. The diverse simulator (T=1.0) maintains robustness, whereas the deterministic simulator (T=0.1) suffers from performance degradation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Module / Agent	Success	Model	Avg.	Avg. Tokens		Avg.
	Rate (%)		Turn	Prompt	Compl.	Cost (¥)
The Architect	68.5	–	–	–	–	1.22
Design Wiki Policy	–	DS-V3.2-Think	2.7	37,410	6,255	0.09
Design Tables	–	DS-V3.2-Think	7.1	157,048	15,906	0.36
Design Triggers	–	DS-V3.2-Think	10.0	327,096	38,654	0.77
The Set Designer	72.8	DS-V3.2	41.4	795,374	5,941	1.61

Appendix

Table 5: Breakdown of Performance and Cost by Agent Module

Synthetic Data Generation for LLM Agents

See all 28 papers

Scaling Agentic Capabilities via Grounded Interaction Synthesis
Jun 1, 2026Wenhang Shi, Jinhao Dong, Yiren Chen +4Synthetic Data Generation Synthetic Data Generation for LLM Agents
Renmin University of China · Peking University · Tencent
SEVerA: Verified Synthesis of Self-Evolving Agents
Mar 26, 2026Debangshu Banerjee, Changming Xu, Eugene Ie +5Agentic Code Generation Self-Evolving Agents
University of Illinois Urbana-Champaign, USA · Google, USA
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Jun 22, 2026Zhanbo Hua, Yifan Yao, Weihao Xie +14LLM Agent Trajectory Synthesis LLM Agents
Nanjing University · StepFun · ZODA +1

Jun 1, 2026J/K move · Enter open · S save

Scaling Agentic Capabilities via Grounded Interaction Synthesis

Wenhang Shi, Jinhao Dong, Yiren Chen, Zhe Zhao+3

Renmin University of China · Peking University · Tencent

Synthetic Data Generation Synthetic Data Generation for LLM Agents LLM Agent Training Synthetic Data Generation for Language Models

Open paper AskPDF arXiv

General agentic intelligence hinges on the ability to interact with diverse real-world tools to complete complex tasks, a capability fundamentally tied to the quality of interaction data. To bypass the prohibitive costs of human annotation, prevailing paradigms depend entirely on Large Language Models (LLMs) to scale the synthesis of agentic environments and tasks. However, such unconstrained generation often degenerates into biased random sampling of LLMs' internal priors, failing to capture the diversity and difficulty of real-world domains or construct high-fidelity, long-horizon tasks. In this work, we introduce Grounded Agentic Interaction Synthesis (GAIS), a framework that automates the scalable construction of diverse environments and complex tasks via a two-phase grounding mechanism. Specifically, we construct protocol-anchored environments derived from real-world Model Context Protocol (MCP) servers to ensure functional diversity and difficulty. Subsequently, we employ structure-guided planning to navigate these environments, actively enforcing logical dependencies and adversarial policies to generate complex tasks. Experiments on BFCL,

τ^2

-Bench, and ACEBench demonstrate that GAIS-synthesized data significantly outperforms state-of-the-art baselines, enabling base models to match or even surpass their official instruction-tuned counterparts. Furthermore, GAIS exhibits superior data efficiency and scalability, achieving exceptional capabilities with significantly less data while maintaining continuous growth where baselines stagnate. Our code and dataset are publicly available at https://github.com/Eric8932/GAIS.

Abstract

Synthetic Data Generation for LLM Agents

Explore similar work

Scaling Agentic Capabilities via Grounded Interaction Synthesis

SEVerA: Verified Synthesis of Self-Evolving Agents

CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents