What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents

Xingshan Zeng; Zishan Xu; Boju Zhang; Yuzhou Wu; Lingzhi Wang; Jianghao Lin; Liangyou Li; Yasheng Wang

cs.AIAug 27, 2026

What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents

Authors: Xingshan Zeng, Zishan Xu, Boju Zhang, Yuzhou Wu, Lingzhi Wang, Jianghao Lin, Liangyou Li, Yasheng Wang,

Organizations: Huawei Technologies Co., Ltd · Shanghai Jiao Tong University · Northwestern University, Chicago, IL · Harbin Institute of Technology, Shenzhen ·

Abstract

LLM agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object $(E,q,τ,v)$ , comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as agents and environments evolve.

Figures & tables

Figure 1: The ACE lens views agentic data generation as constrained distribution design: Accuracy defines valid support, while learner-relative Complexity and Diversity shape useful, non-redundant experience over $(E,q,\tau,v)$ .

Figure 2: Cross-domain instantiations of the common agentic data object

d=(E,q,\tau,v)

. The factors retain the same conceptual roles even though their concrete representations and verification signals differ across domains.

Figure 3: Generation Paradigms: Forward Generation vs Reverse Generation.

Table 1: Representative works in forward agentic data generation. All methods follow the dependency

E\rightarrow q\rightarrow\tau

. Methods are grouped according to the dominant source of environment construction: real or curated environments, LLM-synthesized environments, and programmatic or executable environments. Resource links to public GitHub repositories or Hugging Face websites when available.

Table 2: Representative works in reverse agentic data generation. Methods are grouped into task-first, trajectory-first, and structure-first generation, followed by adaptive and self-evolving approaches as a cross-cutting extension. Domain indicates the primary application setting, while Resource links to public repositories when available.

Figure 4: Four complementary factors of accuracy: environment, task, interaction, and verifier/outcome accuracy.

Figure 5: Four major research directions and emerging trends in accuracy assurance for agentic data generation.

Figure 6: Learner-relative task complexity. Learning utility is maximized within a learnable band between tasks that are too easy and too difficult. This band can also shift with learner capability.

Figure 7: Factor-level complexity in agentic data across environments, tasks, interactions, and completion criteria.

Figure 8: Four complementary factors of diversity: environment, task, interaction, and generator/provenance.

Figure 9: Domain-specific instantiations of diversity across diverse kinds of agents. Meaningful diversity should be characterized by domain-specific behavioral units rather than surface-level variation alone.

Method	Domain	Key Mechanism	Resource
Tool-use and Digital Agents
ToolLLM ( Qin et al., 2024 )	Tool Use	Large-scale coverage of real-world APIs and tool capabilities	GitHub
APIGen ( Liu et al., 2024 )	Tool Use	Verifiable function-calling generation across diverse APIs, tasks, and executions	GitHub
ToolACE ( Liu et al., 2025 )	Tool Use	Diversification of tool pools, function-calling tasks, and action representations	Hugging Face
ToolFlow ( Wang et al., 2025b )	Tool Use	Coherent tool dependencies and multi-turn information flow through structured synthesis	–
ACE-Router ( Yao et al., 2026 )	Tool / MCP	History-aware routing across heterogeneous MCP tools and interaction histories	GitHub

Table 3: Representative works on domain-specific diversity in agentic data generation. Methods are grouped by their primary application domain. Key Mechanism summarizes the domain-specific source of meaningful behavioral diversity, while Resource links to public GitHub repositories or Hugging Face websites when available.

Method	Domain	Key Mechanism	Resource
Embodied and Social Agents
ProcTHOR ( Deitke et al., 2022 )	Embodied	Procedural generation of diverse scenes, objects, layouts, and embodied environments	GitHub
Holodeck ( Yang et al., 2024 )	Embodied	Language-guided generation of interactive 3D environments and scene configurations	GitHub
RoboCasa ( Nasiriany and others, 2024 )	Embodied	Large-scale simulation of everyday scenes, objects, affordances, and manipulation tasks	GitHub
Automatic Cognitive Task Generation ( He et al., 2026 )	Embodied	Generation of activities and task structures under physical and environmental constraints	–
EmbodiedGen V2 ( Wang et al., 2026a )	Embodied	Simulation-ready 3D world generation with diverse scenes and physical interaction conditions	GitHub

Table 3: Representative works on domain-specific diversity in agentic data generation (continued).

Synthetic Data Generation for LLM Agents

See all 28 papers

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI
Sep 8, 2026Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie FuAI Agent Reliability Distribution Shift
School of Computing and Augmented Intelligence, Arizona State University, USA
Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study
Oct 9, 2026Ruomeng Ding, Qianli Shen, Zhaoyang Han +3Autonomous Research Agents LLM Training
University of North Carolina at Chapel Hill · Alibaba Group · Peking University
Scaling Agentic Capabilities via Grounded Interaction Synthesis
Jun 1, 2026Wenhang Shi, Jinhao Dong, Yiren Chen +4Synthetic Data Generation Synthetic Data Generation for LLM Agents
Renmin University of China · Peking University · Tencent

Sep 8, 2026J/K move · Enter open · S save

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie Fu

School of Computing and Augmented Intelligence, Arizona State University, USA

AI Agent Reliability Distribution Shift LLM Interpretability LLM Agent Reliability

Open paper AskPDF arXiv

Agentic AI systems built on large language models fail in two persistent ways that scaling does not fix: they break under distribution shift, and they cannot explain the decisions they make. We argue these are co-symptoms of one structural deficiency in the data lifecycle that governs how agents are trained, evaluated, and deployed. Observational interaction logs record what an agent did, not what it would have done otherwise. They encode spurious correlations without controlled variation, so they lack the counterfactual structure needed to separate causal signal from coincidence or to validate an explanation. No model-centric method can recover invariances the data never contained. We present Data-Centric Anchoring: robustness and interpretability should be engineered into the data environment, not extracted from models after training. Our central contribution is the Data-Centric Agentic Loop, a four-stage framework of Curate, Augment, Constrain, and Attribute. The ordering is structural, not stylistic. Curation precedes augmentation because generative models amplify whatever bias they are trained on. Augmentation precedes constraint because invariance objectives are vacuous without variation across environments to be invariant to. Attribution closes the loop, converting observed failures into targeted data interventions for the next iteration. Each stage manufactures the preconditions of the next, which makes the loop self-correcting rather than merely sequential. We ground the framework in a failure-driven taxonomy that links four core failure modes to the data lifecycle: spurious feature reliance, distribution-shift fragility, uncertainty miscalibration, and explanation unfaithfulness. We close with the limits of this approach and the open problems that stand between it and practical deployment at scale.

Abstract

Synthetic Data Generation for LLM Agents

Explore similar work

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

Scaling Agentic Capabilities via Grounded Interaction Synthesis