cs.AIAug 27, 2026

What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents

Authors: Xingshan Zeng, Zishan Xu, Boju Zhang, Yuzhou Wu, Lingzhi Wang, Jianghao Lin, Liangyou Li, Yasheng Wang, 

Organizations: Huawei Technologies Co., Ltd · Shanghai Jiao Tong University · Northwestern University, Chicago, IL · Harbin Institute of Technology, Shenzhen · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object (E,q,τ,v)(E,q,τ,v), comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as agents and environments evolve.

Figures & tables

Figure 1: The ACE lens views agentic data generation as constrained distribution design: Accuracy defines valid support, while learner-relative Complexity and Diversity shape useful, non-redundant experience over (E,q,τ,v)(E,q,\tau,v) .
Figure 2: Cross-domain instantiations of the common agentic data object d=(E,q,τ,v)d=(E,q,\tau,v) . The factors retain the same conceptual roles even though their concrete representations and verification signals differ across domains.
Figure 3: Generation Paradigms: Forward Generation vs Reverse Generation.
Table 1: Representative works in forward agentic data generation. All methods follow the dependency E→q→τE\rightarrow q\rightarrow\tau . Methods are grouped according to the dominant source of environment construction: real or curated environments, LLM-synthesized environments, and programmatic or executable environments. Resource links to public GitHub repositories or Hugging Face websites when available.
Table 2: Representative works in reverse agentic data generation. Methods are grouped into task-first, trajectory-first, and structure-first generation, followed by adaptive and self-evolving approaches as a cross-cutting extension. Domain indicates the primary application setting, while Resource links to public repositories when available.
Figure 4: Four complementary factors of accuracy: environment, task, interaction, and verifier/outcome accuracy.
Figure 5: Four major research directions and emerging trends in accuracy assurance for agentic data generation.
Figure 6: Learner-relative task complexity. Learning utility is maximized within a learnable band between tasks that are too easy and too difficult. This band can also shift with learner capability.
Figure 7: Factor-level complexity in agentic data across environments, tasks, interactions, and completion criteria.
Figure 8: Four complementary factors of diversity: environment, task, interaction, and generator/provenance.
Figure 9: Domain-specific instantiations of diversity across diverse kinds of agents. Meaningful diversity should be characterized by domain-specific behavioral units rather than surface-level variation alone.
MethodDomainKey MechanismResource
Tool-use and Digital Agents
ToolLLM ( Qin et al., 2024 )Tool UseLarge-scale coverage of real-world APIs and tool capabilitiesGitHub
APIGen ( Liu et al., 2024 )Tool UseVerifiable function-calling generation across diverse APIs, tasks, and executionsGitHub
ToolACE ( Liu et al., 2025 )Tool UseDiversification of tool pools, function-calling tasks, and action representationsHugging Face
ToolFlow ( Wang et al., 2025b )Tool UseCoherent tool dependencies and multi-turn information flow through structured synthesis–
ACE-Router ( Yao et al., 2026 )Tool / MCPHistory-aware routing across heterogeneous MCP tools and interaction historiesGitHub
Table 3: Representative works on domain-specific diversity in agentic data generation. Methods are grouped by their primary application domain. Key Mechanism summarizes the domain-specific source of meaningful behavioral diversity, while Resource links to public GitHub repositories or Hugging Face websites when available.
MethodDomainKey MechanismResource
Embodied and Social Agents
ProcTHOR ( Deitke et al., 2022 )EmbodiedProcedural generation of diverse scenes, objects, layouts, and embodied environmentsGitHub
Holodeck ( Yang et al., 2024 )EmbodiedLanguage-guided generation of interactive 3D environments and scene configurationsGitHub
RoboCasa ( Nasiriany and others, 2024 )EmbodiedLarge-scale simulation of everyday scenes, objects, affordances, and manipulation tasksGitHub
Automatic Cognitive Task Generation ( He et al., 2026 )EmbodiedGeneration of activities and task structures under physical and environmental constraints–
EmbodiedGen V2 ( Wang et al., 2026a )EmbodiedSimulation-ready 3D world generation with diverse scenes and physical interaction conditionsGitHub
Table 3: Representative works on domain-specific diversity in agentic data generation (continued).
More in this topic

Synthetic Data Generation for LLM Agents

See all 28 papers

Explore similar work

CardsList
Sep 8, 2026

Vision: Data-Centric Anchoring for Robust and Interpretable Agentic AI

Agentic AI systems built on large language models fail in two persistent ways that scaling does not fix: they break under distribution shift, and they cannot explain the decisions they make. We argue these are co-symptoms of one structural deficiency in the data lifecycle that governs how agents are trained, evaluated, and deployed. Observational interaction logs record what an agent did, not what it would have done otherwise. They encode spurious correlations without controlled variation, so they lack the counterfactual structure needed to separate causal signal from coincidence or to validate an explanation. No model-centric method can recover invariances the data never contained. We present Data-Centric Anchoring: robustness and interpretability should be engineered into the data environment, not extracted from models after training. Our central contribution is the Data-Centric Agentic Loop, a four-stage framework of Curate, Augment, Constrain, and Attribute. The ordering is structural, not stylistic. Curation precedes augmentation because generative models amplify whatever bias they are trained on. Augmentation precedes constraint because invariance objectives are vacuous without variation across environments to be invariant to. Attribution closes the loop, converting observed failures into targeted data interventions for the next iteration. Each stage manufactures the preconditions of the next, which makes the loop self-correcting rather than merely sequential. We ground the framework in a failure-driven taxonomy that links four core failure modes to the data lifecycle: spurious feature reliance, distribution-shift fragility, uncertainty miscalibration, and explanation unfaithfulness. We close with the limits of this approach and the open problems that stand between it and practical deployment at scale.

Arun Vignesh Malarkkan, Xinyuan Wang, Yanjie Fu

School of Computing and Augmented Intelligence, Arizona State University, USA

AI Agent ReliabilityDistribution ShiftLLM InterpretabilityLLM Agent Reliability
AskPDF
Oct 9, 2026

Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study

LLM data engineering requires iterative experimentation to determine how data selection, rewards, and curricula affect model learning, making it a natural setting for AI-for-AI (AI4AI) research. Can agents use experimental feedback to progressively improve strategies, and do the resulting strategies improve target-model performance on held-out data? We study how agents use experimental feedback to develop data-engineering strategies under fixed training and evaluation protocols. Our Agentic Data Engineering (ADE) framework lets agents pursue, revise, or combine research directions using cumulative findings. Strategies are selected by in-loop scores and evaluated on held-out benchmarks. Across four tasks-math and code data selection, math reward design, and math curriculum learning-selected strategies achieve absolute gains over generic baselines of up to 13.34% on AIME25 Pass@32 and 5.85% averaged across four held-out benchmarks under fixed training conditions. In-loop scores improve over successive experiments, while held-out trajectories show that later in-loop gains do not always improve generalization. Comparisons of research organization and feedback access favor combining evidence analysis with cumulative findings. Research trajectories show how earlier findings inform strategy revisions and combinations across research directions.

Ruomeng Ding, Qianli Shen, Zhaoyang Han +3

University of North Carolina at Chapel Hill · Alibaba Group · Peking University

Autonomous Research AgentsLLM TrainingLLM Agent Self-Improvement
AskPDF
Jun 1, 2026

Scaling Agentic Capabilities via Grounded Interaction Synthesis

General agentic intelligence hinges on the ability to interact with diverse real-world tools to complete complex tasks, a capability fundamentally tied to the quality of interaction data. To bypass the prohibitive costs of human annotation, prevailing paradigms depend entirely on Large Language Models (LLMs) to scale the synthesis of agentic environments and tasks. However, such unconstrained generation often degenerates into biased random sampling of LLMs' internal priors, failing to capture the diversity and difficulty of real-world domains or construct high-fidelity, long-horizon tasks. In this work, we introduce Grounded Agentic Interaction Synthesis (GAIS), a framework that automates the scalable construction of diverse environments and complex tasks via a two-phase grounding mechanism. Specifically, we construct protocol-anchored environments derived from real-world Model Context Protocol (MCP) servers to ensure functional diversity and difficulty. Subsequently, we employ structure-guided planning to navigate these environments, actively enforcing logical dependencies and adversarial policies to generate complex tasks. Experiments on BFCL, τ2τ^2-Bench, and ACEBench demonstrate that GAIS-synthesized data significantly outperforms state-of-the-art baselines, enabling base models to match or even surpass their official instruction-tuned counterparts. Furthermore, GAIS exhibits superior data efficiency and scalability, achieving exceptional capabilities with significantly less data while maintaining continuous growth where baselines stagnate. Our code and dataset are publicly available at https://github.com/Eric8932/GAIS.

Wenhang Shi, Jinhao Dong, Yiren Chen +4

Renmin University of China · Peking University · Tencent

Synthetic Data GenerationSynthetic Data Generation for LLM AgentsLLM Agent TrainingSynthetic Data Generation for Language Models
AskPDF