Towards Generalizable Robotic Data Flywheel: High-Dimensional Factorization and Composition

Yuyang Xiao; Yifei Zhou; Haoran Wang; Wenxuan Ou; Yuxiao Liu

cs.ROMar 26, 2026

Towards Generalizable Robotic Data Flywheel: High-Dimensional Factorization and Composition

Authors: Yuyang Xiao, Yifei Zhou, Haoran Wang, Wenxuan Ou, Yuxiao Liu

Organizations: ByteDance Seed

Abstract

The lack of sufficiently diverse data, coupled with limited data efficiency, remains a major bottleneck for generalist robotic models, yet systematic strategies for collecting and curating such data are not fully explored. Task diversity arises from implicit factors that are sparsely distributed across multiple dimensions and are difficult to define explicitly. To address this challenge, we propose F-ACIL, a heuristic factor-aware compositional iterative learning framework that enables structured data factorization and promotes compositional generalization. F-ACIL decomposes the data distribution into structured factor spaces such as object, action, and environment. Based on the factorized formulation, we develop a factor-wise data collection and an iterative training paradigm that promotes compositional generalization over the high-dimensional factor space, leading to more effective utilization of real-world robotic demonstrations. With extensive real-world experiments, we show that F-ACIL can achieve more than 45% performance gains with 5-10 $\times$ fewer demonstrations comparing to that of which without the strategy. The results suggest that structured factorization offers a practical pathway toward efficient compositional generalization in real-world robotic learning. We believe F-ACIL can inspire more systematic research on building generalizable robotic data flywheel strategies. More demonstrations can be found at: https://f-acil.github.io/

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of data distribution with specific properties. (1)The 3D surfaces (blue-purple) represent different training data distributions, while the contour maps(warm amber) show the shared evaluation distribution. (2)We compare three data distribution: (a) a narrow Gaussian-like distribution with limited coverage, (b) a quasi-uniform distribution with full coverage but low efficiency, and (c) F-ACIL with multiple gaussian modes, which achieves efficient, broad coverage via factor-wise composition.

Figure 2: Overview. (1)F-ACIL explicitly decomposes robotic manipulation into three factors: Object, Action, and Environment. This factorized representation is introduced to systematically expand the distribution of training data in real-world robotic learning. (2)F-ACIL with factorized representation allows principled compositional generalization to out-of-distribution (OOD) instances.

Figure 3: Capabilities . In both Pick-and-Place and Open-and-Close, all cases exhibit robust compositional generalization across out-of-domain objects, actions, and environments.

Factor	Dimension	Description	Example Objects
Texture ( $T$ )	Transparent	Light transmission with refraction	Transparent water cup
	Specular	Mirror-like surface reflection	Aluminum can
	Diffuse	Rough surface with uniform reflection	Metal bowl, Foil tray
	Absorptive	Strong light absorption	Towel, Tape roll
Geometry ( $G$ )	Cylindrical	Axis-symmetric geometry	Cup, Aluminum can
	Dish-like	Roughly concave.	Tray, Plate

Table 1: Object factors

Factor	Dimension	Description	Range
Position	$x$	Forward direction of the robot	workspace
	$y$	Left direction of the robot	workspace
	$z$	Up direction of the robot	workspace
Orientation	roll( $r$ )	Rotation around $x$ -axis	$[-\pi,\pi)$
	pitch( $p$ )	Rotation around $y$ -axis	$[-\pi,\pi)$
	yaw( $y$ )	Rotation around $z$ -axis	$[-\pi,\pi)$

Table 2: Action factors

Factor	Dimension	Description	Examples
Macro	Light Intensity( $I$ )	Overall brightness of the scene	Normal Indoor(200-500 lux)
	Color Temperature( $T$ )	Warm or cool illumination	Warm Light(2700K - 3500K)
	Shadow Direction( $D$ )	Direction of the shadow	Left, Mid, Right
Micro	Surface Material( $M$ )	The material of local scenes	Transparent, Absorptive
	Background Clutter( $C$ )	The level of clutter in the scene	Clean, Light Clutter, Heavy Clutter

Table 3: Environment factors

Figure 4: Demonstrations collected for iteration 1 under $\mathcal{O}$ space The initial demonstrations are sampled along the main diagonal of the factor space, with an equal number of samples per composition. The background color encodes the success rate(SR). Black dots mark the compositions included in the dataset, denoted as $D_{1}$ .

Figure 5: Iterative Search Procedure in Object Space In each iteration, different colors indicate the performance under specific factor compositions. The background color encoding follows the same scheme as in Fig. 4 . Points highlighted in blue mark factor compositions that are newly added into the demonstrations in the current iteration. The number of points reflects the distribution of ratio across different factor compositions.

r\leftarrow\frac{1}{|\mathcal{O}|}\sum_{o\in\mathcal{O}}r(o)

(17)

Algorithm 1 Iterative Factor-Based Data Expansion

Panel of Figure 6 — Figure 6: Empirical Study on Factors Design (a) Shadow direction constitutes a successful factor design, as strong compositional generalization is observed. (b) Light-source position and direction serve as a counterexample, where compositional generalization fails to hold.

Panel of Figure 7 — Figure 7: Generalization evaluation on $f(D_{\mathcal{O}})\mathcal{A}$ and $\mathcal{O}\mathcal{A}$ . (left) The overall success rates evaluated on $f(D_{\mathcal{O}})\mathcal{A}$ and on the complete $\mathcal{O}\mathcal{A}$ space exhibit only a slight difference. (right) Time needed for one iteration of the data flywheel between entire $\mathcal{OA}$ and reduced $f(D_{\mathcal{O}})\mathcal{A}$ .

Figure 8: Comparison of data strategies across factor spaces. Each curve corresponds to a different data collection strategy used. F-ACIL-O, -OA and -OAE represent the iterative subset search under different factor spaces.

Task	Benchmark	Gaussian-32k	Gaussian-16k	Gaussian-8k	Gaussian-4k	Gaussian-2k
Pick-and-Place	F-ACIL-O	73.3	69.2	63.3	51.4	39.8
	F-ACIL-OA	69.2	54.3	41.7	35.4	22.8
	F-ACIL-OAE	40.1	35.2	30.4	24.8	19.4
Open-and-Close	F-ACIL-O	58.3	48.4	40.6	31.7	25.8
	F-ACIL-OA	55.5	41.6	36.2	30.4	23.5
	F-ACIL-OAE	37.3	31.8	29.2	25.8	20.4

Table 4: Changes in success rate with changes in dimensions

Figure 9: Scaling Pattern w/o F-ACIL under Different Benchmark We compare the power-law fits under different benchmark. Each benchmark reflects the model’s generalization ability over the corresponding factor space: $\mathcal{O}$ (object), $\mathcal{OA}$ (object–action), and $\mathcal{OAE}$ (full space).

Data-Efficient Robot Learning

Data-efficient robot learning develops ways for robots to acquire reliable skills with fewer demonstrations, interactions, and human annotations.

See all 187 papers

Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation
Jul 23, 2026Yu Qi, Zhang Ye, Xinyi Xu +6Robot Skill Learning Language-Conditioned Robot Manipulation
Northeastern University · NVIDIA
World-Task Factorization for Robot Learning
Jun 1, 2026Eduardo Sebastián, Adrian Pfisterer, Vito Mengers +2Robotic Control Robot Policy Learning
Department of Computer Science and Technology, University of Cambridge, United Kingdom · Robotics and Biology Laboratory, Technische Universität Berlin, Germany · Robotics and Biology Laboratory, Technische Universität Berlin +1
Diagnosing Compositional Generalization in Sequential Robot Tasks
Jul 31, 2026Yixiao Wang, Cheng-En Wu, Lingfeng Sun +5Language-Conditioned Robot Manipulation Robotic Data Collection
University of California, Berkeley · Tsinghua University

Jul 23, 2026J/K move · Enter open · S save

Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation

Yu Qi, Zhang Ye, Xinyi Xu, Yuxuan Lu+5

Northeastern University · NVIDIA

Robot Skill Learning Language-Conditioned Robot Manipulation Generalization in Robotic Manipulation Robotic Policy Evaluation Data-Efficient Robot Learning Robot Policy Generalization+3

Open paper AskPDF arXiv

Compositional generalization is essential for robot to follow diverse instructions. However, pretrained policies are known to take shortcuts, deferring to salient cues rather than grounding language. We introduce a diagnostic framework that localizes this failure to individual instruction factors, e.g., reusable semantic components such as color, verb, object, size, and spatial attribute. Our framework formalizes instruction factor bias, the tendency of fine-tuned policies to over-rely on dominant factors as shortcuts, and quantifies it through two metrics: Factor Dominance Rate (FDR), capturing pairwise bias between factors, and Factor Dominance Hierarchy (FDH), aggregating these into a global ranking. Evaluation on six foundation policies reveals broadly consistent ordering, i.e., color

\geq

object

\geq

spatial

\geq

verb

\geq

size, with color dominant, and verb and size most under-grounded. We further show the diagnosis is actionable: a bias-aware data collection strategy that reallocates a fixed budget toward under-grounded factors outperforms baselines in simulation and on a real robot using half the demonstrations, thereby enabling more sample-efficient and generalizable policy learning.

Abstract

Data-Efficient Robot Learning

Explore similar work

Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation

World-Task Factorization for Robot Learning

Diagnosing Compositional Generalization in Sequential Robot Tasks