TGM-VLA: Task-Guided Mixup for Sampling-Efficient and Robust Robotic Manipulation
Authors: Fanqi Pu, Lei Jiang, Wenming Yang
Organizations: Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · The National and Local Co-Build Humanoid Robotics Innovation Center, Shanghai, China.
The performance of robotic imitation learning is fundamentally limited by data quality and training strategies. Prevalent sampling strategies on RLBench suffer from severe keyframe redundancy and imbalanced temporal distribution, leading to inefficient memory usage and unstable optimization. Moreover, reprojecting point clouds onto multi-view images with a black background--while more efficient than voxel-based methods--often causes dark objects to be indistinguishable and hard to manipulate. In this work, we propose a novel holistic framework that significantly improves both model performance and training efficiency. First, we redesign and optimize the keyframe sampling strategy, reducing memory consumption by 80% and accelerating training speed by 5x. Second, we augment the model with a color inversion projection branch--a simple yet effective module that resolves the ambiguity of dark objects. Finally, we propose a task-guided mixup technique that dynamically fuses point clouds and action heatmaps according to task instructions, greatly improving robustness to distractors and performance in multi-goal scenarios. Extensive experiments demonstrate that our method achieves state-of-the-art performance with a 90.5% success rate on RLBench and 68.8% on the COLOSSEUM benchmark under challenging interference conditions. Our code and checkpoints are available at https://github.com/PuFanqi23/TGM-VLA.
Figures & tables
Fig. 1: Data Distribution and Sampling. Red dots represent next target poses. Orange dots represent observed states sampled every 10 frames. Yellow arrows depict keyframe samples, while white arrows depict augmented samples. The original sampling strategy causes data redundancy and temporal imbalance, as each augmented sample forces repeated sampling of all subsequent keyframe samples.
Fig. 2: Overview of TGM-VLA . Given the current point cloud and a task instruction, TGM-VLA predicts the next key-frame pose. This model employs data augmentations including 3D transformations, intra-task, and cross-task mixup. At inference, it first renders point cloud into orthogonal views, enhanced with inverted-color images for improved contrast, to coarsely localize the area of interest. The second stage uses zoomed-in views to precisely predict the gripper pose.
Fig. 3: Remedies for Failure Scenarios. Each subfigure pairs a specific failure mode with our corrective measures.
Fig. 4: Low Contrast Problem and Color Inversion.
Fig. 5: Alignment of Different Visual Modalities with Text Instructions.
Overall
Task Success Rate (%)
Models
Avg. SR (%) ↑
Reported Hardware
Close Jar
Drag Stick
Insert Peg
Meat off Grill
Open Drawer
Place Cups
Place Wine
Push Buttons
PerAct [ 12 ]
49.4
8V100 × 16d
55.2 ± 4.7
89.6 ± 4.1
5.6 ± 4.1
70.4 ± 2.0
88.0 ± 5.7
2.4 ± 3.2
44.8 ± 7.8
92.8 ± 3.0
RVT [ 13 ]
62.9
8V100 × 1d
52.0 ± 2.5
99.2 ± 1.6
11.2 ± 3.0
88.0 ± 2.5
71.2 ± 6.9
4.0 ± 2.5
91.0 ± 5.2
100.0 ± 0.0
RVT-2 [ 14 ]
81.4
8V100 × 20h
100.0 ± 0.0
99.0 ± 1.7
40.0 ± 0.0
99.0 ± 1.7
74.0 ± 11.8
38.0 ± 4.5
95.0 ± 3.3
100.0 ± 0.0
SAM2ACT [ 21 ]
86.8
8H100 × 12h or 2A6000 × 5d
99.0 ± 2.0
99.0 ± 2.0
84.0 ± 5.7
98.0 ± 2.3
83.0 ± 6.0
47.0 ± 6.0
93.0 ± 3.8
100.0 ± 0.0
BridgeVLA [ 22 ]
88.2
48H100 × 20h
100.0 ± 0.0
100.0 ± 0.0
88.0 ± 2.8
100.0 ± 0.0
100.0 ± 0.0
58.4 ± 10.0
88.0 ± 2.8
98.4 ± 2.2
TABLE I: Results on RLBench. This table compares our TGM-VLA with prior works and ablations. “Avg. SR” is the average success rate over 18 tasks (higher is better). “Reported Hardware” lists the training resources (GPUs × Type × Time). “w/o CTM” and “w/o ITM” denote ablations without Cross-Task and Intra-Task Mixup, respectively.
Fig. 6: Illustration of Intra-Task Mixup (top row) and Cross-Task Mixup (bottom row).
Overall
Success Rate (%)
Models
Avg. SR (%) ↑
Avg. Rank ↓
All Perturbations
MO-COLOR
RO-COLOR
MO-TEXTURE
RO-TEXTURE
MO-SIZE
PerAct [ 12 ]
27.9
5.71
7.2
24.0
29.2
28.8
17.71
35.6
RVT [ 13 ]
35.4
5.28
6.4
26.0
31.3
44.8
41.1
35.3
RVT-2 [ 14 ]
56.7
3.92
15.6 ± 0.8
53.0 ± 0.9
54.6 ± 0.6
59.7 ± 0.7
56.7 ± 1.4
60.9 ± 0.9
SAM2ACT [ 21 ]
61.9
2.93
18.5 ± 3.5
62.1 ± 0.9
60.9 ± 0.3
58.6 ± 0.6
63.4 ± 2.1
61.8 ± 0.5
BridgeVLA [ 22 ]
64.0
2.14
18.7 ± 2.2
60.5 ± 1.1
63.8 ± 0.1
63.5 ± 1.5
68.4 ± 3.3
69.3 ± 1.0
TABLE II: Results on the COLOSSEUM Benchmark. This table shows the success rates across 14 generalization settings (12 unseen environmental perturbations + all perturbations + RLBench). The “Avg. Rank” column reports the average rank of each method across all perturbations, where lower values indicate better overall performance.
Method
Training
Testing
Stack Cups
Setup Chess
Stack Cups
Setup Chess
w/o CIB
80.0 ± 3.0
16.0 ± 3.8
74.0 ± 4.6
12.0 ± 2.3
Ours
94.0 ± 1.0
46.0 ± 7.3
87.0 ± 2.0
30.0 ± 4.0
Improvement
+14.0
+30.0
+13.0
+18.0
TABLE III: Ablation Study on Color Inversion. ”w/o CIB” denotes ablations without a color inversion branch.
Fig. 7: Real Robot Experimental Setup.
Method
Basic Tasks
Generalization
Stack Blocks
Put in Drawer
Put on Plate
Distractor
Background
w/o TGM
60%
60%
80%
40%
60%
TGM-VLA
80%
70%
90%
90%
80%
Improvement
+20%
+10%
+10%
+50%
+20%
TABLE IV: Results in the real world. ”w/o TGM” denotes ablations without task-guided mixup technique.
Robotic imitation learning typically assumes access to optimal demonstrations, yet real-world data collection often yields suboptimal, exploratory, or even failed trajectories. Discarding such data wastes valuable information about environment dynamics and failure modes, which can instead be leveraged to improve decision-making. While 3D policies reduce reliance on high-quality demonstrations through strong spatial generalization, they still require large-scale data to achieve high task success. To address this, we propose DALI-R, a Data-Asymmetric Latent Imagination and Reranking framework for 3D robotic imitation learning from mixed-quality trajectories. It learns a Latent World Model over 3D point clouds for imagined rollouts and a Task Completion Scorer that reranks candidate action chunks, improving decision-making without additional high-quality demonstrations. We instantiate DALI-R with both diffusion and efficient flow-matching policies and evaluate it on Adroit and MetaWorld benchmarks. Across the two evaluated 3D base policies, DALI-R achieves an average 6.8% improvement in success rate while incurring less than 0.7× additional inference overhead.
Reinforcement Learning (RL) and Imitation Learning (IL) are the standard frameworks for policy acquisition in manipulation. While IL offers efficient policy derivation, it suffers from compounding errors and distribution shift. Conversely, RL facilitates autonomous exploration but is frequently hindered by low sample efficiency and the high cost of trial and error. Since existing hybrid methods often struggle with complex tasks, we introduce Mixture of RL and IL Experts (MoRI). This system dynamically switches between IL and RL experts based on the variance of expert actions to handle coarse movements and fine-grained manipulations. MoRI employs an offline pre-training stage followed by online fine-tuning to accelerate convergence. To maintain exploration safety and minimize human intervention, the system applies IL-based regularization to the RL component. Evaluation across four complex real-world tasks shows that MoRI achieves an average success rate of 97.5% within 2 to 5 hours of fine-tuning. Compared to baseline RL algorithms, MoRI reduces human intervention by 85.8% and shortens convergence time by 21%, demonstrating its capability in robotic manipulation.
Yaohang Xu, Lianjie Ma, Gewei Zuo +3
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China. · DRAGON Lab, The University of Tokyo, Tokyo, Japan. · School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China.
Current Vision-Language-Action (VLA) models predominantly rely on end-to-end fine-tuning. While effective, this paradigm compromises the inherent generalization capabilities of Vision-Language Models (VLMs) and incurs catastrophic forgetting. To address these limitations, we propose M2-VLA, which demonstrates that a generalized VLM is able to serve as a powerful backbone for robotic manipulation directly. However, it remains a key challenge to bridge the gap between the high-level semantic understanding of VLMs and the precise requirements of robotic control. To overcome this, we introduce the Mixture of Layers (MoL) strategy that selectively extracts task-critical information from dense semantic features. Furthermore, to facilitate efficient trajectory learning under constrained model capacity, we propose a Meta Skill Module (MSM) that integrates strong inductive biases. Extensive experiments in both simulated and real-world environments demonstrate the effectiveness of our approach. Furthermore, generalization and ablation studies validate the architecture's zero-shot capabilities and confirm the contribution of each key component. Our code and pre-trained models will be made publicly available.
Siyao Xiao, Yuhong Zhang, Zhifang Liu +9
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · PengCheng Laboratory, Shenzhen, China. · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China. +1