GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes

Huajian Zeng; Abhishek Saroha; Daniel Cremers; Xi Wang

cs.CVMar 18, 2026

GMT: Goal-Conditioned Multimodal Transformer for 6-DOF Object Trajectory Synthesis in 3D Scenes

Authors: Huajian Zeng, Abhishek Saroha, Daniel Cremers, Xi Wang

Organizations: TU München · MBZUAI · MCML · ETH Zürich

Abstract

Synthesizing controllable 6-DOF object manipulation trajectories in 3D environments is essential for enabling robots to interact with complex scenes, yet remains challenging due to the need for accurate spatial reasoning, physical feasibility, and multimodal scene understanding. Existing approaches often rely on 2D or partial 3D representations, limiting their ability to capture full scene geometry and constraining trajectory precision. We present GMT, a multimodal transformer framework that generates realistic and goal-directed object trajectories by jointly leveraging 3D bounding box geometry, point cloud context, semantic object categories, and target end poses. The model represents trajectories as continuous 6-DOF pose sequences and employs a tailored conditioning strategy that fuses geometric, semantic, contextual, and goaloriented information. Extensive experiments on synthetic and real-world benchmarks demonstrate that GMT outperforms state-of-the-art human motion and human-object interaction baselines, such as CHOIS and GIMO, achieving substantial gains in spatial accuracy and orientation control. Our method establishes a new benchmark for learningbased manipulation planning and shows strong generalization to diverse objects and cluttered 3D environments. Project page: https://huajian- zeng.github. io/projects/gmt/.

Figures & tables

Figure 1: Given an observed trajectory, scene context, and action description, our model predicts plausible future 6-DOF object trajectories. The generated trajectories are more efficient than natural human motions.

Figure 2: Pipeline overview. Given an observed trajectory and scene context, our model predicts future 6-DOF object trajectories conditioned on a specified goal state. We encode (a) trajectory dynamics, (b) local geometry propagated from the scene point cloud to the object’s bounding box, (c) semantic fixture boxes and labels, (d) natural language description of the action (e) a goal descriptor. A multimodal transformer performs hierarchical fusion that emphasizes geometric feasibility before semantic preferences. The fused latent is fed directly to the prediction head (no separate decoding stage), which we found more stable for long-horizon control.

Figure 3: Qualitative results on the ADT dataset. The green trajectory represents the input history across all experiments. Only our model produces trajectories that both reach the target and avoid collisions, while also achieving shorter path lengths compared to the ground-truth natural trajectories. Adaptive GIMO fails due to the absence of gaze information, whereas CHOIS accumulates errors over time, ultimately leading to failure.

Method	ADE[m] ↓	FDE[m] ↓	FD[m] ↓	AC[m] ↑	CR ↓
GIMO [ 63 ]	0.982	1.401	1.511	0.140	19.6%
CHOIS [ 30 ]	0.853	1.062	1.209	0.283	9.3%
GMT (Ours)	0.366	0.072	0.438	0.402	13.1%

Table 1: Quantitative results on the ADT dataset. Our method outperforms both baselines in all metrics except collision rate. Note that collision rate is only meaningful when FDE is also low; otherwise, it may decrease due to trivial predictions such as static forecasts.

Figure 4: Qualitative results on the HD-EPIC dataset. Across all examples, the green points indicate the input history. Our model generates trajectories that are more efficient than the ground truth, while all baselines remain stuck in repetitive motions.

Method	ADE[m] ↓	FDE[m] ↓	FD[m] ↓	AC[m] ↑	CR ↓
GIMO [ 63 ]	0.411	0.654	0.780	0.002	11.8%
CHOIS [ 30 ]	0.446	0.589	0.760	0.009	12.0%
GMT (Ours)	0.283	0.034	0.391	0.037	10.3%

Table 2: Quantitative results on the HD-EPIC dataset. Ours achieves the best performance across all metrics, demonstrating superior trajectory prediction accuracy and robustness in real-world scenarios.

Variant	ADE[m] ↓	FDE[m] ↓	FD[m] ↓	AC[m] ↑	CR ↓
w/o pointcloud	0.364	0.062	0.493	0.384	18.7%
w/o semantic	0.360	0.080	0.505	0.391	14.0%
w/o goal	0.531	0.593	0.729	0.311	13.3%
First frame	0.554	0.258	0.696	0.298	83.2%
GMT (Full)	0.366	0.072	0.438	0.402	13.1%

Table 3: Ablation study on the ADT dataset. The best results are highlighted in bold.

Figure 5: 3D bounding box reconstruction in the HD-EPIC dataset. (a): input RGB frame with object mask. (b): mask filtering and sparse 2D-3D correspondences from SLAM and MPS data. (c): monocular depth estimation from UniK3D. (d): final 3D bounding box recovered after depth alignment and scaling. This pipeline enables accurate localization of small objects (e.g., bowls, cups) in cluttered scenes.

Figure 6: Object Position computation using hand-tracking We demonstrate the object positions. depicted as Yellow ⁢ along with the orientation and the hand that is interacting with the object at the particular frame sampled at intervals during the entire period of the moving object.

Figure 7: Qualitative results in the ADT dataset.

Figure 8: Qualitative results in the HD-EPIC dataset.

Figure 9: Failure Cases in the ADT dataset. We can observe that sometimes, despite being goal-conditioned, the generated trajectory may be longer than the ground truth trajectory and may overshoot the destination.

Figure 10: Failure cases in the HD-EPIC dataset. Our method suffers from adding redundant motion for inputs don’t have significant change in their positions. Such motion os observed for small object trajectories that are often interacted with for a very small duration.

Multimodal Transformer

See all 22 papers

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation
Feb 28, 2026Zilong Xie, Jingyu Gong, Xin Tan +2Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation
School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China
Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation
Sep 21, 2025Liang Heng, Jiadong Xu, Yiwen Wang +6Robot Skill Learning Language-Conditioned Robot Manipulation
CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT
AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence
Apr 12, 2026Jiawei Zhang, Kaizhe Hu, Yingqian Huang +3Affordance-Based Robotic Manipulation Robotic Manipulation
Shanghai Qi Zhi Institute · Tsinghua University · Fudan University +1

Feb 28, 2026J/K move · Enter open · S save

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

Zilong Xie, Jingyu Gong, Xin Tan, Zhizhong Zhang+1

School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation VLMs for Robotics Multimodal RAG Object Pose Estimation

Open paper AskPDF arXiv

Existing end-to-end approaches of robotic manipulation often lack generalization to unseen objects or tasks due to limited data and poor interpretability. While recent Multimodal Large Language Models (MLLMs) demonstrate strong commonsense reasoning, they struggle with geometric and spatial understanding required for pose prediction. In this paper, we propose RobMRAG, a 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation (MRAG) framework for zero-shot robotic manipulation. Specifically, we construct a multi-source manipulation knowledge base containing object contact frames, task completion frames, and pose parameters. During inference, a Hierarchical Multimodal Retrieval module first employs a three-priority hybrid retrieval strategy to find task-relevant object prototypes, then selects the geometrically closest reference example based on pixel-level similarity and Instance Matching Distance (IMD). We further introduce a 3D-Aware Pose Refinement module based on 3D Gaussian Splatting into the MRAG framework, which aligns the pose of the reference object to the target object in 3D space. The aligned results are reprojected onto the image plane and used as input to the MLLM to enhance the generation of the final pose parameters. Extensive experiments show that on a test set containing 30 categories of household objects, our method improves the success rate by 7.76% compared to the best-performing zero-shot baseline under the same setting, and by 6.54% compared to the state-of-the-art supervised baseline. Our results validate that RobMRAG effectively bridges the gap between high-level semantic reasoning and low-level geometric execution, enabling robotic systems that generalize to unseen objects while remaining inherently interpretable.

Abstract

Multimodal Transformer

Explore similar work

Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation

AffordGen: Generating Diverse Demonstrations for Generalizable Object Manipulation with Afford Correspondence