Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation

Jianxiang Liu; Gaojing Zhang; Chuan Wen; Qipeng Liu; Yuxuan Zhao; Ning Guo; Wenzhao Lian

cs.ROAug 24, 2026

Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation

Authors: Jianxiang Liu, Gaojing Zhang, Chuan Wen, Qipeng Liu, Yuxuan Zhao, Ning Guo, Wenzhao Lian

Organizations: School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China. · School of Engineering and Informatics, University of Sussex, Brighton, U.K.

Abstract

Ensuring reliability in uncertain environments remains difficult for long-horizon robotic manipulation. End-to-end VLA models are data-heavy and opaque, making diagnosis and verification difficult. Hierarchical pipelines are more interpretable, but their plans are often weakly grounded in observations, weakly aligned with low-level actions, and computed without online feedback, leading to open-loop behavior and hallucinations. To address these issues, we introduce the Triplet-to-Track System (TTS), a closed-loop long-horizon imitation learning system that uses human videos to reduce reliance on robot-collected data. TTS represents high-level subgoals as instance-grounded triplets, translates them into continuous track priors for execution, and monitors task progress from observations for online replanning. Across diverse real-world long-horizon tasks, TTS achieves a 74.8% average success rate and supports object-level and compositional generalization.

Figures & tables

Fig. 4 : The Action Policy consists of a Transformer backbone and an MLP-based Action Head.

Method	Dynamic Sacl-B		Seq-Stacking		Diverse Pick&Place
Method	Seen	Perturbed	Seen	Unseen	Seen	Unseen	Perturbed
FT-Pi0.5 [ 15 ]	13.3	0.0	25.0	0.0	40.0	16.7	40.0
SeeDo [ 34 ]	26.7	0.0	41.7	40.0	50.0	54.5	0.0
PALO [ 25 ]	14.3	0.0	16.7	18.2	42.9	36.4	0.0
TTS	78.6	60.0	83.3	66.7	84.6	70.0	80.0

TABLE I : Success rates on long-horizon tasks (%).

Fig. 5 : Partial visualization. Top: triplet transitions from the Task Planner(GP/R/UR/RD/U/UL=gripper/right/upper-right/right-down/upper/upper-left; G/NG=grasp/no-grasp; Y/R/B/Br+B/W/S=yellow/red/blue/Brown+block/weight/scale). Bottom: two-camera point-track views.

Method	Rollout Time $\downarrow$ (s)	Overall Accuracy $\uparrow$ (%)
Only LTP	$3.4\times 10^{-3}$	84.3
Only VLM	202.6	98.8
LTP + VLM	32.2	98.4

Table 4

Method/Task	DPP	SS	DSB
w/o Track Predictor	45.5 $\pm$ 7.4	46.1 $\pm$ 3.4	43.8 $\pm$ 9.4
w/o Temporal Graph	57.1 $\pm$ 8.4	25.0 $\pm$ 0.0	0.0 $\pm$ 0.0
w/o Relation Predictor	75.0 $\pm$ 0.0	37.5 $\pm$ 5.5	14.3 $\pm$ 0.0
w/o Task Planner (ATM)	7.1 $\pm$ 2.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0
Only Action Policy	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0

Table 5

Fig. 6 : TTS Success rate with 0–40 robot Demos.

Explore similar work

LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Feb 25, 2026Yue Yang, Shuo Cheng, Yu Fang +4Long-Horizon Robotic Manipulation Robotic Manipulation
University of North Carolina at Chapel Hill · Georgia Institute of Technology · Carnegie Mellon University
Long-Horizon Manipulation via Trace-Conditioned VLA Planning
Apr 23, 2026Isabella Liu, An-Chieh Cheng, Rui Yan +7Long-Horizon Robotic Manipulation Language-Conditioned Robot Manipulation
University of California, San Diego · NVIDIA
IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation
May 14, 2026Shijie Lian, Bin Yu, Xiaopeng Lin +8Robot Policy Learning Language-Conditioned Robot Manipulation
1HUST · 2ZGCA · 4HIT +7

Feb 25, 2026J/K move · Enter open · S save

LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

Yue Yang, Shuo Cheng, Yu Fang, Homanga Bharadhwaj+3

University of North Carolina at Chapel Hill · Georgia Institute of Technology · Carnegie Mellon University

Long-Horizon Robotic Manipulation Robotic Manipulation VLMs for Robotics Zero-Shot Robotic Manipulation Robustness of VLA Models

Open paper AskPDF arXiv

General-purpose robots must master long-horizon manipulation, defined as tasks involving multiple kinematic structure changes (e.g., attaching or detaching objects) in unstructured environments. While Vision-Language-Action (VLA) models offer the potential to master diverse atomic skills, they struggle with the combinatorial complexity of sequencing them and are prone to cascading failures due to environmental sensitivity. To address these challenges, we propose LiLo-VLA (Linked Local VLA), a modular framework capable of zero-shot generalization to novel long-horizon tasks without ever being trained on them. Our approach decouples transport from interaction: a Reaching Module handles global motion, while an Interaction Module employs an object-centric VLA to process isolated objects of interest, ensuring robustness against irrelevant visual features and invariance to spatial configurations. Crucially, this modularity facilitates robust failure recovery through dynamic replanning and skill reuse, effectively mitigating the cascading errors common in end-to-end approaches. We introduce a 21-task simulation benchmark consisting of two challenging suites: LIBERO-Long++ and Ultra-Long. In these simulations, LiLo-VLA achieves a 69% average success rate, outperforming Pi0.5 by 41% and OpenVLA-OFT by 67%. Furthermore, real-world evaluations across 8 long-horizon tasks demonstrate an average success rate of 85%. Project page: https://yy-gx.github.io/LiLo-VLA/.

Abstract

Explore similar work

LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

Long-Horizon Manipulation via Trace-Conditioned VLA Planning

IntentVLA: Short-Horizon Intent Modeling for Aliased Robot Manipulation