Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation

Yi Wang; Yang Yang; Guangqi Xu; Sumin Lin; Ning Kang; Pengxiang Lu; Xiaotong Chen; Zeyu Xue

cs.ROOct 3, 2026

Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation

Authors: Yi Wang, Yang Yang, Guangqi Xu, Sumin Lin, Ning Kang, Pengxiang Lu, Xiaotong Chen, Zeyu Xue,

Organizations: H.I.-Robot Lab., School of Automation Science and Engineering, South China University of Technology. · Faculty of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, HKSAR. · School of Astronautics, Northwestern Polytechnical University, Xi’an, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, 999077, HKSAR.

Abstract

Robotic manipulation often requires inferring task-relevant states from past interactions when the current observation alone is insufficient to determine the appropriate action. Despite progress in benchmarking memory-augmented vision-language-action (VLA) models, application-oriented tasks requiring history-dependent semantic inference remain underrepresented. We introduce GiT (Grounded in Time), a dataset and benchmark for grounding manipulation decisions in past events across biolaboratory, household, and industrial scenarios. It includes real-robot and Universal Manipulation Interface (UMI) style demonstrations covering 18 bimanual tasks, together with simulation data and a ManiSkill-based evaluation suite covering nine tasks. Fine-grained subtask annotations and annotated counterfactual task pairs, in which similar current observations require different actions depending on prior events, support policy learning and targeted evaluation of history use. Evaluations of representative end-to-end VLA models in simulation and on selected real-world tasks reveal substantial room for improvement in history-dependent manipulation. The dataset and benchmark are available at the project page.

Figures & tables

Dataset	Year	Focus	Env.	Embodiment	Scale	Hist.	Pre-act.	Snap.	CF
BEHAVIOR-1K [ 1 ]	2024	Daily tasks	Sim / real	Mobile arm	1k tasks	$\times$	$\times$	$\checkmark$	$\times$
RoboEval [ 2 ]	2025	Bimanual tasks	Sim	Dual arm	8 families; 3k demos	$\times$	$\times$	$\checkmark$	$\times$
RoboMME [ 3 ]	2025	Memory tasks	Sim / real	Single arm	16 tasks; 1.6k demos	in-episode/video	$\checkmark$	$\times$	$\times$
RMBench [ 4 ]	2026	Memory tasks	Sim / real	Dual arm	9 tasks; 3 real tasks	in-episode	$\checkmark$	$\times$	$\times$
GiT benchmark (ours)	2026	Memory tasks	Sim / real	Dual arm	18 tasks; 4.6k demos	in-episode/video	$\checkmark$	$\times$	$\checkmark$

TABLE I: Compact comparison of representative robotic manipulation benchmarks.

Scenario	ID	Task	Acquisition Pipeline			History-Dependent Capability
Scenario	ID		Real	UMI	Sim	IHR	OIT	TOR	HG	HSR
BioLab	L1	Tube Selection by Contamination History	$\checkmark$	$\checkmark$	–	$\checkmark$	–	$\checkmark$	–	–
	L2	Container Selection by Filling History	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–	–
	L3	Placement by Wiping History	$\checkmark$	$\checkmark$	–	$\checkmark$	$\checkmark$	$\checkmark$	–	–
	L4	Beaker Selection by Stirring History	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	–
	L5	Sample Selection by Handoff History	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–

TABLE II: Overview of the 18 GiT tasks, acquisition-pipeline coverage, and required history-dependent capabilities.

Fig. 1: (a) Overview of the dual-arm robotic platform, equipped with a global camera, wrist cameras on both arms, and left and right grippers. (b) Close-up view of a UMI gripper, including the wrist camera, tracker, and gripper.

Pipeline	Tasks	Episodes	Time Steps	Duration (h)
Real Robot	18	1,260	639.0K+	5.91+
Simulation	9	1,620	1,026.3K+	9.50+
UMI	18	1,800	1,681.5K+	15.57+
Total	18	4,680	3.34M+	30.99+

TABLE III: Dataset statistics of the three acquisition pipelines.

Fig. 2: (a) Counterfactual (CF) pairs. Each pair shares an identical instruction and an identical final frame but has a completely different demonstration phase, and therefore a completely different correct action. (b) Semi-automated annotation pipeline for robotic-arm data. Videos and robot states processed to identify and refine action segments, which are validated and exported as structured annotations.

Fig. 3: Qualitative examples of successful and failed manipulation in GiT. The top and bottom panels show real-robot and simulated trials, respectively. Each panel presents a failed trial in the upper row and a successful trial in the lower row, with time progressing from left to right. Dashed vertical lines separate pre-episode interaction histories from subsequent policy execution.

Dataset	Metric	Joint	Rotation
RoboMME [ 3 ]	Complexity	0.0756	–
	Diversity	0.2834	–
GiT (ours)	Complexity	0.1782	0.3468
	Diversity	0.4991	0.6825

TABLE IV: Comparative analysis for action complexity and diversity.

Method	L2		L4		L5		H1		H3		H5		I2		I3		I5		AV
	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR	SR	PSR
RDT-1B [ 26 ]	1.67%	4.04%	0%	7.50%	0%	0%	1.67%	6.50%	3.33%	3.17%	0%	5.04%	5.00%	7.50%	1.67%	3.84%	0%	3.54%	1.48%	4.57%
$\pi_{0.5}$ [ 8 ]	5.00 %	5.00%	3.33 %	7.50 %	0%	4.00%	3.33%	8.50 %	1.67 %	3.83 %	0%	3.38 %	3.33%	8.50%	3.33 %	4.27%	0%	4.00%	2.22 %	5.44 %
HAMLET [ 27 ]	3.33%	5.44 %	1.67%	4.32%	1.67 %	4.47 %	5.00 %	3.26%	0%	3.17%	0%	1.17%	5.00 %	8.83 %	3.33%	5.00 %	0%	6.50 %	2.22 %	4.68%

TABLE V: Performance comparison of RDT-1B,

\pi_{0.5}

and HAMLET on simulation tasks.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,593 papers

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
Jun 8, 2026Hao Shi, Weiye Li, Bin Xie +6Long-Horizon Robotic Manipulation World Models for Robotics
Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China +1
TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation
Aug 27, 2026Jiarui Yang, Yehao Lu, Yuning Su +9Robot Learning Long-Horizon Robotic Manipulation
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · Zhejiang University, Hangzhou, China. · Simon Fraser University, Burnaby, BC, Canada. +1
RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies
Mar 4, 2026Yinpei Dai, Hongze Fu, Jayjun Lee +6Memory-Augmented VLMs Robot Manipulation Benchmarks
University of Michigan · Stanford University · Figure AI

Jun 8, 2026J/K move · Enter open · S save

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models

Hao Shi, Weiye Li, Bin Xie, Yulin Wang+5

Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China+1

Long-Horizon Robotic Manipulation World Models for Robotics Temporal Modeling for VLA Models Vision-Language-Action Models Memory-Augmented Agents

Open paper AskPDF arXiv

Temporal modeling is essential for robotic manipulation, as effective control requires both memory of past interactions and imagination of future states. However, most VLA models rely primarily on the current observation and therefore struggle with long-horizon, temporally dependent tasks. Cognitive science suggests that humans rely on working memory to buffer short-lived context, the hippocampal system to preserve episodic memory of past experience, and internal models to imagine possible future state evolution. Inspired by these mechanisms, we propose MemoryVLA++, a full temporal modeling framework that equips VLA models with memory and imagination for robotic manipulation. A pretrained VLM encodes the current observation into perceptual and cognitive tokens, forming working memory. These tokens query a Perceptual-Cognitive Memory Bank to retrieve relevant historical context. This bank stores low-level details and high-level semantics from past interactions, and is updated through redundancy-aware consolidation. A world model imagines future states in a denoising latent space, and the imagined latents are integrated under memory guidance to form full temporal-aware tokens. The resulting tokens condition a diffusion action expert to predict temporally consistent action sequences. We conduct extensive experiments on 5 simulation benchmarks and 3 categories of real-robot tasks across 3 robots, covering general manipulation, long-horizon temporal tasks, robustness, and generalization. Our method achieves strong performance across Libero, SimplerEnv, Mikasa-Robo, Calvin, Libero-Plus, and diverse real-robot tasks, validating the effectiveness of full temporal modeling with memory and imagination. For example, on real robots, it achieves +9%, +26%, +28% gains on general, memory-dependent, and imagination-dependent tasks. Project Page: https://shihao1895.github.io/MemoryVLA-PP-Web

Abstract

Robotic Manipulation

Explore similar work

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models

TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation

RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies